DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of claims: claims 1-5, 7-8, 10-17 and 19-23 are pending below. Claims 6, 9 and 18 are cancelled.
Response to Arguments
Applicant's arguments filed 3/18/2026 have been fully considered but they are not persuasive.
Applicant remark – (page 13-14) Applicant argue lack of teaching regarding claim language “one or more an image sensor that detect electromagnetic radiation and read out image data wherein the image data read out by the one or more image sensor.” Please see Remarks for more detail.
Examiner response – Examiner respectfully disagree. As pointed out in the office action before NGO DINH et al detail in figure 1 and 0080 the list of image sensors such as X-ray, MRI, CT, endoscopy that produce videos or one or more images, which all used different wavelength for imaging tissues. Please see Office Action below for more detail.
Applicant remark – (page 13-17) Applicant argue lack of teaching regarding claim language “…read out by the one or more image sensors comprises: color image data; and advanced image data comprising one or more of fluorescence image data or multispectral image data…” Please see Remarks for more detail.
Examiner response – Examiner respectfully disagree. Examiner view the claim requiring selecting one of the two output from the sensor(s): one output of color image data and fluorescence image data, OR another output of color image data and multispectral image data. Examiner selected the output of color image data and multispectral image data for examination and taught by NGO DINH et al in 0074 for first and second and third color, where each color is a different spectral, figure 7 and 0121-0122 for this color overlay/overlapping. Paragraph 00080 mentioned above also detail list of image sensors X-ray, MRI, CT, endoscopy…etc which use different spectral for image processing as well such view as multispectral image data.
However from the Remarks it seems that applicant require the combination of all image data of color, fluorescence image data AND multispectral image data as part of image processing, thus for compact prosecution Examiner have updated the search and found Tirupathi et al (US 2023/0206431) that uses all three claim element for such processing in figure 12 and 0062. Please review Office Action below for detail.
Claim Objections
Claim 10 dependence to cancelled claim 9 will be view as claim 10 dependent to claim 8 for compact prosecution and examination. Please correct to clarify.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 7, 11-13 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over NGO DINH et al (US 2019/0385302) in view of Tirupathi et al (US 2023/0206431).
Claim 1:
NGO DINH et al (US 2019/0385302) teach the following subject matter:
A system comprising:
an emitter comprising a plurality of electromagnetic sources (figure 1 and 0080-0082 teaches different type such as X-ray machine, a computed tomography (CT) machine, a magnetic resonance imaging (MRI use multispectral imaging to measure light across multiple wavelengths including visible and non-visible spectrums) machine, an endoscopy machine, or other medical imaging device that produces videos or one or more images of a human body or a portion);
one or more an image sensor that detect electromagnetic radiation and read out image data wherein the image data read out by the one or more image sensor (figure 1 and 0080 detail image sensors such as X-ray, MRI, CT, endoscopy that produce videos or one or more images):
color image data; and advanced image data; and an image signal processor that receives the image data read out by the image sensor; wherein the image signal processor generates an overlay frame based upon(0074 teaches overlaid with color and border around target region: color for border, and second/third color (multispectral image data) for detected object for true and false due to classification of detected object (lesion or other abnormality to be cancerous or otherwise abnormal which advance image data); figure 7 and 0121-0122 teaches image processed for feature of interest (target object) with first, second, third…colors for different features and positive/false classification, where 0122 teaches example where third color is lesion or cancerous lesion (advance data, since data is further processing is required to be classified to be cancerous); figure 8 and paragraph 0128-130 teaches object detection with different color overlay base on particular classification determined based on adversarial brand or network/trained neural network (advance data frame processing));
wherein the overlay frame comprises a false color overlay (0074 teaches overlaid with false color; figure 7-8 and paragraph 0121-0122 and 0128-0130, where false color is just different color assignment due to computer processing to help highlight, alert or enhance viewing for the user); and wherein the image signal processor adjusts the false color overlay based upon a user-requested sensitivity level and further based upon pixel values in the advanced image data (0013 teaches real-time video processing with trained neural network on frames to detect objects in frame with sensitivity setting from user).
NGO DINH et al teaches color image data and multispectral image data from image sensors of X-ray, MRI, CT, endoscopy, but do not teach the from the Remarks applicant is claiming requirement of color AND fluorescence AND multispectral image data all together:
advanced image data comprising one or more of fluorescence image data or multispectral image data.
Tirupathi et al (US 2023/0206431) teaches the following subject matter: advanced image data comprising one or more of fluorescence image data or multispectral image data (figure 12 and 0062 detail tumor boundary from the image data of color, dye administration and multispectral video generation 1100 and inversion and tissue classification, where the dye is fluorescence over time and specific regions).
NGO DINH et al and Tirupathi et al are both in the field of image analysis, especially using image data, specifically combination of image of color and multispectral or fluorescence to determine boundary of region or tumor of interest.
Therefore it would have been obvious to one having ordinary skill before the effective filing date to modify NGO DINH et al by Tirupathi et al regarding combination of different image data to generate live data for region monitoring as disclosed by Tirupathi et al in 0062
Claim 2:
Tirupathi et al teach:
The system of claim 1, wherein the advanced image data frame comprises one or more of:
the fluorescence image dataOR [[a]] the multispectral image data(figure 12 and 0062).
Claim 3:
NGO DINH et al teach:
The system of claim 1, wherein the image signal processor communicates with a deep learning algorithm configured to identify a target object within a scene based on the advanced data frame (figure 8 and paragraph 0128-130 teaches object detection with different color overlay base on particular classification determined based on adversarial brand or network/trained neural network/deep learning (advance data frame processing)).
Claim 4:
NGO DINH et al teach:
The system of claim 3, wherein the image signal processor generates the overlay frame based upon an output from the deep learning algorithm, and wherein the false color overlay comprises a heatmap that comprises:
a first false color overlay highlighting a region having a high likelihood of comprising the target object; and a second false color overlay highlighting a region having a lesser likelihood of comprising the target object relative to the region highlighted by the first false color overlay; wherein the first false color overlay comprises a different color than the second false color overlay (figure 7 and 0120-0122, especially 0121-0122 teaches first, second and third color adjustment (highlight/false color) for feature-of-interest (target object) that are classified to be true or false positive (high or low likelihood)).
Claim 5:
NGO DINH et al teach:
The system of claim 1, wherein the advanced overlay data frame comprises: a first false color overlay highlighting a first target object; and a second false color overlay highlighting a second target object, wherein the first target object is different than the second target object (figure 7 and 0121-0122 teaches image processed for feature of interest (target object) with first, second, third…colors for different features and positive/false classification, where 0122 teaches example where third color is lesion or cancerous lesion (advance data, since data is further processing is required to be classified to be cancerous); figure 8 and paragraph 0128-130 teaches object detection with different color overlay base on particular classification);
wherein each of the first target object and the second target object is identified based upon the one or more of the fluorescence OR multispectral imaging data of the advanced data frame (0074 teaches overlaid with color and border around target region: color for border, and second/third color (multispectral image data) for detected object for true positive and false positive due to classification of detected object (lesion or other abnormality to be cancerous or otherwise abnormal which advance image data)).
Claim 7:
NGO DINH et al teach:
The system of claim 5,
wherein each of the first false color overlay and the second false color overlay comprises a heatmap indicating a likelihood that a region comprises the first target object or the second target object (figure 7 and 0120-0122, especially 0121-0122 teaches first, second and third color adjustment (highlight) for feature-of-interest (target object) that are classified to be true or false positive (high or low likelihood), where heatmap is just different colors assign to different body parts); and
wherein the heatmap for each of the first false color overlay and the second false color overlay comprises a plurality of colors for representing varying likelihoods of comprising the first target object or the second target object (figure 7 and 0120-0122, especially 0121-0122 teaches first, second and third color adjustment (highlight) for feature-of-interest (target object) that are classified to be true or false positive (high or low likelihood)).
Claim 11:
NGO DINH et al teach:
The system of claim 1, wherein the image signal processor renders a video stream (real-time video obtained from a medical image device, a first bus for transferring the received real-time video, and at least one processor configured to receive the real-time video from the first bus, perform object detection by applying a trained neural network on frames of the received real-time video), and wherein the image signal processor executes an adaptive persistence algorithm to stabilize movement of the false color overlay (0065 teaches graphical indicator (classification, feature of interest, color, false/positive, shape, pattern…etc) may depend on classification and vibration).
Claim 12:
NGO DINH et al teach:
The system of claim 11, wherein the image signal processor stabilizes the movement of the false color overlay to compensate for movement of the image sensor (0064-0067 teaches consideration of vibration consideration of image/frame/video capture).
Claim 13:
NGO DINH et al teach:
The system of claim 11, wherein the adaptive persistence algorithm comprises applying persistence on movement detections within the advanced data frame to stabilize the movement of the false color overlay (0064-0067 teaches consideration of vibration consideration of image/frame/video capture, especially 0065 teaches graphical indicator (classification, feature of interest, color, false/positive, shape, pattern…etc) may depend on classification and vibration).
Claim 19:
The system of claim 1, wherein the plurality of electromagnetic sources of the emitter comprises:
a white light source; a plurality of excitation sources configured to emit electromagnetic radiation only within a waveband selected for fluorescing a tissue or reagent (Tirupathi et al teach in figure 1 and 0033 detail fluorescence of light at a certain wavelength when the light is shone onto the tissue); and
a plurality of multispectral sources configured to emit electromagnetic radiation only within a waveband selected for eliciting a spectral response from a tissue (NGO DINH et al detail in figure 14 and 0080 teaches X-ray and MRI are multispectral).
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over NGO DINH et al (US 2019/0385302) in view of Tirupathi et al (US 2023/0206431) as applied to claim 13 above, and further in view of Voss-Wolffet al et al (US 2021/0264661).
Claim 14:
NGO DINH et al and Tirupathi et al teach all the subject above, but not the following:
The system of claim 13, wherein applying the persistence on the movement detections comprises executing one or more of: a hysteresis-based tracking algorithm; a Kalman filter-based tracking algorithm; a particle filter-based tracking algorithm; OR a SLAM (simultaneous localization and mapping) based tracking algorithm
The following is taught by Voss-Wolffet al et al (US 2021/0264661) teach:
The system of claim 13, wherein applying the persistence on the movement detections comprises executing one or more of: a hysteresis-based tracking algorithm; a Kalman filter-based tracking algorithm; a particle filter-based tracking algorithm; OR a SLAM (simultaneous localization and mapping) based tracking algorithm (0003 teaches use of SLAM for tracking of the movement with visual rendering, where 0010 detail application of overlapping with color for identification process where color identify areas of content regions and border of content and non-content regions to distinguishing between essential and non-essential as disclosed in 0065).
NGO DINH et al and Tirupathi et al and Voss-Wolffet al et al are in the field of image analysis, especially in imaging with different color overlay such that the combine outcome is predictable.
Therefore it would have been obvious to one having ordinary skill before the effective filing date to modify NGO DINH et al and Tirupathi et al by Voss-Wolffet et al where such tracking are used to augment or supplement the camera data to provide a more reliable position and pose determination as disclosed by Voss-Wolffet al et al in 0002-0003.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over NGO DINH et al (US 2019/0385302) in view of Tirupathi et al (US 2023/0206431) as applied to claim 19 above, and further in view of Wolter et al (US 2003/0158470).
Claim 20:
NGO DINH et al and Tirupathi et al teach all the subject above, but the following is taught by Wolter et al:
The system of claim 19, wherein the plurality of excitation sources is tuned to emit electromagnetic radiation within a narrow waveband of 20 nm or less; wherein the plurality of multispectral sources is tuned to emit electromagnetic radiation within a narrow waveband of 20 nm or less; and wherein the plurality of excitation sources is tuned to emit electromagnetic radiation within a near infrared range of the electromagnetic spectrum (0051 teaches 20 or less nm imaging for differentiating portion or spectra between normal and abnormal tissues; and claim 1-3 teaches imaging of tissues with false color overlay).
NGO DINH et al and Tirupathi et al and Wolter et al are in the field of image analysis, especially in medical/tissue imaging with different color overlay such that the combine outcome is predictable.
Therefore it would have been obvious to one having ordinary skill before the effective filing date to modify NGO DINH et al and Tirupathi et al by Wolter et al such imaging would provide faster would proportionately reduce the examination times and blurring, as compared to push broom or other technologies that collect full spectral data as disclosed by Wolter et al in paragraph 0051.
Allowable Subject Matter
Claim 8, and dependent claims 10 and 21, are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. At the time of examination unable to find prior art teaching “wherein the one or more threshold ranges comprises one or more of: a first threshold range, wherein pixel values within the first threshold range indicate the corresponding pixel has no likelihood of comprising the target object; a second threshold range, wherein pixel values within the second threshold range indicate the corresponding pixel has a low likelihood of comprising the target object; a third threshold range, wherein pixel values within the third threshold range indicate the corresponding pixel has a medium likelihood of comprising the target object; and a fourth threshold range, wherein pixel values within the fourth threshold range indicate the corresponding pixel has a high likelihood of comprising the target object.”
Claim 15, and dependent claims 16-17, are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. At the time of examination unable to find prior art teaching “an inertial measurement unit (IMU) associated with the image sensor, wherein the IMU measures movement of the image sensor in real-time; and wherein the image signal processor further receives sensor data from the IMU; and wherein the adaptive persistence algorithm comprises stabilizing movement of the false color overlay at least based on the sensor data from the IMU.”
Claim 22 is allowed.
Claims 23 is allowed.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Pinho (US 2014/0187968) teaches SPECTRAL IMAGING WITH A COLOR WHEEL – paragraph 0075 allow for real-time video projection of false-color overlay images onto the target being examined. Although the main example is described with respect to tissue oxygenation in the medical arts, the present invention may find use in other fields where real-time overlay images are needed.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TSUNG-YIN TSAI whose telephone number is (571)270-1671. The examiner can normally be reached 7am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bhavesh Mehta can be reached at (571) 272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TSUNG YIN TSAI/Primary Examiner, Art Unit 2656
Prosecution Insights