DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are pending.
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claim(s) 1-3, 5-12 and 14-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Puigvert et al (WO2024260942A1) in view of Sonnenborg et al (US20200405124A1).
Regarding claims 1, 10 and 17, Puigvert teaches a system comprising:
an endoscope including a camera module and a light source module, and
(Puigvert, "As disclosed above, the invention relates to a method for obtaining data of a space (1) illuminated by a light source comprised by a joint camera-light source system (2), in which a neural network (3) is trained using the illumination decline profile of every pixel in a set of training 2D image data (4).", [p13:10-15]; "its application is particularly relevant for exploring or inspecting shallow, inaccessible spaces (1) such as cavities or pipes. Because of this, it will be shown, in the following, and according to Figs. 1-10, a particular embodiment of the method of the invention applied to the field of endoscopic imaging, described for illustrative, but not limiting purposes.", [p13:15-25]; Sonnenborg; "In addition to the exit holes, a camera sensor, such as a CMOS sensor or any other image capturing device, as well as one or several light sources, such as light emitting diodes (LEDs) or any other light emitting devices, may be placed in the tip part 108.", [0074]; Puigvert teaches a joint camera-light source system used for endoscopic imaging. Sonnenborg provides the conventional structural disclosure of an endoscope tip carrying both a CMOS camera and LEDs)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the endoscope tip structure with a camera sensor and LEDs of Sonnenborg into the joint camera-light source system for endoscopic imaging of Puigvert in order to provide a practical hardware implementation for Puigvert's illumination-decline method. The combination of Puigvert and Sonnenborg also teaches other enhanced capabilities.
a console including a processor to:
(Sonnenborg, "Data processing operations closely related to e.g. operation of the camera sensor, such as reading out image data, may be performed in the endoscope itself, while more complex data processing operations, requiring more computational power, may be made in the monitor 200. Since most of the more complex data processing operations are related to image data processing an Image Signal Processor (ISP) may be provided in the monitor and used for image data processing operations.", [0078]; Puigvert; "first, the camera-light source system (2) obtains first 2D image data of a space (1) illuminated by the light source (10) comprised by said camera-light source system (2). The first 2D image data are delivered to a neural network (3) which calculates a depth estimate (5) and an albedo estimate (6) for each pixel", [p13:25-30]; rationale: Sonnenborg teaches the split architecture where heavy image processing resides in a monitor/console ISP, Puigvert teaches delivering the captured image to a processor/neural network for subsequent steps)
receive, from the endoscope, an image of a scene, obtained by the camera module, that is illuminated by the light source;
(Puigvert, "first, the camera-light source system (2) obtains first 2D image data of a space (1) illuminated by the light source (10) comprised by said camera-light source system (2).", [p13:25-30]; Sonnenborg; "method for processing image data obtained using a medical visual aid system comprising an endoscope and a monitor, wherein the endoscope is configured to be inserted into a body cavity and comprises an image capturing device for capturing image data and a light emitting device", [0007]; both Puigvert and Sonnenborg describe capturing image data with an endoscopic camera while the integrated light source illuminates the cavity)
perform image linearization for the image based on a stored gamma curve;
(Sonnenborg, "The non-linear scaling model may be a non-linear intensity scaling model, such as a non-linear gamma correction model.", [0017]; "The non-linear intensity scaling model may be configured to, in the image displayed on the monitor, increase the contrast in the dark parts of the image and reduce the contrast in the parts of the image having an intermediate light intensity in a manner whereby pixels having low pixel intensity values are scaled up significantly and pixels having mid-range pixel intensity values are only slightly adjusted or not adjusted at all.", [0018]; Puigvert; "The parameter g denotes the gain applied by the camera and y is the gamma correction commonly applied by cameras to adapt images to human perception.", [p15:25-30]; Puigvert acknowledges gamma correction as part of the imaging pipeline but does not implement it as a stored model; Sonnenborg fills the gap by teaching a stored non-linear gamma correction model applied to endoscopic image data. Incorporating Sonnenborg into Puigvert to linearize the input before illumination-decline calculations would have been obvious to improve photometric accuracy)
estimate tissue colors in the image based on stored tissue color estimation data;
(Puigvert, "In this way, the neural network (3) can estimate the depth (5), albedo (6) and surface orientation (7) of each pixel of input 2D image data (8) allowing for a posterior 3D reconstruction (9) of the space (1) depicted in said 2D input image data (8), including its structures, shapes and colors.", [p13:10-20]; "pi represents the albedo of the surface at that point.", [p15:25-30]; albedo is the intrinsic reflectivity/color of tissue; Puigvert's network learns and stores albedo parameters from training data, which is the stored tissue color estimation data)
correct the image for incident light intensity based on the estimated tissue color;
(Puigvert, "the illumination of a surface point comprised by said space (1) decays with a known function of the distance from said surface point to the light source (10).", [p14:1-5]; "for each pixel, the following rendering equation can be written:... pi represents the albedo of the surface at that point. The parameter g denotes the gain applied by the camera and y is the gamma correction commonly applied by cameras to adapt images to human perception. The resulting l(dj, pi, g) is the color captured by the camera.", [p10:20-30]; rendering equation separates measured intensity into depth-dependent illumination decline and albedo, thereby correcting incident intensity using the estimated tissue albedo)
correct the image for light beam pattern intensity, based on a calibration image, to obtain corrected light intensity for the image;
(Puigvert, "the method of the invention requires precise photometric and geometric calibration of the camera-light source system (2) or endoscope. This calibration method (16) involves the acquisition of calibration 2D image data (17) from a known calibration pattern as well as solving an optimization problem to obtain the best values for the photometric and geometric calibration parameters (18).", [p15:30-p16:5]; "Hence, a spotlight model (SLS) is adopted according to the configuration shown in Fig.2, in which, for surface point xi with off-axis angle ψi, the radiance reads...", [p15:1-5]; acquiring calibration image data from a known pattern to derive photometric parameters that model the non-uniform spotlight beam (radial attenuation), which is then used to correct per-pixel intensity)
generate a depth map for the image based on the corrected light intensity; and
(Puigvert, "In this way, the neural network (3) can estimate the depth (5), albedo (6) and surface orientation (7) of each pixel of input 2D image data (8)", [p13:10-15]; "first synthesized 2D image data (13) based on illumination decline, i.e., the fact that, in a space (1) illuminated by a direct light source (10), the illumination of a surface point comprised by said space (1) decays with a known function of the distance from said surface point to the light source (10).", [p14:1-5]; generating a per-pixel depth estimate directly from the illumination-decline-corrected intensity via the neural network)
provide a measurement of an object in the image based on the depth map.
(Puigvert, "allowing for a posterior 3D reconstruction (9) of the space (1) depicted in said 2D input image data (8), including its structures, shapes and colors.", [p13:10-20]; a 3D reconstruction derived from the depth map inherently provides geometric measurements of structures within the endoscopic scene; obtaining linear measurements from such reconstructions was well-known)
Regarding claims 2 and 11, the combination of Puigvert and Sonnenborg teaches its/their respective base claim(s).
The combination further teaches the system of claim 1, wherein the processor is further configured to store in a memory of the console:
the gamma curve for the camera module, and
the calibration image for the light source module.
(Sonnenborg, "the first non-linear scaling model and the second non-linear scaling model are stored in the monitor.", [0034]; Sonnenborg, "The non-linear scaling model may be a non-linear intensity scaling model, such as a non-linear gamma correction model.", [0043]; Puigvert, "This calibration method (16) involves the acquisition of calibration 2D image data (17) from a known calibration pattern as well as solving an optimization problem to obtain the best values for the photometric and geometric calibration parameters (18).", [p15:35-p16:5]; Sonnenborg teaches storing gamma-type non-linear models in the monitor/console, Puigvert teaches acquiring and retaining calibration image data for the light source. Together they render obvious storing both items in console memory)
Regarding claims 3 and 12, the combination of Puigvert and Sonnenborg teaches its/their respective base claim(s).
The combination further teaches the system of claim 2, wherein the calibration image includes a beam pattern for the light source module at a known distance.
(Puigvert, "This calibration method (16) involves the acquisition of calibration 2D image data (17) from a known calibration pattern", [P15:35-p16:1]; "Hence, a spotlight model (SLS) is adopted according to the configuration shown in Fig.2, in which, for surface point xi with off-axis angle ψi, the radiance reads...", [p15:1-5]; "where σ0 is the maximum radiance and R(ψi) is the radial attenuation controlled by a spread factor µ. It is to be noted that the light reaching the surface is subject to the inverse-square law and decays with the propagation distance from xi (light source position) to xj.", [p15:5-10]; the calibration uses a known pattern at a fixed geometry and explicitly models the light-source beam as a spotlight with radial attenuation versus distance, which is the beam-pattern calibration image)
Regarding claim 5, the combination of Puigvert and Sonnenborg teaches its/their respective base claim(s).
The combination further teaches the system of claim 1, further comprising:
a data cable configured to transfer data between the endoscope and the console.
(Sonnenborg, "The image data captured by the camera sensor and optionally also other data captured by other sensors placed in the tip part can be transferred via a connection cable 114 and a connector 116 to a monitor 200 illustrated in FIG. 2.", [0076]; teaching the conventional wired link between endoscope and monitor/console)
Regarding claims 6, 14 and 19, the combination of Puigvert and Sonnenborg teaches its/their respective base claim(s).
The combination further teaches the system of claim 1, wherein, when generating the depth map, the processor is further configured to:
calculate a depth value for each pixel in the image.
(Puigvert, "the neural network (3) can estimate the depth (5), albedo (6) and surface orientation (7) of each pixel of input 2D image data (8)", [p13:10-15]; computing a depth estimate per pixel, which is the depth map)
Regarding claims 7 and 15, the combination of Puigvert and Sonnenborg teaches its/their respective base claim(s).
The combination further teaches the system of claim 1, wherein, when receiving the image of a scene, the processor is further configured to receive:
a camera gain value and a light intensity value at a time of capturing the image.
(Puigvert, "The parameter g denotes the gain applied by the camera and y is the gamma correction commonly applied by cameras to adapt images to human perception.", [p15:25-30]; "where σ0 is the maximum radiance and R(ψi) is the radial attenuation controlled by a spread factor µ. It is to be noted that the light reaching the surface is subject to the inverse-square law and decays with the propagation distance from xi (light source position) to xj.", [p15:5-10]; "the auto-gain values of the camera-light source system (2) or endoscope are not known, so radiance measurements of the camera are unitless. Thus, g = 1 and σ0 = 1 can be arbitrarily set", [p16:5-10]; rendering model requires both camera gain (g) and source radiance (σ0) as inputs; the reference discusses receiving/estimating those values at capture time)
Regarding claim 8, the combination of Puigvert and Sonnenborg teaches its/their respective base claim(s).
The combination further teaches the system of claim 1, wherein the processor is further configured to store in a memory of the console:
the tissue color estimation data for known biological tissues.
(Puigvert, "neural network (3) is trained using the illumination decline profile of every pixel in a set of training 2D image data (4).", [p13:10-15]; "pi represents the albedo of the surface at that point.", [p15:25-30]; training on endoscopic image sets stores albedo parameters that encode tissue color for known biological tissues within the network weights/memory)
Regarding claims 9, 16 and 20, the combination of Puigvert and Sonnenborg teaches its/their respective base claim(s).
The combination further teaches the system of claim 1, wherein, when estimating the tissue colors, the processor is further configured to:
estimate a tissue color for each pixel in the image.
(Puigvert, "the neural network (3) can estimate the depth (5), albedo (6) and surface orientation (7) of each pixel of input 2D image data (8)", [p13:10-15]; albedo is the per-pixel tissue color estimate)
Regarding claim 18, the combination of Puigvert and Sonnenborg teaches its/their respective base claim(s).
The combination further teaches the non-transitory computer-readable storage medium of claim 17, wherein the instructions are further to:
store, in a memory, the gamma curve for the camera module, the calibration image for the light source module, and the tissue color estimation data for known biological tissues.
(Sonnenborg, "the first non-linear scaling model and the second non-linear scaling model are stored in the monitor.", [0034]; "The non-linear scaling model may be a non-linear intensity scaling model, such as a non-linear gamma correction model.", [0017]; Puigvert, "This calibration method (16) involves the acquisition of calibration 2D image data (17) from a known calibration pattern", [p15:35-p16:1]; "neural network (3) is trained using the illumination decline profile of every pixel in a set of training 2D image data (4).", [p13:10-15]; the combination teaches storing gamma models (Sonnenborg), calibration images (Puigvert), and learned tissue-albedo data (Puigvert))
Claim(s) 4 and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Puigvert et al (WO2024260942A1) in view of Sonnenborg et al (US20200405124A1) and further in view of Li et al (WO2022253093A1).
Regarding claims 4 and 13, the combination of Puigvert and Sonnenborg teaches its/their respective base claim(s).
The combination does not expressly disclose but Li teaches the system of claim 1, wherein, when performing the image linearization, the processor is further configured to:
convert the image from color to grayscale.
(Li, "method for processing images in an endoscope observation video, the method for processing images in the endoscope observation video includes:", p3; "Convert each frame of video image to a grayscale image; In the grayscale image of each frame of video image, according to the effective grayscale value of the pre-determined intestinal observation area, the pixel points in the grayscale image that are smaller than the effective grayscale value are", p3; Li teaches the conventional preprocessing step for endoscopic video of converting each color frame to grayscale before further analysis. Sonnenborg, "The non-linear scaling model may be a non-linear intensity scaling model, such as a non-linear gamma correction model.", [0017]; "the first non-linear scaling model and the second non-linear scaling model are stored in the monitor.", [0034]; Sonnenborg teaches performing linearization with a stored gamma model in an endoscopic monitor)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the color-to-grayscale conversion for endoscopic video frames of Li into the image linearization process using stored gamma models of Sonnenborg in order to simplify intensity processing and reduce data volume before applying the non-linear gamma correction. The combination of Puigvert, Sonnenborg and Li also teaches other enhanced capabilities.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANXUN YANG whose telephone number is (571)272-9874. The examiner can normally be reached on MON-FRI: 8AM-5PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272- 1000.
/JIANXUN YANG/
Primary Examiner, Art Unit 2662 7/12/2026
Prosecution Insights