DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This Office Action is in response to the Preliminary Amendment filed on 11/27/2023.
Status of the Claims:
Claim(s) 18-29 has/have been canceled.
Claim(s) 1-17 and 30-32 is/are pending in this Office Action.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-8, 10-17 and 30-32 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2019/0099081 to HORESH et al. (US version of IDS cited reference EP 3469978, hereinafter HORESH) in view of US 2014/0267626 to Lilagan et al. (hereinafter Lilagan).
Regarding claim 1, HORESH teaches an apparatus (imaging device 100, see Fig. 1) comprising:
one or more processors (processing unit 108, see Fig. 1); and
memory storing executable instructions that, when executed by the one or more processors (memory 112, see Fig. 1), cause the apparatus to:
obtain a depth map corresponding to an image frame captured by an image capture system in accordance with an auto-exposure parameter set to a first setting, the depth map indicating depth values for pixel units of the image frame (a distance sensor configured to measure a distance to the fluorescing sample and provide the measured distance to the processing unit, wherein the processing unit is configured to determine a distance between the optical detector and the fluorescing sample from the measured distance, match the determined distance to one of a plurality of distance values, see par. [0012]);
obtain an object map corresponding to the image frame, the object map indicating which of the pixel units depict an object of a predetermined object type (image frame obtained contains tissue sample 130, hence the Tissue sample 130 is the object type to be obtained, see par. [0036]);
determine, based on the depth map and the object map, an auto-exposure gain associated with the image frame (obtain the respective gain values range stored in association with the matched distance value of the sample, see pars. [0012 and 0037]); and
But HORESH fails to clearly specify “determine, based on the auto-exposure gain, a second setting for the auto-exposure parameter, the second setting configured to be used by the image capture system to capture a subsequent image frame”.
However, Lilagan teaches and image control method that “determine, based on the auto-exposure gain (as combined HORESH provides the auto exposure gain as indicated above), a second setting for the auto-exposure parameter, the second setting configured to be used by the image capture system to capture a subsequent image frame (adjust a control element as a function of a depth value of the target area, see par. [0031]. The control element can be the automatic exposure functions on a camera, see par. [0056], implied to be applied to the subsequent image frame capture.)”.
References are analogous art because they are from the same field of endeavor and/or are reasonably pertinent to the particular problem with which the applicant was concerned because they relate to imaging device control.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the above imaging device, as taught by HORESH, by combining the teachings of Lilagan.
One of ordinary skill in the art would have been motivated to do this modification in order to provide assistance in achieving a desirable adjustment of the control element that is a function of a depth value of a target area relative to the image capturing device as suggested by Lilagan (see par. [0001]).
Regarding claim 2, HORESH in view of Lilagan teaches the apparatus of claim 1, wherein:
the image capture system comprises a stereoscopic image capture system and the image frame comprises a stereoscopic image frame that includes a left-side version and a right-side version of the image frame (uses a stereoscopic camera with left and right optical lens 101, 102, see Lilagan par. [0038]); and
the obtaining the depth map includes generating the depth map based on differences between corresponding features that are depicted in both the left-side and right-side versions of the image frame (depth at which the left and right images are focus, see Lilagan par. [0038]).
Regarding claim 3, HORESH in view of Lilagan teaches (original) The apparatus of claim 1, wherein:
the depth map includes, for each pixel unit represented in the depth map, a depth value and a confidence value associated with the depth value (Processing unit 108 is operative to query the multiple distance values stored in memory unit 112 using the determined distance as a search query, and identify the associated range of gain values, accordingly. On finding a match, processing unit 108 is operative to apply an automatic gain controller (AGC) algorithm that determines which gain value within the gain values range to apply, by optimizing the brightness of the display image pixels subsequently displayed on UI 104 within the gain limits imposed by the matched gain values range. The AGC operation can be considered a confidence value, see HORESH par. [0038]); and
the determining the auto-exposure gain is based on both the depth values and the confidence values included in the depth map (see HORESH par. [0038]).
Regarding claim 4, HORESH in view of Lilagan teaches the apparatus of claim 1, wherein:
the image capture system includes an endoscopic image capture device configured to capture internal imagery of a body on which a medical procedure is performed and the image frame depicts a scene internal to the body during the medical procedure (medical robotic system 7000 uses an endoscope, see Lilagan Figs. 7-10 and par. [0052]);
the predetermined object type comprises tissue of the body and the object comprises a tissue sample from the body that is more proximate to the endoscopic image capture device than is other content at the scene (plural target areas can be imaged and selected, any one specific can be selected, see Lilagan par. [0042]); and
the determining the auto-exposure gain includes allotting more weight to the tissue sample than is allotted to the other content at the scene (a specific sample can be selected and hence uses to determine the gain, see Lilagan par. [0042]).
Regarding claim 5, HORESH in view of Lilagan teaches the apparatus of claim 1, wherein:
the image capture system includes an endoscopic image capture device configured to capture internal imagery of a body on which a medical procedure is performed and the image frame depicts a scene internal to the body during the medical procedure (medical robotic system 7000 uses an endoscope, see Lilagan Figs. 7-10 and par. [0052]);
the predetermined object type comprises instrumentation for tissue manipulation and the object comprises an instrument at the scene that is more proximate to the endoscopic image capture device than is other content at the scene (when tools, which are being used to interact with objects of the surface topology, appear above the surface topology, depth values for the tools which occlude part of the surface topology may be included or excluded from the calculation, see Lilagan par. [0037]); and
the determining the auto-exposure gain includes allotting less weight to the instrument than is allotted to the other content at the scene (the tools can be excluded, see Lilagan par. [0037]).
Regarding claim 6, HORESH in view of Lilagan teaches the apparatus of claim 1, wherein:
the image capture system includes an endoscopic image capture device configured to capture internal imagery of a body on which a medical procedure is performed and the image frame depicts a scene internal to the body during the medical procedure (medical robotic system 7000 uses an endoscope, see Lilagan Figs. 7-10 and par. [0052]);
the predetermined object type comprises instrumentation for tissue manipulation and the object comprises an instrument at the scene (when tools, which are being used to interact with objects of the surface topology, appear above the surface topology, depth values for the tools which occlude part of the surface topology may be included or excluded from the calculation, see Lilagan par. [0037]); and
the obtaining the object map includes obtaining instrument tracking data and generating the object map based on the instrument tracking data (depth values for the tools which occlude part of the surface topology may be included or excluded from the calculation, hence are tracked in order to include or exclude the tool, see Lilagan par. [0037]).
Regarding claim 7, HORESH in view of Lilagan teaches the apparatus of claim 6, wherein the instrument tracking data is based on kinematic parameters used to control the instrument at the scene and is obtained from a computer-assisted medical system controlling the instrument using the kinematic parameters (uses kinematics, see Lilagan par. [0052]).
Regarding claim 8, HORESH in view of Lilagan teaches the apparatus of claim 6, wherein the instrument tracking data is based on computer-vision tracking of the instrument and is obtained from a computer-assisted medical system performing the computer-vision tracking of the instrument (the Surgeon views real-time images of a work site in three-dimensions ("3D") on a stereo vision display 45 of the Console. A stereoscopic endoscope 37 (having left and right cameras for capturing left and right stereo views) captures stereo images of the work site. The processor 43 processes the stereo images so that they may be properly displayed on the stereo vision display 45.see Lilagan par. [0059]).
Regarding claim 10, HORESH in view of Lilagan teaches the apparatus of claim 1, wherein:
the instructions further cause the apparatus to generate, based on the depth map, a specular threshold map corresponding to the image frame, the specular threshold map indicating specular thresholds for the pixel units of the image frame (generates the specular depth map, see HORESH par. [0036]); and
the determining the auto-exposure gain associated with the image frame is further based on the specular threshold map (the specular depth map to obtain the respective gain values range stored in association with the matched distance value of the sample, see HORESH pars. [0012 and 0036-0037]).
Regarding claim 11, HORESH in view of Lilagan teaches the apparatus of claim 1, wherein the determining the auto-exposure gain associated with the image frame includes designating, for each pixel unit of the image frame, a respective weight value determined based on:
a depth of the pixel unit according to the depth map, an object correspondence of the pixel unit according to the object map, and a spatial status of the pixel unit within the image frame (obtain the respective gain values range stored in association with the matched distance value of the sample, see HORESH pars. [0012 and 0037] and as combined with Lilagan the user may define the center of the target area using a gaze tracker, see Lilagan par. [0035]).
Regarding claim 12, HORESH in view of Lilagan teaches the apparatus of claim 11, wherein the spatial status of the pixel unit indicates a centrality of a position of the pixel unit within the image frame (the center of the target area indicates the centrality of the gaze point, see Lilagan par. [0035]).
Regarding claim 13, HORESH in view of Lilagan teaches the apparatus of claim 11, wherein the spatial status of the pixel unit indicates a proximity of a position of the pixel unit to a gaze area within the image frame to which a user is determined to be focusing attention ((the center of the target area indicates the centrality of the gaze point, see Lilagan par. [0035]).
Regarding independent claim(s) 14, claim(s) is/are drawn to the method used by the corresponding apparatus in claim(s) 1 and 10 and is/are rejected for the same reasons used above.
Regarding claim(s) 15-17, claim(s) is/are drawn to the method used by the corresponding apparatus in claim(s) 2-4 and is/are rejected for the same reasons used above.
Regarding independent claim(s) 30, claim(s) is/are drawn to the system corresponding apparatus in claim(s) 1 and 10 and is/are rejected for the same reasons used above. Claim 30 however also recites the following limitations “an illumination source configured to illuminate tissue within a body during a performance of a medical procedure (HORESH teaches the use of light source 116 in Fig. 1B and Lilagan teaches the use of an illuminator for the camera, see Lilagan par. [0047])”.
Regarding claim 31, HORESH teaches the system of claim 30, wherein the generating the specular threshold map is further based on an illuminance fall-off map that indicates how illuminance within the internal view of the body depicted by the image frame decreases as a function of distance from the illumination source (Distance sensor 110 may emit a light beam onto the tissue, and detect any of specular or diffuse reflectance off tissue sample 130. Processing unit 108 may analyze one or more properties of the reflectance to determine the distance between distance sensor 110 and tissue sample 130, see HORESH par. [0036]).
Regarding independent claim(s) 32, claim(s) is/are drawn to the system corresponding apparatus in claim(s) 1 and is/are rejected for the same reasons used above.
Allowable Subject Matter
Claim(s) 9 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 9, none of the prior art cited alone or in combination provides the motivation to teach the following claimed limitations, with emphasize that it is each claim, taken as a whole, including the interrelationships and interconnections between various claimed elements make them allowable over the prior art of record, the apparatus of claim 1, wherein:
the determining the auto-exposure gain associated with the image frame includes:
determining, based on the image frame, a frame auto-exposure value and a frame auto-exposure target for the image frame, and
computing, based on the frame auto-exposure value and the frame auto-exposure target, the auto-exposure gain associated with the image frame;
the determining of at least one of the frame auto-exposure value or the frame auto-exposure target is based on at least one of the depth map or the object map; and
the instructions further cause the apparatus to update the auto-exposure parameter to the second setting such that the subsequent image frame will be captured in accordance with the auto-exposure parameter set to the second setting.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANGEL L GARCES-RIVERA whose telephone number is (571)270-7268. The examiner can normally be reached Mon-Fri 9AM-5PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sinh Tran can be reached at 571-727-7564. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANGEL L GARCES-RIVERA/Examiner, Art Unit 2637
/SINH TRAN/Supervisory Patent Examiner, Art Unit 2637