DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 10/28/2025 has been entered.
Response to Amendment
This Office Action is in response to the Amendment filed on 08/25/2025.
Status of the Claims:
Claim(s) 1, 9 and 17 has/have been amended.
Claim(s) 5-6 and 13-14 has/have been canceled.
Claim(s) 21-24 has/have been newly added.
Claim(s) 1-4, 7-12 and 15-24 is/are pending in this Office Action.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1, 9 and 17 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 7-12 and 15-24 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2022/0303446 to Smith et al. (hereinafter Smith) in view of US 2020/0138534 to Garcia Kilroy et al. (hereinafter Garcia).
Regarding claim 1, Smith teaches an apparatus comprising:
one or more processors (processors 1560, see Fig 15 and par. [0254]); and
memory storing executable instructions that, when executed by the one or more processors (disk drives 1580, see Fig. 15 and par. [0254]), cause the apparatus to:
determine, for an image captured by an image capture system, auto-exposure-related characteristics of pixel units into which the image is divided (process 100 divides the image into pixel groups (120), computes the average luma for each group (130) and performs automatic exposure control (AEC) on the average luma pixel group values (140), see Fig. 1A and par. [0063, 0065-066]); and
update, based on the auto-exposure-related characteristics determined for the pixel units and on eye tracking data for a viewer viewing the image, one or more auto-exposure parameters for use by the image capture system to capture an additional image within an image sequence (the AEC method may include applying weights to the average luma pixel group and a weight array may be created using eye gaze, see pars. [0066-0067]).
wherein the updating comprises:
determining a gaze direction based on the eye tracking data (To implement the eye gaze weight array 690, a system may include one or more eye capture devices. An eye capture device may be used to capture one or more images and/or one or more videos of one or more eyes (e.g., eyes 660) of a user. The images and/or videos may be further processed to determine a gaze 670 of the eyes 660., see par. [0101]);
assigning weight values to the pixel units based on the gaze direction and distances of the pixel units to the gaze direction (For example, the center of the location that the gaze 670 is looking may have the highest weight. Pixel groups around the center may also have the highest weight. But, as distance increases from the center, the pixel groups may decrease in amount of weight, see par. [0102]); and
updating the one or more auto-exposure parameters based on the auto-exposure characteristics determined for the pixel units and on the weight values assigned to the pixel units (perform AEC 140 based on the determine average luma pixel group and weight array, see pars. [0066-0067]).
Smith fails to clearly specify “wherein the eye tracking data is received from an eye tracking system integrated with a computer-assisted medical system comprising one or more robotic manipulator arms and a user control apparatus, the user control apparatus configured to provide the image to the viewer and facilitate control by the viewer of the one or more robotic manipulator arms”.
However, Garcia teaches a surgical robotic system 100 (see Fig. 1) “wherein the eye tracking data is received from an eye tracking system integrated with a computer-assisted medical system (the surgical robotic system 100 comprises a user console 110 that provides eye gaze tracking, see pars. [0041, 0050-0051]) comprising one or more robotic manipulator arms (the surgical robotic system 100 comprises a surgical robot 120 having one or more surgical robotic arms 122, see Fig. 1 and par. [0019]) and a user control apparatus (user console 110, see Fig. 1 and pars. [0019-0020]), the user control apparatus configured to provide the image to the viewer and facilitate control by the viewer of the one or more robotic manipulator arms (user console 110 comprise an open display 118 and user interface devices 116, see par. [0020])”.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the above camera, as taught by Smith, by implementing them in the surgical system as taught by Garcia.
One of ordinary skill in the art would have been motivated to do this modification in order to provide a robotically-assisted surgical system, which is a software-controlled, electro-mechanical system, designed for surgeons to perform minimally-invasive surgery. The surgical robotic system may be comprised of three major sub-systems: a surgeon subsystem-the user console (surgeon console or surgeon bridge), a central control subsystem-the control tower, and a patient subsystem-the table and robotic arms as suggested by Garcia (see par. [0006]).
Regarding claim 2, Smith in view of Garcia teaches the apparatus of claim 1, wherein the image capture system includes an endoscopic image capture device configured to capture the image sequence (the surgical robotic system can be used with an endoscope, compatible endoscopic instruments, see Garcia par. [0018]).
Regarding claim 3, Smith in view of Garcia teaches the apparatus of claim 1, wherein the one or more auto-exposure parameters include one or more of:
an exposure time parameter;
a shutter aperture parameter; or
a luminance gain parameter (In some examples, a setting (e.g., an exposure setting and/or a gain setting) may be applied to one or more portions based upon the weight array. In such examples, the exposure setting may be a shutter speed, an ISO speed, or any combination thereof. In some examples, the shutter speed may be a global shutter or a rolling shutter, see Smith pars. [0070, 0072]).
Regarding claim 4, Smith in view of Garcia teaches the apparatus of claim 1, wherein:
the instructions, when executed by the one or more processors, cause the apparatus to:
determine auto-exposure targets for the pixel units into which the image is divided (FIG. 1B illustrates an example of a process 160 for determining how to update one or more settings of a content capture device. The process may include comparing a target luma value to a luma value (e.g., a weighted luma average) (step 170), see Smith par. [0073]), and
determine, based on the auto-exposure-related characteristics and auto-exposure targets for the pixel units into which the image is divided, auto-exposure gains for the pixel units into which the image is divided (If the exposure setting is at the maximum allowed exposure setting, a gain setting of the one or more settings may be increased (e.g., incrementally increased) up to a maximum allowed gain setting (steps 174 and 178), see Smith pars. [0019-0020]); and
the updating the one or more auto-exposure parameters is further based on the auto-exposure gains for the pixel units (see blocks 184 and 186 based on gain setting, see Smith Fig. 1B).
Regarding claim 7, Smith in view of Garcia teaches the apparatus of claim 1, wherein the eye tracking data comprises real-time eye tracking data received from the eye tracking system that tracks one or more eyes of the viewer viewing the image (a system may include one or more eye capture devices. An eye capture device may be used to capture one or more images and/or one or more videos of one or more eyes (e.g., eyes 660) of a user. The images and/or videos may be further processed to determine a gaze 670 of the eyes 660., see Smith par. [0101]).
Regarding claim 8, Smith in view of Garcia teaches the apparatus of claim 7, wherein the user control apparatus comprises a set of master controls configured to be manipulated by the viewer to teleoperate the one or more robotic manipulator arms (as shown in the exemplary user console 110, a surgeon siting in the seat 112 and viewing the open display 118 may manipulate the pedals 114 and/or handheld user interface devices 116 to remotely control the robotic arms 122 and/or surgical instruments mounted to the distal ends of the arms 122, see Garcia Fig. 1 and par. [0020]).
Regarding independent claim(s) 9-12 and 15-16, claim(s) has/have limitations similar to those treated in the above rejection(s), and are met by the references as discussed above. Claim 9 however also recites the following limitations taught by Smith in view of Garcia “an image capture device configured to capture an image sequence of a scene (video capture, see Smith par. [0002] and provides video output, see Garcia par. [0023]);
a display device configured to present the image sequence (monitor 1570, see Smith Fig. 15, display 18, see Garcia Fig. 1).
Regarding independent claim(s) 17-20, claim(s) is/are drawn to the method used by the corresponding apparatus in claim(s) 1-8 and is/are rejected for the same reasons used above.
Regarding claim 21, Smith in view of Garcia teaches the apparatus of claim 1, wherein the gaze direction comprises an average gaze direction of the viewer over a defined period of time (the period of time of gaze can be defined as for example for safety reasons, see Garcia par. [0051]).
Regarding claim 22, Smith in view of Garcia teaches the apparatus of claim 1, wherein assigning the weight values to the pixel units based on the distances of the pixel units to the gaze direction is further based on distances of the pixel units to a center of the image (In some examples, similar to the center metering weight array 230, weights of the eye gaze weight array 690 may decrease as distance increases from a location that the gaze 670 is looking. For example, the center of the location that the gaze 670 is looking may have the highest weight. Pixel groups around the center may also have the highest weight. But, as distance increases from the center, the pixel groups may decrease in amount of weight. See Smith par. [0102]).
Regarding claim 23, Smith in view of Garcia teaches the apparatus of claim 1, wherein assigning the weight values to the pixel units based on the distances of the pixel units to the gaze direction is further based on relationships between the pixel units and an object depicted in the image (To illustrate the eye gaze weight array 690, the gaze 670 may be pointing to a second set of pixel groups 630, which may include a second object 632. The pixel groups that include the second object 632 may be assigned the highest weight, see Smith par. [0103]).
Regarding claim 24, Smith in view of Garcia teaches the apparatus of claim 23, wherein the object depicted in the image comprises a shaft of an instrument coupled to one of the one or more robotic manipulator arms (upon a distance away from the second object 632 and/or based upon other objects that are identified. as combined any object gazed can be assigned weight, see Smith par. [0103], as combined Garcia’s robotic arms 112 are present in the scene).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANGEL L GARCES-RIVERA whose telephone number is (571)270-7268. The examiner can normally be reached Mon-Fri 9AM-5PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sinh Tran can be reached at 571-727-7564. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANGEL L GARCES-RIVERA/Examiner, Art Unit 2637
/SINH TRAN/Supervisory Patent Examiner, Art Unit 2637