DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of claims: claims 21, 23-24, 26, 28-43. Claims 1-20, 22, 25 and 27 are cancelled.
Response to Arguments
Applicant's arguments filed 6/1/2026 have been fully considered but they are not persuasive.
Applicant remark – (page 11-12) Applicant argued regarding lack of teaching of new claim amendment dated 6/1/2026. Please see claim amendment for new claim amendment and claim elements for more detail.
Examiner response – Examiner respectfully disagree. Wolf et al addresses some of the claim elements and with the combine teaching of MOTTRAM et al specifically paragraph 0036 and 0049 teaches the instant invention as a whole. Please see the Office Action below for detail.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 21, 23-24, 26, 28-43 are rejected under 35 U.S.C. 103 as being unpatentable over Wolf et al (US 2021/0012868) in view of MOTTRAM et al (US 2023/0149101).
Claim 21, similarly claim 29 and 40:
Wolf et al (US 2021/0012868) teach the following subject matter:
An inference device connected between a surgical robot and a console controlling the surgical robot, the inference device comprising:
one or more processors; and a storage storing instructions causing any of the one or more processors to execute processing of (figure 14 and 0290 teaches of structure of processor, storage/database/memory, non-transitory computer readable medium to contain and execute instruction):
acquiring an operative field image shot by an imaging unit of the surgical robot (0285 teaches surgical event with image frames, where 0537,0538 and 0595 detail the use of surgical robot with camera for capturing and transmitting images, video footage over network directly wired and/or wirelessly; 0090 teaches camera for moving to object with field of view and function such as adjust zoom, direction, speed);
performing an inference process on the acquired operative field image (0285 detail inferred base on set of images from surgical area (operative field image)); [[and]]
inferring at least one of a position of an object to be recognized in the operative field image and an event occurring in the operative field image (0080 detail inferred output with classification of item depicted in the image (operative field image) with output including inferred value, size, volume, age, cost, product);
computing an area of the recognized object (0080 detail inferred value such as volume (area or 3D object); paragraph 0172 detail calculating characteristic with numerical value: a dimension, a length, an area, a volume);
generating the control information in order to move the imaging unit following a portion in which the computed area increases or decreases (0086 detail control application positioning camera to capture video/image ROI (region of interest) with identify structure, surgical tool, hand of surgeon, motion, particular location, tracking location with camera 115; 0087 teaches camera to image ROI (area) to larger using camera 115 to zoom in or out (increase or decrease)); and
transmitting at least one of the acquired operative field image and information based on an inference result to the console according to transmission settings by the console (0285-0286, specifically 0286 teaches additional surgeon assistance and/or guidance is requested base on set of frames (field image) that are inferred, where figure 14 and 0290 teaches transmission to 1410 (paragraph 0309-0311)); and
transmitting the control information generated by the control unit to the console (figure 4 and 0110 detail control through user interface 700 (console); 0086 teaches control of camera as well as infrared laser by operator; 0553 detail provide support for surgical procedures to user with recommendation and control commands to surgical robots).
Wolf et al teaches all the subject matter above but do not teach the following subject matter: generating control information for controlling an operation of the surgical robot according to the inference result.
MOTTRAM et al (US 2023/0149101) teaches the following subject matter: generating control information for controlling an operation of the surgical robot according to the inference result (paragraph 0049 detail control configuration of robot based on inferred means, where 0036 detail such function for surgical robot).
Wolf et al and MOTTRAM et al are both in the field of image analysis, especially generating control information for surgical robot based on ROI imaging such that the combine outcome is predictable.
Therefore it would have been obvious to one having ordinary skill before the effective filing date to modify Wolf et al by MOTTRAM et al such using that information, the control system can model the effect of gravity on the components of the robot arm for the current configuration of the robot arm and estimate a force or torque due to gravity on each joint of the robot arm as disclosed by MOTTRAM et al in 0049.
Regarding claim 39, Wolf et al addresses method (figure 5).
Regarding claim 40, Wolf et al addresses non-transitory computer readable recording medium (figure 14 and 0290 teaches of structure of processor, storage/database/memory, non-transitory computer readable medium)
Claim 23:
Wolf et al teach:
The inference device according claim 21, wherein instructions causing the any of the one or more processors to execute processing of:
generating image data indicating the position of the object; and transmitting the image data using one-way communication with the console (0086 detail identify position/location of object (surgical tool, hand of surgeon, bleeding, motion, anatomical structure), where human operator viewing the ROI (region of interest) which is one way).
Claim 24:
Wolf et al teach:
The inference device according claim 23, wherein instructions causing the any of the one or more processors to execute processing of:
generating positional information indicating the position of the object; and transmitting the positional information using two-way communication with the console (0086 teaches position data, tracking of different object and ROI to human operator (one way); figure 14 teaches network 1418 with communication going both ways).
Claim 26:
Wolf et al teach:
The inference device according claim 21, wherein instructions causing the any of the one or more processors to execute processing of:
generating the control information in order to move the imaging unit following a tip portion of the recognized surgical device (0086 detail identify position/location of object such as surgical tool).
Claim 28:
Wolf et al teach:
The inference device according to claim 21, wherein the instructions causing the any of the one or more processors to execute processing of:
generating the control information in order to move the imaging unit to a designated position on the object (0086 teaches control information to move camera to zoom in and out of object (surgical tool, hand of surgeon, bleeding, motion, anatomical structure)).
Claim 29:
Wolf et al teach:
The inference device according to claim 21, wherein the instructions causing the any of the one or more processors to execute processing of:
generating the control information in order to move the imaging unit according to a distance between the imaging unit and the object (0086 teaches control information to move camera to zoom in and out of object (surgical tool, hand of surgeon, bleeding, motion, anatomical structure), where 0091 detail distance consideration between object and camera).
Claim 30:
Wolf et al teach:
The inference device according to claim 21, wherein the instructions causing the any of the one or more processors to execute processing of:
inferring a motion or gesture of an operator operating the console; and generating control information in order to control an operation of the imaging unit or a surgical device according to an inference result (0089 detail control motions such as orientation, zoom of camera in given surgical procedure to track motion of surgeon hands).
Claim 31:
Wolf et al teach:
The inference device according to claim 21, wherein the instructions causing the any of the one or more processors to execute processing of:
computing an area or shape of the recognized object; and generating control information for selecting or controlling a surgical device to be used according to the area or shape of the object (0182 teaches consideration such as color, shape, structure and condition cause changes to surgical procedure (control of surgery, which above is the camera as well as surgical robot)).
Claim 32:
Wolf et al teach:
The inference device according to claim 21, wherein the instructions causing the any of the one or more processors to execute processing of:
computing a confidence of the inference result (0629 and 0630 teaches calculating confidence probability or score); and changing a resolution of the operative field image according to the computed confidence (0089 teaches camera setting such as resolution is controlled; 0124 teaches playing of resolution based on decision marker, where 0560 and 0579 detail decision marker is based on confident level).
Claim 33:
Wolf et al teach:
The inference device according to claim 21, wherein the instructions causing the any of the one or more processors to execute processing of:
computing a confidence of the inference result (0629 and 0630 teaches calculating confidence probability or score); acquiring information of the surgical robot from the console; and computing a score of a surgery performed by the surgical robot on the basis of the computed confidence and the acquired information (0579 teaches surgical robot with confidence level).
Claim 34:
Wolf et al teach:
The inference device according to claim 21, wherein the inference device according to wherein the imaging unit of the surgical robot is configured to output an operative field image for a left eye and an operative field image for a right eye, and wherein the instructions causing the any of the one or more processors to execute processing of:
acquiring the operative field image for the left eye and the operative field image for the right eye output from the imaging unit, and inferring each of the acquired operative field image for the left eye and the acquired operative field image for the right eye (0091 teaches stereo camera for ROI, where stereo image provide field of view each for left eye and right eye individually).
Claim 35:
Wolf et al teach:
The inference device according to claim 34, wherein the instructions causing the any of the one or more processors to execute processing of:
computing a confidence of the inference result; and outputting an alert on the basis of a difference between the confidence of the inference result computed for the operative field image for the left eye and the confidence of the inference result computed for the operative field image for the right eye (0637-0638 teaches image-related data structure with correlated from other data that change the probability of confidence would generated alertness to undertake other surgical actions, where above teaches use of stereo images (left and right eye images)).
Claim 36:
Wolf et al teach:
The inference device according to claim 34, wherein the instructions causing the any of the one or more processors to execute processing of:
computing a confidence of the inference result; generating control information for moving the imaging unit according to a difference between the confidence of the inference result computed for the operative field image for the left eye and the confidence of the inference result computed for the operative field image for the right eye; and transmitting the generated control information to the console (0637-0638 teaches image-related data structure with correlated from other data that change (difference) the probability of confidence would generated alertness to undertake other surgical actions (generated controls information), where above teaches use of stereo images (left and right eye images)).
Claim 37:
Wolf et al teach:
The inference device according to claim 34, wherein the instructions causing the any of the one or more processors to execute processing of:
computing depth information on the basis of the operative field image for the left eye and the operative field image for the right eye; and transmitting the computed depth information to the console (0691 teaches frames consideration to depth of incision during surgery, where neural network configure to identify more specific intraoperative in 0692, above teaches use of image/frame from stereo camera for left and right eye image).
Claim 38:
Wolf et al teach:
The inference device according to claim 34, wherein the instructions causing the any of the one or more processors to execute processing of:
computing depth information on the basis of the operative field image for the left eye and the operative field image for the right eye; generating control information for controlling an operation of the surgical robot according to the computed depth information, and transmitting the generated control information to the console (teaches frames consideration to depth of incision during surgery, where neural network configure to identify more specific intraoperative (generated control information) in 0692, above teaches use of image/frame from stereo camera for left and right eye image).
Claim 41, similarly claim 42-43:
Wolf et al (US 2021/0012868) teach the following subject matter:
An inference device connected between a surgical robot and a console controlling the surgical robot, the inference device comprising:
one or more processors; and a storage storing instructions causing any of the one or more processors to execute processing of (figure 14 and 0290 teaches of structure of processor, storage/database/memory, non-transitory computer readable medium to contain and execute instruction):
acquiring an operative field image shot by an imaging unit of the surgical robot (0285 teaches surgical event with image frames, where 0537,0538 and 0595 detail the use of surgical robot with camera for capturing and transmitting images, video footage over network directly wired and/or wirelessly; 0090 teaches camera for moving to object with field of view and function such as adjust zoom, direction, speed);
performing an inference process on the acquired operative field image (0285 detail inferred base on set of images from surgical area (operative field image));
inferring at least one of a position of an object to be recognized in the operative field image and an event occurring in the operative field image (80 detail inferred output with classification of item depicted in the image (operative field image) with output including inferred value, size, volume, age, cost, product);
generating the control information in order to move the imaging unit according to a distance between the imaging unit and the object (0086 detail control application positioning camera to capture video/image ROI (region of interest) with identify structure, surgical tool, hand of surgeon, motion, particular location, tracking location with camera 115; 0087 teaches camera to image ROI (area) to larger using camera 115 to zoom in or out (increase or decrease));
transmitting at least one of the acquired operative field image and information based on an inference result to the console according to transmission settings by the console (0285-0286, specifically 0286 teaches additional surgeon assistance and/or guidance is requested base on set of frames (field image) that are inferred, where figure 14 and 0290 teaches transmission to 1410 (paragraph 0309-0311)); and
transmitting the control information generated by the control unit to the console (figure 4 and 0110 detail control through user interface 700 (console); 0086 teaches control of camera as well as infrared laser by operator; 0553 detail provide support for surgical procedures to user with recommendation and control commands to surgical robots).
Wolf et al teaches all the subject matter above but do not teach the following subject matter:
generating control information for controlling an operation of the surgical robot according to the inference result
MOTTRAM et al (US 2023/0149101) teaches the following subject matter:
generating control information for controlling an operation of the surgical robot according to the inference result (paragraph 0049 detail control configuration of robot based on inferred means, where 0036 detail such function for surgical robot).
Wolf et al and MOTTRAM et al are both in the field of image analysis, especially generating control information for surgical robot based on ROI imaging such that the combine outcome is predictable.
Therefore it would have been obvious to one having ordinary skill before the effective filing date to modify Wolf et al by MOTTRAM et al such using that information, the control system can model the effect of gravity on the components of the robot arm for the current configuration of the robot arm and estimate a force or torque due to gravity on each joint of the robot arm as disclosed by MOTTRAM et al in 0049.
Regarding claim 42, Wolf et al addresses method (figure 5).
Regarding claim 43, Wolf et al addresses non-transitory computer readable recording medium (figure 14 and 0290 teaches of structure of processor, storage/database/memory, non-transitory computer readable medium)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Shelton et al (2022/0233241) teaches SURGICAL PROCEDURE MONITORING - surgical computing system may receive usage data associated with movement of a surgical instrument and user inputs to the surgical instrument. The surgical computing system may receive motion and biomarker sensor data from sensing systems applied to the operator of the surgical instrument. The surgical computing system may determine, based on at least one of the usage data and/or the sensor data, an evaluation of the actions of the operator of the surgical instrument. The surgical computing system may determine, based on the evaluation, to provide feedback. The feedback may comprise instructions for the surgical instrument to provide haptic feedback and/or to modify its configuration. The feedback may comprise instructions for a display unit to present notifications instructing the healthcare professional.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TSUNG-YIN TSAI whose telephone number is (571)270-1671. The examiner can normally be reached 7am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bhavesh Mehta can be reached at (571) 272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TSUNG YIN TSAI/Primary Examiner, Art Unit 2656
Prosecution Insights