DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments in combination with amendments, see Remarks and Claims, filed 05/07/2026, with respect to the rejection(s) of claim(s) under 35 USC 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of KR 20140104586 to Joo et al. See details below.
Applicant’s arguments in combination with amendments, see Remarks and Claims, filed 05/07/2026, with respect to the rejection(s) of claim(s) under double patenting rejection have been fully considered and are persuasive. Therefore, the rejection has been withdrawn.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3, 5-12, 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over US Patent Publication Number 20190090969A1 granted to Jarc et al. (hereinafter “Jarc” – previously published as WO2017083768A1. – both on IDS) in view of KR 20140104586 to Joo et al. (hereinafter “Joo”).
Regarding claims 1 and 11, Jarc discloses a system/method for robot-assisted surgery (e.g. Para 0043 “the system…for performing surgical procedure” Fig. 1, 5A, 20), comprising: an image sensor (e.g. para 0046 and 0049 “image capture device”); a display (e.g. Fig. 4, para 0048 “display”); and a controller coupled to the image sensor and the display (e.g. Fig. 4, para 0043 “processor”), wherein the controller (e.g. Fig. 4, para 0043 “processor”), wherein the controller is configured to execute instructions that cause (e.g. “algorithm”) the system to perform operations (e.g. para 0076-0078 “algorithm and pattern matching” please note that the claim does not recite any details regarding the logic and “when” it is executed by the controller in order to perform certain operations. The examiner understands the processor’s operation of pattern matching and analysis using the algorithm to performing the same functions), including: acquiring first images of a surgical procedure with the image sensor (e.g. para 0047 “capture images of a surgical site and output the captured images to a computer processor”); analyzing the first images to identify a surgical step in the surgical procedure (e.g. para 0048 “image processing” para 0110 “a parallel-running simulation may be autonomously analyzed to determine the current surgical stage using segmented 1004,” and para 0137 “Technique advisor 1614 may recommend a particular surgical technique to be used in a given scenario, in response to the current or upcoming surgical stage”); displaying, in a first mode of operation, second images on the display in response to identifying the surgical step (e.g. para 0137 “Technique advisor 1614 may recommend a particular surgical technique to be used in a given scenario, in response to the current or upcoming surgical stage”), wherein the second images include at least one of a diagram of human anatomy, a preoperative image, an intraoperative image, or an annotated image of one of the first images (e.g. para 0140 “loading of pre-op images for the current/upcoming step from current patient or related patients.).
Jarc, fails to disclose displaying, in a second mode of operation, third images corresponding to the identified surgical step in response to receiving a user-initiated request from a user interface of the system during the surgical procedure.
Joo, from a similar field of endeavor teaches the interface to be configured to allow a user to perform various operations including the operation of the surgical apparatus [], the user inputs various commands [] to select various contents by using an image to be displayed (Description). It would have been obvious to one of ordinary skill in the art to modify the disclosure of Jarc with the teachings of Joo to provide the predictable result of allowing the user to select various content to be displayed during the operation.
Regarding claim 2, Jarc as modified by Joo renders obvious the system of claim 1, further comprising: a plurality of arms coupled to the controller and configured to hold surgical instruments (e.g. Para .0049 “surgical procedures using one or more mechanical support arms 510”); and a tactile user interface coupled to the controller (e.g. Fig. 1-5a, para 0065 “graphic user interface overlaid onto the displayed video”), wherein the operations further include: in response to receiving user input from the tactile user interface, manipulating the plurality of arms (e.g. para 0006 “The TSS generally includes a surgeon input interface that accepts surgical control input for effecting an electromechanical surgical system to carry out a surgical procedure”).
Regarding claim 3, Jarc as modified by Joo renders obvious the system of claim 1, further comprising: a microphone coupled to the controller to send voice commands from a user to the controller; and a speaker coupled to the controller to output audio (e.g. Para 0065 “the graphic user interface can include a QWERTY keyboard, a pointing device such as a mouse and an interactive screen display, a touch-screen display, or other means for data or text entry or voice annotation/or speech to text conversion via a microphone and processor.”).
Regarding claim 5, Jarc as modified by Joo renders obvious the system of claim 1, further comprising annotating the one of the first images to form the annotated image by at least one of highlighting a piece of anatomy, highlighting the location of a surgical step, or highlighting where a surgical instrument should be placed (e.g. para 0065-0066 “highlights or annotates certain patient anatomy shown in the displayed video using an input device of surgeon's console 52”).
Regarding claim 6, Jarc as modified by Joo renders obvious the system of claim 1, wherein the preoperative image includes a magnetic resonance image, computerized tomography scan, or an X-ray (e.g. para 0108).
Regarding claim 7, Jarc as modified by Joo renders obvious the system of claim 1, wherein analyzing the first images to identify the surgical step in the surgical procedure includes using a machine learning algorithm trained to recognize the surgical steps from the first images (e.g. para 0094).
Regarding claim 8, Jarc as modified by Joo renders obvious the system of claim 7, wherein the machine learning algorithm includes at least one of a convolutional neural network (CNN) or long short-term memory (LSTM) (e.g. para 0094 “convolutional neural networks”).
Regarding claim 9, Jarc as modified by Joo renders obvious the system of claim 1, wherein the operations further include: estimating a remaining duration of the surgical procedure, in response to identifying the surgical step (e.g. para 0116).
Regarding claim 10, Jarc as modified by Joo renders obvious the system of claim 1, wherein the image sensor is disposed surgical instrument (e.g. Fig. 5A, para 0042 “endoscope that includes a camera to view a surgical site within a patient's body”).
Regarding claim 12, Jarc as modified by Joo renders obvious the method of claim 11, further comprising estimating a remaining duration of the surgical procedure, in response to identifying the surgical step (e.g. para 0116).
Regarding claim 15, Jarc as modified by Joo renders obvious the method of claim 11, further comprising annotating the one of the first images to form the annotated image by at least one of highlighting a piece of anatomy, highlighting the location of a surgical step, or highlighting where a surgical instrument should be placed (e.g. para 0065-0066 “highlights or annotates certain patient anatomy shown in the displayed video using an input device of surgeon's console 52”).
Regarding claim 16, Jarc as modified by Joo renders obvious the method of claim 11, wherein the preoperative image includes a magnetic resonance image, computerized tomography scan, or an X-ray (e.g. para 0108).
Regarding claim 17, Jarc as modified by Joo renders obvious the method of claim 11, wherein identifying the surgical step in the surgical procedure from the first images includes using the machine learning algorithm trained to recognize surgical steps from the first images (e.g. para 0094).
Regarding claim 18, Jarc as modified by Joo renders obvious the method of claim 17, wherein the machine learning algorithm includes at least one of a convolutional neural network (CNN) or long short-term memory (LSTM) (e.g. para 0094 “convolutional neural networks”).
Regarding claim 19, Jarc as modified by Joo renders obvious the method of claim 11, wherein the image sensor is disposed in a surgical instrument(e.g. Fig. 5A, para 0042 “endoscope that includes a camera to view a surgical site within a patient's body”).
Regarding claim 20, Jarc as modified by Joo renders obvious the method of claim 11, further comprising: capturing voice commands with a microphone; and in response to capturing the voice commands, displaying the preoperative image (e.g. Para 0065 “the graphic user interface can include a QWERTY keyboard, a pointing device such as a mouse and an interactive screen display, a touch-screen display, or other means for data or text entry or voice annotation/or speech to text conversion via a microphone and processor.”).
Claims 4 and 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Jarc as modified by Joo and in view of US Patent Number 9788907B1 issued to Alvi et al. (hereinafter “Alvi” –IDS).
Regarding claim 4, Jarc as modified by Joo renders obvious the system of claim 3, but fails to disclose wherein the operations further include: in response to identifying the surgical step, outputting audio commands to a user of the system from the speaker.
Alvi teaches wherein the controller further includes logic that when executed by the controller causes the system to perform operations, including: in response to identifying the surgical step, outputting audio commands to a user of the system from the speaker (e.g. Col. 29, lines 20-26 “the electronic data may include an audio signal (e.g., of spoken words) that is to be presented during a visual display of the part of the video data). It would have been obvious to one of ordinary skill in the art at the time to modify the disclosure of Jarc as modified by Joo with the teachings of Alvi to include audio commands to a user to provide the predictable result of allowing the user to follow instructions without having to read or detect marks on the screen using auditory senses.
Regarding claim 13, Jarc as modified by Joo renders obvious the method of claim 11, but fails to disclose further comprising outputting audio commands from a speaker coupled to the controller, in response to determining the surgical step.
Alvi teaches comprising outputting audio commands from a speaker coupled to the controller, in response to determining the surgical step (e.g. Col. 29, lines 20-26 “the electronic data may include an audio signal (e.g., of spoken words) that is to be presented during a visual display of the part of the video data). It would have been obvious to one of ordinary skill in the art at the time to modify the disclosure of Jarc as modified by Joo with the teachings of Alvi to include audio commands to a user to provide the predictable result of allowing the user to follow instructions without having to read or detect marks on the screen using auditory senses.
Regarding claim 14, Jarc as modified by Joo renders obvious the method of claim 13, discloses outputting the duration of the surgical procedure (e.g. para 0116) but fails to disclose outputting the duration of the surgical procedure from the speaker.
Alvi teaches outputting the duration of the surgical procedure from the speaker (e.g. Col. 29, lines 20-26 “the electronic data may include an audio signal (e.g., of spoken words) that is to be presented during a visual display of the part of the video data). It would have been obvious to one of ordinary skill in the art at the time to modify the disclosure of Jarc as modified by Joo with the teachings of Alvi to include audio commands to a user to provide the predictable result of allowing the user to follow instructions without having to read or detect marks on the screen using auditory senses.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SANA SAHAND whose telephone number is (571)272-6842. The examiner can normally be reached M-Th 8:30 am -5:30 pm; F 9 am-3 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer S McDonald can be reached at (571) 270- 3061. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SANA SAHAND/Examiner, Art Unit 3796
Prosecution Insights