DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
1. Applicant’s arguments and amendments, filed 12/23/2025, with respect to the rejection(s) of claim(s) 1-3, 10-12, 14, and 53-56 under U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Schmidt (US 20130185325 A1).
Regarding claim 49, the Applicant argues that an Office Action mailed on June 16, 20205 indicated that Wang did not teach “provide a command to the at least one of the plurality of components of the teleoperational surgical system to implement a response to the voice communication based on the evaluated voice communication”. They then go on to argue that in the most recent office action (mailed 09/25/2025), that the Examiner “minimizes” these features down to “implementing the voice communication”. Regarding the first argument, the Examiner notes that the art of Brant was dropped in the most recent office action after further reviewing Wang. The office action mailed in June is not of significance when there has been a more recent action mailed in September. Regarding the argument that the Examiner has “minimized” the claims, the Examiner respectfully disagrees. The paragraphs cited when pointing to the implementation (Par. 0035 and 0041) give examples of clinicians/radiologists using the voice control system to have it dictate their commands, i.e. providing a command and implementing it based on the voice communication. Therefore, this portion of the rejection still stands.
Further regarding claim 49, the Applicant argues that the art of Sadfi does not teach “deactivate the voice recognition system upon receipt of a signal indicating that the input device at the operator input system is displaced in the one or more degrees of freedom beyond a threshold displacement value” since they do not teach controlling a manipulate arm. The Examiner respectfully disagrees. Sadfi is being used to show that pressing a button (i.e. displacing the input system by one or more degrees of freedom beyond a threshold vale) can activate and deactivate a voice control system. Based on the combination of Sadfi/Wang/ DiMaio, it would be inherent that deactivating/activating the voice system (which controls the manipulator arm – per Wang) would deactivate/control the motion of the teleoperational manipulator. This portion of the rejection still stands.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
2. Claim(s) 1-3, 10-12, 14, and 53-56 is/are rejected under 35 U.S.C. 103 as being obvious over Wang (US 20050242919 A1) in view of Balicki (US 20160361125 A1) and in further view of Schmidt (US 20130185325 A1).
In regards to claim 1, Wang teaches a teleoperational surgical system (Abstract) comprising:
an operator input system (Par. 0017 teaches an input system [10]);
a teleoperational manipulator system configured for operation in a procedure by the operator input system, the teleoperational manipulator system including a manipulator arm and a medical instrument coupled to the manipulator arm in a surgical environment (Par. 0032 teaches a robotic manipulator arm 20 that is connected to a medical instrument, such as an endoscope), wherein the operator input system, the manipulator arm, and the medical instrument are a plurality of components of the teleoperational surgical system (Fig 1 shows the interconnection of the devices); and
a processing unit including one or more processors (Fig 1, element 12 the shows the master controller), wherein the processing unit is configured to
recognize a voice communication (Par. 0035-0036 teach a voice control interface that receives and recognizes speech from the user);
assess a plurality of surgical environment state variables associated with at least one of the plurality of components of the teleoperational surgical system to obtain a current status of the manipulator arm (Par. 0035 teaches the voice control interface includes means for receiving selection commands from a user wherein each selection command is associated with one specific device in electrical communication with the master controller. This is accomplished by providing the master controller a list of the devices that are in electrical communication therewith upon start-up of the control system, i.e. the “current status” of the robotic arm is whether or not it’s connected to the master controller)
evaluate the voice communication in a context of the obtained current status of the manipulator arm by: referencing a word recognition search space having a plurality of words associated with at least one of the plurality of components of the teleoperational surgical system and constraining the word recognition search space (Par. 0035 and 0041 teaches there is a lexicon of words that can be recognized); and
determine a response to the voice communication based on the evaluated voice communication; and provide a command to the at least one of the plurality of components of the teleoperational surgical system to implement the response. (Par. 0036 teaches evaluating the voice commands to make a proper response).
Wang does not teach wherein the current status of the manipulator arm includes motion a set of failure modes or an obstruction of the manipulator arm. However, in the same field of endeavor, Balicki, teaches a controlled surgical robot system (Abstract) wherein the system can obtain motion control information about the manipulator arm, including if the arm is going to collide with itself or another object (i.e. obstruction) (“arm collision detection” Pars. 0005 and 0041-0045) and communicate this information or implement it into an algorithm in order to avoid arm collisions and add safety to the system.
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to have taken the teachings of Wang and modified them by having the system detect arm collision, as taught and suggested by Balicki, in order to add safety to the system
The combined teachings of Wang and Balicki do not disclose prioritizing, within the word recognition search space, a first plurality of words as more likely to match the voice communication than a second plurality words. However, it is known in voice recognition systems to prioritize certain commands over others based on what a user is currently doing – as can be seen in Par. 0060 of Schmidt, in order to make sure the current preferences are being taken into consideration (Par. 0060).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to have taken the teachings of Wang and Balicki and modified them by having the system prioritize a first plurality of words as more likely to match the voice communication than a second plurality words, as taught and suggested by Schmidt, in order to make sure the current preferences are being taken into consideration (Par. 0060 of Schmidt).
In regards to claim 2, the combined teachings of Wang, Balicki, and Schmidt as applied to claim 1 disclose the teleoperational surgical system of claim 1 except for wherein the plurality of surgical environment state variables includes an instrument state variable for the medical instrument.
The Applicant discloses an instrument state variable can be information about what a surgical instrument is doing. Balicki goes on to further teach in Par. 0058-0061 that surgical state environments can also be surgical instrument states, in order to make sure the command matches up to what the surgical instruments are actually doing.
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to have taken the teachings of Wang, Balicki, and Schmid and modified them by having the system have surgical state environments as surgical instrument states, as taught by Balicki, in order to make sure the command matches up to what the surgical instruments are actually doing.
In regards to claim 3, the combined teachings of Wang, Balicki, and Schmidt as applied to claim 1 discloses the teleoperational surgical system of claim 1 except for wherein the plurality of surgical environment state variables includes a manipulator state variable for the teleoperational manipulator (Par. 0041-0045 of Balicki).
However, Balicki goes on to further teach in Par. 0041-0045 that the plurality of surgical environment state variables includes a manipulator state variable for the teleoperational manipulator, in order to make sure the commands matches up to what the manipulators are actually doing.
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to have taken the teachings of Wang, Balicki, and Schmid and modified them by having the system have environment state variables includes a manipulator state variable for the teleoperational manipulator, as taught by Balicki, in order to make sure the command matches up to what the surgical instruments are actually doing.
In regards to claim 10, the combined teachings of Wang, Balicki, and Schmidt as applied to claim 1 discloses the teleoperational surgical system of claim 1 wherein providing a command includes displaying a textual message to a user with at least one proposed course of action based on at least one surgical environment state variable (Abstract of Wang teaches providing the user with a message about the course of action to be taken).
In regards to claims 11 and 12, the combined teachings of Wang, Balicki, and Schmidt as applied to claim 1 rant discloses the teleoperational surgical system of claim 1 wherein providing a command includes changing a location of a visualization device in the surgical environment and changing an illumination setting on an illumination device in the surgical environment (Par. 0034 of Wang discloses the system can be used for adjusting the lighting of the surgical environment).
In regards to claim 14, the combined teachings of Wang, Balicki, and Schmidt as applied to claim 1 discloses the teleoperational surgical system of claim 1 wherein evaluating the voice communication includes modifying the word recognition search space by eliminating at least one of the plurality of words of the word recognition search space (Par. 0041 of Wang discloses the command lexicon can be added to or subtracted from based on the instruments connected)
In regards to claim 53, Wang discloses a method comprising:
recognizing a voice communication (Par. 0035-0036 teach a voice control interface that receives and recognizes speech from the user);
assessing a plurality of surgical environment state variables associated with at least one of a plurality of components of a teleoperational surgical system to obtain a current status of a manipulator arm of the teleoperational surgical system (Par. 0035 teaches the voice control interface includes means for receiving selection commands from a user wherein each selection command is associated with one specific device in electrical communication with the master controller. This is accomplished by providing the master controller a list of the devices that are in electrical communication therewith upon start-up of the control system, i.e. the “current status” of the robotic arm is whether or not it’s connected to the master controller);
evaluating the voice communication in a context of the obtained current status the manipulator arm by: referencing a word recognition search space having a plurality of words associated with at least one of the plurality of components of the teleoperational surgical system ; and based on the obtained current status of the manipulator arm, constraining the word recognition search space space (Par. 0035 and 0041 teaches there is a lexicon of words that can be recognized;
determining and implementing a response to the voice communication based on at least one of the plurality of surgical environment state variables associated with the at least one of the plurality of components of the teleoperational surgical system (Par. 0036 teaches evaluating the voice commands to make a proper response).
Wang does not teach wherein the current status of the manipulator arm includes motion a set of failure modes or an obstruction of the manipulator arm. However, in the same field of endeavor, Balicki, teaches a controlled surgical robot system (Abstract) wherein the system can obtain motion control information about the manipulator arm, including if the arm is going to collide with another arm (i.e. obstruction) (Par. 0041-0045) and communicate this information or implement it into an algorithm in order to avoid arm collisions and add safety to the system.
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to have taken the teachings of Wang and modified them by having the system detect arm collision, as taught and suggested by Balicki, in order to add safety to the system. The combined teachings of Wang and Balicki do not disclose prioritizing, within the word recognition search space, a first plurality of words as more likely to match the voice communication than a second plurality words. However, it is known in voice recognition systems to prioritize certain commands over others based on what a user is currently doing – as can be seen in Par. 0060 of Schmidt, in order to make sure the current preferences are being taken into consideration (Par. 0060).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to have taken the teachings of Wang and Balicki and modified them by having the system prioritize a first plurality of words as more likely to match the voice communication than a second plurality words, as taught and suggested by Schmidt, in order to make sure the current preferences are being taken into consideration (Par. 0060 of Schmidt).
In regards to claim 54, the combined teachings of Wang, Balicki, and Schmidt as applied to claim 53 discloses the method of claim 53, wherein the plurality of surgical environment state variables includes an instrument state variable for a medical instrument of the teleoperational surgical system.
However, Balicki goes on to further teach in Par. 0041-0045 that the plurality of surgical environment state variables includes a manipulator state variable for the teleoperational manipulator, in order to make sure the commands matches up to what the manipulators are actually doing.
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to have taken the teachings of Wang, Balicki, and Schmid and modified them by having the system have environment state variables includes a manipulator state variable for the teleoperational manipulator, as taught by Balicki, in order to make sure the command matches up to what the surgical instruments are actually doing.
In regards to claim 55, the combined teachings of Wang, Balicki, and Schmidt as applied to claim 53 discloses the method of claim 53, wherein the plurality of surgical environment state variables includes a manipulator state variable for a teleoperational manipulator system of the teleoperational surgical system (Par. 0035 of Wang teaches determining the electrical communication between the manipulator and control system).
In regards to claim 56, the combined teachings of Wang, Balicki, and Schmidt as applied to claim 53discloses the method of claim 53, wherein constraining the word recognition search space includes modifying the word recognition search space by eliminating at least one of the plurality of words of the word recognition search space or prioritizing at least one of the plurality of words of the word recognition search space (Par. 0041 of Wang discloses the command lexicon can be added to or subtracted from based on the instruments connected)
3. Claim(s) 4, 8, 15-16, and 57 is/are rejected under 35 U.S.C. 103 as being obvious over Wang, Balicki, and Schmidt and in further view of Brant (US 6278975 B1)
In regards to claim 4, 8, 15-16, and 57, the combined teachings of Wang, Balicki, and Schmidt as applied to claim 1 discloses the teleoperational surgical system of claim 1 except for wherein the plurality of surgical environment state variables includes an operator console variable, wherein the part of speech is at least one of a noun, a pronoun, a verb, and, wherein the operator console state variable is an operator input at an operator input device of the operator input system and wherein the plurality of surgical environment state variables includes a procedure state variable and wherein the plurality of surgical environment state variables includes an identified subsystem variable and wherein providing a command to implement the response includes providing a command to implement the response relative to a subsystem associated with the identified subsystem variable.
However, in the same field of endeavor, Brant teaches a voice operated surgical control system (Abstract) wherein the speaker volume and speech is evaluated (Applicant discloses the operator console variable can be the volume of the speaker. Col. 4, lines 47-54 of Brant disclose evaluating the speaker volume), steps of the procedure are accounted for (Applicant describes the procedure state variable as any information about the procedure such as steps. Col 5 of Brant discloses the steps of the procedure are accounted for), and identifying a subsystem (Applicant discloses that the subsystem can be a surgeon console and Col 5, lines 15-25 of Brant disclose a console subsystem), in order to better aid the system in accurate voice control and implementing the desired result.
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to have taken the teachings of Wang, Balicki, and Schmidt and modified them by having various elements and surgical enviorment state variables evaluated, as taught and suggested by Brant, in order to better aid the system in accurate voice control and implementing the desired result.
4. Claims 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang, Balicki, and Schmidt and in further view of DIY Life Tech (How to Change the Volume on Amazon Echo, NPL)
The combined teachings of Wang, Balicki, and Schmidt as applied to claim 1 discloses the teleoperational surgical system of claim 1 except for wherein providing a command includes changing a volume setting at the operator input system.
However, solving the same problem, DIY Life Tech shows in their video that using voice recognition with a smart device, in this case an Amazon Echo, to adjust the volume of the device is not novel. The reason for having a robot/smart system able to recognize a voice and adjust volume settings would be to reduce the need for manual adjustment.
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to have taken the art of Wang, Balicki, and Schmidt and modified it by having the system change a volume setting, as taught and suggested by DIY Life Tech, for the purpose of reducing the need for manual adjustment and keeping a surgeon’s hands free.
5. Claims 49-51 is/are rejected under 35 U.S.C. 103 as being unpatentable Wang and in further view of DiMaio (US 20090036902 A1) and in further view of Safdi (US 20100036676 A1)
In regards to claim 49, Wang disclose a teleoperational surgical system (Abstract) comprising:
an operator input system (Par. 0017 teaches an input system [10]);
a teleoperational manipulator configured for operation by the operator input systems wherein the operator input system and the teleoperational manipulator are a plurality of components of the teleoperational surgical system (Par. 0032 teaches a robotic manipulator arm that is connected to a medical instrument, such as an endoscope), and
a processing unit including one or more processors (Fig 1, element 12 the shows the master controller), wherein the processing unit is configured to:
recognize a voice communication (Par. 0035-0036 teach a voice control interface that receives and recognizes speech from the user);
evaluating and implementing the voice communication in a context of a current status of a plurality of surgical environment state variables associated with at least one of the plurality of components of the teleoperational system by constraining a word recognition search space to a plurality of words associated with the at least one of the plurality of components of the teleoperational system (Par. 0035 and 0041 teaches there is a lexicon of words that can be recognized in context of the environmental state);
Wang not disclose wherein the processing unit is configured to receive a signal from a master clutch switch to enable voice signals at the operator input system.
However, DiMaio discloses a robotic surgical system (Abstract) that comprises a master clutch that is used to switch on the control device (Claim 17) showing that it is known to use master clutches to switch on and off a device in order to add a layer of safety and prevent the voice command signals from being processed at an unwanted time.
Therefore, it would be obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to have taken the teachings of Wang and modified them by having the system configured to receive a voice enable signal from a master clutch switch at the operator input system, as taught and suggested by DiMaio, showing that it is known to use master clutches to switch on and off a device in order to add a layer of safety and prevent the voice command signals from being processed at an unwanted time.
The combined teachings of Wang and DiMaio do not disclose deactivating the voice recognition system upon receipt of a signal indicating that an input device at the operator input system is displaced in one or more degrees of freedom beyond a threshold displacement value, wherein displacement of the input device in one or more degrees of freedom controls motion of the teleoperational manipulator. However, in the same field of endeavor, Sadfi discloses medical treatment system that employs a voice recognition system (Abstract and Par. 0013) wherein the voice activation can be deactivated by pressing a button (Par. 0215; the movement threshold would be the threshold the button is depressed in order to turn the voice recognition off which would in turn stop the system and prevent the manipulator arm from moving) in order to add an additional layer of security to the system by making sure the voice activation is only used when desired.
Therefore, it would be obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to have taken the teachings of Wang and DiMaio and modified them by having the system configured to deactivate the voice recognition system if a signal exceeds a threshold, as taught and suggested by Sadfi, in order to add an additional layer of security to the system by making sure the voice activation is only used when desired.
In regards to claim 50, the combined teachings of Wang, DiMaio, and Sadfi as applied to claim 49 discloses the teleoperational surgical system of claim 49 except for wherein the processing unit is further configured to recognize an uncharacteristic master clutch activity prior to recognizing the voice communication.
However, DiMaio does go on to teach recognizing an uncharacteristic master clutch activity prior to recognizing the voice communication (Par. 0105-0108 of DiMaio), in order to have an added safety feature to the device before beginning an operation.
Therefore, it would be obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to have taken the teachings of Wang, DiMaio, and Sadfi and modified them by having the system configured to recognize an uncharacteristic master clutch activity prior to recognizing the voice communication, as taught and suggested by DiaMaio, in order to have an added safety feature to the device before beginning an operation.
In regards to claim 51, the combined teachings of Wang, DiMaio, and Sadfi as applied to claim 49 discloses the teleoperational surgical system of claim 49 wherein the processing unit is further configured to issue a tone indicative of operation of a speech recognition system for evaluating the voice communication (Par. 0035-0036 of Wang disclose that the processor takes in the voice and evaluates it).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SKYLAR LINDSEY CHRISTIANSON whose telephone number is (571)272-0533. The examiner can normally be reached Monday-Friday, 7:30-5:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Niketa Patel can be reached on (571) 272-4156. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.L.C./Examiner, Art Unit 3792
/NIKETA PATEL/Supervisory Patent Examiner, Art Unit 3792
Prosecution Insights