DETAILED ACTION
Receipt is acknowledged of applicant’s request for continued examination and argument/remarks filed on June 28, 2026, claims 1, 4-8, and 13-27 are pending and an action on the merits is as follows.
Applicant's arguments with respect to amended claims have been fully considered but are moot in view of the same ground(s) of rejection. Applicant has amended claims 1, 13, 17 and 23.
Previously, claims 2-3 and 9-12 has been canceled.
Per amendment of the claims, the rejection of the claim 1 and 17 under 35 U.S.C. 112(a) and drawing objection had been withdrawn.
Response to Argument
Regarding applicant’s first arguments: “However, this "tracking" in Devam refers to visually tracking the instrument's position for displaying it in augmented reality space—not causing a physical endoscope to autonomously move within a body cavity to maintain visualization of the instrument … This is fundamentally different from the claimed invention, which requires that tagging a surgical tool causes a robotically controlled surgical endoscope to constantly track the movement of the tagged tool by autonomously moving within the body cavity” (page 3, par. 3-4), the examiner respectfully disagreed with applicant statement. Applicant statement is not persuasive because argument(s) attack Devam et al.’s references individually when the rejections are over a combination of three references. Applicant may consider that the argued feature “autonomously moving the surgical endoscope within the body cavity as said at least one robotically controlled surgical tool moves within the body cavity” is taught by Wang et al. (‘361) – third prior art. It is the examiner position that Wang et al. in view of Devam et al. and Wang et al. (‘361) disclosed the claimed invention. The rejection is maintained. Applicant is kindly invited to consider the Office Action below to view the ground of rejection, cited prior arts section and motivation.
Regarding applicant’s second arguments: “Furthermore, the Examiner's motivation to combine is deficient. The Examiner has alleged that the combination would achieve “the benefit of allowing a surgeon to view and accurately locate patient organ and other features prior to and during surgery via laparoscopic camara.’ .. A person of ordinary skill in the art would not have been motivated to modify Wang's voice-controlled endoscope system with Devam's AR display system to arrive at the claimed invention of tagging a tool via vocal command to cause an endoscope to constantly track that tagged tool by autonomously moving within the body cavity” (page 3, par. 4 – page 4, par. 1), the examiner respectfully disagreed with applicant statement. First, applicant’s statements are not persuasive because argument(s) attack Devam et al.’s references individually when the rejections are over a combination of three references. Second, applicant made a conclusory statement to allege that the motivation to combine is defective because Devam’s reference disclose augmented reality (AR) overlay display system without specifically and distinctly point out supposed error in the motivation to combine the cited references. It should be considered that none of the references’ disclosure preclude(s) the combination of the references as set forth in the Office Action and it is the examiner's position that a prima facie case of obviousness had been properly presented.
Regarding applicant’s third arguments: “While Wang '361 teaches a ‘track instrument’ voice command that moves an endoscope to track a second instrument, the combination still fails to teach the specific claim limitation of ‘tagging’ a surgical tool via vocal command, where the tagging causes the endoscope to "constantly track" the tagged tool. Wang '361's general tracking functionality does not teach the distinct vocal-command-driven tagging step that establishes the constant tracking relationship recited in claims 1 and 17 as amended” (page 4, par. 2), the examiner respectfully disagreed with applicant statement. Applicant statement is not persuasive because argument(s) attack Wang ('361)’s references individually when the rejections are over a combination of three references. Applicant may consider that the argued feature is taught by Devam et al.’s references – second prior art. It is the examiner position that Wang et al. in view of Devam et al. and Wang et al. (‘361) disclosed the claimed invention. The rejection is maintained. Applicant is kindly invited to consider the Office Action below to view the ground of rejection, cited prior arts section and motivation.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 6-8, 14-17, 20-22, and 24-27 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (Pat. No.: US 6,063,095) in view of Devam et al. (Pub. No.: US 2016/0249989 A1) and Wang et al. (Pat. No.: US 6,463,361 B1).
Regarding claim 1, Wang et al. disclose a method for performing minimally invasive surgical procedures using voice instruction, the method comprising:
introducing a robotically controlled surgical endoscope and at least one robotically controlled surgical tool into a surgical environment in a body cavity (e.g., endoscope 28 and surgical instruments 22 inserted into a chest cavity of a patient to perform surgical procedure (col. 5, lines 41-46, col. 17, lines 34-36 and lines 62-65 and Figures 9 and 10A-10C));
detecting said at least one vocal command generated by at least one surgeon in a surgical setting (e.g., a surgeon inputs vocabulary / voice instruction(s) via a microphone to move an endoscope – see col. 6, lines 45-52 and Figure 1);
in response to said vocal command (e.g., a controller 46 moves the endoscope 28 and robotic arm assembly 26 in accordance with the input vocabulary / voice instruction(s) of the surgeon – col. 6, line 66 – col. 7, line 2 and col. 6, lines 34-35 and Figure 1).
Wang et al. disclose the controller 46 configured to move the endoscope 28 and robotic arm assembly 26 in accordance with the input vocabulary / voice instruction(s) of the surgeon – col. 6, line 66 – col. 7, line 2 and col. 6, lines 34-35 and Figure 1).
However, Wang et al. failed to specifically disclose tagging at least one of said at least one robotically controlled surgical tools, said tagging causing the robotically controlled surgical endoscope to constantly track the movement of said at least one tagged robotically controlled surgical tool to maintain visualization of said at least one tagged robotically controlled surgical tools within the body cavity.
However, Devam et al. teach a technique to identify a surgical instrument based on user request (limitation: tagging at least one of said at least one robotically controlled surgical tools) and tracking the identified instrument via a laparoscopic camera based on user vocal command during surgical procedure (par. 43, 26 and 37) to display the surgical instrument during a surgical procedure (par. 26, 43, 50), which requires the camera to move, for instance, within the patient body.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the controller for operating surgical equipment (e.g., endoscope) taught by Wang et al., such that the controller identifies a surgical instrument based on user request (e.g., vocal command) and tracking the identified instrument via a laparoscopic camara to display the surgical instrument during a surgical procedure based on user vocal command during surgical procedure, in view of Devam et al., with reasonable expectation of success, since doing so would have achieved the benefit of allowing a surgeon to view and accurately locate patient organ and other features prior to and during surgery via laparoscopic camara thereby reducing morbidity and mortality due to surgical error associated with variance in patient feature location (par. 25).
However, Wang et al., as modified by Devam et al., failed to specifically disclose autonomously moving the surgical endoscope within the body cavity as said at least one robotically controlled surgical tool moves within the body cavity.
However, Wang et al (‘361) teach an automated endoscope system configured to move an endoscope to automatically track a movement of an instrument that is inserted into a patient based on voice command (col. 4, lines 9-15 and col. 3, lines 36-41).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to further modify the controller for operating surgical equipment (e.g., endoscope) as taught by the combination of Wang et al. in view of Devam et al., such that the controller moves the endoscope to automatically track a movement of an instrument that is inserted into a patient based on voice command, in view of Wang et al (‘361), with reasonable expectation of success, since doing so would have achieved the benefit of moving a surgical instrument in response to voice command from a user (col. 1, lines 6-8) while avoiding holding an endoscope (e.g., by an assistance) during a surgical procedure (col. 1, lines 21-27).
Regarding claim 6, Wang et al. disclose a method for performing minimally invasive surgical procedures using voice instruction wherein said at least one vocal command can be unqualified or can comprise at least one qualifier (e.g., voice instruction(s): “up,” “down,” “back,” and “in”), said at least one qualifier configured to modify at least one component of a response (col. 6, lines 45-48), and wherein at least one of the following:
said qualifier is selected from a group consisting of an amount, a surgical tool (e.g., endoscope), an apparatus, and any combination thereof (col. 6, lines 45-52 and Figure 1); and
said amount is either a fixed-term value or a fractional value.
Regarding claim 7, the claim limitations recited features on alternative form of rejected claim 6; therefore, Wang et al. in view of Devam et al. and Wang et al (‘361) still read on the claimed combination alternative form.
Regarding claims 8 and 14, Wang et al., as modified by Devam et al., teach a vocabulary / voice instructions to be input by a surgeon to control the movement of the endoscope to perform different task(s) (Wang et al.’s col. 6, lines 40-52), wherein the surgeon defines specific voice command for the endoscope to perform particular surgical procedure (Wang et al.’s col. 14, 8-11), which covers voice command to stop tracking / tagging the tool and/or tracking / tagging the tool for a predetermined time.
Regarding claims 15-16, Wang et al. failed to specifically disclose wherein tagging at least one of said at least one robotically controlled surgical tools and causing said at least one robotically controlled surgical endoscope to track movement of said at least one tagged robotically controlled surgical tools comprises (i) tagging only one of said at least one robotically controlled surgical tools, and causing said robotically controlled surgical endoscope to track movement of said only one tagged robotically controlled surgical tool (claim 15) and (ii) tagging a set of robotically controlled surgical tools, and causing said robotically controlled surgical endoscope to track movement of said set of tagged robotically controlled surgical tool (claim 16).
However, Devam et al. teach the technique of identifying a surgical instrument(s) based on user request and tracking the identified instrument(s) via a laparoscopic camera based on user vocal command during surgical procedure (par. 43, 26 and 37) – which covers only one or a set of robotically controlled surgical tool.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the controller for operating surgical equipment (e.g., endoscope) taught by Wang et al., such that the controller identifies a surgical instrument(s) based on user request (e.g., vocal command) and tracking the identified instrument(s) via a laparoscopic camara based on user vocal command during surgical procedure, in view of Devam et al., with reasonable expectation of success, since doing so would have achieved the benefit of allowing a surgeon to view and accurately locate patient organ and other features prior to and during surgery via laparoscopic camara thereby reducing morbidity and mortality due to surgical error associated with variance in patient feature location (par. 25).
Regarding claim 17, Wang et al. disclose a method for performing minimally invasive surgical procedures using voice instruction, the method comprising:
introducing a robotically controlled surgical endoscope and at least one surgical tool into a surgical environment in a body cavity (e.g., endoscope 28 and surgical instruments 22 inserted into a chest cavity of a patient to perform surgical procedure (col. 5, lines 41-46, col. 17, lines 34-36 and lines 62-65 and Figures 9 and 10A-10C));
detecting said at least one vocal command generated by at least one surgeon in a surgical setting (e.g., a surgeon inputs vocabulary / voice instruction(s) via a microphone to move an endoscope – see col. 6, lines 45-52 and Figure 1);
in response to said vocal command (e.g., a controller 46 moves the endoscope 28 and robotic arm assembly 26 in accordance with the input vocabulary / voice instruction(s) of the surgeon – col. 6, line 66 – col. 7, line 2 and col. 6, lines 34-35 and Figure 1).
Wang et al. disclose the controller 46 configured to move the endoscope 28 and robotic arm assembly 26 in accordance with the input vocabulary / voice instruction(s) of the surgeon – col. 6, line 66 – col. 7, line 2 and col. 6, lines 34-35 and Figure 1).
However, Wang et al. failed to specifically disclose tagging at least one of said at least one surgical tools, said tagging causing the robotically controlled surgical endoscope to constantly track the movement of said at least one tagged surgical tool to maintain visualization of said at least one tagged surgical tools within the body cavity.
However, Devam et al. teach the technique of identifying a surgical instrument based on user request and tracking the identified instrument via a laparoscopic camera based on user vocal command during surgical procedure (par. 43, 26 and 37) to display the surgical instrument during a surgical procedure(par. 26, 43, 50), which requires the camera to move, for instance, within the patient body.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the controller for operating surgical equipment (e.g., endoscope) taught by Wang et al., such that the controller identifies a surgical instrument based on user request (e.g., vocal command) and tracking the identified instrument via a laparoscopic camara to display the surgical instrument during a surgical procedure based on user vocal command during surgical procedure, in view of Devam et al., with reasonable expectation of success, since doing so would have achieved the benefit of allowing a surgeon to view and accurately locate patient organ and other features prior to and during surgery via laparoscopic camara thereby reducing morbidity and mortality due to surgical error associated with variance in patient feature location (par. 25).
However, Wang et al., as modified by Devam et al., failed to specifically disclose autonomously moving the surgical endoscope within the body cavity as said at least one surgical tool moves within the body cavity.
However, Wang et al (‘361) teach an automated endoscope system configured to move an endoscope to automatically track a movement of an instrument that is inserted into a patient based on voice command (col. 4, lines 9-15 and col. 3, lines 36-41).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to further modify the controller for operating surgical equipment (e.g., endoscope) as taught by the combination of Wang et al. in view of Devam et al., such that the controller moves the endoscope to automatically track a movement of an instrument that is inserted into a patient based on voice command, in view of Wang et al (‘361), with reasonable expectation of success, since doing so would have achieved the benefit of moving a surgical instrument in response to voice command from a user (col. 1, lines 6-8) while avoiding holding an endoscope (e.g., by an assistance) during a surgical procedure (col. 1, lines 21-27).
Regarding claim 20, Regarding claim 6, Wang et al. disclose a method for performing minimally invasive surgical procedures using voice instruction wherein said at least one vocal command can be unqualified or can comprise at least one qualifier (e.g., voice instruction(s): “up,” “down,” “back,” and “in”), said at least one qualifier configured to modify at least one component of a response (col. 6, lines 45-48), and wherein at least one of the following:
said qualifier is selected from a group consisting of an amount, a surgical tool (e.g., endoscope), an apparatus, and any combination thereof (col. 6, lines 45-52 and Figure 1); and
said amount is either a fixed-term value or a fractional value.
Regarding claim 21, the claim limitations recited features on alternative form of rejected claim 20; therefore, Wang et al. in view of Devam et al. and Wang et al (‘361) still read on the claimed combination alternative form.
Regarding claims 22 and 24, Wang et al., as modified by Devam et al., teach a vocabulary / voice instructions to be input by a surgeon to control the movement of the endoscope to perform different task(s) (Wang et al.’s col. 6, lines 40-52), wherein the surgeon defines specific voice command for the endoscope to perform particular surgical procedure (Wang et al.’s col. 14, 8-11), which covers voice command to stop tracking / tagging the tool and/or tracking / tagging the tool for a predetermined time.
Regarding claim 25-26, Wang et al. failed to specifically disclose wherein tagging at least one of said at least one robotically controlled surgical tools and causing said at least one robotically controlled surgical endoscope to track movement of said at least one tagged robotically controlled surgical tools comprises (i) tagging only one of said at least one robotically controlled surgical tools, and causing said robotically controlled surgical endoscope to track movement of said only one tagged robotically controlled surgical tool (claim 15) and (ii) tagging a set of robotically controlled surgical tools, and causing said robotically controlled surgical endoscope to track movement of said set of tagged robotically controlled surgical tool (claim 16).
However, Devam et al. teach the technique of identifying a surgical instrument(s) based on user request and tracking the identified instrument(s) via a laparoscopic camera based on user vocal command during surgical procedure (par. 43, 26 and 37) – which covers only one or a set of robotically controlled surgical tool.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the controller for operating surgical equipment (e.g., endoscope) taught by Wang et al., such that the controller identifies a surgical instrument(s) based on user request (e.g., vocal command) and tracking the identified instrument(s) via a laparoscopic camara based on user vocal command during surgical procedure, in view of Devam et al., with reasonable expectation of success, since doing so would have achieved the benefit of allowing a surgeon to view and accurately locate patient organ and other features prior to and during surgery via laparoscopic camara thereby reducing morbidity and mortality due to surgical error associated with variance in patient feature location (par. 25).
Regarding claim 27, Wang et al. disclose a method for performing minimally invasive surgical procedures using voice instruction, wherein said at least one surgical tool is a robotically controlled surgical tool (e.g., endoscope 28 and surgical instruments 22 inserted into a chest cavity of a patient to perform surgical procedure (col. 5, lines 41-46, col. 17, lines 34-36 and lines 62-65 and Figures 9 and 10A-10C));
Claims 4 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (Pat. No.: US 6,063,095) in view of Devam et al. (Pub. No.: US 2016/0249989 A1, Wang et al. (Pat. No.: US 6,463,361 B1) and Scheib et al. (Pub. No.: US 2020/0015901 A1).
Regarding claims 4 and 18, further modified Wang et al. failed to specifically disclose wherein at least one response to at least one said at least one vocal command is a smart response, said smart response being configured to take into account at least one feature of the an environment surrounding said robotically controlled endoscope, said feature selected from a group consisting of a hazard or obstacle, an interference with a second surgical tool, an interference between two parts of an apparatus, interference between two apparatus, possibility of damage to a portion of a patient, and any combination thereof.
However, Scheib et al. teach a process for applying control motion to a surgical tool (e.g., gasper) to automatically navigate around a tissue structure during a surgical procedure (par. 188).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to furthermore modify the controller for operating surgical equipment (e.g., the endoscope and robotic arm assembly) via a voice instruction(s) as taught by the combination of Wang et al. in view of Devam et al. and Wang et al. ( ‘361), such that the controller applies control motion to surgical tool (e.g., endoscope) to automatically navigate around a tissue structure during a surgical procedure, in view of Scheib et al., with reasonable expectation of success, since doing so would have achieved the benefit of avoiding certain areas in order prevent inadvertent dissection of a critical structure of patient’s organ during a surgical procedure (par. 70).
Claims 5 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (Pat. No.: US 6,063,095) in view of Devam et al. (Pub. No.: US 2016/0249989 A1), Wang et al. (Pat. No.: US 6,463,361 B1) and Jung (Pub. No.: US 2009/0299751 A1).
Regarding claims 5 and 19, Wang et al. disclose a method for performing minimally invasive surgical procedures using voice instruction, wherein a surgeon inputs command (e.g., voice instruction: “up,” “down,” “back,” and “in”) via a microphone to control an endoscope (col. 6, lines 40-48).
However, further modified Wang et al. failed to specifically disclose wherein said at least one vocal command comprises at least one complex sentence, and wherein the method further includes recognizing said at least one vocal command from said complex sentence.
However, Jung teaches a robot configured to received voice command from a user for controlling the robot to perform a predetermined task; wherein the voice command comprise a complex sentence – e.g., “get me a glass of water from a dining room” (par. 38 and 7).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to furthermore modify the controller for operating surgical equipment (e.g., the endoscope and robotic arm assembly) as taught by the combination of Wang et al. in view of Devam et al. and Wang et al. ( ‘361), such that the controller received and processes user complex voice command, in view of Jung, with reasonable expectation of success, since doing so would have achieved the benefit of controlling a robot using complex voice command (par. 43).
Claims 13 and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (Pat. No.: US 6,063,095) in view of Devam et al. (Pub. No.: US 2016/0249989 A1), Wang et al. (Pat. No.: US 6,463,361 B1) and Lee et al. (Pub. No.: US 2018/0249973 A1).
Regarding claims 13 and 23, further modified Wang et al. failed to specifically disclose estimating a 3D spatial position of said at least one surgical tool based on data from a real-time image captured by the surgical endoscope.
However, Lee et al. teach estimating a three-dimensional location of a surgical tool based on image during a surgical operation (par. 43, 26).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to furthermore modify the controller for operating surgical equipment (e.g., the endoscope and robotic arm assembly) as taught by the combination of Wang et al. in view of Devam et al. and Wang et al. (‘361), such that the controller estimating a three-dimensional location of a surgical tool based on image, in view of Lee et al., with reasonable expectation of success, since doing so would have achieved the benefit of estimating the posture of a surgical instrument by using tracking technique and markers while reducing large incision on patient during surgical operation (par. 7 and 5).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Kudo (US 6,036,637 A) is directed to a treading system utilizing an endoscope for tracking an instrument via endoscope wherein the instrument contains color markers
Pandya et al. (US 2013/0331644) is directed to an intelligent autonomous camera configured to follow a tool and keep the tool in the center of view.
Wang et al. (US 6,965,812 B2) is directed to an endoscope system configured to automatically track the movement of a second instrument that is inserted into the patient based on voice command.
This application is a continuation application of U.S. application no. 18/084,477 filed on December 19, 2022, now U.S. Patent 11,886,772 (“Parent Application”) which is a continuation application of U.S. application no. 17/064,549 filed on October 6, 2020, now U.S. Patent 11,561,762 (“Parent Application”). See MPEP §201.07. In accordance with MPEP §609.02 A. 2 and MPEP §2001.06(b) (last paragraph), the Examiner has reviewed and considered the prior art cited in the Parent Application. Also, in accordance with MPEP §2001.06(b) (last paragraph), all documents cited or considered ‘of record’ in the Parent Application are now considered cited or ‘of record’ in this application. Additionally, Applicant(s) are reminded that a listing of the information cited or ‘of record’ in the Parent Application need not be resubmitted in this application unless Applicant(s) desire the information to be printed on a patent issuing from this application. See MPEP §609.02 A. 2. Finally, Applicant(s) are reminded that the prosecution history of the Parent Application is relevant in this application.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jorge O. Peche whose telephone number is (571)270-1339. The examiner can normally be reached Monday-Friday 8:30 AM - 5:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Khoi H. Tran can be reached on 571 272 6919. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Jorge O Peche/Examiner, Art Unit 3656
Prosecution Insights