DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Applicant’s election without traverse of Group I in the reply filed on 01/26/2026 is acknowledged.
Claims 17-20 are withdrawn from further consideration pursuant to 37 CFR 1.142(b) as being drawn to a nonelected Group II, there being no allowable generic or linking claim. Election was made without traverse in the reply filed on 01/26/2026.
Claim Objections
Claims 5-10, and 12-16 are objected to because of the following informalities:
In claims 5-7 and 12-14, “the instrument” should read –the surgical instrument–.
In claim 10, “selecting an overlay for the instrument” should read – selecting an overlay for the surgical instrument–.
In claims 8 and 15, “further tracks a position of the surgical instrument and updates” should read –is further configured to track a position of the surgical instrument and update–.
In claims 9 and 16, “tracks” should read –is configured to track–.
In claims 9 and 16, “the plurality of plurality” should read –the plurality–.
Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-3 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Azizian et al (US 20190282307), hereinafter Azizian.
Regarding claim 1, Azizian teaches a surgical robotic system (Figs. 1-3) comprising:
a robotic arm (“a robot arm” [0110]) including an instrument drive unit (108) (201) (“The visual servoing controller 108 can receive the tracking information from the image-based tracking module 107, combine these with the intraoperative commands from the surgeon 100, and send appropriate commands to the robot in real-time in order to control the surgical robot 101 and the surgical tool(s) 110 to obtain a predetermined goal (e.g. anastomosis).” [0109]; “the surgeon controls the surgical tool … through master-slave control 201 of a robot arm.” [0110]; Figs. 1-2;);
a surgical instrument (110) including a plurality of fluorescent fiducial markers (“The system is based on deploying fluorescent markers on the organ under surgery and/or on the surgical tool, allowing for tracking of the fluorescent markers in real-time and controlling the surgical tool via visually servoing.” [0100]), wherein the surgical instrument is coupled to and actuatable by the instrument drive unit (“the surgeon controls the surgical tool … through master-slave control 201 of a robot arm.” [0110]; Figs. 2-3);
a laparoscopic camera for capturing a video feed of the surgical instrument (“ the camera may be mounted on a robotic arm and the marker position and orientation could in turn control the position and orientation of the camera. This may allow the marker to always be in a suitable field-of-view or angle relative to the camera. Additionally, the tracking of markers on tools could prevent tool-camera occlusions or collisions.” [0176]);
an image processing device (106) coupled to the laparoscopic camera, the image processing device operatable in a white light imaging mode (104) and a low visibility imaging mode (102) (“The light sources 102, 104, … and the image capturing by the CCD 105 are controlled and synchronized by the image acquisition and, control module 106.” [0106] “NIR and visual light can be split by using either a beam-splitting or a dichromatic prism, with two CCDs capturing images, one for the visual spectrum and one for the NIR spectrum. In another embodiment, there may be separate light paths for NIR and visual light and separate CCDs corresponding thereto.” [0107]; Figs. 1-3);
a controller (107) for:
processing the video feed of the surgical instrument in the low visibility imaging mode to detect the plurality of fluorescent fiducial markers (“the tracking of markers on tools could prevent tool-camera occlusions or collisions.” [0176]; “the image-based tracking module further identifies … the surgical tool based on the detected fluorescent markers.” [0205]);
generating an overlay of the surgical instrument (“The stereoscopic display 109t can also display fluorescent images as a color coded overlay or display an augmented reality image by overlaying the target points detected by the image-based tracking module 107… to detect the fluorescent markers in order to track the tools” [0107]; “The surgeon 100 can be provided with visual feedback, audio feedback, or haptic feedback 110 while viewing the stereoscopic display.” [0109]); and
rendering the overlay, while in the low visibility imaging mode, on a portion of the video feed including the surgical instrument, based on locations of the identified plurality of fluorescent fiducial markers (“The stereoscopic display 109 provides the acquired visual images. The stereoscopic display 109t can also display fluorescent images as a color coded overlay or display an augmented reality image by overlaying the target points detected by the image-based tracking module 107. The image-based tracking module 107 can apply image processing algorithms to detect the fluorescent markers in order to track the tools” [0107]; “a tracking module that performs pre-processing of the NIR image and visual tracking based on the processed image information.” [0108]); and
a screen (109) for displaying the video feed in the low visibility imaging mode with the overlay (“The surgeon receives visual feedback through the stereoscopic display 109” [0109]; Figs. 1-3).
Regarding claim 2, Azizian teaches the surgical robotic system according to claim 1, wherein the laparoscopic camera captures white light and near infrared (NIR) light images (“The light sources 102, 104, … and the image capturing by the CCD 105 are controlled and synchronized by the image acquisition and, control module 106.” [0106] “NIR and visual light can be split by using either a beam-splitting or a dichromatic prism, with two CCDs capturing images, one for the visual spectrum and one for the NIR spectrum. In another embodiment, there may be separate light paths for NIR and visual light and separate CCDs corresponding thereto.” [0107]; Figs. 1-3).
Regarding claim 3, Azizian teaches the surgical robotic system according to claim 2, wherein the low visibility imaging mode is a monochromatic NIR mode (“The other light source 102 is a narrow-band source of light (e.g. in the near infrared range) that is chosen according to the excitation wavelength of the fluorescent material.” [0105]; Figs. 1-3).
Claim Rejections - 35 USC § 103
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Azizian as applied to claim 1, and further in view of McLeod et al (US 20220409300), hereinafter, McLeod.
Regarding claim 4, Azizian teaches the surgical robotic system according to claim 1.
Azizian does not teach that the overlay is a virtual overlay and includes at least one of a line model, a mesh model, or a 3D surface model of the instrument.
However, in the surgical systems and methods field of endeavor, McLeod discloses systems and methods for providing surgical assistance based on operational context, which is analogous art. McLeod teaches that the overlay is a virtual overlay and includes at least one of a line model, a mesh model, or a 3D surface model of the instrument (504) (“In each of images 500-1 and 500-2 (collectively referred to herein as images 500), a surgical site including various surface objects is shown. Specifically, various anatomical structures 502 are shown (some of which are explicitly labeled while others are not) together with two surgical instruments 504 that are configured to manipulate tissue at the surgical site during a surgical procedure.” [0060] “In procedural phase 3, system 100 may continue with the subsurface anatomy model augmentation, while also adding an overlay of a 3D model of surface objects present at the surgical site such as instruments that may be occluded by smoke or another obscurant (“Show 3D model (instrument)”). [0085]; Figs. 5A-B).
Therefore, based on McLeod’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Azizian to employ the overlay that is a virtual overlay and that includes at least one of a line model, a mesh model, or a 3D surface model of the instrument, as taught by McLeod, in order to facilitate tracking of the medical instrument.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Azizian as applied to claim 1, and further in view of Eunjin et al (KR 102473037), hereinafter, Eunjin.
Regarding claim 5, Azizian teaches the surgical robotic system according to claim 1.
Azizian does not teach that the overlay is a masked overlay of the instrument.
However, in the surgical navigation systems and methods field of endeavor, Eunjin discloses a method and system for automatically controlling navigation of surgical tool based on reinforcement learning, which is analogous art. Eunjin teaches that the overlay is a masked overlay (820) of the instrument (“The information processing system 110 may extract a mask image obtained by segmenting the surgical tools located in the blood vessels from the image 120 in which the blood vessels including the surgical tools are photographed. Then, the information processing system 110 may overlay the mask image of the extracted surgical tool on the blood vessel path map considering only the calculated path.”; p. 5, 3rd para. “Then, the information processing system may generate an image 870 by overlaying the mask image 820 of the extracted surgical tool on the blood vessel path map 840 considering only the calculated path. In this way, by overlaying the blood vessel path map 840 and the mask image 820 from which the surgical tool is extracted, the blood vessel path and the location of the surgical tool can be accurately known in real time during the interventional procedure.”; p. 17, 5th para.; Fig. 8).
Therefore, based on Eunjin’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Azizian to employ the overlay that is a masked overlay of the instrument, as taught by Eunjin, in order to facilitate tracking of the medical instrument.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Azizian and Eunjin as applied to claim 5, and further in view of Geric et al (EP 4134011), hereinafter, Geric.
Regarding claim 6, Azizian modified by Eunjin teaches the surgical robotic system according to claim 5.
Azizian modified by Eunjin does not teach that the controller further generates the masked overlay of the instrument from the video feed while in the white light imaging mode.
However, in the surgical methods and systems field of endeavor, Geric discloses endoscopic vessel harvesting with thermal management and augmented reality display, which is analogous art. Geric teaches that the controller further generates the overlay from the video feed while in the white light imaging mode (“rendering a video stream including the visible-light images and an overlay depicting the temperatures present on at least some of the respective surfaces when applying the thermal energy; and presenting the video stream and overlay to a user on a display.” [0044]).
Therefore, based on Geric’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Azizian and Eunjin to employ the controller that further generates the overlay from the video feed while in the white light imaging mode, as taught by Geric, in order to improve tracking capabilities of the surgical system. In the invention of Azizian, Eunjin, and Geric, the overlay is the masked overlay of the instrument.
Claims 7-8 are rejected under 35 U.S.C. 103 as being unpatentable over Azizian as applied to claim 1, and further in view of Rockrohr et al (US 20230092980), hereinafter, Rockrohr.
Regarding claim 7, Azizian teaches the surgical robotic system according to claim 1, further comprising:
a surgeon console for receiving user input, wherein the instrument is actuated by the instrument drive unit in response to the user input (“In the semi-autonomous mode (FIG. 1) the surgeon 100 also provides commands to the visual servoing controller 108 during the operation.” [0109]; “ in manual mode (FIG. 2), the surgeon controls the surgical tool manually, as in conventional laparoscopic surgery, or through master-slave control 201 of a robot arm. The surgeon receives visual feedback through the stereoscopic display 109 and may also be provided with other visual feedback, audio feedback, or haptic feedback, but the control loop is solely closed through the surgeon.” [0110]).
Azizian does not teach a surgeon console including a handle controller.
However, in the surgical systems and methods field of endeavor, Rockrohr discloses a surgical robotic system setup, which is analogous art. Rockrohr teaches a surgeon console including a handle controller (“The surgeon console 30 also includes a plurality of user interface devices, such as foot pedals 36 and a pair of handle controllers 38a and 38b, which are used by a user to remotely control robotic arms 40.” [0039]; Fig. 1).
Therefore, based on Rockrohr’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Azizian to employ a surgeon console including a handle controller, as taught by Rockrohr, in order to improve the control of the medical instrument.
Regarding claim 8, Azizian modified by Rockrohr teaches the surgical robotic system according to claim 7, wherein Azizian teaches that the controller further tracks a position of the surgical instrument and updates a location of the overlay on the video feed (110) (210) based on the position of the surgical instrument (“The stereoscopic display 109t can also display fluorescent images as a color coded overlay or display an augmented reality image by overlaying the target points detected by the image-based tracking module 107… to detect the fluorescent markers in order to track the tools” [0107]; “The visual servoing controller 108 can receive the tracking information from the image-based tracking module 107, combine these with the intraoperative commands from the surgeon 100, and send appropriate commands to the robot in real-time in order to control the surgical robot 101 and the surgical tool(s) 110 to obtain a predetermined goal (e.g. anastomosis). The surgeon 100 can be provided with visual … feedback 110 while viewing the stereoscopic display.” [0109]; “ in manual mode (FIG. 2), the surgeon controls the surgical tool manually, as in conventional laparoscopic surgery, or through master-slave control 201 of a robot arm. The surgeon receives visual feedback through the stereoscopic display 109 and may also be provided with other visual feedback” [0110]; Figs. 1-2).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Azizian and Rockrohr as applied to claim 8, and further in view of Soto (US 20230240793), hereinafter, Soto.
Regarding claim 9, Azizian modified by Rockrohr teaches the surgical robotic system according to claim 8, wherein Azizian teaches that the controller tracks movement of the surgical instrument based on the plurality of fluorescent fiducial markers (“The stereoscopic display 109t can also display fluorescent images as a color coded overlay or display an augmented reality image by overlaying the target points detected by the image-based tracking module 107… to detect the fluorescent markers in order to track the tools” [0107]; “The visual servoing controller 108 can receive the tracking information from the image-based tracking module 107, combine these with the intraoperative commands from the surgeon 100, and send appropriate commands to the robot in real-time in order to control the surgical robot 101 and the surgical tool(s) 110 to obtain a predetermined goal (e.g. anastomosis). The surgeon 100 can be provided with visual … feedback 110 while viewing the stereoscopic display.” [0109]; “ in manual mode (FIG. 2), the surgeon controls the surgical tool manually, as in conventional laparoscopic surgery, or through master-slave control 201 of a robot arm. The surgeon receives visual feedback through the stereoscopic display 109 and may also be provided with other visual feedback” [0110]; Figs. 1-2).
Azizian modified by Rockrohr does not teach that the controller tracks movement of the surgical instrument based on kinematics data of the robotic arm.
However, in the surgical systems field of endeavor, Soto discloses quick-connect mounting system for surgical components, which is analogous art. Soto teaches that the controller tracks movement of the surgical instrument based on kinematics data of the robotic arm (“using the navigation system 32, the pose of the tool 26 can be determined by tracking the location of the base 16 and the associated manipulator coordinate system via the manipulator tracker 52B and calculating the pose of the tool 26 based on joint encoder data from the joint encoders 22 (and/or motor encoders) at the joints J1-J6 (using kinematic data) and based on a known geometric relationship between the tool 26 and the manipulator 14. Ultimately, the localizer 44 and the trackers 52A, 52B, 54, 56, PT enable the determination of the pose of the tool 26 and the patient's anatomy so the navigation system 32 knows the relative relationship between the tool 26 and the patient's anatomy.” [0071]; Fig. 1).
Therefore, based on Soto’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Azizian and Rockrohr to employ the controller that tracks movement of the surgical instrument based on kinematics data of the robotic arm, as taught by Soto, in order to improve tracking of the surgical instrument during a surgery.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Azizian et al (US 20190282307), hereinafter Azizian, in view of Podhajsky (US 20080015664), hereinafter Podhajsky.
Regarding claim 10, Azizian teaches a surgical robotic system (Figs. 1-3) comprising:
a robotic arm (“a robot arm” [0110]) including an instrument drive unit (108) (201) (“The visual servoing controller 108 can receive the tracking information from the image-based tracking module 107, combine these with the intraoperative commands from the surgeon 100, and send appropriate commands to the robot in real-time in order to control the surgical robot 101 and the surgical tool(s) 110 to obtain a predetermined goal (e.g. anastomosis).” [0109]; “the surgeon controls the surgical tool … through master-slave control 201 of a robot arm.” [0110]; Figs. 1-2;);
a surgical instrument (110) including a plurality of fluorescent fiducial markers (“The system is based on deploying fluorescent markers on the organ under surgery and/or on the surgical tool, allowing for tracking of the fluorescent markers in real-time and controlling the surgical tool via visually servoing.” [0100]), wherein the surgical instrument is coupled to and actuatable by the instrument drive unit (“the surgeon controls the surgical tool … through master-slave control 201 of a robot arm.” [0110]; Figs. 2-3);
a laparoscopic camera for capturing a video feed of the surgical instrument (“ the camera may be mounted on a robotic arm and the marker position and orientation could in turn control the position and orientation of the camera. This may allow the marker to always be in a suitable field-of-view or angle relative to the camera. Additionally, the tracking of markers on tools could prevent tool-camera occlusions or collisions.” [0176]);
an image processing device (106) coupled to the laparoscopic camera, the image processing device operatable in a white light imaging mode (104) and a low visibility imaging mode (102) (“The light sources 102, 104, … and the image capturing by the CCD 105 are controlled and synchronized by the image acquisition and, control module 106.” [0106] “NIR and visual light can be split by using either a beam-splitting or a dichromatic prism, with two CCDs capturing images, one for the visual spectrum and one for the NIR spectrum. In another embodiment, there may be separate light paths for NIR and visual light and separate CCDs corresponding thereto.” [0107]; Figs. 1-3);
a controller (107) for:
processing the video feed of the surgical instrument in the low visibility imaging mode to detect the plurality of fluorescent fiducial markers (“the tracking of markers on tools could prevent tool-camera occlusions or collisions.” [0176]; “the image-based tracking module further identifies … the surgical tool based on the detected fluorescent markers.” [0205]);
generating the overlay of the surgical instrument (“The stereoscopic display 109t can also display fluorescent images as a color coded overlay or display an augmented reality image by overlaying the target points detected by the image-based tracking module 107… to detect the fluorescent markers in order to track the tools” [0107]; “The surgeon 100 can be provided with visual feedback, audio feedback, or haptic feedback 110 while viewing the stereoscopic display.” [0109]); and
rendering the overlay, while in the low visibility imaging mode, on a portion of the video feed including the surgical instrument, based on locations of the identified plurality of fluorescent fiducial markers (“The stereoscopic display 109 provides the acquired visual images. The stereoscopic display 109t can also display fluorescent images as a color coded overlay or display an augmented reality image by overlaying the target points detected by the image-based tracking module 107. The image-based tracking module 107 can apply image processing algorithms to detect the fluorescent markers in order to track the tools” [0107]; “a tracking module that performs pre-processing of the NIR image and visual tracking based on the processed image information.” [0108]); and
a screen (109) for displaying the video feed in the low visibility imaging mode with the overlay (“The surgeon receives visual feedback through the stereoscopic display 109” [0109]; Figs. 1-3).
Azizian does not teach selecting an overlay for the instrument from a plurality of overlays.
However, in the surgical systems and methods field of endeavor, Podhajsky discloses systems and methods for thermally profiling radiofrequency electrodes, which is analogous art. Podhajsky teaches selecting an overlay for the instrument from a plurality of overlays (“The method may also include superimposing one or more overlays atop the tissue to ascertain a desired treatment area 512 for the tissue based on tissue type, generator settings, type of surgical device, tip configuration of the surgical device and/or depth of penetration of the surgical device. The method also includes the step of selecting a desired overlay and configuring the generator and surgical instrument accordingly to treat tissue.” [0111]).
Therefore, based on Podhajsky’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Azizian to employ the step of selecting an overlay for the instrument from a plurality of overlays, as taught by Podhajsky, in order to improve the visualization and tracking of the surgical instrument thereby facilitating surgical treatments.
Claims 11-12 are rejected under 35 U.S.C. 103 as being unpatentable over Azizian and Podhajsky as applied to claim 10, and further in view of McLeod et al (US 20220409300), hereinafter, McLeod, and Eunjin et al (KR 102473037), hereinafter, Eunjin.
.
Regarding claim 11, Azizian modified by Podhajsky teaches the surgical robotic system according to claim 10.
Azizian modified by Podhajsky does not teach that the plurality of overlays includes a virtual overlay and a masked overlay.
However, in the surgical systems and methods field of endeavor, McLeod discloses systems and methods for providing surgical assistance based on operational context, which is analogous art. McLeod teaches that the overlay is a virtual overlay (“In each of images 500-1 and 500-2 (collectively referred to herein as images 500), a surgical site including various surface objects is shown. Specifically, various anatomical structures 502 are shown (some of which are explicitly labeled while others are not) together with two surgical instruments 504 that are configured to manipulate tissue at the surgical site during a surgical procedure.” [0060] “In procedural phase 3, system 100 may continue with the subsurface anatomy model augmentation, while also adding an overlay of a 3D model of surface objects present at the surgical site such as instruments that may be occluded by smoke or another obscurant (“Show 3D model (instrument)”). [0085]; Figs. 5A-B).
Therefore, based on McLeod’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Azizian and Podhajsky to employ the plurality of overlays that includes a virtual overlay, as taught by McLeod, in order to facilitate tracking of the medical instrument.
Azizian modified by Podhajsky and McLeod does not teach that the overlay is a masked overlay.
However, in the surgical navigation systems and methods field of endeavor, Eunjin discloses a method and system for automatically controlling navigation of surgical tool based on reinforcement learning, which is analogous art. Eunjin teaches that the overlay is a masked overlay (820) (“The information processing system 110 may extract a mask image obtained by segmenting the surgical tools located in the blood vessels from the image 120 in which the blood vessels including the surgical tools are photographed. Then, the information processing system 110 may overlay the mask image of the extracted surgical tool on the blood vessel path map considering only the calculated path.”; p. 5, 3rd para. “Then, the information processing system may generate an image 870 by overlaying the mask image 820 of the extracted surgical tool on the blood vessel path map 840 considering only the calculated path. In this way, by overlaying the blood vessel path map 840 and the mask image 820 from which the surgical tool is extracted, the blood vessel path and the location of the surgical tool can be accurately known in real time during the interventional procedure.”; p. 17, 5th para.; Fig. 8).
Therefore, based on Eunjin’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Azizian, Podhajsky, and McLeod to employ the overlay that is a masked overlay, as taught by Eunjin, in order to facilitate tracking of the medical instrument.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Azizian, Podhajsky, McLeod, and Eunjin as applied to claim 11, and further in view of Geric et al (EP 4134011), hereinafter, Geric.
Regarding claim 13, Azizian modified by Podhajsk, McLeod, and Eunjin teaches the surgical robotic system according to claim 11.
Azizian modified by Podhajsk, McLeod, and Eunjin does not teach that the controller further generates the masked overlay of the instrument from the video feed while in the white light imaging mode.
However, in the surgical methods and systems field of endeavor, Geric discloses endoscopic vessel harvesting with thermal management and augmented reality display, which is analogous art. Geric teaches that the controller further generates the overlay from the video feed while in the white light imaging mode (“rendering a video stream including the visible-light images and an overlay depicting the temperatures present on at least some of the respective surfaces when applying the thermal energy; and presenting the video stream and overlay to a user on a display.” [0044]).
Therefore, based on Geric’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Azizian, Podhajsky, McLeod, and Eunjin to employ the controller that further generates the overlay from the video feed while in the white light imaging mode, as taught by Geric, in order to improve tracking capabilities of the surgical system. In the invention of Azizian, Eunjin, and Geric, the overlay is the masked overlay of the instrument.
Claims 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Azizian and Podhajsky as applied to claim 10, and further in view of Rockrohr et al (US 20230092980), hereinafter, Rockrohr.
Regarding claim 14, Azizian modified by Podhajsky teaches the surgical robotic system according to claim 10.
Azizian teaches a surgeon console for receiving user input, wherein the instrument is actuated by the instrument drive unit in response to the user input (“In the semi-autonomous mode (FIG. 1) the surgeon 100 also provides commands to the visual servoing controller 108 during the operation.” [0109]; “ in manual mode (FIG. 2), the surgeon controls the surgical tool manually, as in conventional laparoscopic surgery, or through master-slave control 201 of a robot arm. The surgeon receives visual feedback through the stereoscopic display 109 and may also be provided with other visual feedback, audio feedback, or haptic feedback, but the control loop is solely closed through the surgeon.” [0110]).
Azizian modified by Podhajsky does not teach a surgeon console including a handle controller.
However, in the surgical systems and methods field of endeavor, Rockrohr discloses a surgical robotic system setup, which is analogous art. Rockrohr teaches a surgeon console including a handle controller (“The surgeon console 30 also includes a plurality of user interface devices, such as foot pedals 36 and a pair of handle controllers 38a and 38b, which are used by a user to remotely control robotic arms 40.” [0039]; Fig. 1).
Therefore, based on Rockrohr’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Azizian and Podhajsky to employ a surgeon console including a handle controller, as taught by Rockrohr, in order to improve the control of the medical instrument.
Regarding claim 15, Azizian modified by Podhajsky and Rockrohr teaches the surgical robotic system according to claim 14, wherein Azizian teaches that the controller further tracks a position of the surgical instrument and updates a location of the overlay on the video feed (110) (210) based on the position of the surgical instrument (“The stereoscopic display 109t can also display fluorescent images as a color coded overlay or display an augmented reality image by overlaying the target points detected by the image-based tracking module 107… to detect the fluorescent markers in order to track the tools” [0107]; “The visual servoing controller 108 can receive the tracking information from the image-based tracking module 107, combine these with the intraoperative commands from the surgeon 100, and send appropriate commands to the robot in real-time in order to control the surgical robot 101 and the surgical tool(s) 110 to obtain a predetermined goal (e.g. anastomosis). The surgeon 100 can be provided with visual … feedback 110 while viewing the stereoscopic display.” [0109]; “ in manual mode (FIG. 2), the surgeon controls the surgical tool manually, as in conventional laparoscopic surgery, or through master-slave control 201 of a robot arm. The surgeon receives visual feedback through the stereoscopic display 109 and may also be provided with other visual feedback” [0110]; Figs. 1-2).
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Azizian, Podhajsky, and Rockrohr as applied to claim 15, and further in view of Soto (US 20230240793), hereinafter, Soto.
Regarding claim 16, Azizian modified by Podhajsky and Rockrohr teaches the surgical robotic system according to claim 15, wherein Azizian teaches that the controller tracks movement of the surgical instrument based on the plurality of fluorescent fiducial markers (“The stereoscopic display 109t can also display fluorescent images as a color coded overlay or display an augmented reality image by overlaying the target points detected by the image-based tracking module 107… to detect the fluorescent markers in order to track the tools” [0107]; “The visual servoing controller 108 can receive the tracking information from the image-based tracking module 107, combine these with the intraoperative commands from the surgeon 100, and send appropriate commands to the robot in real-time in order to control the surgical robot 101 and the surgical tool(s) 110 to obtain a predetermined goal (e.g. anastomosis). The surgeon 100 can be provided with visual … feedback 110 while viewing the stereoscopic display.” [0109]; “ in manual mode (FIG. 2), the surgeon controls the surgical tool manually, as in conventional laparoscopic surgery, or through master-slave control 201 of a robot arm. The surgeon receives visual feedback through the stereoscopic display 109 and may also be provided with other visual feedback” [0110]; Figs. 1-2).
Azizian modified by Podhajsky and Rockrohr does not teach that the controller tracks movement of the surgical instrument based on kinematics data of the robotic arm.
However, in the surgical systems field of endeavor, Soto discloses quick-connect mounting system for surgical components, which is analogous art. Soto teaches that the controller tracks movement of the surgical instrument based on kinematics data of the robotic arm (“using the navigation system 32, the pose of the tool 26 can be determined by tracking the location of the base 16 and the associated manipulator coordinate system via the manipulator tracker 52B and calculating the pose of the tool 26 based on joint encoder data from the joint encoders 22 (and/or motor encoders) at the joints J1-J6 (using kinematic data) and based on a known geometric relationship between the tool 26 and the manipulator 14. Ultimately, the localizer 44 and the trackers 52A, 52B, 54, 56, PT enable the determination of the pose of the tool 26 and the patient's anatomy so the navigation system 32 knows the relative relationship between the tool 26 and the patient's anatomy” [0071]; Fig. 1).
Therefore, based on Soto’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Azizian, Podhajsky, and Rockrohr to employ the controller that tracks movement of the surgical instrument based on kinematics data of the robotic arm, as taught by Soto, in order to improve tracking of the surgical instrument during a surgery.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEXEI BYKHOVSKI whose telephone number is (571)270-1556. The examiner can normally be reached on Monday-Friday: 8:30am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pascal Bui Pho can be reached on 571-272-2714. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALEXEI BYKHOVSKI/
Primary Examiner, Art Unit 3798
Prosecution Insights