DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments, See Remarks filed 25 June 2026 at pages 7-10, with respect to the rejection under 35 U.S.C. 103 have been fully considered but are not persuasive.
Applicant’s amendment necessitated new ground of rejection and Seita is no longer relied upon to disclose a mechanical end limit. Notwithstanding the new ground of rejection, Examiner respectfully disagrees with Applicant’s position that “Seita’s disclosure of movement within pre-determined safe ranges does not disclose or suggest “commanding the end effector to move to a mechanical end limit” and that a “software- or experiment-defined safe sampling range, however, is not a mechanical end limit of the end effector” as stated on page 8 of Remarks. A “mechanical end limit” is a physical limit or boundary that is a stopping point in a mechanical system. Seita uses a computer to control a robotic arm to move to specific locations within a pre-defined safe range of locations. The range sets a limit that the mechanical arm will not ideally exceed, thereby constituting a mechanical end limit.
Applicant’s remaining arguments are directed to the newly-added subject matter that prompted the new ground of rejection herein. Therefore, those arguments are considered moot because U.S. Pat. Appl. Pub. No. 20050251110 is now relied upon to disclose, among other subject matter, a mechanical end limit of the jaws of a gripper mechanism where the jaws are commanded to move to fully open and fully closed positions, which are mechanical end points of the jaws.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed inventions absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1 and 3-5 are rejected under 35 U.S.C. 103 as being unpatentable over Fast and Reliable Autonomous Surgical Debridement with Cable-Driven Robots Using a Two-Phase Calibration Procedure to Seita in view of U.S. Pat. No. 8,620,473 to Diolaiti, in view of Autonomous High Precision Positioning of Surgical Instruments in Robot-Assisted Minimally Invasive Surgery under Visual Guidance to Staub, and in further view of U.S. Pat. Appl. Pub. No. 20050251110 to Nixon (hereinafter “Nixon”).
Regarding claim 1, Seita teaches a surgical robotic system comprising:
a robotic arm (Seita, pg. 3, Fig. 3, “dVRK arm”); and
a camera (Seita, pg. 3, Fig. 3, “Endoscopic Stereo Camera”) that is configured to output a video stream (Seita, pg. 7, section VI.B, “To investigate runtime in more detail, we ran a frame-by-frame analysis of a 25fps video of a pumpkin seed trial. We considered six phases for each fragment: to seed, lower, grip, raise, to receptacle, and drop & rotate”);
a surgical instrument (Seita, pg. 3, section III, “We use one dVRK arm with a gripper end-effector, called a large needle driver (Figure 2)”; pg. 1, Fig. 1, “the dVRK, wrapped with red tape, randomly explores its workspace”) including an end effector having at least one fiducial marker (Seita, pg. 1, Abstract, “we place a red calibration marker on the end effector and let it randomly move through a set of open-loop trajectories to obtain a large sample set of camera pixels and internal robot end-effector configurations.”); and
a controller (Seita, pg. 1, Fig. 1 caption, “we train a Deep Neural Network (DNN)”; Training a neural network requires a computer.; section I, “the dVRK can achieve positional control in the workspace”; The dVRK is a controller.) configured to:
receive the video stream from the camera (Seita, pg. 1, Fig. 1 caption, “The robot positions and orientations are recorded by a camera and its internal joint sensors. After data cleaning, we train a Deep Neural Network (DNN) to map from camera position (not images) and robot orientation (extracted from the images) to base position. Phase II: we apply the DNN on a calibration grid and a human directly corrects the resulting end effector positions.”; pg. 7, section VI.B, “To investigate runtime in more detail, we ran a frame-by-frame analysis of a 25fps video of a pumpkin seed trial.”);
detect the at least one fiducial marker on the end effector in the video stream using a machine learning algorithm (Seita, pg. 1, section I, “we train a Deep Neural Network (DNN) to map from camera position (not images) and robot orientation (extracted from the images) to base position.”; pg. 4, section IV.A, “Forming XDNN also requires the camera position xc at these points, so before executing the trajectories we apply red tape to the end-effector, thus allowing the dVRK to use HSV thresholding to detect the location of red contours.”); and
calculate a calibration factor (Seita, pg. 1, Abstract, “transformation bias.”), and correlating the positional information derived from the video stream with kinematic positional information of commanded movement (random trajectories) of the end effector (Seita, pg. 1, section I, “robot orientation”) with kinematic positional information of commanded movement (Seita, pg. 1, section I, “commanded to some position and orientation”), but does not teach that which is explicitly taught by Diolaiti.
Diolaiti teaches a robotic arm including (Diolaiti, FIG. 1; col. 12, ll. 11-29, “In one embodiment a set-up arm assembly 2114 is a modified da Vinci® Surgical System arm assembly.”):
a camera (Diolaiti, FIG. 8; col. 15, ll. 15-48, “a stereoscopic endoscopic image capture component 1756”); and
a surgical instrument (Diolaiti, FIG. 8; “end effectors 1748a, 1748b”. Each end effector instrument receives motion commands (control signal). See id. at col. 15, ll. 21-48).
Seita discloses a surgical robotic system that uses a stereo endoscopic camera and a robotic arm with red tape attached to an end effector as a fiducial tracking marker to calibrate the positional accuracy of the robotic system. Thus, Seita shows that it was known in the art before the effective filing date of the claimed invention to calibrate a surgical robotic system using a fiducial marker attached to an end effector at a distal end of a robotic arm, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, providing an accurate representation of a surgical instrument in the precise area that the surgeon is operating on. Diolaiti discloses a surgical robotic system that uses a single robotic arm with a stereoscopic endoscope and end effectors at a distal end thereof (see Diolaiti at col. 15, ll. 34-48). Thus, Diolaiti shows that it was known in the art before the effective filing date of the claimed invention to use a robotic arm having both a camera and a surgical instrument, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, providing an accurate representation of a surgical instrument in the precise area that the surgeon is operating on.
A person of ordinary skill in the art would have been motivated to change the location of the stereo endoscopic camera disclosed by Seita to be attached to the same robotic arm as the surgical instrument as disclosed by Diolaiti, to thereby acquire the images of an end effector and any fiducials thereof. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of providing a closer and more detailed image of the calibration target and the area of a patient being operated upon.
Seita in view of Diolaiti does not teach that which is explicitly taught by Staub.
Staub teaches determine whether the end effector is viewable by the camera (Staub, pg. 67, section II.C, “a practical difficulty during the alignment of surgical instruments with a desired position lies in the fact that the instrument is not necessarily in the field of view of the camera and no image-features can be extracted.”), including determining whether the end effector is within a field of view of the camera (Staub, pg. 67, section II.C.1, “PBVS is used to drive the instrument to a reconstructed point which is located within the view of the camera”. Before PBVS is used, the instrument has not yet reached the reconstructed point within the camera’s field of view.); and
command movement of the surgical instrument until the instrument is viewable (detectable via image feature extraction) by the camera in response to determining that the end effector is not viewable by the camera (Staub, pg. 67, section II.C, “In order to command surgical instruments with a high precision to a desired position, we propose a switching servoing scheme. First, the instrument has to be driven to the target Cartesian coordinate which is in sight of the camera, employing position-based servoing ... The proposed switching scheme is not only necessary to drive the instruments into the field of view of the camera, but also has a positive effect on the convergence characteristic of the image-based part.”), wherein commanding movement is repeated until the controller confirms that the end effector is viewable by the camera (After the PBVS brings the end-effector into view of the camera, the image-based visual servoing (IBVS) incrementally moves the instrument until it reaches a final position when the error is sufficiently minimized. See Staub at FIG. 3 and section II.C. ).
Seita and Diolaiti are analogous to the claimed invention for the reasons provided above. Staub discloses a multi-arm robotic system (See Fig. 2(a)) that commands movement of an end effector until it is viewable by the camera to then precisely calibrate the position of the end-effector. The calibration includes a visual servoing switching scheme that uses position-based visual servoing (PBVS) if the instrument is not in the field of view of the camera and image-based visual servoing (IBVS) once the instrument has been driven into the field of view. Thus, Staub shows that it was known in the art before the effective filing date of the claimed invention to calibrate a surgical robotic system using a camera attached to a robotic arm and a moveable (controllable) end effector by commanding the end effector to move towards a particular reconstructed position, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, obtaining precise actuation of surgical instruments to reflect movement commands.
A person of ordinary skill in the art would have been motivated to modify the calibration procedure for the robotic arm of Seita in view of Diolaiti to incorporate the visual servoing switch scheme disclosed by Staub, to thereby detect when the end effector and/or its fiducial marker are not viewable by the camera and in response, command movement of the end effector and/or the robotic arm until end effector is within the camera’s field of view at the desired reconstructed final position. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of requiring less manual effort to calibrate the system.
Seita in view of Diolaiti and in further view of Staub does not teach that which is explicitly taught by Nixon.
Nixon teaches calculate a calibration factor (The calibration offset is a calibration factor. See Nixon at par. 67, “To facilitate identification of the end effector mutual engagement position, it is helpful to limit the analysis of the data to a limited range. The data may be restricted to a position range within which engagement is expected to occur, thereby assuming that the calibration offset will be within a predetermined range. For example, it may be assumed that the calibration offset for the end effector mutual engagement configuration will be between a nominal or initially commanded configuration of the jaws being open by 5 degrees (+5 degrees), and the jaws being squeezed past closed by 20 degrees (-20 degrees).”) for each degree of freedom of the end effector (The jaws of the end effector have one degree of freedom that controls their separation angle. See Nixon at par. 56, “the gripping members of handle 226 again define a separation angle that is substantially equal to the separation angle defined by the end effector elements.”) and correlating positional information with kinematic positional information of the movement of the end effector with kinematic positional information of commanded movement of the end effector to the mechanical end limit (Nixon, par. 69, “Referring now to FIGS. 11 and 12A-12C, the mutual engagement configuration of the end effector should correlate with the position of greatest curvature in the negative direction of the commanded torque data. This can be identified from the minima of the second derivative of the commanded torque.”); and
apply the calibration factor to adjust inverse kinematics used to generate motor commands for the robotic arm (Nixon, par. 69, “Once the mutual engagement configuration is known, calibration of the mounted tool can then be effected by applying a difference between the actual position and the commanded position of the end effectors as an offset to the grip controllers described above, for example, so that the closed configuration of the handle 226 (at which the handle members first engage the resilient bumper 60 a) corresponds to the mutual engagement configuration of the end effectors. This difference can be stored as an offset in grip calibration data table 405 for the tool/manipulator combination.”).
Seita in view of Diolaiti and in further view of Staub is analogous to the claimed invention for the reasons provided above. Nixon discloses a surgical robot with an articulated surgical robot arm that holds an endoscope connected to a display. See Nixon at par. 36. The surgical robot includes an end effector having a pair of mechanical jaws and a mechanical end limit where the jaws are closed. See id. at par. 67. The jaws are commanded to move via a commanded torque to change their position of engagement, which is assumed to be within a range of “an initially commanded configuration of the jaws”. Id. Once the calibration offset is determined, the difference between the actual and commanded position is stored as an offset in a table for subsequent use. See id. at pars. 69-70. Thus, Nixon shows that it was known in the art before the effective filing date of the claimed invention to command a pair of mechanical jaws to their mechanical end limit and recording the difference between the actual and commanded positions as a calibration offset for the jaws, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, obtaining precise actuation of surgical instruments to reflect movement commands.
A person of ordinary skill in the art would have been motivated to modify the calibration procedure for the robotic arm of Seita in view of Diolaiti and in further view of Staub to command the jaws to move each degree of freedom of the end effector to a mechanical end limit as disclosed by Nixon, to thereby determine the calibration offset of each degree of freedom by commanding the respective degree of freedom to move the end effector to a nominal mechanical end limit where the jaws are visible to the camera including the configuration in the closed configuration. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of mitigating commanded movement of the jaws to grip tissue during a surgery from applying excessive torque and harming the patient.
Regarding claim 3, Seita in view of Diolaiti, in further view of Staub and in further view of Nixon teaches the surgical robotic system according to claim 1, wherein the controller is further configured to determine whether the surgical instrument is detected by the camera (Seita, pg. 1, Abstract, “we place a red calibration marker on the end effector”).
Regarding claim 4, Seita in view of Diolaiti, in further view of Staub and in further view of Nixon teaches the surgical robotic system according to claim 3, wherein the controller is further configured to determine whether the surgical instrument is detected by the camera based on a distance of the surgical instrument from the camera (The dataset XDNN is collected, which includes the three-dimensional camera position coordinates xc of the instrument position with respect to the camera frame, the three-dimensional coordinates xb of the instrument with respect to the base, and Φ, the yaw, pitch, and roll of the instrument. See Seita at pgs. 2-4, sections III and IV. The difference between a coordinate of the base and a coordinate of the instrument defines a distance from the instrument to the camera. Thus, the detection, which uses this model, is based on said distance.).
Regarding claim 5, Seita in view of Diolaiti, in further view of Staub and in further view of Nixon teaches the surgical robotic system according to claim 3, wherein the controller is further configured to move the surgical instrument until the end effector is within a field of view of the camera (Staub, pg. 67, section II.C, “The proposed switching scheme is not only necessary to drive the instruments into the field of view of the camera, but also has a positive effect on the convergence characteristic of the image-based part.”).
The rationale for obviousness is the same as provided for claim 1.
Claims 6, 8 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Seita in view of Diolaiti, in further view of Staub and in further view of Nixon, and in further view of U.S. Pat. No. 9,014,856 to Manzo.
Regarding claim 6, Seita in view of Diolaiti, in further view of Staub and in further view of Nixon teaches the surgical robotic system according to claim 1, wherein the surgical instrument includes an end effector including at least one degree of freedom (Seita, Fig. 2 caption, “three axes of rotation”), but does not teach that which is explicitly taught by Manzo.
Manzo teaches the controller is further configured to identify a type of the surgical instrument (Manzo, col. 9, ll. 62-67, “the data from the tool memory will often include a tool-type to signal to the master control console how it is to be controlled. In some cases, the data will also include tool calibration information. The data may be provided in response to a request signal from the computer 151.”).
Seita in view of Diolaiti, in further view of Staub and in further view of Nixon is analogous to the claimed invention for the reasons provided above. Manzo discloses a surgical robotic system with multiple robotic arms (Manzo, FIG. 2A) that use a surgical instrument having an integrated circuit (Manzo, FIG. 4B; col. 9, ll. 45-67) including a memory that stores data describing the type of instrument and corresponding calibration data. Thus, Manzo shows that it was known in the art before the effective filing date of the claimed invention to calibrate a surgical robotic system for specific instruments based on a type of instrument, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, obtaining precise actuation of surgical instruments to reflect movement commands.
A person of ordinary skill in the art would have been motivated to add the tool type and calibration data in a memory as disclosed by Manzo to the surgical instrument of Seita in view of Diolaiti, in further view of Staub and in further view of Nixon for the computer that controls the instrument to thereby read the memory of the instrument and obtain initial calibration data as part of a calibration routine based on the type of instrument. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of improving the starting point of calibration, thereby reducing the amount of time needed to fully calibrate the system when a new tool is attached to one of the robotic arms.
Regarding claim 8, Seita in view of Diolaiti, in view of Staub, in view of Nixon and in further view of Manzo teaches the surgical robotic system according to claim 6, wherein the controller is further configured to select a calibration routine based on the type of the surgical instrument (Manzo, col. 9, ll. 45-67).
The rationale for obviousness is the same as provided for claim 6.
Regarding claim 9, Seita in view of Diolaiti, in view of Staub, in view of Nixon and in further view of Manzo teaches the surgical robotic system according to claim 8, wherein the controller is further configured to calibrate the surgical instrument by:
moving the end effector in the at least one degree of freedom to a calibration position based on the calibration routine (Seita, pg. 7, section VI.B, “To investigate runtime in more detail, we ran a frame-by-frame analysis of a 25fps video of a pumpkin seed trial. We considered six phases for each fragment: to seed, lower, grip, raise, to receptacle, and drop & rotate”);
receiving the video stream of the end effector being moved to the calibration position, the video stream including positional information of the end effector (Seita, pg. 1, Abstract, “In Phase I (coarse), we place a red calibration marker on the end effector and let it randomly move through a set of open-loop trajectories to obtain a large sample set of camera pixels and internal robot end-effector configurations.”);
correlating the positional information to the calibration position (Seita, pg. 1, Abstract, “This coarse data is then used to train a Deep Neural Network (DNN) to learn the coarse transformation bias.”); and
calculating the calibration factor based on a difference between the positional information and the calibration position (Seita, pg. 1, Abstract, “transformation bias.”; pg. 1, Fig. 1 caption, “The robot positions and orientations are recorded by a camera and its internal joint sensors. After data cleaning, we train a Deep Neural Network (DNN) to map from camera position (not images) and robot orientation (extracted from the images) to base position. Phase II: we apply the DNN on a calibration grid and a human directly corrects the resulting end effector positions.”).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Seita in view of Diolaiti, in view of Staub, in view of Nixon and in further view of Manzo, and in further view of U.S. Pat. No. 8,147,503 to Zhao.
Regarding claim 7, Seita in view of Diolaiti, in view of Staub, in view of Nixon and in further view of Manzo teaches the surgical robotic system according to claim 6, but does not teach that which is explicitly taught by Zhao.
Zhao teaches wherein the controller is further configured to identify the type of the surgical instrument from the video stream (Zhao, col. 26, ll. 16-32, “FIG. 5B illustrates video images 101AV and 101BV for a respective pair of tools 101A and 101B in the field of view 510. FIG. 5B further illustrates pose information 101AK and 101BK based on kinematics for the respective tools 101A and 101B in and around the field of view 510. The video images 101AV and 101BV and the pose information 101AK and 101BK for the respective tools 101A and 101B may be adaptively fused together to improve the overall pose information for each. A plurality of marker dots 502A and 502B or other types of markers, may be affixed to the respective tools 101A and 101B.”; col. 32, ll. 24-36, “Another example is that surgeon can use one type of tracked tool (e.g., an ultrasound tool) to draw marks to indicate regions of interest and then use a different type of tracked tool (e.g., a cautery tool) to operate or perform a surgical or other medical procedure in selected regions of interest.”. Identifying the specific tool thereby identifies what type of tool it is.).
Seita in view of Diolaiti, in view of Staub, in view of Nixon and in further view of Manzo is analogous to the claimed invention for the reasons provided above. Zhao discloses a surgical robotic system with multiple robotic arms that identifies specific tools in video according to markers on each tool. Thus, Zhao shows that it was known in the art before the effective filing date of the claimed invention to use visual markers on surgical instruments to improve pose estimation, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, obtaining precise actuation of surgical instruments to reflect movement commands.
A person of ordinary skill in the art would have been motivated to add a unique marker to a surgical instrument as disclosed by Zhao and configure the controller disclosed by Seita in view of Diolaiti, in view of Staub, in view of Nixon and in further view of Manzo to determine the type of instrument based on the marker if present in a video stream and based on the data in the memory of the instrument if the data is present. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of improving the starting point of calibration, thereby reducing the amount of time needed to calibrate the system when a new tool is attached to a robotic arm.
Claims 10-12, 16 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Seita in view of Staub and in further view of Nixon.
Regarding claim 10, Seita teaches a method for calibrating a surgical instrument, the method comprising:
transmitting from a camera (Seita, pg. 3, Fig. 3, “Endoscopic Stereo Camera”) to a controller (Seita, pg. 1, Fig. 1 caption, “we train a Deep Neural Network (DNN)”; Training a neural network requires a computer.; section I, “the dVRK can achieve positional control in the workspace”. The dVRK is a controller.) a video stream of a surgical instrument coupled to a robotic arm (Seita, pg. 3, section III, “We use one dVRK arm with a gripper end-effector, called a large needle driver (Figure 2)”; pg. 7, section VI.B, “To investigate runtime in more detail, we ran a frame-by-frame analysis of a 25fps video of a pumpkin seed trial. We considered six phases for each fragment: to seed, lower, grip, raise, to receptacle, and drop & rotate”), the surgical instrument including an end effector (Seita, pg. 3, section III, “one dVRK arm with a gripper end-effector”) having at least one fiducial marker (Seita, pg. 1, Abstract, “we place a red calibration marker on the end effector and let it randomly move through a set of open-loop trajectories to obtain a large sample set of camera pixels and internal robot end-effector configurations.”);
detecting, at the controller, the at least one fiducial marker on the end effector in the video stream using a machine learning algorithm (Seita, pg. 1, section I, “we train a Deep Neural Network (DNN) to map from camera position (not images) and robot orientation (extracted from the images) to base position.”; pg. 4, section IV.A, “Forming XDNN also requires the camera position xc at these points, so before executing the trajectories we apply red tape to the end-effector, thus allowing the dVRK to use HSV thresholding to detect the location of red contours.”);
calculating, by the controller, a calibration factor (Seita, pg. 1, Abstract, “transformation bias.”), and correlating the positional information derived from the video stream with kinematic positional information of commanded movement (random trajectories) of the end effector (Seita, pg. 1, section I, “robot orientation”) with kinematic positional information of commanded movement (Seita, pg. 1, section I, “commanded to some position and orientation”); and
applying, by the controller, the calibration factor to adjust inverse kinematics used to generate motor commands for the robotic arm (see Seita at Section IV regarding the coarse calibration in Phase I. The transformation bias learned by the Deep Neural Network is applied to correct inverse kinematics during the fine calibration of Phase II.), to control movement of the surgical instrument (Seita, Abstract, “In Phase II (fine), the bias from Phase I is applied to move the end-effector toward a small set of specific target points on a printed sheet. For each target, a human operator manually adjusts the end-effector position by direct contact (not through teleoperation) and the residual compensation bias is recorded. This fine data is then used to train a Random Forest (RF) to learn the fine transformation bias. Subsequent experiments suggest that without calibration, position errors average 4.55mm.”), but does not teach that which is explicitly taught by Staub.
Staub teaches determining whether the end effector is viewable by the camera (Staub, pg. 67, section II.C, “a practical difficulty during the alignment of surgical instruments with a desired position lies in the fact that the instrument is not necessarily in the field of view of the camera and no image-features can be extracted.”), including determining whether the end effector is within a field of view of the camera (Staub, pg. 67, section II.C.1, “PBVS is used to drive the instrument to a reconstructed point which is located within the view of the camera”. Before PBVS is used, the instrument has not yet reached the reconstructed point within the camera’s field of view.);
commanding movement of the surgical instrument until the instrument is viewable (detectable via image feature extraction) by the camera in response to determining that the end effector is not viewable by the camera (Staub, pg. 67, section II.C, “In order to command surgical instruments with a high precision to a desired position, we propose a switching servoing scheme. First, the instrument has to be driven to the target Cartesian coordinate which is in sight of the camera, employing position-based servoing ... The proposed switching scheme is not only necessary to drive the instruments into the field of view of the camera, but also has a positive effect on the convergence characteristic of the image-based part.”), wherein commanding movement is repeated until the controller confirms that the end effector is viewable by the camera (After the PBVS brings the end-effector into frame, the image-based visual servoing (IBVS) incrementally moves the instrument until the final position is reached when the error in equation (4) is sufficiently minimized. See Staub at FIG. 3 and section II.C. ).
Seita discloses a surgical robotic system that uses an endoscope and a robotic arm with a surgical instrument attached to calibrate the instrument. Thus, Seita shows that it was known in the art before the effective filing date of the claimed invention to calibrate a surgical robotic system using image data acquired through an endoscope that is separated from a robotic arm to accurately track a surgical tool attached to the robotic arm, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, obtaining precise actuation of surgical instruments to reflect movement commands. Staub discloses a multi-arm robotic system (See Fig. 2(a)) that commands movement of an end effector until it is viewable by the camera to then precisely calibrate the position of the end-effector. The calibration includes a visual servoing switching scheme that uses position-based visual servoing (PBVS) if the instrument is not in the field of view of the camera and image-based visual servoing (IBVS) once the instrument has been driven into the field of view. Thus, Staub shows that it was known in the art before the effective filing date of the claimed invention to calibrate a surgical robotic system using a camera attached to a robotic arm and a moveable (controllable) end effector by commanding the end effector to move towards a particular reconstructed position, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, obtaining precise actuation of surgical instruments to reflect movement commands.
A person of ordinary skill in the art would have been motivated to modify the calibration procedure for the robotic arm of Seita to incorporate the visual servoing switch scheme disclosed by Staub, to thereby detect when the end effector and/or its fiducial marker are not viewable by the camera and in response, command movement of the end effector and/or the robotic arm until end effector is within the camera’s field of view at the desired reconstructed final position. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of requiring less manual effort to calibrate the system.
Seita in view of Staub does not teach that which is explicitly taught by Nixon.
Nixon teaches calculate a calibration factor (The calibration offset is a calibration factor. See Nixon at par. 67, “To facilitate identification of the end effector mutual engagement position, it is helpful to limit the analysis of the data to a limited range. The data may be restricted to a position range within which engagement is expected to occur, thereby assuming that the calibration offset will be within a predetermined range. For example, it may be assumed that the calibration offset for the end effector mutual engagement configuration will be between a nominal or initially commanded configuration of the jaws being open by 5 degrees (+5 degrees), and the jaws being squeezed past closed by 20 degrees (-20 degrees).”) for each degree of freedom of the end effector (The jaws of the end effector have one degree of freedom that controls their separation angle. See Nixon at par. 56, “the gripping members of handle 226 again define a separation angle that is substantially equal to the separation angle defined by the end effector elements.”) and correlating the positional information with kinematic positional information of the movement of the end effector with kinematic positional information of commanded movement of the end effector to the mechanical end limit (Nixon, par. 69, “Referring now to FIGS. 11 and 12A-12C, the mutual engagement configuration of the end effector should correlate with the position of greatest curvature in the negative direction of the commanded torque data. This can be identified from the minima of the second derivative of the commanded torque.”); and
apply the calibration factor to adjust inverse kinematics used to generate motor commands for the robotic arm (Nixon, par. 69, “Once the mutual engagement configuration is known, calibration of the mounted tool can then be effected by applying a difference between the actual position and the commanded position of the end effectors as an offset to the grip controllers described above, for example, so that the closed configuration of the handle 226 (at which the handle members first engage the resilient bumper 60 a) corresponds to the mutual engagement configuration of the end effectors. This difference can be stored as an offset in grip calibration data table 405 for the tool/manipulator combination.”).
Seita, Staub and Nixon are analogous to the claimed invention for the reasons provided above.
A person of ordinary skill in the art would have been motivated to modify the calibration procedure for the robotic arm of Seita in view of Staub to command the jaws to move each degree of freedom of the end effector to a mechanical end limit as disclosed by Nixon, to thereby determine the calibration offset of each degree of freedom by commanding the respective degree of freedom to move the end effector to a nominal mechanical end limit where the jaws are visible to the camera including the configuration in the closed configuration. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of mitigating commanded movement of the jaws to grip tissue during a surgery from applying excessive torque and harming the patient.
Regarding claim 11, Seita in view of Staub and in further view of Nixon teaches the method according to claim 10, further comprising:
determining, at the controller, whether the surgical instrument is detected by the camera based on a distance of the surgical instrument from the camera (Seita, pgs. 2-4, sections III and IV; The dataset XDNN is collected, which includes the three-dimensional camera position coordinates xc of the instrument position with respect to the camera frame, the three-dimensional coordinates xb of the instrument with respect to the base, and Φ, the yaw, pitch, and roll of the instrument. The difference between a coordinate of the base and a coordinate of the instrument defines a distance from the instrument to the camera. Thus, the detection, which uses this model, is based on said distance.).
Regarding claim 12, Seita in view of Staub and in further view of Nixon teaches the method according to claim 11, further comprising:
commanding, by the controller, movement of the surgical instrument until the end effector is within a field of the camera (Seita, pg. 1, Abstract, “we place a red calibration marker on the end effector and let it randomly move through a set of open-loop trajectories to obtain a large sample set of camera pixels and internal robot end-effector configurations. This coarse data is then used to train a Deep Neural Network (DNN) to learn the coarse transformation bias.”; Each trajectory is a movement to places the end effector in position to be observed by the camera so that the robot’s internal kinematics can be used with the camera data to refine the accuracy of is predicted target positions.).
Regarding claim 16, Seita teaches a surgical robotic system comprising:
a camera configured to output a video stream (Seita, pg. 3, Fig. 3, “Endoscopic Stereo Camera”);
a second robotic arm (Seita, Fig. 3 caption, “dVRK arm”) including a surgical instrument having an end effector with at least one fiducial marker (Seita, Fig. 2 caption, “dVRK end-effector”); and
a controller (Seita, pg. 1, Fig. 1 caption, “we train a Deep Neural Network (DNN)”; section I, “the dVRK can achieve positional control in the workspace”; The dVRK is a controller. Training a neural network requires a computer) configured to:
receive the video stream from the camera (Seita, pg. 1, Abstract, “Phase I and Phase II”);
detect the at least one fiducial marker on the end effector in the video stream using a machine learning algorithm (Seita, pg. 1, section I, “we train a Deep Neural Network (DNN) to map from camera position (not images) and robot orientation (extracted from the images) to base position.”; pg. 4, section IV.A, “Forming XDNN also requires the camera position xc at these points, so before executing the trajectories we apply red tape to the end-effector, thus allowing the dVRK to use HSV thresholding to detect the location of red contours.”);
calculate a calibration factor (Seita, pg. 1, Abstract, “transformation bias.”), and correlating the positional information derived from the video stream with kinematic positional information of commanded movement (random trajectories) of the end effector (Seita, pg. 1, section I, “robot orientation”) with kinematic positional information of commanded movement (Seita, pg. 1, section I, “commanded to some position and orientation”); and
apply the calibration factor to adjust inverse kinematics used by the controller to generate motor commands for the second robotic arm, to control movement of the surgical instrument (Seita, Abstract, “In Phase II (fine), the bias from Phase I is applied to move the end-effector toward a small set of specific target points on a printed sheet. For each target, a human operator manually adjusts the end-effector position by direct contact (not through teleoperation) and the residual compensation bias is recorded. This fine data is then used to train a Random Forest (RF) to learn the fine transformation bias. Subsequent experiments suggest that without calibration, position errors average 4.55mm.”), but does not teach that which is explicitly taught by Staub.
Staub teaches a first robotic arm including a camera (See Staub at Fig. 2(a));
determine whether the end effector is viewable by the camera (Staub, pg. 67, section II.C, “a practical difficulty during the alignment of surgical instruments with a desired position lies in the fact that the instrument is not necessarily in the field of view of the camera and no image-features can be extracted.”), including determining whether the end effector is within a field of view of the camera (Staub, pg. 67, section II.C.1, “PBVS is used to drive the instrument to a reconstructed point which is located within the view of the camera”. Before PBVS is used, the instrument has not yet reached the reconstructed point within the camera’s field of view.);
command movement of the surgical instrument until the instrument is viewable (detectable via image feature extraction) by the camera in response to determining that the end effector is not viewable by the camera (Staub, pg. 67, section II.C, “In order to command surgical instruments with a high precision to a desired position, we propose a switching servoing scheme. First, the instrument has to be driven to the target Cartesian coordinate which is in sight of the camera, employing position-based servoing ... The proposed switching scheme is not only necessary to drive the instruments into the field of view of the camera, but also has a positive effect on the convergence characteristic of the image-based part.”), wherein commanding movement is repeated until the controller confirms that the end effector is viewable by the camera (After the PBVS brings the end-effector into frame, the image-based visual servoing (IBVS) incrementally moves the instrument until the final position is reached when the error in equation (4) is sufficiently minimized. See Staub at FIG. 3 and section II.C. ).
Seita and Staub are analogous to the claimed invention for the same reasons provided above.
A person of ordinary skill in the art would have been motivated to modify the camera position, number of used robotic arms, and calibration procedure disclosed by Seita to incorporate the two-armed visual servoing switch scheme disclosed by Staub, to thereby move the camera of Seita to a second robotic arm and detect when the end effector and/or its fiducial marker are not viewable by the camera and in response, command movement of the end effector and/or the robotic arm until it is within the camera’s field of view at the desired reconstructed final position. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of greater range of motion and viewing angles of the camera.
Seita in view of Staub does not teach that which is explicitly taught by Nixon.
Nixon teaches calculate a calibration factor (The calibration offset is a calibration factor. See Nixon at par. 67, “To facilitate identification of the end effector mutual engagement position, it is helpful to limit the analysis of the data to a limited range. The data may be restricted to a position range within which engagement is expected to occur, thereby assuming that the calibration offset will be within a predetermined range. For example, it may be assumed that the calibration offset for the end effector mutual engagement configuration will be between a nominal or initially commanded configuration of the jaws being open by 5 degrees (+5 degrees), and the jaws being squeezed past closed by 20 degrees (-20 degrees).”) for each degree of freedom of the end effector (The jaws of the end effector have one degree of freedom that controls their separation angle. See Nixon at par. 56, “the gripping members of handle 226 again define a separation angle that is substantially equal to the separation angle defined by the end effector elements.”) and correlating positional information with kinematic positional information of the movement of the end effector with kinematic positional information of commanded movement of the end effector to the mechanical end limit (Nixon, par. 69, “Referring now to FIGS. 11 and 12A-12C, the mutual engagement configuration of the end effector should correlate with the position of greatest curvature in the negative direction of the commanded torque data. This can be identified from the minima of the second derivative of the commanded torque.”); and
apply the calibration factor to adjust inverse kinematics used to generate motor commands for the robotic arm (Nixon, par. 69, “Once the mutual engagement configuration is known, calibration of the mounted tool can then be effected by applying a difference between the actual position and the commanded position of the end effectors as an offset to the grip controllers described above, for example, so that the closed configuration of the handle 226 (at which the handle members first engage the resilient bumper 60 a) corresponds to the mutual engagement configuration of the end effectors. This difference can be stored as an offset in grip calibration data table 405 for the tool/manipulator combination.”).
Seita and Staub are analogous to the claimed invention for the same reasons provided above. Nixon is analogous to the claimed invention for the same reasons provided above.
A person of ordinary skill in the art would have been motivated to modify the calibration procedure for the robotic arm of Seita in view of Staub to command the jaws to move each degree of freedom of the end effector to a mechanical end limit as disclosed by Nixon, to thereby determine the calibration offset of each degree of freedom by commanding the respective degree of freedom to move the end effector to a nominal mechanical end limit where the jaws are visible to the camera including the configuration in the closed configuration. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of mitigating commanded movement of the jaws to grip tissue during a surgery from applying excessive torque and harming the patient.
Claim 17 substantially corresponds to the limitations of claims 4 and 5 as a whole by reciting a surgical robotic system wherein the controller is further configured to perform the functions recited in claims 4 and 5 (The robotic system of Seita).
Therefore, claim 17 is rejected for the same reasons of obviousness provided for claims 4 and 5.
Claims 13-15 are rejected under 35 U.S.C. 103 as being unpatentable over Seita in view of Staub, in view of Nixon, in view of Manzo, and in further view of Zhao.
Regarding claim 13, Seita in view of Staub and in further view of Nixon teaches the method according to claim 11, but does not teach that which is explicitly taught by Manzo.
Manzo teaches selecting, at the controller, a calibration routine based on the type of the surgical instrument (Manzo, col. 9, ll. 45-67; See claim 6).
Seita, Staub and Nixon are analogous to the claimed invention for the same reasons provided above. Manzo is analogous to the claimed invention for the same reasons provided above.
A person of ordinary skill in the art would have been motivated to add the tool type and calibration data in a memory as disclosed by Manzo to the surgical instrument disclosed by Seita in view of Staub and in further view of Nixon for the computer that controls the instrument to thereby read the memory of the instrument and obtain initial calibration data as part of a calibration routine based on the type of instrument. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of improving the starting point of calibration, thereby reducing the amount of time needed to fully calibrate the system when a new tool is attached to one of the robotic arms.
Seita in view of Staub, in view of Nixon and in further view of Manzo does not teach that which is explicitly taught by Zhao.
Zhao teaches identifying, at the controller, a type of the surgical instrument from the video stream (Zhao, col. 26, ll. 16-32; col. 32, ll. 24-36).
Seita, Staub, Nixon and Manzo are analogous to the claimed invention for the same reasons provided above. Zhao is analogous to the claimed invention for the same reasons provided above.
The rationale for obviousness is the same as applied to claim 7, where the difference of Diolaiti being applied to claim 7 and not being applied to claim 13 does not change the provided reasoning.
Claim 14 corresponds to the “moving” and “receiving” functions (taught by Seita) of the surgical robotic system of claim 9 by reciting a method performing those functions.
Claim 14 is therefore rejected for the same reasons as claim 9.
Claim 15 corresponds to the “correlating” and “calculating” functions (taught by Seita) of the surgical robotic system recited in claim 9 by reciting a method for performing those functions.
Claim 15 is therefore rejected for the same reasons as claim 9.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Seita in view of Staub, in view of Nixon and in further view of Zhao.
Regarding claim 18, Seita in view of Staub and in further view of Nixon teaches the surgical robotic system according to claim 16, but does not teach that which is explicitly taught by Zhao.
Zhao teaches wherein the controller is further configured to identify a type of the surgical instrument from the video stream (Zhao, col. 26, ll. 16-32; col. 32, ll. 24-36; See claim 7).
Seita, Staub, Nixon and Zhao are analogous to the claimed invention for the reasons provided above.
A person of ordinary skill in the art would have been motivated to add a unique marker to a surgical instrument as disclosed by Zhao and configure the controller disclosed by Seita in view of Staub and in further view of Nixon to determine the type of instrument based on the marker in the video stream. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of improving the starting point of calibration by knowing what type of instrument is being used, thereby reducing the amount of time needed to calibrate the system when a new tool is attached to one of the robotic arms.
Claims 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Seita in view of Staub, in view of Nixon, in view of Zhao, and in further view of Manzo.
Regarding claim 19, Seita in view of Staub, in view of Nixon and in further view of Zhao teaches the surgical robotic system according to claim 18, but does not teach that which is explicitly taught by Manzo.
Manzo teaches wherein the controller is further configured to select a calibration routine based on the type of the surgical instrument (Manzo, col. 9, ll. 62-67).
Seita, Staub, Nixon, Zhao and Manzo are analogous to the claimed invention for the reasons provided above.
A person of ordinary skill in the art would have been motivated to add the tool type and calibration data in a memory as disclosed by Manzo to the surgical instrument of Seita in view of Staub, in view of Nixon and in further view of Zhao to thereby determine the type of instrument based on the marker if present in a video stream, and based on the data in the memory of the instrument if the data is present. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of improving the starting point of calibration, thereby reducing the amount of time needed to calibrate the system when a new tool is attached to one of the robotic arms.
Claim 20 substantially corresponds to claim 9 by reciting the same steps to calibrate the surgical instrument (taught by Seita).
Claim 20 is therefore rejected for the same reasons as claim 9.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. U.S. Pat. Appl. Pub. No. 20100331855 is considered pertinent because it is analogous to the applied art above and discloses a calibration method in Figure 9 that is suitable for use with the robotic systems of the applied art.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN P POTTS whose telephone number is (571)272-6351. The examiner can normally be reached M-F, 9am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 571-272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN P POTTS/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights