DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 10/16/2024, 01/08/2025, and 03/20/2026 have been considered by the examiner.
Double Patenting
The non-statutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A non-statutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on non-statutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP §§ 706.02(l)(1) - 706.02(l)(3) for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 1 and 11 are rejected on the ground of non-statutory double patenting as being unpatentable over claims 23 and 37 respectively of U.S. Pat. 12,108,992. Although the claims at issue are not identical, they are not patentably distinct from each other because the broader claims of the present application are anticipated by the narrower claims of the patent.
Table 1 illustrates the conflicting claim pairs:
Present Application
1
11
U.S. Pat. 12,108,992
23
37
Table 2 illustrates a mapping between the limitations claim 1 of the present application and claim 23 of U.S. Pat. 12,108,992.
Claim 1 of present App.
Claim 23 of U.S. Pat. 12,108,992
A surgical instrument tracking system comprising: a memory storing instructions; a processor communicatively coupled to the memory and configured to execute the instructions to:
22. A surgical instrument tracking system comprising: a memory storing instructions; a processor communicatively coupled to the memory and configured to execute the instructions to:
determine a kinematics-based physical location of a robotically-manipulated instrument at a surgical area based on kinematics of the robotically-manipulated instrument; determine, based on image data representative of one or more images of the surgical area, an observation for an object of interest; associate the observation to the robotically-manipulated instrument at the surgical area;
determine, based on imagery of a surgical area, an observation for an object of interest depicted in the imagery; determine a probability of an association of the observation to a robotically- manipulated surgical instrument located at the surgical area based on kinematics of the robotically-manipulated surgical instrument; associate the observation for the object of interest to the robotically-manipulated surgical instrument based on the probability; and determine a physical position of the robotically-manipulated surgical instrument at the surgical area based on the kinematics of the robotically-manipulated surgical instrument and the observation associated with the robotically-manipulated surgical instrument.
23. The system of claim 22, wherein: the imagery comprises a left endoscopic image and a right endoscopic image that is stereoscopic with the left endoscopic image; the observation for the object of interest comprises a left observation and a right observation for the object of interest; and the processor is configured to execute the instructions to: determine the observation for the object of interest by using a trained neural network to determine the left observation for the object of interest based on the left endoscopic image and to determine the right observation for the object of interest based on the right endoscopic image; associate the observation for the object of interest to the robotically-manipulated surgical instrument by associating the left observation to the robotically-manipulated surgical instrument and associating the right observation to the robotically-manipulated surgical instrument;
and determine a corrected physical location of the robotically-manipulated instrument at the surgical area by adjusting the kinematics-based physical location based on the observation associated with the robotically-manipulated instrument.
and determine the physical position of the robotically-manipulated surgical instrument at the surgical area by using the left observation and the right observation to determine a three- dimensional ("3D") position of the object of interest in a 3D space and adjusting a 3D kinematic- based position of the robotically-manipulated surgical instrument in the 3D space based on the 3D position of the object of interest in the 3D space.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-2, 4-11, and 13-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Reiter et al. (US Pub. No. 2015/0297313 A1).
Regarding claim 1, Reiter discloses, a surgical instrument tracking system comprising: a memory storing instructions; (See Reiter claim 15, “A non-transient computer readable medium for use with a robotic surgical tool tracking system, comprising: instructions.”)
a processor communicatively coupled to the memory and configured to execute the instructions to: (See Reiter ¶76, “can be implemented on a GPU processor.”)
determine a kinematics-based physical location of a robotically-manipulated instrument at a surgical area based on kinematics of the robotically-manipulated instrument; (See Reiter ¶7, “joints of a robotic surgical system can be equipped with encoders so that the pose of the instruments can be computed through forward kinematics.” Further see Reiter ¶73, “For each, superposed lines 701-710 portray the raw kinematics estimates as given by the robot, projected into the image frames.”)
determine, based on image data representative of one or more images of the surgical area, an observation for an object of interest; (See Reiter ¶37, “Class-labeled features are then localized in the image. Next, these candidate feature detections are stereo matched and triangulated to localize as 3D coordinates.”)
associate the observation to the robotically-manipulated instrument at the surgical area; (See Reiter ¶62, “Using the forward kinematics estimate from each PSM, the marker patterns are rotated to achieve the estimated orientations of each PSM.” Further see Reiter ¶63, “Next, given N detected feature observations using the classification method described above, both an N×N distance matrix between each 3D feature observation and an N×N×3 matrix of unit vectors are computed, similar to those computed for the marker patterns using the kinematics estimates from the robot. Finally, any feature observations which do not adhere to one of the pre-processed marker distance and rotation configurations according to the PSMs are rejected. Using empirically determined distance (e.g., ˜3-5 mm) and orientation (e.g., ˜10°-20°) thresholds, the PSM associated with each feature is determined, allowing only one assignment per feature class to each PSM.”)
and determine a corrected physical location of the robotically-manipulated instrument at the surgical area by adjusting the kinematics-based physical location based on the observation associated with the robotically-manipulated instrument. (See Reiter ¶66, “Because features detected are not guaranteed to always be visible in any given frame, the robot kinematics are combined with the vision estimates in Fusion and Tracking Module 105 to provide the final articulated pose across time.”)
Regarding claim 2, Reiter discloses, the surgical instrument tracking system of claim 1, wherein the observation is associated to the robotically-manipulated instrument based on a determined probability of an association of the observation to the robotically-manipulated instrument. (See Reiter ¶35, “FIG. 2 shows an example result of the pixel labeling routine described with reference to FIG. 1. FIG. 2A shows the original image 201 from an in-vivo porcine sequence of first and second robotic tools 203 and 204 performing a suturing procedure using the da Vinci® Surgical System. FIG. 2B shows the metal likelihood (e.g., tool tip, clevis), with mask regions 205 and 206 corresponding to the highest probability locations of metal. FIG. 2C shows the shaft likelihood, with mask regions 207 and 208 corresponding to the highest probability locations of shaft. FIG. 2D shows the background likelihood, with mask region 209 corresponding to the highest probability location of the background. The metal class represents all pixels located at the distal tip of the tool, from the clevis to the grippers. All of the features to be detected by the Feature Classification Module 102 are located in this region.”)
Regarding claim 4, Reiter discloses, the surgical instrument tracking system of claim 1, wherein the adjusting the kinematics-based physical location based on the observation associated with the robotically- manipulated instrument comprises: determining, based on the observation, an image-based physical location of the robotically-manipulated instrument at the surgical area; (See Reiter ¶37, “Class-labeled features are then localized in the image. Next, these candidate feature detections are stereo matched and triangulated to localize as 3D coordinates.”)
and adjusting the kinematics-based physical location based on the image-based physical location. (See Reiter ¶66, “Because features detected are not guaranteed to always be visible in any given frame, the robot kinematics are combined with the vision estimates in Fusion and Tracking Module 105 to provide the final articulated pose across time.”)
Regarding claim 5, Reiter discloses, the surgical instrument tracking system of claim 4, wherein: the kinematics-based physical location comprises a first three-dimensional (3D) position in a 3D space; (See Reiter ¶68, “where p.sup.K is a 3D point location in the kinematics remote center coordinate frame KCS.”)
and the image-based physical location comprises a second 3D position in the 3D space. (See Reiter ¶37, “Class-labeled features are then localized in the image. Next, these candidate feature detections are stereo matched and triangulated to localize as 3D coordinates.”)
Regarding claim 6, Reiter discloses, the surgical instrument tracking system of claim 4, wherein: the one or more images comprises a sequence of video frames; (See Reiter ¶72, “Overall, testing included 6 different video sequences, totaling 6876 frames (458 seconds worth of video).”)
and the image-based physical location is determined based on a motion of the object of interest represented in the sequence of video frames. (See Reiter ¶39, “some embodiments of the presently disclosed subject matter coast through small temporal spaces using Lucas-Kanade optical flow (KLT) to predict ground truth locations between user clicks as follows: the user drags a bounding-box around a feature of interest; the software uses KLT optical flow to track this feature from frame-to-frame (keeping the same dimensions of the box);”)
Regarding claim 7, Reiter discloses, the surgical instrument tracking system of claim 1, wherein the processor is configured to execute the instructions to: determine, based on the image data, an additional observation for an additional object of interest; (See Reiter ¶63, “Next, given N detected feature observations using the classification method described above, both an N×N distance matrix between each 3D feature observation and an N×N×3 matrix of unit vectors are computed, similar to those computed for the marker patterns using the kinematics estimates from the robot.”)
associate the additional observation for the additional object of interest to a false positive designation; and refrain from using the additional observation for the additional object of interest to determine the corrected physical location of the robotically-manipulated instrument. (See Reiter ¶63, “Finally, any feature observations which do not adhere to one of the pre-processed marker distance and rotation configurations according to the PSMs are rejected. Using empirically determined distance (e.g., ˜3-5 mm) and orientation (e.g., ˜10°-20°) thresholds, the PSM associated with each feature is determined, allowing only one assignment per feature class to each PSM.”)
Regarding claim 8, Reiter discloses, the surgical instrument tracking system of claim 1, wherein the observation for the object of interest indicates an image-based physical location of the robotically-manipulated instrument at the surgical area (See Reiter ¶37, “Class-labeled features are then localized in the image. Next, these candidate feature detections are stereo matched and triangulated to localize as 3D coordinates.”)
and a motion cue associated with the object of interest. (See Reiter ¶39, “some embodiments of the presently disclosed subject matter coast through small temporal spaces using Lucas-Kanade optical flow (KLT) to predict ground truth locations between user clicks as follows: the user drags a bounding-box around a feature of interest; the software uses KLT optical flow to track this feature from frame-to-frame (keeping the same dimensions of the box);”)
Regarding claim 9, Reiter discloses, the surgical instrument tracking system of claim 1, wherein the observation for the object of interest indicates an image-based physical location of the robotically-manipulated instrument at the surgical area (See Reiter ¶37, “Class-labeled features are then localized in the image. Next, these candidate feature detections are stereo matched and triangulated to localize as 3D coordinates.”)
and a type of instrument associated with the object of interest. (See Reiter ¶31, “Multiple tools are handled simultaneously by applying a tool association algorithm and the system of the presently disclosed subject matter is able to detect features on different types of tools. … The learning system of the presently disclosed subject matter is extends to multiple tool types and multiple tools tracked simultaneously as well as various types of surgical data.”)
Regarding claim 10, Reiter discloses, the surgical instrument tracking system of claim 1, wherein the processor is configured to execute the instructions to output data representative of the corrected physical location of the robotically-manipulated instrument at the surgical area for use by a robotic surgical system to display user interface content collocated with a visual representation of the robotically-manipulated instrument displayed by a display device. (See Reiter ¶6, “For example, accurate tool localization can be used as a Virtual Ruler capable of measuring the distances between various points in the anatomical scene, such as the sizes of anatomical structures. Graphical overlays can indicate the status of a particular tool, for example, in the case of the firing status of an electro-cautery tool. These indicators can be placed at the tip of the tool in the visualizer which is close to the surgeon's visual center of attention, enhancing the overall safety of using such tools.”
Further see Reiter ¶32, “The da Vinci® surgical robot is a tele-operated, master-slave robotic system. The main surgical console is separated from the patient, whereby the surgeon sits in a stereo viewing console and controls the robotic tools with two Master Tool Manipulators (MTM) while viewing stereoscopic high-definition video.”
Further see Reiter ¶81, “In FIG. 10B, an example scenario of a lost tool (e.g., outside the camera's field-of-view) is shown, whereby the endoscopic image (top) shows only two tools, and with fixed kinematics and a graphical display (bottom), the surgeon can accurately be shown where the third tool 1003 (out of the left-bottom corner) is located and posed so they can safely manipulate the tool back into the field-of-view.”)
Regarding claim 11, Reiter discloses, a method comprising: determining, by a surgical instrument tracking system, a kinematics-based physical location of a robotically-manipulated instrument at a surgical area based on kinematics of the robotically-manipulated instrument; (See Reiter ¶7, “joints of a robotic surgical system can be equipped with encoders so that the pose of the instruments can be computed through forward kinematics.” Further see Reiter ¶73, “For each, superposed lines 701-710 portray the raw kinematics estimates as given by the robot, projected into the image frames.”)
determining, by the surgical instrument tracking system and based on image data representative of one or more images of the surgical area, an observation for an object of interest; (See Reiter ¶37, “Class-labeled features are then localized in the image. Next, these candidate feature detections are stereo matched and triangulated to localize as 3D coordinates.”)
associating, by the surgical instrument tracking system, the observation to the robotically- manipulated instrument at the surgical area; (See Reiter ¶62, “Using the forward kinematics estimate from each PSM, the marker patterns are rotated to achieve the estimated orientations of each PSM.” Further see Reiter ¶63, “Next, given N detected feature observations using the classification method described above, both an N×N distance matrix between each 3D feature observation and an N×N×3 matrix of unit vectors are computed, similar to those computed for the marker patterns using the kinematics estimates from the robot. Finally, any feature observations which do not adhere to one of the pre-processed marker distance and rotation configurations according to the PSMs are rejected. Using empirically determined distance (e.g., ˜3-5 mm) and orientation (e.g., ˜10°-20°) thresholds, the PSM associated with each feature is determined, allowing only one assignment per feature class to each PSM.”)
and determining, by the surgical instrument tracking system, a corrected physical location of the robotically-manipulated instrument at the surgical area by adjusting the kinematics-based physical location based on the observation associated with the robotically-manipulated instrument. (See Reiter ¶66, “Because features detected are not guaranteed to always be visible in any given frame, the robot kinematics are combined with the vision estimates in Fusion and Tracking Module 105 to provide the final articulated pose across time.”)
Regarding claim 13, Reiter discloses, the method of claim 11, wherein the adjusting the kinematics-based physical location based on the observation associated with the robotically-manipulated instrument comprises: determining, based on the observation, an image-based physical location of the robotically-manipulated instrument at the surgical area; and adjusting the kinematics-based physical location based on the image-based physical location. (See the rejection of claim 4 as it is equally applicable for claim 13 as well.)
Regarding claim 14, Reiter discloses, the method of claim 13, wherein: the kinematics-based physical location comprises a first three-dimensional (3D) position in a 3D space; and the image-based physical location comprises a second 3D position in the 3D space. (See the rejection of claim 5 as it is equally applicable for claim 14 as well.)
Regarding claim 15, Reiter discloses, the method of claim 13, wherein: the one or more images comprises a sequence of video frames; and the image-based physical location is determined based on a motion of the object of interest represented in the sequence of video frames. (See the rejection of claim 6 as it is equally applicable for claim 15 as well.)
Regarding claim 16, Reiter discloses, the method of claim 11, further comprising: determining, by the surgical instrument tracking system and based on the image data, an additional observation for an additional object of interest; associating, by the surgical instrument tracking system, the additional observation for the additional object of interest to a false positive designation; and refraining, by the surgical instrument tracking system, from using the additional observation for the additional object of interest to determine the corrected physical location of the robotically-manipulated instrument. (See the rejection of claim 7 as it is equally applicable for claim 16 as well.)
Regarding claim 17, Reiter discloses, the method of claim 11, wherein the observation for the object of interest indicates an image-based physical location of the robotically-manipulated instrument at the surgical area and a motion cue associated with the object of interest. (See the rejection of claim 8 as it is equally applicable for claim 17 as well.)
Regarding claim 18, Reiter discloses, the method of claim 11, wherein the observation for the object of interest indicates an image-based physical location of the robotically-manipulated instrument at the surgical area and a type of instrument associated with the object of interest. (See the rejection of claim 9 as it is equally applicable for claim 18 as well.)
Regarding claim 19, Reiter discloses, the method of claim 11, further comprising: outputting, by the surgical instrument tracking system, data representative of the corrected physical location of the robotically-manipulated instrument at the surgical area to a robotic surgical system, the robotic surgical system configured to use the data representative of the corrected physical location of the robotically-manipulated instrument at the surgical area to display user interface content collocated with a visual representation of the robotically- manipulated instrument. (See the rejection of claim 10 as it is equally applicable for claim 19 as well.)
Regarding claim 20, Reiter discloses, a non-transitory computer-readable medium storing instructions executable by a processor to: (See Reiter claim 15, “A non-transient computer readable medium for use with a robotic surgical tool tracking system, comprising: instructions.”)
determine a kinematics-based physical location of a robotically-manipulated instrument at a surgical area based on kinematics of the robotically-manipulated instrument; (See Reiter ¶7, “joints of a robotic surgical system can be equipped with encoders so that the pose of the instruments can be computed through forward kinematics.” Further see Reiter ¶73, “For each, superposed lines 701-710 portray the raw kinematics estimates as given by the robot, projected into the image frames.”)
determine, based on image data representative of one or more images of the surgical area, an observation for an object of interest; (See Reiter ¶37, “Class-labeled features are then localized in the image. Next, these candidate feature detections are stereo matched and triangulated to localize as 3D coordinates.”)
associate the observation to the robotically-manipulated instrument at the surgical area; (See Reiter ¶62, “Using the forward kinematics estimate from each PSM, the marker patterns are rotated to achieve the estimated orientations of each PSM.” Further see Reiter ¶63, “Next, given N detected feature observations using the classification method described above, both an N×N distance matrix between each 3D feature observation and an N×N×3 matrix of unit vectors are computed, similar to those computed for the marker patterns using the kinematics estimates from the robot. Finally, any feature observations which do not adhere to one of the pre-processed marker distance and rotation configurations according to the PSMs are rejected. Using empirically determined distance (e.g., ˜3-5 mm) and orientation (e.g., ˜10°-20°) thresholds, the PSM associated with each feature is determined, allowing only one assignment per feature class to each PSM.”)
and determine a corrected physical location of the robotically-manipulated instrument at the surgical area by adjusting the kinematics-based physical location based on the observation associated with the robotically-manipulated instrument. (See Reiter ¶66, “Because features detected are not guaranteed to always be visible in any given frame, the robot kinematics are combined with the vision estimates in Fusion and Tracking Module 105 to provide the final articulated pose across time.”)
Allowable Subject Matter
Claim 3 and 12 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claim 3, The surgical instrument tracking system of claim 2, wherein the determined probability is determined based on the kinematics of the robotically-manipulated instrument. (The disclosed prior art of record fails to disclose the limitations of this claim.)
Regarding claim 12, the method of claim 11, wherein the associating the observation to the robotically-manipulated instrument at the surgical area comprises: determining a probability of an association of the observation to the robotically- manipulated instrument based on the kinematics of robotically-manipulated instrument; associating the observation to the robotically-manipulated instrument based on the probability. (The disclosed prior art of record fails to disclose the limitations of this claim.)
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant’s disclosure.
Kwak et al. (US Pub. No. 2014/0241577 A1) A method of tracking a moving object includes measuring displacement of an object to be tracked, obtaining a particle of the object to be tracked using the measured displacement, and tracking the object using pose information of the object in an image thereof and the obtained particle. A control apparatus includes an imaging module to perform imaging of an object and generates an image, and a tracking unit to acquire displacement and pose information of the object using the generated image of the object, to set a particle of the object using the acquired displacement of the object, and to track the object using the pose information of the object and the particle.
Rafii-Tari et al. (US Pub No. 2021/0059764 A1) Provided are systems and methods for weight-based registration of location sensors. In one aspect, a system includes an instrument and a processor configured to provide a first set of commands to drive the instrument along a first branch of the luminal network, the first branch being outside a path to a target within a model. The processor is also configured to track a set of one or more registration parameters during the driving of the instrument along the first branch and determine that the set of registration parameters satisfy a registration criterion.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID PERLMAN whose telephone number is (571) 270-1417.
The examiner can normally be reached on Monday - Friday; 10:00am -6:30pm.
Examiner interviews are available via telephone and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is
(571) 273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at (866) 217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call (800) 786-9199 (IN USA OR CANADA) or (571) 272-1000.
/DAVID PERLMAN/Primary Examiner, Art Unit 2673
Prosecution Insights