DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-10 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 4 and 6-8 of U.S. Patent No. US 12213745 B2 (reference patent). Although the claims at issue are not identical, they are not patentably distinct from each other because the claimed scope of the instant application is broader than the reference patent, which comprises limitation of weighted anatomical features. Therefore, the reference patent anticipates the instant application.
The following table illustrates the conflicting claim pairs:
Instant Application
1
2
3
4
5
6
7
8
9
10
Reference Patent
1, 4
5
1, 6
7
7
7
7
7
8
6
Claims of the instant application are compared to claims of Reference Patent in the following tables.
Instant Application
Reference Patent
1. A camera tracking system comprising:
at least one processor configured to perform the steps of:
obtain remote operator-gesture tracking information from a remote reference tracking camera indicating movement of an object relative to a remote extended reality (XR) headset reference frame by a remote operator wearing a remote XR headset;
determine a remote gesture path relative to the remote XR headset reference frame based on processing the remote operator-gesture tracking information through a remote XR headset view pose transform; and
transform the remote gesture path to a local gesture path relative to a local XR headset reference frame using a local XR headset view pose transform.
1. A method by a camera tracking system, the method comprising:
receiving patient reference tracking information indicating pose of a patient reference array tracked by a patient tracking camera relative to a patient reference frame;
determining a local extended reality (XR) headset view pose transform between a local XR headset reference frame of a local XR headset and the patient reference frame using the patient reference tracking information;
receiving remote reference tracking information indicating pose of a remote reference array tracked by a remote reference tracking camera;
determining a remote XR headset view pose transform between a remote XR headset reference frame of a remote XR headset and the remote reference array using the remote reference tracking information;
transforming a three-dimensional (3D) computer image from a local pose determined using the local XR headset view pose transform to a remote pose determined using the remote XR headset view pose transform which outputs a transformed 3D computer image; and
providing the transformed 3D computer image to the remote XR headset for display with the remote pose relative to the remote XR headset reference frame.
4. The method of claim 1, further comprising:
obtaining remote operator-gesture tracking information from the remote reference tracking camera indicating movement of an object relative to the remote XR headset reference frame by a remote operator wearing the remote XR headset;
determining a remote gesture path relative to the remote XR headset reference frame based on processing the remote operator-gesture tracking information through the remote XR headset view pose transform; and
transforming the remote gesture path to a local gesture path relative to the local XR headset reference frame using the local XR headset view pose transform.
Instant Application
Reference Patent
2. The system of claim 1, wherein the at least one processor is further configured to provide the local gesture path to a local XR headset for display relative to the local XR headset reference frame.
5. The method of claim 4, further comprising:
provide the local gesture path to the local XR headset for display relative to the local XR headset reference frame.
Instant Application
Reference Patent
3. The system of claim 1, wherein the at least one processor is further configured to determine the remote gesture path relative to the remote XR headset reference frame based on tracking movement indicated by the remote operator-gesture tracking information of the object moved by the remote operator while the remote operator is concurrently viewing a transformed three-dimensional computer image through the remote XR headset relative to the object being moved.
1. A method by a camera tracking system, the method comprising:
receiving patient reference tracking information indicating pose of a patient reference array tracked by a patient tracking camera relative to a patient reference frame;
determining a local extended reality (XR) headset view pose transform between a local XR headset reference frame of a local XR headset and the patient reference frame using the patient reference tracking information;
receiving remote reference tracking information indicating pose of a remote reference array tracked by a remote reference tracking camera;
determining a remote XR headset view pose transform between a remote XR headset reference frame of a remote XR headset and the remote reference array using the remote reference tracking information;
transforming a three-dimensional (3D) computer image from a local pose determined using the local XR headset view pose transform to a remote pose determined using the remote XR headset view pose transform which outputs a transformed 3D computer image; and
providing the transformed 3D computer image to the remote XR headset for display with the remote pose relative to the remote XR headset reference frame.
6. The method of claim 4, further comprising:
determining the remote gesture path relative to the remote XR headset reference frame based on tracking movement indicated by the remote operator-gesture tracking information of a hand and/or a stylus which is moved by the remote operator while concurrently viewing the transformed 3D computer image through the remote XR headset relative to the hand and/or stylus being moved.
Instant Application
Reference Patent
4. The system of claim 1, wherein the at least one processor is further configured to recognize a gesture of the object performed by the remote operator as corresponding to a defined operational command.
7. The method of claim 4, further comprising:
selecting an operational command from among a set of operational commands based on the remote gesture path corresponding to defined gesture associated with the operational command, wherein the operational commands in the set are associated with different shaped gesture paths; and
providing the operational command to an equipment which is local to the local XR headset.
Instant Application
Reference Patent
5. The system of claim 4, wherein the defined operational command is configured to control equipment in a local environment.
7. The method of claim 4, further comprising:
selecting an operational command from among a set of operational commands based on the remote gesture path corresponding to defined gesture associated with the operational command, wherein the operational commands in the set are associated with different shaped gesture paths; and
providing the operational command to an equipment which is local to the local XR headset.
Instant Application
Reference Patent
6. The system of claim 1, wherein the at least one processor is further configured to select an operational command from among a set of operational commands based on the remote gesture path corresponding to a defined gesture associated with the operational command.
7. The method of claim 4, further comprising:
selecting an operational command from among a set of operational commands based on the remote gesture path corresponding to defined gesture associated with the operational command, wherein the operational commands in the set are associated with different shaped gesture paths; and
providing the operational command to an equipment which is local to the local XR headset.
Instant Application
Reference Patent
7. The system of claim 6, wherein each operational command in the set of operational commands is associated with a different shaped gesture path.
7. The method of claim 4, further comprising:
selecting an operational command from among a set of operational commands based on the remote gesture path corresponding to defined gesture associated with the operational command, wherein the operational commands in the set are associated with different shaped gesture paths; and
providing the operational command to an equipment which is local to the local XR headset.
Instant Application
Reference Patent
8. The system of claim 6, wherein the at least one processor is further configured to provide the selected operational command to an equipment local to the local XR headset.
7. The method of claim 4, further comprising:
selecting an operational command from among a set of operational commands based on the remote gesture path corresponding to defined gesture associated with the operational command, wherein the operational commands in the set are associated with different shaped gesture paths; and
providing the operational command to an equipment which is local to the local XR headset.
Instant Application
Reference Patent
9. The system of claim 1, wherein the at least one processor is further configured to select an operational command for relocating an end effector connected to a surgical robot arm that is movable under control of a surgical robot system from among a set of operational commands based on the remote gesture path corresponding to a defined gesture associated with the operational command for relocating the end effector.
8. The method of claim 7, further comprising:
selecting the operational command for relocating an end effector connected to a surgical robot arm that is movable under control of a surgical robot system, from among the set of operational commands based on the remote gesture path corresponding to the defined gesture associated with the operational command for relocating the end effector;
determining a present pose of the end effector based on end effector tracking information indicating pose of the end effector tracked by the patient tracking camera relative to the patient reference frame; and
controlling movement of the end effector by the surgical robot system from the present pose to a target pose relative to the patient reference frame based on the operational command for relocating the end effector.
Instant Application
Reference Patent
10. The system of claim 1, wherein the object is a tracked stylus or a hand.
6. The method of claim 4, further comprising:
determining the remote gesture path relative to the remote XR headset reference frame based on tracking movement indicated by the remote operator-gesture tracking information of a hand and/or a stylus which is moved by the remote operator while concurrently viewing the transformed 3D computer image through the remote XR headset relative to the hand and/or stylus being moved.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over GARCIA et al. (US 20190005848 A1), referred herein as GARCIA in view of NIKOU et al. (US 20220151705 A1), referred herein as NIKOU.
Regarding Claim 1, GARCIA in view of NIKOU teaches a camera tracking system comprising (GARCIA [0003] The surgical procedures may then be performed using the introduced surgical instruments, with the visualization aid provided by the camera):
at least one processor configured to perform the steps of (FIG. 3.210: virtual reality processor):
obtain remote operator-gesture tracking information from a remote reference tracking camera indicating movement of an object relative to a remote extended reality (XR) headset reference frame by a remote operator wearing a remote XR headset (GARCIA [0042] a user (such as a surgeon or other operator) may use a user console 100 to remotely manipulate the robotic arms 160 and/or surgical instruments (e.g., tele-operation)… the user console 100 may be located in an adjacent or nearby room, or tele-operated from a remote location in a different building, city, or country; [0062] the API may include one or more data structures that specify how to communicate information about … aspects relating to the virtual environment (e.g., reference frame for displaying the virtual environment, control system framework, etc.); [0069] the virtual reality system may further include one or more tracking emitters 212 that emit infrared light into a workspace for the user U. The tracking emitters 212 may, for example, be mounted on a wall, ceiling, fixture, or other suitable mounting surface. Sensors may be coupled to outward-facing surfaces of the head-mounted display 220 and/or handheld controllers 230 for detecting the emitted infrared light; [0127] the system may include a flight mode that enables the user to quickly navigate the virtual environment in a “flying” manner at different elevations and/or speeds, and at different angles. For example, the user may navigate in flight mode by directing one or more handheld controllers and/or the headset in a desired direction for flight);
GARCIA teaches a virtual reality environment. However, NIKOU teaches a NIKOU [0009] The augmented reality display may show various guide markers to assist the clinician in positioning the one or more surgical tools; [0072] a QR code may be placed in a corner of a tool tray, allowing the orientation and identity of that tray to be tracked; [0073] impinging the tip of the tool against the surface of the bone, a three-dimensional surface can be mapped for that bone that is associated with a position and orientation relative to the frame of reference of that fiducial mark)
GARCIA in view of NIKOU further teaches
determine a remote gesture path relative to the remote XR headset reference frame based on processing the remote operator-gesture tracking information through a remote XR headset view pose transform (GARCIA [0042] the user console 100 comprises a seat 110, foot-operated controls 120, one or more handheld user interface devices 122, and at least one user display 130 configured to display, for example, a view of the surgical site inside a patient. For example, as shown in the exemplary user console shown in FIG. 1C, a user located in the seat 110 and viewing the user display 130 may manipulate the foot-operated controls 120 and/or handheld user interface devices to remotely control the robotic arms 160 and/or surgical instruments; NIKOU [0216] the operator is detected and may interact with the CASS 100 by way of hand or body gestures, which may be used to control a surgical workflow); and
transform the remote gesture path to a local gesture path relative to a local XR headset reference frame using a local XR headset view pose transform (GARCIA [0042] in FIG. 1C, a user located in the seat 110 and viewing the user display 130 may manipulate the foot-operated controls 120 and/or handheld user interface devices to remotely control the robotic arms 160 and/or surgical instruments; [0047] the virtual reality system 200 may additionally or alternatively include an external display 240 for displaying the virtual robotic surgical environment. The immersive display 222 and the external display 240, if both are present, may be synchronized to show the same or similar content; NIKOU [0057] As an alternative or supplement to the Display 125, one or more members of the surgical staff may wear an Augmented Reality (AR) Head Mounted Device (HMD). For example, in FIG. 1 the Surgeon 111 is wearing an AR HMD 155 that may, for example, overlay pre-operative image data on the patient or provide surgical planning suggestions).
NIKOU discloses computer-assisted surgical systems and methods for assisting with tool alignment using one or more mixed reality displays, which is analogous to the present patent application.
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified GARCIA to incorporate the teachings of NIKOU, and apply the computer-assisted surgical system that track the position and orientation of a patient's anatomy and one or more surgical tools using fiducial markers or image analysis and optical recognition into the virtual robotic surgical environment.
Doing so would improve and enhances the ability of a surgeon to perform an ongoing surgery in real time including the ability to provide additional information in an augmented reality display system.
Regarding Claim 2, GARCIA in view of NIKOU teaches the system of claim 1, and further teaches wherein the at least one processor is further configured to provide the local gesture path to a local XR headset for display relative to the local XR headset reference frame (GARCIA [0042] The user console 100 may be located in the same procedure room as the robotic system 150, as shown in FIG. 1A; [0043] In some variations, a user may operate the robotic surgical system 150 in an “over the bed” (OTB) mode, in which the user is at the patient's side and simultaneously manipulating a robotically-driven tool driver/end effector attached thereto (e.g., with a handheld user interface device 122 held in one hand) and a manual laparoscopic tool. For example, the user's left hand may be manipulating a handheld user interface device 122 to control a robotic surgical component, while the user's right hand may be manipulating a manual laparoscopic tool. Thus, in these variations, the user may perform both robotic-assisted MIS and manual laparoscopic techniques on a patient).
Regarding Claim 3, GARCIA in view of NIKOU teaches the system of claim 1, and further teaches wherein the at least one processor is further configured to determine the remote gesture path relative to the remote XR headset reference frame based on tracking movement indicated by the remote operator-gesture tracking information of the object moved by the remote operator while the remote operator is concurrently viewing a transformed three-dimensional computer image through the remote XR headset relative to the object being moved (GARCIA [0047] As shown in FIG. 2B, the head-mounted display 220 may include an immersive display 222 for displaying the virtual robotic surgical environment to the user U (e.g., with a first-person perspective view of the virtual environment). The immersive display may, for example, be a stereoscopic display provided by eyepiece assemblies. In some variations, the virtual reality system 200 may additionally or alternatively include an external display 240 for displaying the virtual robotic surgical environment. The immersive display 222 and the external display 240, if both are present, may be synchronized to show the same or similar content; [0060] For example, the kinematics application 420 may allow for a description or definition of one or more virtual control modes, such as for the virtual robotic arms or other suitable virtual components in the virtual environment. Generally, for example, a control mode for a virtual robotic arm may correspond to a function block that enables the virtual robotic arm to perform or carry out a particular task).
Regarding Claim 4, GARCIA in view of NIKOU teaches the system of claim 1, and further teaches wherein the at least one processor is further configured to recognize a gesture of the object performed by the remote operator as corresponding to a defined operational command (GARCIA [0022] the control mode to be tested may be a trajectory following control mode for a robotic arm. In trajectory following, movement of the robotic arm may be programmed then emulated using the virtual reality system. Accordingly, when the system is used to emulate a trajectory following control mode, the actuation command generated by a kinematics application may include generating an actuated command for each of a plurality of virtual joints in the virtual robotic arm).
Regarding Claim 5, GARCIA in view of NIKOU teaches the system of claim 4, and further teaches wherein the defined operational command is configured to control equipment in a local environment (GARCIA [0041] the robotic arm 160 may include a plurality of links that are actuated so as to position and orient the tool driver 170, which actuates the surgical instrument 190. The robotic surgical system may further include a control tower 152 (e.g., including a power supply, computing equipment, etc.) and/or other suitable equipment for supporting functionality of the robotic components; [0042] a user located in the seat 110 and viewing the user display 130 may manipulate the foot-operated controls 120 and/or handheld user interface devices to remotely control the robotic arms 160 and/or surgical instruments).
Regarding Claim 6, GARCIA in view of NIKOU teaches the system of claim 1, and further teaches wherein the at least one processor is further configured to select an operational command from among a set of operational commands based on the remote gesture path corresponding to a defined gesture associated with the operational command (GARCIA [0061] Examples of primitive virtual control modes include, but are not limited to, a joint command mode (which allows a user to directly actuate a single virtual joint individually, and/or multiple virtual joints collectively), a gravity compensation mode (in which the virtual robotic arm holds itself in a particular pose, with particular position and orientation of the links and joints, without drifting downward due to simulated gravity), and trajectory following mode (in which the virtual robotic arm may move to follow a sequence of one or more Cartesian or other trajectory commands); [0065] This set of actuated commands may be implemented by a virtual operating environment application to move the virtual robotic arm in the virtual environment, thereby allowing testing for collision, volume or workspace of movement, etc.).
Regarding Claim 7, GARCIA in view of NIKOU teaches the system of claim 6, and further teaches wherein each operational command in the set of operational commands is associated with a different shaped gesture path (GARCIA [0057] a configuration file in Unified Robot Description Format (URDF) may store a configuration of a particular robotic arm, including definitions or values for fields such as number of arm links, number of arm joints connecting the arm links, length of each arm link, diameter or girth of each arm link, mass of each arm link, type of arm joint (e.g., roll, pitch, yaw etc.), etc. Additionally, kinematic constraints may be loaded as a “wrapper” over a virtual robotic component (e.g., arm) to further define the kinematic behavior of the virtual robotic component. In other variations, the virtual reality processor 210 may receive any suitable descriptions of virtual components to load and generate in the virtual robotic surgical environment. Accordingly, the virtual reality processor 210 may receive and utilize different combinations of configuration files and/or other descriptions of virtual components to generate particular virtual robotic surgical environments; [0063] the virtual operating environment application 410 passes status information to the kinematics application 420, and the kinematics application 420 passes commands to the virtual operating environment application 410 via the API, where the commands are generated based on the status information and subsequently used by the virtual reality processor 210 to generate changes in the virtual robotic surgical environment).
Regarding Claim 8, GARCIA in view of NIKOU teaches the system of claim 6, and further teaches wherein the at least one processor is further configured to provide the selected operational command to an equipment local to the local XR headset (NIKOU [0091] FIG. 5A shows examples of some of the control instructions that the Surgical Computer 150 provides to other components of the CASS 100, according to some embodiments. Note that the example of FIG. 5A assumes that the components of the Effector Platform 105 are each controlled directly by the Surgical Computer 150. In embodiments where a component is manually controlled by the Surgeon 111, instructions may be provided on the Display 125 or AR HMD 155 instructing the Surgeon 111 how to move the component).
Regarding Claim 9, GARCIA in view of NIKOU teaches the system of claim 1, and further teaches wherein the at least one processor is further configured to select an operational command for relocating an end effector connected to a surgical robot arm that is movable under control of a surgical robot system from among a set of operational commands based on the remote gesture path corresponding to a defined gesture associated with the operational command for relocating the end effector (GARCIA [0148] a user may operate the virtual reality system to plan surgical workflow. Configuration files of virtual objects (e.g., robotic surgical system including arms and tool drivers, user console, end effectors, other equipment, patient bed, patient, personnel, etc.) may be loaded into a virtual robotic surgical environment as representative of actual objects that will be in the actual (i.e., non-virtual, or real) operating room; NIKOU [0064] the CASS 100 can employ a manual or powered impactor that is attached or connected to the robotic arm 105A or end effector 105B to impact trial implants and final implants into the acetabulum. The robotic arm 105A and/or end effector 105B can be used to guide the impactor to impact the trial and final implants into the acetabulum in accordance with the surgical plan).
Regarding Claim 10, GARCIA in view of NIKOU teaches the system of claim 1, and further teaches wherein the object is a tracked stylus or a hand (GARCIA [0050] The handheld controller may include, for example, a carried device (e.g., wand, remote device, etc.) and/or a garment worn on the user's hand (e.g., gloves, rings, wristbands, etc.) and including sensors and/or configured to cooperate with external sensors to thereby provide tracking of the user's hand(s), individual finger(s), wrist(s), etc.).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Samantha (Yuehan) Wang whose telephone number is (571)270-5011. The examiner can normally be reached Monday-Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571)272-7440. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Samantha (YUEHAN) WANG/
Primary Examiner
Art Unit 2617
Prosecution Insights