DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are pending.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-3, 7-10 and 18-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Calloway et al. (US 2021/0338337 A1).
As to claim 1, Calloway discloses a surgical system (Calloway, FIG. 1, [0042], “surgical system 2”), comprising:
a navigation system (Calloway, FIGS. 3B-3C, [0065], “computer platform 910 … provide surgical navigation information to a display device, e.g., XR headset”) comprising a localizer (Calloway, FIG. 12, [0102], “tracking subsystem 830”) including a localizer coordinate system (Calloway, FIG. 12, [0102], “to determine the pose of the DRA relative to one or more defined coordinate systems, e.g., of the XR headsets 920, the tracking cameras 46, and/or another coordinate system defined for the patient, table, and/or room”), wherein the localizer (Calloway, FIG. 12, [0102], “tracking subsystem 830”) is configured to track a physical registration object (Calloway, e.g., FIG. 18, [0148], “T=NIR coordinate system of a tracked tool 604”) such that a pose of the physical registration object is configured to track a physical registration object is known in the localizer coordinate system (Calloway, e.g., FIG. 14, [0162], “registration of a direction of curvature of a surgical tool relative to an identified reference array enables the navigation controller 828 to display an accurate graphical representation of the surgical tool through the XR headset 920 and accurately posed relative to a tracked anatomical structure during the surgical procedure”);
a head-mounted device (HMD) (Calloway, e.g., FIG. 18, [0144], “HMD1 920”) comprising an HMD coordinate system (Calloway, e.g., FIG. 18, [0144], “N2=NIR coordinate system of the primary headset HMD1 920”), a display (Calloway, FIG. 13, [0104], “display screen 1302”) positionable in front of eyes of a user of the HMD (Calloway, FIG. 13, [0104], “XR headset 920”), and a sensing system (Calloway, [0101], “Each XR headset 920 can include one or more cameras”; FIG. 14, [0113], “cameras 1440”) configured to sense a biomechanical control input from the user (Calloway, FIG. 14, [0113], “may be configured operate as the gesture sensor 1444 by capturing for identification user hand gestures performed within the field of view of the camera(s) 1440”), wherein the HMD (Calloway, e.g., FIG. 18, [0144], “HMD1 920”) is configured to:
enable the physical registration object to be observed on, or through, the display (Calloway, e.g., FIG. 19, [0165], “FIG. 19 illustrates a graphical representation 1910 of surgical tool characteristics that can be displayed by the XR headset when a reference array 1902 is being registered to characteristics of a surgical tool 1900 for use in computer assisted navigation of the surgical tool 1900 during surgery”); and
sense, with the sensing system (Calloway, [0101], “Each XR headset 920 can include one or more cameras”; FIG. 14, [0113], “cameras 1440”), the biomechanical control input from the user (Calloway, FIG. 14, [0113], “may be configured operate as the gesture sensor 1444 by capturing for identification user hand gestures performed within the field of view of the camera(s) 1440”) to command acquisition of a virtual registration object (Calloway, FIG. 19, [0171], “the camera tracking system may be configured to determine the user's selection among the displayed list 1910”) relative to the physical registration object (Calloway, FIGS. 19-21, [0176], “enable a surgeon to confirm that the correct mode among the list 1910 (FIG. 19) has been selected by comparing the tool tip bend direction shown in the graphical representation 2112 to that of the physical surgical tool 1900”); and
one or more controllers coupled to the navigation system and/or the HMD and being configured to utilize the virtual registration object (Calloway, FIG. 19, [0171], “displayed list 1910”) to co-register the HMD coordinate system and the localizer coordinate system (Calloway, [0136], “Mechanically calibrating the visible and near infrared tracking coordinate systems enables the coordinate systems to be aligned sufficiently to perform 3D DRA fiducials triangulation operations using stereo matching to jointly identify pose of the DRA fiducials between the visible and near infrared tracking coordinate systems. Using both visible and near infrared tracking coordinate systems can enable any one or more of: (a) identifying tools that would not be identified using a single coordinate system; (b) increased pose tracking accuracy; (c) enabling a wider range of motion without losing tracking of surgical instruments, patient anatomy, and/or a robotic end effector; and (d) naturally track an XR headset in the same coordinate system as the navigated surgical instruments”).
As to claim 2, Calloway discloses the surgical system of claim 1, wherein the virtual registration object (Calloway, FIG. 19, [0171], “displayed list 1910”) is a point, a line, a plane (Calloway, see FIG. 19), a volume, or a point cloud.
As to claim 3, Calloway discloses The surgical system of claim 1, wherein:
the physical registration object (Calloway, FIGS. 19-21, [0176], “physical surgical tool 1900”) has a reference datum (Calloway, e.g., FIG. 19, [0170], “four modes A B C D”), wherein a pose of the reference datum is known in the localizer coordinate system (Calloway, e.g., FIG. 19, [0170], e.g., “the four modes include Mode A corresponding to +X curvature, Mode B corresponding to +Z curvature, Mode C corresponding to −X curvature, and Mode D corresponding to −Z curvature”); and
the HMD is configured to sense, with the sensing system, the biomechanical control input from the user to command acquisition of the virtual registration object that corresponds to the reference datum (Calloway, FIG. 19, [0171], “the camera tracking system may be configured to determine the user's selection among the displayed list 1910, based on tracking information from the set of tracking cameras indicating pose of a hand of the user in XR space relative to the displayed list 1910”).
As to claim 7, Calloway discloses the surgical system of claim 1, wherein:
the physical registration object (Calloway, FIGS. 19-21, [0176], “physical surgical tool 1900”) is a surgical tool (Calloway, FIGS. 19-21, [0176], “physical surgical tool 1900”) that is detectable by the localizer (Calloway, e.g., FIG. 14, [0162], “navigation controller 828”); and
the virtual registration object (Calloway, FIG. 19, [0171], “displayed list 1910”) is acquired relative to a feature of the surgical tool (Calloway, FIG. 19, [0165], “graphical representation 1910 of surgical tool characteristics”).
As to claim 8, Calloway discloses the surgical system of claim 7, wherein:
the feature of the surgical tool comprises a tool tip (Calloway, FIG. 19, [0170], e.g., “the tip of the surgical tool 1900 curves off to one side and requires a user to select one of four modes”); and
the virtual registration object is acquired relative to the tool tip (Calloway, FIG. 19, [0170], “displayed as a list 1910 during registration based on which direction the tool tip is curving with respect to the reference array 1902”).
As to claim 9, Calloway discloses the surgical system of claim 1, wherein:
the physical registration object (Calloway, FIGS. 1-2, [0055], “dynamic reference array 52”) is a tracker (Calloway, FIGS. 1-2, [0055], “markers, or other indicia which may be attached or formed on … a surgical tool in a navigated surgical procedure”) that is detectable by the localizer (Calloway, FIGS. 1-2, [0055], “The computer platform 910 in combination with the camera tracking system component 6 or other 3D localization system are configured to track in real-time the pose (e.g., positions and rotational orientations) of the DRA”); and
the virtual registration object (Calloway, e.g., FIG. 16, [0121], “textual information and other objects 1640”) is acquired relative to a feature of the tracker (Calloway, [0121], e.g., FIG. 16, “dynamic reference arrays 1630 and 1632, connected to the surgical tool 1602, become within the field of view of the cameras 1440”).
As to claim 10, Calloway discloses the surgical system of claim 9, wherein the tracker is:
coupled to a surgical tool; coupled to a base of a robotic manipulator; coupled to an end effector of the robotic manipulator; coupled to an anatomy; coupled to the localizer (Calloway, FIG. 4, [0055] “Dynamic reference arrays, also referred to as “DRAB” and “reference arrays” herein, can be rigid bodies, markers, or other indicia which may be attached or formed on one or more XR headsets being worn by personnel in the operating room, the end effector, the surgical robot, a surgical tool in a navigated surgical procedure, and an anatomical structure (e.g., bone) of a patient”); or a standalone tracker that is not coupled to any object.
As to claim 18, Calloway discloses the surgical system of claim 1, wherein the localizer (Calloway, e.g., FIG. 14, [0162], “navigation controller 828”) comprises a camera unit (Calloway, FIG. 17, [1032], “auxiliary tracking bar 46 having two pairs of stereo tracking cameras”) that comprises one or more optical sensors configured to detect visible light (Calloway, FIG. 17, [1032], “visible light capturing cameras”) and infrared light (Calloway, FIG. 17, [1032], “near infrared capturing cameras”) and
wherein the localizer (Calloway, e.g., FIG. 14, [0162], “navigation controller 828”) is configured to track the pose of the physical registration object using the one or more optical sensors (Calloway, FIG. 18, [0139], “the combination of XR headsets HMD1 920 and HMD2 920 and the tracking cameras 46 on the auxiliary tracking bar can, in operation with the computer platform 910, more robustly track the example objects of a patient reference array (R), robotic end effector (E), and surgical tool (T) or instrument”).
As to claim 19, it differs from claim 1 only in that it is the head-mounted device (HMD) configured for use with the surgical system of claim 1. It recites substantially the same limitations as in claim 1, and Calloway discloses them. Please see claim 1 for detailed analysis.
As to claim 20, it differs from claim 1 only in that it is the computer-implemented method of co-registering the head-mounted device (HMD) to the navigation system of claim 1. It recites substantially the same limitations as in claim 1, and Calloway discloses them. Please see claim 1 for detailed analysis.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office Action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 4-6 are rejected under 35 U.S.C. 103 as being unpatentable over Calloway et al. (US 2021/0338337 A1) in view of Recker et al. (US 2020/0388177 A1).
As to claim 4, Calloway does not teach the surgical system of claim 1, wherein the HMD senses the biomechanical control input by the sensing system being configured to detect a gaze of the user focused on the physical registration object.
However, Recker teaches the concept that the HMD (Recker, FIG. 18, [0097], “HMD device 1810”) senses the biomechanical control input by the sensing system being configured to detect a gaze of the user focused on the physical registration object (Recker, FIG. 18, [0120], “as will be described in relation to FIG. 18, the system using a mixed reality hardware platform, may monitor the position of a real object in the environment and the user's position/change in position relative to the object. For example, hand movement and eye movement may be measured relative to one or more real objects”).
At the time of effective filing date, it would have been obvious to one of ordinary skill in the art to modify the “HMD1 920” taught by Calloway to further measure the “eye movement relative to the real objects”, as taught by Recker, in order to provide more intuitive user interactions with tools by gaze detection in addition to the touch detection.
As to claim 5, Calloway in view of Recker teaches the surgical system of claim 1, wherein the HMD (Calloway, e.g., FIG. 18, [0144], “HMD1 920”) senses the biomechanical control input by the sensing system being configured to detect a hand or finger of the user physically touching the physical registration object (Recker, FIG. 18, [0098], “the camera 1825 may detect that the user's actual hands are touching a real object”). Examiner renders the same motivation as in claim 4.
As to claim 6, Calloway in view of Recker teaches the surgical system of claim 1, wherein the HMD (Calloway, e.g., FIG. 18, [0144], “HMD1 920”) commands acquisition of the virtual registration object (Calloway, FIG. 19, [0171], “displayed list 1910”) in response to sensing, with the sensing system, the biomechanical control input for a threshold amount of time (Recker, FIGS. 16-18, [0096], e.g., “over object OBJ-F for a period of time to select the object”). Examiner renders the same motivation as in claim 4.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Calloway et al. (US 2021/0338337 A1) in view of Kim (KR 20190099999 A, hereinafter English translation by Clarivate Analytics).
As to claim 11, Calloway teaches the surgical system of claim 1, wherein the virtual registration object (Calloway, e.g., FIG. 16, [0121], “textual information and other objects 1640”) is acquired relative to a feature of a tool (Calloway, e.g., see FIG. 16).
Calloway does not teach “the localizer comprises a camera unit; the physical registration object is the camera unit”, i.e., the “tool” is the “camera unit”.
However, Kim teaches the concepts that the localizer comprises a camera unit; the physical registration object is the camera unit (Kim, FIG. 3, S131, “the position of the virtual camera of the virtual image and the position of a camera (e.g., a camera included in a surgical tool such as an endoscope) of the actual surgical site image may be matched”), i.e., the “tool” is the “camera unit”.
At the time of effective filing date, it would have been obvious to one of ordinary skill in the art to modify the “navigation controller 828” taught by Calloway to further comprise the “camera included in a surgical tool”, as taught by Kim, in order to providing “optimizing the surgical process by minimizing unnecessary processes in performing surgery using a three-dimensional medical image” (Kim, BACKGROUND-ART).
Claims 12-15 are rejected under 35 U.S.C. 103 as being unpatentable over Calloway et al. (US 2021/0338337 A1) in view of Weinstein et al. (US 2018/0185100 A1).
As to claim 12, Calloway teaches the surgical system of claim 1, wherein the virtual registration object (Calloway, e.g., FIG. 16, [0121], “textual information and other objects 1640”) is acquired relative to a feature of a tool (Calloway, e.g., see FIG. 16).
Calloway does not teach “the physical registration object is an anatomy that is registered to the localizer coordinate system”, i.e., the “tool” is the “anatomy”.
However, Weinstein teaches the concept that the physical registration object is an anatomy that is registered to the localizer coordinate system (Weinstein, FIG. 12, [0134], “the HMD 200 may be used to assist with registration of the three-dimensional model (model coordinate system MODEL2 in FIG. 3) of the patient's anatomy to the associated bone tracker 44 (bone tracker coordinate system BTRK1) via the associated registration of the actual bone (bone coordinate system FBONE) to the bone tracker 44, e.g., see transforms T5, T11 in FIG. 3”)”, i.e., the “tool” is the “anatomy”.
At the time of effective filing date, it would have been obvious to one of ordinary skill in the art to modify the “navigation controller 828” taught by Calloway to further comprise the “the patient's anatomy”, as taught by Weinstein, in order to provide “registering the HMD to the common coordinate system, along with the surgical tools and/or the anatomy” (Weinstein, [0005]).
As to claim 13, Weinstein teaches the surgical system of claim 1, wherein: prior to co-registration between the HMD coordinate system and the localizer coordinate system, the pose of the physical registration object is unknown in the HMD coordinate system (Weinstein, FIG. 31, [0159], “in step 300, registering the HMD coordinate system and the localizer coordinate system LCLZ such that images displayed by the HMD 200 can be associated with real objects tracked by the localizer (e.g., the surgical tool 22, the femur F, the tibia T, etc.)” → “registration error is indicated to the user in step 302”; it is reasonably inferred that the pose of the physical registration object is unknown, e.g., “registration error”); and
after co-registration between the HMD coordinate system and the localizer coordinate system, the pose of the physical registration object is known in the HMD coordinate system (Weinstein, FIG. 34, [0162], “Registration of the robotic coordinate system and the localizer coordinate system LCLZ is finalized in step 326 in response to the user moving the surgical tool in accordance with the protocol images 258 displayed by the HMD 200”). Examiner renders the same motivation as in claim 12.
As to claim 14, Weinstein teaches the surgical system of claim 1, wherein: prior to co-registration between the HMD coordinate system and the localizer coordinate system, a pose the HMD is unknown in the localizer coordinate system (Weinstein, FIG. 31, [0159], “in step 300, registering the HMD coordinate system and the localizer coordinate system LCLZ such that images displayed by the HMD 200 can be associated with real objects tracked by the localizer (e.g., the surgical tool 22, the femur F, the tibia T, etc.)” → “registration error is indicated to the user in step 302”; it is reasonably inferred that the pose of the HMD is unknown, e.g., “registration error”); and
after co-registration between the HMD coordinate system and the localizer coordinate system, the pose of the HMD is known in the localizer coordinate system (Weinstein, FIG. 34, [0162], “Registration of the robotic coordinate system and the localizer coordinate system LCLZ is finalized in step 326 in response to the user moving the surgical tool in accordance with the protocol images 258 displayed by the HMD 200”). Examiner renders the same motivation as in claim 12.
As to claim 15, Weinstein teaches the surgical system of claim 14, wherein the HMD is trackerless and not directly trackable by the navigation system prior to co-registration between the HMD coordinate system and the localizer coordinate (Weinstein, FIGS. 1-2, [0100], “HMD 200 then utilizes the camera 214 alone, or in conjunction with the depth camera, to find objects in the HMD coordinate system”; [0101], “In another embodiment, a separate HMD tracker 218 (see FIG. 3), similar to the trackers 44, 46, 48, could be mounted to the HMD 200 (e.g., fixed to the support structure 202). In this version, the HMD tracker 218 would have its own HMD tracker coordinate system HMDTRK that is in a known position/orientation relative to the local coordinate system LOCAL”; it is reasonably inferred that HMD 200 may be with tracker such as “a separate HMD tracker 218” or without tracker). Examiner renders the same motivation as in claim 12.
Claims 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Calloway et al. (US 2021/0338337 A1) in view of Healy et al. (US 2021/0251716 A1).
As to claim 16, Calloway does not teach the surgical system of claim 1, wherein: the virtual registration object is acquired relative to a real-world view of the physical registration object; the HMD comprises a camera configured to capture a video of the real-world view; and the HMD is configured to present the video of the real-world view on the display.
However, Healy teaches the concepts that the virtual registration object (Healy, FIG. 18A, [0137], e.g., “virtual model 1830”) is acquired relative to a real-world view of the physical registration object (Healy, FIG. 18A, [0137], “physical instrument attachment 1820”);
the HMD (Healy, FIGS. 15, 18A-18C, [0137], “XR headset 920”) comprises a camera (Healy, FIG. 15, [0106], “cameras 1440”) configured to capture a video of the real-world view (Healy, FIG. 15, [0106], “real-world scene 1502”); and
the HMD is configured to present the video of the real-world view on the display (Healy, see FIGS. 18A-18C, [0034], “the XR headset may be configured to provide a virtual reality (“VR”) viewing environment by preventing or substantially preventing light from the real-world scene from being directly viewed by the user while the user is viewing the computer generated AR images on a display screen”).
At the time of effective filing date, it would have been obvious to one of ordinary skill in the art to modify the “HMD1 920” taught by Calloway to further comprise the “cameras 1440” to display the “real-world scene 1502”, as taught by Healy, in order to “improve the tracking performance of navigation systems” (Healy, [0003]).
As to claim 17, Healy teaches the surgical system of claim 1, wherein:
the virtual registration object (Healy, FIG. 18A, [0137], e.g., “virtual model 1830”) is acquired relative to a real-world view of the physical registration object (Healy, FIG. 18A, [0137], “physical instrument attachment 1820”); and
the display (Healy, FIGS. 13 and 18A-18C, [0098], “display screen 1302” of “XR headset 920”) is a transparent or semi-transparent display (Healy, FIGS. 13 and 18A-18C, [0098], “may be a see-through LCD display device or a semi-reflective lens that reflects images projected by a display device toward the wearer's eyes”) such that the real-world view is visible by the eyes of the user based on light passing directly through the display (Healy, see FIGS. 13 and 18A-18C, [0099], “display screen 1302 operates as a see-through display screen, also referred to as a combiner, that reflects light from display panels of a display device toward the user's eyes”). Examiner renders the same motivation as in claim 16.
Conclusion
The prior arts made of record and not relied upon are considered pertinent to applicant’s disclosure:
Crawford et al. (US 2022/0061921 A1) teaches the concept of “camera tracking system for computer assisted navigation during surgery” (Abs.);
Calloway et al. (US 2021/0346098 A1) teaches the concept of “surgical tool marking array including a first marker holder, a second marker holder, and a tool holder” (Abs.); and
Healy et al. (US 2021/0386503 A1) teaches the concept of “camera tracking system for computer assisted navigation during surgery” (Abs.).
Inquiry
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RICHARD J HONG whose telephone number is (571) 270-7765. The examiner can normally be reached on 9:00 AM to 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chanh Nguyen can be reached on (571) 272-7772. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Oct. 4, 2024
/RICHARD J HONG/Primary Examiner, Art Unit 2623
***
Prosecution Insights