DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
2. The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory obviousness-type double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); and In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on a nonstatutory double patenting ground provided the conflicting application or patent either is shown to be commonly owned with this application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement.
Effective January 1, 1994, a registered attorney or agent of record may sign a terminal disclaimer. A terminal disclaimer signed by the assignee must fully comply with 37 CFR 3.73(b).
3. Claims 1-20 are rejected on the ground of nonstatutory obviousness-type double patenting as being unpatentable over claims 1-20 of Patent No. 12,295,798. Although the conflicting claims are not identical, they are not patentably distinct from each other because they are essentially the same except that claims 1 and 13 of the instant application additionally recites “A method of providing a planned instrument attachment”, “a physical instrument attachment” “determining a pose of an instrument relative to an XR headset;”. However, Sauer et al. (US 2005/0203380 A1) teaches these limitations, as discussed below. Thus, claims 1-20 of the instant application is obvious in view of claims 1-20 of Patent No. 12,295,798.
4. Regarding claim 1, the application claim discloses A method of providing a virtual model of a planned instrument attachment to ensure correct selection of a physical instrument attachment, the method comprising: providing a surgical system, the surgical system including: a tracking system, an extended reality (XR) headset configured to be worn by a user during a surgical procedure and including a plurality of markers for tracking a pose of the XR headset by the tracking system, a display screen configured to display a virtual model of a selected implant attached to an inserter instrument for viewing by the user, and an XR headset controller; determining a pose of an instrument relative to an XR headset; generating a shape and a pose of the virtual model of the planned instrument attachment, via the XR controller, based on predetermined information associated with the planned instrument attachment and based on the pose of the instrument relative to the XR headset; and displaying the virtual model on the display screen. Claim 1 of Patent No. 12,295,798 discloses A surgical system comprising: a tracking system; an extended reality (XR) headset configured to be worn by a user during a surgical procedure and including a plurality of markers for tracking a pose of the XR headset by the tracking system and a display screen configured to display a virtual model of a selected implant attached to an inserter instrument that is overlaid on a real-world scene that has passed through the display screen for viewing by the user, the inserter instrument being trackable by the tracking system such that the virtual model of the selected implant is superimposed on the selected implant physically attached to the inserter instrument prior to insertion of the insertion instrument into a patient body such that both the physical implant attached to the inserter instrument and the virtual implant overlaid on the attached physical implant are simultaneously visible to the user from the display screen so as to allow the user to check for correct selection of the selected implant; and an XR headset controller configured to generate a shape and a pose of the virtual model of the selected implant based on predetermined information associated with the selected implant and based on a pose of the inserter instrument relative to the XR headset. Regarding claim 1, the only difference is that claim 1 of the instant application recites “A method of providing a planned instrument attachment”, “a physical instrument attachment” “determining a pose of an instrument relative to an XR headset;” and does not recite “the selected implant is superimposed on the selected implant physically attached to the inserter instrument” “a virtual model of a selected implant attached to an inserter instrument that is overlaid on a real-world scene that has passed through the display screen” “the inserter instrument being trackable by the tracking system” “prior to insertion of the insertion instrument into a patient body such that both the physical implant attached to the inserter instrument and the virtual implant overlaid on the attached physical implant are simultaneously visible”; while claim 1 of Patent No. 12,295,798 recites. For the additional limitation, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Sauer into Patent No. 12,295,798, in order to facilitate spatially registered visualization of the planned instrument attachment relative to the physical surgical instrument. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention by applicant to modify claim 1 of Patent No. 12,295,798 to include “A method of providing a planned instrument attachment”, “a physical instrument attachment” “determining a pose of an instrument relative to an XR headset;” ([0020, 0027-0028, 0032]) Regarding claim 13, the analysis is similar to that of claim 1, the rationale of claim 1 rejection is applied in rejecting claim 13. Therefore, the claims in the present application disclosing similar limitations with the claims in the Patent No. 12,295,798 recite.
5. The following table shows the claims of the current application being examined and the conflicting claims of Patent No. 12,295,798.
Current Application No.
19/199,517
Patent No.
12,295,798
1
1
2-12
2-12
13
13
14-20
14-20
The following table shows an example of the corresponding conflicting claims of the current application and Patent No. 12,295,798.
Current Application No.
19/199,517
Claim 1
Patent No.
12,295,798
Claim 1
A method of providing a virtual model of a planned instrument attachment to ensure correct selection of a physical instrument attachment, the method comprising:
the virtual model of the selected implant is superimposed on the selected implant physically attached to the inserter instrument ... so as to allow the user to check for correct selection of the selected implant;
providing a surgical system, the surgical system including: a tracking system,
A surgical system comprising: a tracking system;
an extended reality (XR) headset configured to be worn by a user during a surgical procedure and including a plurality of markers for tracking a pose of the XR headset by the tracking system,
an extended reality (XR) headset configured to be worn by a user during a surgical procedure and including a plurality of markers for tracking a pose of the XR headset by the tracking system
a display screen configured to display a virtual model of a selected implant attached to an inserter instrument for viewing by the user,
a display screen configured to display a virtual model of a selected implant attached to an inserter instrument that is overlaid on a real-world scene that has passed through the display screen for viewing by the user,
the inserter instrument being trackable by the tracking system
determining a pose of an instrument relative to an XR headset;
Sauer teaches these limitations, as discussed above.
and an XR headset controller; generating a shape and a pose of the virtual model of the planned instrument attachment, via the XR controller, based on predetermined information associated with the planned instrument attachment and based on the pose of the instrument relative to the XR headset; and displaying the virtual model on the display screen.
prior to insertion of the insertion instrument into a patient body such that both the physical implant attached to the inserter instrument and the virtual implant overlaid on the attached physical implant are simultaneously visible to the user from the display screen; and an XR headset controller configured to generate a shape and a pose of the virtual model of the selected implant based on predetermined information associated with the selected implant and based on a pose of the inserter instrument relative to the XR headset.
Claim Rejections - 35 USC § 103
6. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
7. The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103(a) are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
8. Claims 1-8, 10 and 12 are rejected under 35 U.S.C. 103(a) as being unpatentable over Sauer et al. (US 2005/0203380 A1) in view of White et al. (US 2018/0082480 A1) .
9. With reference to claim 1, Sauer teaches A method of providing a virtual model of a planned instrument attachment, a physical instrument attachment, , the method comprising: providing a surgical system, (“Exemplary embodiments of the invention as described herein generally include systems and methods for augmented reality navigation systems and setup for MR-guided interventions.” [0020] “To visualize a hand-held surgical instrument, such as a biopsy needle, inside the patient's body in the AR video view, the patients body pose needs to be tracked. A cluster of retro-reflective markers 14 can be attached to the biopsy needle from which the rigid body transformation between this cluster 14 and the tracker camera for each video frame can be estimated. The pose of this hand-held instrument with respect to the scanner table can then easily be deduced from the estimated head pose from the same video frame, a procedure referred to as instrument tracking.” [0027] “FIG. 3 depicts a monoscopic augmented view of a needle insertion guide, according to an embodiment of the invention. In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path,” [0032]) Sauer also teaches the surgical system including: a tracking system, an extended reality (XR) headset configured to be worn by a user during a surgical procedure and including a plurality of markers for tracking a pose of the XR headset by the tracking system, (“there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor.” [0005] “FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20. The monochrome tracking camera 11 works in conjunction with retro-reflective optical markers 13 that are placed around the intervention workspace. The flash 22 can be synchronized with the tracker camera and permits the use of a high speed electronic shutter.” [0024] “To visualize a hand-held surgical instrument, such as a biopsy needle, inside the patient's body in the AR video view, the patients body pose needs to be tracked. A cluster of retro-reflective markers 14 can be attached to the biopsy needle from which the rigid body transformation between this cluster 14 and the tracker camera for each video frame can be estimated. The pose of this hand-held instrument with respect to the scanner table can then easily be deduced from the estimated head pose from the same video frame, a procedure referred to as instrument tracking.” [0027]) Sauer further teaches a display screen configured to display a virtual model of a selected implant attached to an inserter instrument for viewing by the user, (“In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path, allowing the user to aim precisely at the target from a distance, before and during insertion of the needle. The large concentric ring 32 around the intersection of the extended needle path 33 with the target plane indicates the distance between needle tip 34 and the target plane, enhancing the user's ability to judge the depth correctly.” [0032]) Sauer teaches an XR headset controller; determining a pose of an instrument relative to an XR headset; (“FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20.” [0024] “To visualize a hand-held surgical instrument, such as a biopsy needle, inside the patient's body in the AR video view, the patients body pose needs to be tracked. A cluster of retro-reflective markers 14 can be attached to the biopsy needle from which the rigid body transformation between this cluster 14 and the tracker camera for each video frame can be estimated. The pose of this hand-held instrument with respect to the scanner table can then easily be deduced from the estimated head pose from the same video frame, a procedure referred to as instrument tracking.” [0027] “In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path, allowing the user to aim precisely at the target from a distance, before and during insertion of the needle. The large concentric ring 32 around the intersection of the extended needle path 33 with the target plane indicates the distance between needle tip 34 and the target plane, enhancing the user's ability to judge the depth correctly.” [0032]) Sauer also teaches generating a shape and a pose of the virtual model of the planned instrument attachment, via the XR controller, based on predetermined information associated with the planned instrument attachment and based on the pose of the instrument relative to the XR headset; and displaying the virtual model on the display screen. (“there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor. … determining the pose of said medical instrument with respect to the table from the patient's body pose and said instrument markers, and displaying in the guidance display a visual representation of said patient, said instrument, and a path for guiding said instrument to perform said medical intervention” [0005] “FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20. The monochrome tracking camera 11 works in conjunction with retro-reflective optical markers 13 that are placed around the intervention workspace.” [0024] “The pose of this hand-held instrument with respect to the scanner table can then easily be deduced from the estimated head pose from the same video frame, a procedure referred to as instrument tracking.” [0027] “the tracked instrument, such as the biopsy needle, includes a marker attachment containing set of retro-reflective markers. The size and shape of the markers are designed so that they are differentiable from the scanner table marker set within the tracking camera images.” [0029] “the pose of the instrument being used for the intervention procedure is determined with respect to the scanning table from optical markers attached to the instrument, and, at step 57, the respiration of the patient is monitored. This information regarding the patient's body pose, the instrument's pose, and patient's respiration is used, along with an MRI or other modality image to generate a visual representation of the patient, the intervention target, the instrument, and tracking path to guide the instrument in the stereoscopic guidance display of the HMD at step 58.” [0035])
PNG
media_image1.png
591
500
media_image1.png
Greyscale
Sauer does not explicitly teach a virtual model of a planned instrument to ensure correct selection of a physical instrument; This is what White teaches (“The surgical assistant 114 may be presented with a virtual instrument through the AR display 116. The surgical assistant 114 may select a surgical instrument corresponding to the virtual instrument, which may be detected by a detection device, such as an AR camera 118. After detecting that the surgical instrument has been identified, selected, or moved by the surgical assistant 114, the AR display 116 may cease display of the virtual instrument.” [0027]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of White into Sauer, in order to facilitate identification and selection of the physical instrument and improve efficiency during the surgical procedure.
10. With reference to claim 2, Sauer teaches the inserter instrument includes a plurality of second markers trackable by the tracking system and the XR headset controller uses the plurality of the second markers to track the pose of the inserter instrument relative to the XR headset. (“there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor.” [0005] “FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20. The monochrome tracking camera 11 works in conjunction with retro-reflective optical markers 13 that are placed around the intervention workspace. … a bridge 16 with retro-reflective markers 14 can be attached to the MR scanner table 17, thereby establishing a reference for the patient coordinate system. In an offline calibration step, the rigid body transformations between the tracker camera and the scene cameras can be determined. During the interventional procedure the rigid body transformation between the marker frame 16 and the tracker camera can be estimated. Thus, the poses of both scene cameras for each video frame with respect to the marker frame can be deduced, a procedure referred to as head tracking. … To visualize a hand-held surgical instrument, such as a biopsy needle, inside the patient's body in the AR video view, the patients body pose needs to be tracked. A cluster of retro-reflective markers 14 can be attached to the biopsy needle from which the rigid body transformation between this cluster 14 and the tracker camera for each video frame can be estimated. The pose of this hand-held instrument with respect to the scanner table can then easily be deduced from the estimated head pose from the same video frame, a procedure referred to as instrument tracking. As shown in FIG. 2, the marker set 15 according to an embodiment of the invention can be attached to the MR scanner table and includes a bridge 16 which is double oblique cut of the surface of cylinder with the radius equal to that of MR bore. The markers 15 have a 3D distribution implemented using small posts 18 attached to the main bridge. The design provides a clear line sight for the tracking camera, while maximizing the workspace volume. Furthermore, the marker set is clear to go inside the MR bore, while the patient is on the scanner table for the scan.” [0024-0028])
11. With reference to claim 3, Sauer teaches the XR headset controller generate a virtual model of the inserter instrument for display on the display screen of the XR headset. (“there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor.” [0005] “FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20. The monochrome tracking camera 11 works in conjunction with retro-reflective optical markers 13 that are placed around the intervention workspace.” [0024] “FIG. 3 depicts a monoscopic augmented view of a needle insertion guide, according to an embodiment of the invention. In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path, allowing the user to aim precisely at the target from a distance, before and during insertion of the needle. The large concentric ring 32 around the intersection of the extended needle path 33 with the target plane indicates the distance between needle tip 34 and the target plane, enhancing the user's ability to judge the depth correctly.” [0032] “With a patient positioned on the scanning table, the intervention procedure is started at step 53. During the intervention, the rigid body transformation between tracking camera and the scanning table is determined at step 54 from optical markers attached to the frame that is part of the table. The relationship between the tracking camera and table can then be used to determine the patient's body pose at step 55. In addition, at step 56, the pose of the instrument being used for the intervention procedure is determined with respect to the scanning table from optical markers attached to the instrument, and, at step 57, the respiration of the patient is monitored. This information regarding the patient's body pose, the instrument's pose, and patient's respiration is used, along with an MRI or other modality image to generate a visual representation of the patient, the intervention target, the instrument, and tracking path to guide the instrument in the stereoscopic guidance display of the HMD at step 58. … The composite stereoscopic images (i.e., video plus the graphics from the image data) according to an embodiment of the invention provide an intuitive guidance to the interventional radiologist and/or surgeon to perform the procedure more accurately, with more confidence, and in less time. In addition, video-see-through AR allows greater control over the augmented video stream. An HMD gives the user the freedom to move around in a natural way and to inspect the scene from a variety of viewpoints” [0035-0036])
Sauer does not explicitly teach such that the user sees both the virtual model of the inserter instrument and the actual inserter instrument through the display screen. This is White teaches (“The surgical assistant 114 may be presented with a virtual instrument through the AR display 116. The surgical assistant 114 may select a surgical instrument corresponding to the virtual instrument, which may be detected by a detection device, such as an AR camera 118. After detecting that the surgical instrument has been identified, selected, or moved by the surgical assistant 114, the AR display 116 may cease display of the virtual instrument. In an example, the surgeon 102 or the nurse 120 may cause an instruction to be sent to the AR device 116 of the surgical assistant 114 to display the virtual instrument. … AR display 200A includes a physical surgical instrument case 205 and virtual representation of a surgical instruments 202. The virtual representation of the surgical instruments 202 may be shown to a surgical assistant in order to identify a physical surgical instrument to be retrieved for use in a surgical procedure. … the virtual representation of a surgical instrument 202 may include a particular location in the AR display 200A. For example, the virtual representation of a surgical instrument 202 may hover over the case 205 or tray containing the physical surgical instrument (corresponding to the virtual representation of the surgical instrument 202). In an example, the case 205 may be sealed (e.g., sterilized) and may contain the physical surgical instrument. The case 205 may have an identification component 215 (e.g., a barcode, a QR code, a radio frequency identification (RFID) tag, etc.) to identify that the case 205 contains the physical surgical instrument. The identification component 215 may be used to identify the surgical instrument using a detection device (e.g., a camera, a scanner, an RFID reader, etc.). Using the identification, the AR display 200A may automatically place the virtual representation of the surgical instrument 202 over, within, or on the case 205.” [0027-0029] “the AR display 300 may include sample displays of other team members, such as a nurse display 312 or a surgical assistant display 314. In an example, the surgeon using the AR display 300 may identify a surgical instrument to be used and virtually drag the surgical instrument from a selection location to the surgical assistant display 314 (or may select the surgical instrument from within the surgical assistant display 314). A virtual representation of the surgical instrument may then be displayed in the surgical assistant display 314 automatically.” [0050]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of White into Sauer, in order to facilitate identification and selection of the physical instrument and improve efficiency during the surgical procedure.
12. With reference to claim 4, Sauer teaches the XR headset controller is further configured to display the virtual model over where the selected implant would have been attached to the inserter instrument for comparison between the actual selected implant and the virtual model by the user. (“there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor.” [0005] “The camera triplet (i.e. a stereo scene camera 12 and tracking camera 11) is rigidly mounted on top of the HMD 10. Both scene cameras have fixed focus lenses and provide sharp images in about arm's length distance. The monochrome tracking camera 11, which is only sensitive in the near-infrared spectrum, is equipped with a wide-angle lens 21 and a ring shaped infrared flash 22. The user, who is wearing the HMD 10, sees in guidance display viewer 19 a stereo view of a patient overlaid with diagnostic information as provided by an MRI or other imaging modality, and with guiding graphics. … FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20.” [0023-0024] “FIG. 3 depicts a monoscopic augmented view of a needle insertion guide, according to an embodiment of the invention. In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path, allowing the user to aim precisely at the target from a distance, before and during insertion of the needle. The large concentric ring 32 around the intersection of the extended needle path 33 with the target plane indicates the distance between needle tip 34 and the target plane, enhancing the user's ability to judge the depth correctly.” [0032] “With a patient positioned on the scanning table, the intervention procedure is started at step 53. During the intervention, the rigid body transformation between tracking camera and the scanning table is determined at step 54 from optical markers attached to the frame that is part of the table. The relationship between the tracking camera and table can then be used to determine the patient's body pose at step 55. In addition, at step 56, the pose of the instrument being used for the intervention procedure is determined with respect to the scanning table from optical markers attached to the instrument, and, at step 57, the respiration of the patient is monitored. This information regarding the patient's body pose, the instrument's pose, and patient's respiration is used, along with an MRI or other modality image to generate a visual representation of the patient, the intervention target, the instrument, and tracking path to guide the instrument in the stereoscopic guidance display of the HMD at step 58. … The composite stereoscopic images (i.e., video plus the graphics from the image data) according to an embodiment of the invention provide an intuitive guidance to the interventional radiologist and/or surgeon to perform the procedure more accurately, with more confidence, and in less time. In addition, video-see-through AR allows greater control over the augmented video stream. An HMD gives the user the freedom to move around in a natural way and to inspect the scene from a variety of viewpoints” [0035-0036])
13. With reference to claim 5, Sauer teaches the XR headset controller generates the virtual model of the selected implant with a partial transparency such that the physical selected implant is visible through the virtual model 1830 on the display screen. (“there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor.” [0005] “An augmented reality (AR) image guidance system can map medical imaging information onto a patient's body. Anatomical structures can be visually displayed at the location where they actually are, a concept referred to as in-situ visualization. With this visual display, a physician can observe directly a patient's internal anatomy. The patient's body appears transparent. In-situ visualization is a direct way of presenting medical imaging information, and has the potential to guide minimally invasive procedures very intuitively.” [0003] “FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20.” [0024] “FIG. 3 depicts a monoscopic augmented view of a needle insertion guide, according to an embodiment of the invention. In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path, allowing the user to aim precisely at the target from a distance, before and during insertion of the needle. The large concentric ring 32 around the intersection of the extended needle path 33 with the target plane indicates the distance between needle tip 34 and the target plane, enhancing the user's ability to judge the depth correctly.” [0032] “the pose of the instrument being used for the intervention procedure is determined with respect to the scanning table from optical markers attached to the instrument, and, at step 57, the respiration of the patient is monitored. This information regarding the patient's body pose, the instrument's pose, and patient's respiration is used, along with an MRI or other modality image to generate a visual representation of the patient, the intervention target, the instrument, and tracking path to guide the instrument in the stereoscopic guidance display of the HMD at step 58.” [0035])
14. With reference to claim 6, Sauer teaches the XR headset controller is further configured to generate the virtual model of the selected implant to have partially transparent graphical features that at least one of which is posed to substantially align with at least one corresponding feature of the physical selected implant when the virtual model of the selected implant is overlaid on the physical selected implant while viewed through the display. (“there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor.” [0005] “An augmented reality (AR) image guidance system can map medical imaging information onto a patient's body. Anatomical structures can be visually displayed at the location where they actually are, a concept referred to as in-situ visualization. With this visual display, a physician can observe directly a patient's internal anatomy. The patient's body appears transparent. In-situ visualization is a direct way of presenting medical imaging information, and has the potential to guide minimally invasive procedures very intuitively.” [0003] “The camera triplet (i.e. a stereo scene camera 12 and tracking camera 11) is rigidly mounted on top of the HMD 10. Both scene cameras have fixed focus lenses and provide sharp images in about arm's length distance. The monochrome tracking camera 11, which is only sensitive in the near-infrared spectrum, is equipped with a wide-angle lens 21 and a ring shaped infrared flash 22. The user, who is wearing the HMD 10, sees in guidance display viewer 19 a stereo view of a patient overlaid with diagnostic information as provided by an MRI or other imaging modality, and with guiding graphics.” [0023] “FIG. 3 depicts a monoscopic augmented view of a needle insertion guide, according to an embodiment of the invention. In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path, allowing the user to aim precisely at the target from a distance, before and during insertion of the needle. The large concentric ring 32 around the intersection of the extended needle path 33 with the target plane indicates the distance between needle tip 34 and the target plane, enhancing the user's ability to judge the depth correctly.” [0032] “the pose of the instrument being used for the intervention procedure is determined with respect to the scanning table from optical markers attached to the instrument, and, at step 57, the respiration of the patient is monitored. This information regarding the patient's body pose, the instrument's pose, and patient's respiration is used, along with an MRI or other modality image to generate a visual representation of the patient, the intervention target, the instrument, and tracking path to guide the instrument in the stereoscopic guidance display of the HMD at step 58.” [0035])
15. With reference to claim 7, Sauer teaches the XR headset controller being further configured to selectively render graphical features of the virtual model of the selected implant as one of partially transparent and opaque based on user input. (“there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor.” [0005] “FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20.” [0024] “An augmented reality (AR) image guidance system can map medical imaging information onto a patient's body. Anatomical structures can be visually displayed at the location where they actually are, a concept referred to as in-situ visualization. With this visual display, a physician can observe directly a patient's internal anatomy. The patient's body appears transparent. In-situ visualization is a direct way of presenting medical imaging information, and has the potential to guide minimally invasive procedures very intuitively.” [0003] “The camera triplet (i.e. a stereo scene camera 12 and tracking camera 11) is rigidly mounted on top of the HMD 10. Both scene cameras have fixed focus lenses and provide sharp images in about arm's length distance. The monochrome tracking camera 11, which is only sensitive in the near-infrared spectrum, is equipped with a wide-angle lens 21 and a ring shaped infrared flash 22. The user, who is wearing the HMD 10, sees in guidance display viewer 19 a stereo view of a patient overlaid with diagnostic information as provided by an MRI or other imaging modality, and with guiding graphics.” [0023] “FIG. 3 depicts a monoscopic augmented view of a needle insertion guide, according to an embodiment of the invention. In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path, allowing the user to aim precisely at the target from a distance, before and during insertion of the needle. The large concentric ring 32 around the intersection of the extended needle path 33 with the target plane indicates the distance between needle tip 34 and the target plane, enhancing the user's ability to judge the depth correctly.” [0032] “the pose of the instrument being used for the intervention procedure is determined with respect to the scanning table from optical markers attached to the instrument, and, at step 57, the respiration of the patient is monitored. This information regarding the patient's body pose, the instrument's pose, and patient's respiration is used, along with an MRI or other modality image to generate a visual representation of the patient, the intervention target, the instrument, and tracking path to guide the instrument in the stereoscopic guidance display of the HMD at step 58.” [0035])
Sauer does not explicitly teach one of opaque. This is what White teaches (“The virtual object 110 may be made visible to the surgeon 102 by projecting light. The virtual object 110 may appear to have a degree of transparency or may be opaque (i.e., blocking aspects of the real environment).” [0023]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of White into Sauer, in order to facilitate identification and selection of the physical instrument and improve efficiency during the surgical procedure.
16. With reference to claim 8, Sauer teaches the XR headset controller being further configured to selectively determine the pose of the virtual model of the selected implant to be displayed as one of an overlay on the physical selected implant, as behind the physical selected implant, and apart from the physical selected implant based on user input. (“there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor.” [0005] “FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20.” [0024] “FIG. 3 depicts a monoscopic augmented view of a needle insertion guide, according to an embodiment of the invention. In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path, allowing the user to aim precisely at the target from a distance, before and during insertion of the needle. The large concentric ring 32 around the intersection of the extended needle path 33 with the target plane indicates the distance between needle tip 34 and the target plane, enhancing the user's ability to judge the depth correctly.” [0032] “With a patient positioned on the scanning table, the intervention procedure is started at step 53. During the intervention, the rigid body transformation between tracking camera and the scanning table is determined at step 54 from optical markers attached to the frame that is part of the table. The relationship between the tracking camera and table can then be used to determine the patient's body pose at step 55. In addition, at step 56, the pose of the instrument being used for the intervention procedure is determined with respect to the scanning table from optical markers attached to the instrument, and, at step 57, the respiration of the patient is monitored. This information regarding the patient's body pose, the instrument's pose, and patient's respiration is used, along with an MRI or other modality image to generate a visual representation of the patient, the intervention target, the instrument, and tracking path to guide the instrument in the stereoscopic guidance display of the HMD at step 58. … The composite stereoscopic images (i.e., video plus the graphics from the image data) according to an embodiment of the invention provide an intuitive guidance to the interventional radiologist and/or surgeon to perform the procedure more accurately, with more confidence, and in less time. In addition, video-see-through AR allows greater control over the augmented video stream. An HMD gives the user the freedom to move around in a natural way and to inspect the scene from a variety of viewpoints” [0035-0036])
17. With reference to claim 10, Sauer teaches the XR headset controller being further configured to generate the virtual model to have graphical features representing an instrument extender. (“there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor.” [0005] “FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20.” [0024] “FIG. 3 depicts a monoscopic augmented view of a needle insertion guide, according to an embodiment of the invention. In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path, allowing the user to aim precisely at the target from a distance, before and during insertion of the needle. The large concentric ring 32 around the intersection of the extended needle path 33 with the target plane indicates the distance between needle tip 34 and the target plane, enhancing the user's ability to judge the depth correctly.” [0032])
18. With reference to claim 12, Sauer teaches the display screen (“there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor.” [0005])
Sauer does not explicitly teach both optically passes through real world objects and displays the virtual model to the user. This is what White teaches (“The AR device 104 may include one or more screens, such as a single screen or two screens (e.g., one per eye of a user). The screens may allow light to pass through the screens such that aspects of the real environment are visible while displaying the virtual object 110. The virtual object 110 may be made visible to the surgeon 102 by projecting light.” [0023]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of White into Sauer, in order to facilitate identification and selection of the physical instrument and improve efficiency during the surgical procedure.
19. Claims 9 and 11 are rejected under 35 U.S.C. 103(a) as being unpatentable over Sauer et al. (US 2005/0203380 A1) and White et al. (US 2018/0082480 A1), as applied to claim 1 above, and further in view of Ryan et al. (US 2018/0049622 A1).
20. With reference to claim 9, Sauer teaches the XR headset controller being further configure to the shape of the virtual model of the selected implant based on a distance determined from the pose of the physical selected implant relative to the XR headset. (“there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor.” [0005] “FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20.” [0024] “the tracked instrument, such as the biopsy needle, includes a marker attachment containing set of retro-reflective markers. The size and shape of the markers are designed so that they are differentiable from the scanner table marker set within the tracking camera images.” [0029] “FIG. 3 depicts a monoscopic augmented view of a needle insertion guide, according to an embodiment of the invention. In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path, allowing the user to aim precisely at the target from a distance, before and during insertion of the needle. The large concentric ring 32 around the intersection of the extended needle path 33 with the target plane indicates the distance between needle tip 34 and the target plane, enhancing the user's ability to judge the depth correctly.” [0032]).
The combination of Sauer and White does not explicitly teach scale the model. This is what Ryan teaches (“FIG. 20 depicts an exemplary embodiment of a MXUI shown to the user 106 via the display device 104 of a virtual pelvis 2000 and a virtual femur 2002 during a hip replacement procedure. If patient-specific models had been uploaded into the display device 104 then virtual models of these would be displayed along with any other virtual features of interest such as neurovascular structures. If not, the virtual pelvis and virtual femur could be gender-specific models, which have been scaled to best match the spacing of the registered landmarks.” [0104] “the ultrasound probe 3104 is battery operated, cordless, and can communicate with the AR headset 3600 via radio. The software has geometric and other information necessary to be able to position and scale the 2D ultrasound image relative to the marker's 1300 position.” [0117]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Ryan into the combination of Sauer and White, in order to assist medical procedures.
21. With reference to claim 11, Sauer teaches the XR headset controller being further configure to generate the virtual model indicating characteristics of the selected implant, based on the predetermined information associated with the selected implant. (“there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor.” [0005] “FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20.” [0024] “To visualize a hand-held surgical instrument, such as a biopsy needle, inside the patient's body in the AR video view, the patients body pose needs to be tracked. A cluster of retro-reflective markers 14 can be attached to the biopsy needle from which the rigid body transformation between this cluster 14 and the tracker camera for each video frame can be estimated. The pose of this hand-held instrument with respect to the scanner table can then easily be deduced from the estimated head pose from the same video frame, a procedure referred to as instrument tracking. As shown in FIG. 2, the marker set 15 according to an embodiment of the invention can be attached to the MR scanner table and includes a bridge 16 which is double oblique cut of the surface of cylinder with the radius equal to that of MR bore. The markers 15 have a 3D distribution implemented using small posts 18 attached to the main bridge. The design provides a clear line sight for the tracking camera, while maximizing the workspace volume. Furthermore, the marker set is clear to go inside the MR bore, while the patient is on the scanner table for the scan.” [0027-0028] “In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path, allowing the user to aim precisely at the target from a distance, before and during insertion of the needle. The large concentric ring 32 around the intersection of the extended needle path 33 with the target plane indicates the distance between needle tip 34 and the target plane, enhancing the user's ability to judge the depth correctly.” [0032])
The combination of Sauer and White does not explicitly teach virtual text. This is what Ryan teaches (“FIG. 20 depicts an exemplary embodiment of a MXUI shown to the user 106 via the display device 104 of a virtual pelvis 2000 and a virtual femur 2002 during a hip replacement procedure. If patient-specific models had been uploaded into the display device 104 then virtual models of these would be displayed along with any other virtual features of interest such as neurovascular structures. If not, the virtual pelvis and virtual femur could be gender-specific models, which have been scaled to best match the spacing of the registered landmarks. A first virtual trajectory 2004 and a second virtual trajectory 2006 for each of two fixation pins are displayed. In other embodiments, these may be tube-shaped or cone shaped. A drill 2008 is shown which includes a plurality of fiducials 2010 defining markers on a plurality of surfaces, which allows its pose to be tracked from various vantage points. Insertion of each pin can be guided either by lining up an actual pin 2012 with the virtual trajectory 2004 in the case where the drill is not tracked or by lining up a virtual pin (not shown) with the virtual trajectory in the case where the drill is tracked. If the drill is tracked, the angle of the drill relative to the pelvic reference frame is displayed numerically for additional augmentation. Virtual text 2014 is located on a surface 2016 of the actual drill and moves with the drill making it intuitive to the user the object to which the angles represented by the virtual text are associated.” [0104]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Ryan into the combination of Sauer and White, in order to assist medical procedures.
22. Claims 13-17 and 19 are rejected under 35 U.S.C. 103(a) as being unpatentable over Sauer et al. (US 2005/0203380 A1) in view of White et al. (US 2018/0082480 A1) and Crawford et al. (US 2016/0242849 A1).
23. With reference to claim 13, Sauer teaches A method of providing a virtual model of a planned instrument attachment, a physical instrument attachment, , the method comprising: providing a surgical system, (“Exemplary embodiments of the invention as described herein generally include systems and methods for augmented reality navigation systems and setup for MR-guided interventions.” [0020] “To visualize a hand-held surgical instrument, such as a biopsy needle, inside the patient's body in the AR video view, the patients body pose needs to be tracked. A cluster of retro-reflective markers 14 can be attached to the biopsy needle from which the rigid body transformation between this cluster 14 and the tracker camera for each video frame can be estimated. The pose of this hand-held instrument with respect to the scanner table can then easily be deduced from the estimated head pose from the same video frame, a procedure referred to as instrument tracking.” [0027] “FIG. 3 depicts a monoscopic augmented view of a needle insertion guide, according to an embodiment of the invention. In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path,” [0032]) Sauer also teaches the surgical system including: a camera tracking system; an inserter instrument containing a plurality of first optical markers for tracking a pose of the inserter instrument by the camera tracking system; (“FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20. The monochrome tracking camera 11 works in conjunction with retro-reflective optical markers 13 that are placed around the intervention workspace. The flash 22 can be synchronized with the tracker camera and permits the use of a high speed electronic shutter.” [0024] “the tracked instrument, such as the biopsy needle, includes a marker attachment containing set of retro-reflective markers. The size and shape of the markers are designed so that they are differentiable from the scanner table marker set within the tracking camera images.” [0029] “During the intervention, the rigid body transformation between tracking camera and the scanning table is determined at step 54 from optical markers attached to the frame that is part of the table. The relationship between the tracking camera and table can then be used to determine the patient's body pose at step 55. In addition, at step 56, the pose of the instrument being used for the intervention procedure is determined with respect to the scanning table from optical markers attached to the instrument, and, at step 57, the respiration of the patient is monitored. This information regarding the patient's body pose, the instrument's pose, and patient's respiration is used, along with an MRI or other modality image to generate a visual representation of the patient, the intervention target, the instrument, and tracking path to guide the instrument in the stereoscopic guidance display of the HMD at step 58.” [0035]) Sauer further teaches an extended reality (XR) headset configured to be worn by a user during a surgical procedure and containing a plurality of second optical markers for tracking a pose of the XR headset by the camera tracking system, (“there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor.” [0005] “FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20. The monochrome tracking camera 11 works in conjunction with retro-reflective optical markers 13 that are placed around the intervention workspace. … a bridge 16 with retro-reflective markers 14 can be attached to the MR scanner table 17, thereby establishing a reference for the patient coordinate system. In an offline calibration step, the rigid body transformations between the tracker camera and the scene cameras can be determined. During the interventional procedure the rigid body transformation between the marker frame 16 and the tracker camera can be estimated. Thus, the poses of both scene cameras for each video frame with respect to the marker frame can be deduced, a procedure referred to as head tracking. … To visualize a hand-held surgical instrument, such as a biopsy needle, inside the patient's body in the AR video view, the patients body pose needs to be tracked. A cluster of retro-reflective markers 14 can be attached to the biopsy needle from which the rigid body transformation between this cluster 14 and the tracker camera for each video frame can be estimated. The pose of this hand-held instrument with respect to the scanner table can then easily be deduced from the estimated head pose from the same video frame, a procedure referred to as instrument tracking. As shown in FIG. 2, the marker set 15 according to an embodiment of the invention can be attached to the MR scanner table and includes a bridge 16 which is double oblique cut of the surface of cylinder with the radius equal to that of MR bore. The markers 15 have a 3D distribution implemented using small posts 18 attached to the main bridge. The design provides a clear line sight for the tracking camera, while maximizing the workspace volume. Furthermore, the marker set is clear to go inside the MR bore, while the patient is on the scanner table for the scan.” [0024-0028]) Sauer teaches a display screen configured to display a virtual model of a selected implant attached to an inserter instrument for viewing by the user, (“In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path, allowing the user to aim precisely at the target from a distance, before and during insertion of the needle. The large concentric ring 32 around the intersection of the extended needle path 33 with the target plane indicates the distance between needle tip 34 and the target plane, enhancing the user's ability to judge the depth correctly.” [0032]) Sauer also teaches an XR headset controller; determining a pose of an instrument relative to an XR headset; (“FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20.” [0024] “To visualize a hand-held surgical instrument, such as a biopsy needle, inside the patient's body in the AR video view, the patients body pose needs to be tracked. A cluster of retro-reflective markers 14 can be attached to the biopsy needle from which the rigid body transformation between this cluster 14 and the tracker camera for each video frame can be estimated. The pose of this hand-held instrument with respect to the scanner table can then easily be deduced from the estimated head pose from the same video frame, a procedure referred to as instrument tracking.” [0027] “In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path, allowing the user to aim precisely at the target from a distance, before and during insertion of the needle. The large concentric ring 32 around the intersection of the extended needle path 33 with the target plane indicates the distance between needle tip 34 and the target plane, enhancing the user's ability to judge the depth correctly.” [0032]) Sauer further teaches generating a shape and a pose of the virtual model of the planned instrument attachment, via the XR controller, based on predetermined information associated with the planned instrument attachment and based on the pose of the instrument relative to the XR headset; and displaying the virtual model on the display screen. (“there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor. … determining the pose of said medical instrument with respect to the table from the patient's body pose and said instrument markers, and displaying in the guidance display a visual representation of said patient, said instrument, and a path for guiding said instrument to perform said medical intervention” [0005] “FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20. The monochrome tracking camera 11 works in conjunction with retro-reflective optical markers 13 that are placed around the intervention workspace.” [0024] “The pose of this hand-held instrument with respect to the scanner table can then easily be deduced from the estimated head pose from the same video frame, a procedure referred to as instrument tracking.” [0027] “the tracked instrument, such as the biopsy needle, includes a marker attachment containing set of retro-reflective markers. The size and shape of the markers are designed so that they are differentiable from the scanner table marker set within the tracking camera images.” [0029] “the pose of the instrument being used for the intervention procedure is determined with respect to the scanning table from optical markers attached to the instrument, and, at step 57, the respiration of the patient is monitored. This information regarding the patient's body pose, the instrument's pose, and patient's respiration is used, along with an MRI or other modality image to generate a visual representation of the patient, the intervention target, the instrument, and tracking path to guide the instrument in the stereoscopic guidance display of the HMD at step 58.” [0035])
PNG
media_image1.png
591
500
media_image1.png
Greyscale
Sauer does not explicitly teach a virtual model of a planned instrument to ensure correct selection of a physical instrument; a first dynamic reference array (DRA), and a second DRA. This is what White teaches. White teaches a virtual model of a planned instrument to ensure correct selection of a physical instrument (“The surgical assistant 114 may be presented with a virtual instrument through the AR display 116. The surgical assistant 114 may select a surgical instrument corresponding to the virtual instrument, which may be detected by a detection device, such as an AR camera 118. After detecting that the surgical instrument has been identified, selected, or moved by the surgical assistant 114, the AR display 116 may cease display of the virtual instrument.” [0027]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of White into Sauer, in order to facilitate identification and selection of the physical instrument and improve efficiency during the surgical procedure.
The combination of Sauer and White does not explicitly teach a first dynamic reference array (DRA), and a second DRA. This is what Crawford teaches (“FIG. 36 illustrates an example embodiment 3600 of surgical robot system 1 that utilizes a surveillance marker 710 in accordance with one or more aspects of the invention. As illustrated, the example embodiment 3600 comprises a 4-marker tracker array 3610 attached to the patient 18 and having a surveillance marker, and a 4-marker tracker array 3620 on the robot 15. … the vector (3D) distances between the surveillance marker 710, and each of the markers 3611, 3612, 3613, and 3614 on the primary tracker array 3610 can be acquired and retained in computer 100 memory (such as a memory of a computing device 3401 executing the control software application). In an embodiment in which a 4-marker tracker array 3610 is utilized (FIG. 36), four distances 3611a, 3612a, 3613a, and 3614a can be acquired and retained, representing the distances between the surveillance marker 710 and markers 3611, 3612, 3613, and 3614.” [0354-0355]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Crawford into the combination of Sauer and White, in order to arrive at an accurately targeted location.
24. With reference to claim 14, Sauer teaches the XR headset controller is further configured to display the virtual model over where the selected implant would have been attached to the inserter instrument for comparison between the actual selected implant and the virtual model by the user. (“there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor.” [0005] “The camera triplet (i.e. a stereo scene camera 12 and tracking camera 11) is rigidly mounted on top of the HMD 10. Both scene cameras have fixed focus lenses and provide sharp images in about arm's length distance. The monochrome tracking camera 11, which is only sensitive in the near-infrared spectrum, is equipped with a wide-angle lens 21 and a ring shaped infrared flash 22. The user, who is wearing the HMD 10, sees in guidance display viewer 19 a stereo view of a patient overlaid with diagnostic information as provided by an MRI or other imaging modality, and with guiding graphics. … FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20.” [0023-0024] “FIG. 3 depicts a monoscopic augmented view of a needle insertion guide, according to an embodiment of the invention. In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path, allowing the user to aim precisely at the target from a distance, before and during insertion of the needle. The large concentric ring 32 around the intersection of the extended needle path 33 with the target plane indicates the distance between needle tip 34 and the target plane, enhancing the user's ability to judge the depth correctly.” [0032] “With a patient positioned on the scanning table, the intervention procedure is started at step 53. During the intervention, the rigid body transformation between tracking camera and the scanning table is determined at step 54 from optical markers attached to the frame that is part of the table. The relationship between the tracking camera and table can then be used to determine the patient's body pose at step 55. In addition, at step 56, the pose of the instrument being used for the intervention procedure is determined with respect to the scanning table from optical markers attached to the instrument, and, at step 57, the respiration of the patient is monitored. This information regarding the patient's body pose, the instrument's pose, and patient's respiration is used, along with an MRI or other modality image to generate a visual representation of the patient, the intervention target, the instrument, and tracking path to guide the instrument in the stereoscopic guidance display of the HMD at step 58. … The composite stereoscopic images (i.e., video plus the graphics from the image data) according to an embodiment of the invention provide an intuitive guidance to the interventional radiologist and/or surgeon to perform the procedure more accurately, with more confidence, and in less time. In addition, video-see-through AR allows greater control over the augmented video stream. An HMD gives the user the freedom to move around in a natural way and to inspect the scene from a variety of viewpoints” [0035-0036])
25. With reference to claim 15, Sauer teaches the XR headset controller generates the virtual model of the selected implant with a partial transparency such that the physical selected implant is visible through the virtual model 1830 on the display screen. (“An augmented reality (AR) image guidance system can map medical imaging information onto a patient's body. Anatomical structures can be visually displayed at the location where they actually are, a concept referred to as in-situ visualization. With this visual display, a physician can observe directly a patient's internal anatomy. The patient's body appears transparent. In-situ visualization is a direct way of presenting medical imaging information, and has the potential to guide minimally invasive procedures very intuitively.” [0003] “there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor.” [0005] “FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20.” [0024] “FIG. 3 depicts a monoscopic augmented view of a needle insertion guide, according to an embodiment of the invention. In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path, allowing the user to aim precisely at the target from a distance, before and during insertion of the needle. The large concentric ring 32 around the intersection of the extended needle path 33 with the target plane indicates the distance between needle tip 34 and the target plane, enhancing the user's ability to judge the depth correctly.” [0032] “the pose of the instrument being used for the intervention procedure is determined with respect to the scanning table from optical markers attached to the instrument, and, at step 57, the respiration of the patient is monitored. This information regarding the patient's body pose, the instrument's pose, and patient's respiration is used, along with an MRI or other modality image to generate a visual representation of the patient, the intervention target, the instrument, and tracking path to guide the instrument in the stereoscopic guidance display of the HMD at step 58.” [0035])
26. With reference to claim 16, Sauer teaches the XR headset controller is further configured to generate the virtual model of the selected implant to have partially transparent graphical features that at least one of which is posed to substantially align with at least one corresponding feature of the physical selected implant when the virtual model of the selected implant is overlaid on the physical selected implant while viewed through the see-through display. (“An augmented reality (AR) image guidance system can map medical imaging information onto a patient's body. Anatomical structures can be visually displayed at the location where they actually are, a concept referred to as in-situ visualization. With this visual display, a physician can observe directly a patient's internal anatomy. The patient's body appears transparent. In-situ visualization is a direct way of presenting medical imaging information, and has the potential to guide minimally invasive procedures very intuitively.” [0003] “there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor.” [0005] “The camera triplet (i.e. a stereo scene camera 12 and tracking camera 11) is rigidly mounted on top of the HMD 10. Both scene cameras have fixed focus lenses and provide sharp images in about arm's length distance. The monochrome tracking camera 11, which is only sensitive in the near-infrared spectrum, is equipped with a wide-angle lens 21 and a ring shaped infrared flash 22. The user, who is wearing the HMD 10, sees in guidance display viewer 19 a stereo view of a patient overlaid with diagnostic information as provided by an MRI or other imaging modality, and with guiding graphics.” [0023] “FIG. 3 depicts a monoscopic augmented view of a needle insertion guide, according to an embodiment of the invention. In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path, allowing the user to aim precisely at the target from a distance, before and during insertion of the needle. The large concentric ring 32 around the intersection of the extended needle path 33 with the target plane indicates the distance between needle tip 34 and the target plane, enhancing the user's ability to judge the depth correctly.” [0032] “the pose of the instrument being used for the intervention procedure is determined with respect to the scanning table from optical markers attached to the instrument, and, at step 57, the respiration of the patient is monitored. This information regarding the patient's body pose, the instrument's pose, and patient's respiration is used, along with an MRI or other modality image to generate a visual representation of the patient, the intervention target, the instrument, and tracking path to guide the instrument in the stereoscopic guidance display of the HMD at step 58.” [0035])
27. With reference to claim 17, Sauer teaches the XR headset controller being further configured to selectively determine the pose of the virtual model of the selected implant to be displayed as one of an overlay on the physical selected implant. (“there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor.” [0005] “FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20.” [0024] “FIG. 3 depicts a monoscopic augmented view of a needle insertion guide, according to an embodiment of the invention. In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path, allowing the user to aim precisely at the target from a distance, before and during insertion of the needle. The large concentric ring 32 around the intersection of the extended needle path 33 with the target plane indicates the distance between needle tip 34 and the target plane, enhancing the user's ability to judge the depth correctly.” [0032] “With a patient positioned on the scanning table, the intervention procedure is started at step 53. During the intervention, the rigid body transformation between tracking camera and the scanning table is determined at step 54 from optical markers attached to the frame that is part of the table. The relationship between the tracking camera and table can then be used to determine the patient's body pose at step 55. In addition, at step 56, the pose of the instrument being used for the intervention procedure is determined with respect to the scanning table from optical markers attached to the instrument, and, at step 57, the respiration of the patient is monitored. This information regarding the patient's body pose, the instrument's pose, and patient's respiration is used, along with an MRI or other modality image to generate a visual representation of the patient, the intervention target, the instrument, and tracking path to guide the instrument in the stereoscopic guidance display of the HMD at step 58. … The composite stereoscopic images (i.e., video plus the graphics from the image data) according to an embodiment of the invention provide an intuitive guidance to the interventional radiologist and/or surgeon to perform the procedure more accurately, with more confidence, and in less time. In addition, video-see-through AR allows greater control over the augmented video stream. An HMD gives the user the freedom to move around in a natural way and to inspect the scene from a variety of viewpoints” [0035-0036])
28. With reference to claim 19, Sauer teaches the XR headset controller being further configured to generate the virtual model to have graphical features representing an instrument extender. (“there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor.” [0005] “FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20.” [0024] “FIG. 3 depicts a monoscopic augmented view of a needle insertion guide, according to an embodiment of the invention. In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path, allowing the user to aim precisely at the target from a distance, before and during insertion of the needle. The large concentric ring 32 around the intersection of the extended needle path 33 with the target plane indicates the distance between needle tip 34 and the target plane, enhancing the user's ability to judge the depth correctly.” [0032])
29. Claims 18 and 20 are rejected under 35 U.S.C. 103(a) as being unpatentable over Sauer et al. (US 2005/0203380 A1), White et al. (US 2018/0082480 A1) and Crawford et al. (US 2016/0242849 A1), as applied to claims 13 and 14 above, and further in view of Ryan et al. (US 2018/0049622 A1).
30. With reference to claim 18, Sauer teaches the XR headset controller being further configure to the shape of the virtual model of the selected implant based on a distance determined from the pose of the physical selected implant relative to the XR headset. (“there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor.” [0005] “FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20.” [0024] “the tracked instrument, such as the biopsy needle, includes a marker attachment containing set of retro-reflective markers. The size and shape of the markers are designed so that they are differentiable from the scanner table marker set within the tracking camera images.” [0029] “FIG. 3 depicts a monoscopic augmented view of a needle insertion guide, according to an embodiment of the invention. In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path, allowing the user to aim precisely at the target from a distance, before and during insertion of the needle. The large concentric ring 32 around the intersection of the extended needle path 33 with the target plane indicates the distance between needle tip 34 and the target plane, enhancing the user's ability to judge the depth correctly.” [0032]).
The combination of Sauer, White and Crawford does not explicitly teach scale the model. This is what Ryan teaches (“FIG. 20 depicts an exemplary embodiment of a MXUI shown to the user 106 via the display device 104 of a virtual pelvis 2000 and a virtual femur 2002 during a hip replacement procedure. If patient-specific models had been uploaded into the display device 104 then virtual models of these would be displayed along with any other virtual features of interest such as neurovascular structures. If not, the virtual pelvis and virtual femur could be gender-specific models, which have been scaled to best match the spacing of the registered landmarks.” [0104] “the ultrasound probe 3104 is battery operated, cordless, and can communicate with the AR headset 3600 via radio. The software has geometric and other information necessary to be able to position and scale the 2D ultrasound image relative to the marker's 1300 position.” [0117]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Ryan into the combination of Sauer, White and Crawford, in order to assist medical procedures.
31. With reference to claim 20, Sauer teaches the XR headset controller being further configure to generate the virtual model indicating characteristics of the selected implant, based on the predetermined information associated with the selected implant. (“there is provided a system for augmented reality navigation during a medical intervention, including a stereoscopic head mounted display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance monitor, a plurality of markers on a frame attached to a table, a medical instrument for performing said medical intervention, said instrument including a plurality of markers, and a control component, said control component accepting signals from said viewing cameras and tracking camera to generate an image for display in said stereoscopic guidance monitor.” [0005] “FIG. 2 depicts a video-see-through augmented reality system setup according to an embodiment of the invention, including a stereo head mounted display (HMD) 10 with tracking 11 and scene cameras 12 and a marker set 13 surrounding an interventional workspace and a calibration phantom 20.” [0024] “To visualize a hand-held surgical instrument, such as a biopsy needle, inside the patient's body in the AR video view, the patients body pose needs to be tracked. A cluster of retro-reflective markers 14 can be attached to the biopsy needle from which the rigid body transformation between this cluster 14 and the tracker camera for each video frame can be estimated. The pose of this hand-held instrument with respect to the scanner table can then easily be deduced from the estimated head pose from the same video frame, a procedure referred to as instrument tracking. As shown in FIG. 2, the marker set 15 according to an embodiment of the invention can be attached to the MR scanner table and includes a bridge 16 which is double oblique cut of the surface of cylinder with the radius equal to that of MR bore. The markers 15 have a 3D distribution implemented using small posts 18 attached to the main bridge. The design provides a clear line sight for the tracking camera, while maximizing the workspace volume. Furthermore, the marker set is clear to go inside the MR bore, while the patient is on the scanner table for the scan.” [0027-0028] “In this picture, a needle is being inserted into a gel phantom that contains a set of six targets 30. A slice of an MR image is overlaid onto the video view, while both the location of six targets and the extended path of the needle are modeled and overlaid graphically onto the video image. The user would actually see a stereo image in the head-mounted display for good spatial perception. Target 31 is the currently selected target. The shaded cylinder 35 indicates the inserted portion of the actual needle, the sphere 34 indicates the location of the actual needle tip, and the cylinder 33 is an extrapolation of the needle along a straight path, allowing the user to aim precisely at the target from a distance, before and during insertion of the needle. The large concentric ring 32 around the intersection of the extended needle path 33 with the target plane indicates the distance between needle tip 34 and the target plane, enhancing the user's ability to judge the depth correctly.” [0032])
The combination of Sauer, White and Crawford does not explicitly teach virtual text. This is what Ryan teaches (“FIG. 20 depicts an exemplary embodiment of a MXUI shown to the user 106 via the display device 104 of a virtual pelvis 2000 and a virtual femur 2002 during a hip replacement procedure. If patient-specific models had been uploaded into the display device 104 then virtual models of these would be displayed along with any other virtual features of interest such as neurovascular structures. If not, the virtual pelvis and virtual femur could be gender-specific models, which have been scaled to best match the spacing of the registered landmarks. A first virtual trajectory 2004 and a second virtual trajectory 2006 for each of two fixation pins are displayed. In other embodiments, these may be tube-shaped or cone shaped. A drill 2008 is shown which includes a plurality of fiducials 2010 defining markers on a plurality of surfaces, which allows its pose to be tracked from various vantage points. Insertion of each pin can be guided either by lining up an actual pin 2012 with the virtual trajectory 2004 in the case where the drill is not tracked or by lining up a virtual pin (not shown) with the virtual trajectory in the case where the drill is tracked. If the drill is tracked, the angle of the drill relative to the pelvic reference frame is displayed numerically for additional augmentation. Virtual text 2014 is located on a surface 2016 of the actual drill and moves with the drill making it intuitive to the user the object to which the angles represented by the virtual text are associated.” [0104]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Ryan into the combination of Sauer, White and Crawford, in order to assist medical procedures.
Conclusion
32. Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michelle Chin whose telephone number is (571)270-3697. The examiner can normally be reached on Monday-Friday 8:00 AM-4:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http:/Awww.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Kent Chang can be reached on (571)272-7667. The fax phone number for the organization where this application or proceeding is assigned is (571)273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https:/Awww.uspto.gov/patents/apply/patent- center for more information about Patent Center and https:/Awww.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE CHIN/
Primary Examiner, Art Unit 2614
Prosecution Insights