DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
The double patenting rejections over U.S. Patent No. 11766296, U.S. Patent No. 11980429 B2, and U.S. Patent No. 12201384 B2, have been withdrawn in view of the terminal disclaimers filed 09/21/2026.
Applicant’s arguments with respect to the rejection of claims 41-60 under 35 U.S.C. 103, in Applicant’s responses filed 09/21/2026 have been fully considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Specifically, newly found prior art, Ryan, et al., US 20180049622 A1 has been applied in the stead of Siemionow et al., US 20190053851 A1 (disqualified as prior art). See pages 7-8 of Applicant’s remarks filed 09/21/2026.
Therefore, the claims stand rejected.
Terminal Disclaimer
The terminal disclaimers filed on 09/21/2021 disclaiming the terminal portion of any patent granted on this application which would extend beyond the expiration date of U.S. Pat. No. 11,766296, U.S. Pat. No. 11,980429 B2, and U.S. Pat. No. 12,201384 B2 has been reviewed and is accepted. The terminal disclaimer has been recorded.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 41-60 are rejected under 35 U.S.C. 103 as being unpatentable over Lang, P. K., US 20170258526 A1 in view of Ryan, et al., US 20180049622 A1.
Regarding claim 41, Lang teaches a method for use with a tool configured to be placed within a portion of a body of a patient ([0004] states that “Aspects of the invention provides, among other things, for a simultaneous visualization of live data of the patient, e.g. a patient's spine or joint, and digital representations of virtual data such as virtual cuts and/or virtual surgical guides including cut blocks or drilling guides through an optical head mounted display (OHMD). In some embodiments, the surgical site including live data of the patient, the OHMD, and the virtual data are registered in a common coordinate system. In some embodiments, the virtual data are superimposed onto and aligned with the live data of the patient.”),
the method comprising:
tracking, using a first tracking device (optical head mounted display (OHMD) 1 of [0147]) from a first line of sight ([0119] describes generation of the projection information onto the OHMD 1 along a viewpoint and view direction), at least a portion of the tool ([0071] states “With surgical navigation, a first virtual instrument can be displayed on a computer monitor which is a representation of a physical instrument tracked with navigation markers, e.g. infrared or RF markers”) and a patient marker that is configured to be placed upon the body of the patient ([0032] states that “the one or more intraoperative measurements include detecting one or more optical markers attached to the patient's joint, the operating room table, fixed structures in the operating room or combinations thereof”),
wherein the first tracking device is disposed upon a first head-mounted device that is configured to be worn by a first person, the first head-mounted device including a first head-mounted display (OHMD 1 is worn by a surgeon, surgical assistant and/or nurses according to [0147]);
tracking, using a second tracking device (OHMD 2 of [0147]) from a second line of sight ([0119] describes generation of the projection information onto the OHMD 2 along a viewpoint and view direction), at least the portion of the tool ([0071] states “With surgical navigation, a first virtual instrument can be displayed on a computer monitor which is a representation of a physical instrument tracked with navigation markers, e.g. infrared or RF markers”) and the patient marker ([0032] states that “the one or more intraoperative measurements include detecting one or more optical markers attached to the patient's joint, the operating room table, fixed structures in the operating room or combinations thereof”), wherein the second tracking device is disposed upon a second head-mounted device that is configured to be worn by a second person (OHMD 2 is worn by a surgeon, surgical assistant and/or nurses other than the primary wearer of OHMD 1 according to [0147]); and
using at least one computer processor (processors of [0103] and computer graphics system of [0116],[0117]),
tracking the location and the orientation of the first head-mounted device with respect to the patient marker using data from the first tracking device ([0228] describes how optical, radiofrequency (RF) trackers, or other tracking devices known in the art can be applied to the OHMDs and the patient for tracking the movement, pose, location and orientation of both the OHMDs and the patient during the surgical procedure. This improves the accuracy of moving or re-orienting virtual data according to [0226]-[0227]);
generating an augmented reality image upon the first head-mounted display ([0147] discloses generating shared digital holographic experience of the surgical scene on the OHMDs), based upon the tracking of the location and the orientation of the first head-mounted device with respect to the patient marker ([0227]-[0228] describe that the virtual data is reoriented to follow changes in the surgical scene including movements of the surgeon and the patient) in combination with data received from the second tracking device ([0116]-[0117] describe registering different objects to be displayed (display information) their individual coordinates to a common global coordinate system and [0119] describes projecting the display information onto the display plane using the different viewpoints and view directions and updating the projections with updated viewpoints and view directions. Meaning that the shared holographic experience 30 of [0147] includes a combination of the viewpoints and view directions of the OHMD 1 and OHMD 2. Of note, [0147] discloses additional OHMDs), the augmented reality image including (a) a virtual image of the tool and anatomy of the patient, overlaid upon (b) the body of the patient ([0004] states that “the virtual data are superimposed onto and aligned with the live data of the patient. Unlike virtual reality head systems that blend out live data, the OHMD allows the surgeon to see the live data of the patient, e.g. the surgical field, while at the same time observing virtual data of the patient and/or virtual surgical instruments or implants with a predetermined position and/or orientation using the display of the OHMD unit”).
Lang fails to teach wherein the patient marker comprises an array of elements configured to be visible to the first tracking device to enable determination of a location and an orientation of the first head-mounted device relative to the patient marker.
However, within the same field of endeavor, Ryan teaches a mixed reality surgical navigation system (10) comprising: a display device (104) comprising a processor unit (102), a display generator (204), a sensor suite (210) having at least one camera (206); and at least one marker (600) fixedly attached to a surgical tool (608); wherein the system (10) maps three-dimensional surfaces of partially exposed surfaces of an anatomical object of interest (604); tracks a six-degree of freedom post of the surgical tool (608), and provides a mixed reality user interface comprising stereoscopic virtual images of desired features of the surgical tool (608) and desired features of the anatomical object (604) in the user's (106) field of view (see abstract) wherein the patient marker comprises an array of elements configured to be visible to the first tracking device to enable determination of a location and an orientation of the first head-mounted device relative to the patient marker ([0103] FIG. 19 depicts an exemplary embodiment of a MXUI shown to the user 106 via the display device 104 of a patient 1900 at the beginning of a hip replacement procedure. A femur marker 1902, having a plurality of fiducials 1904 for tracking, is attached to the skin of the patient's 1900 thigh with adhesive tape such as Ioban. Alternatively, the femur marker 1902 could be fixated directly to the bone of the femur by use of pins and a clamp assembly like that depicted in FIG. 13B. The user 106 registers the anterior landmarks of the pelvis using the tip 1804 of the stylus 1800 to determine the location of the pelvis in the reference frame of the femur marker 1902 to establish a temporary pelvic reference frame. In another embodiment, this registration can be in the body reference frame defined by SLAM scanning of the visible surface of the patient. In another embodiment, the anterior landmarks of the pelvis can be registered by generating a surface map with SLAM and having the user 106 identify each point by positioning a virtual point 1910 on each landmark in turn by motion of his head. In another embodiment, a single fiduciary 1906 can be placed at the location to be registered. A virtual circle 1908 can be used to define a mask whose position is controlled by the gaze of the user 106. The machine vision algorithm only looks for a single fiduciary 1906 within the virtual circle 1908. Registration steps may be triggered with a voice command by the user 106 such as “register point”. The user 106 may also register a point representing the distal femur such as the center of the patella or the medial and lateral epicondyles. When each point is registered, a virtual marker, such as a small sphere, may be positioned and remain at the location of the tip at the time of registration and beyond to provide the user 106 a visual confirmation to the user 106 and check on the quality of the registration).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to configure Lang, wherein the patient marker comprises an array of elements configured to be visible to the first tracking device to enable determination of a location and an orientation of the first head-mounted device relative to the patient marker, as taught by Ryan, as such modification would provide a more accurate way to register the anatomy during surgery [0004] and positions of objects within the surgical scene [0080].
Regarding claim 42, Lang in view of Ryan teaches all the limitations of claim 41 above.
Lang further teaches wherein the tracking, using the first tracking device from the first line of sight, at least the portion of the tool comprises tracking a tool marker ([0071] states that “With surgical navigation, a first virtual instrument can be displayed on a computer monitor which is a representation of a physical instrument tracked with navigation markers, e.g. infrared or RF markers”).
Regarding claim 43, Lang in view of Ryan teaches all the limitations of claim 42 above.
Lang further teaches wherein the tracking, using the second tracking device from the second line of sight, at least the portion of the tool comprises tracking the tool marker ([0071] states that “With surgical navigation, a first virtual instrument can be displayed on a computer monitor which is a representation of a physical instrument tracked with navigation markers, e.g. infrared or RF markers”).
Regarding claim 44, Lang in view of Ryan teaches all the limitations of claim 41 above.
Lang further teaches wherein the second head-mounted device includes a second head-mounted display (OHMD 2 of [0147]), and the method further comprises generating a further augmented reality image upon the second head-mounted display ([0147] states that “The OHMD's 11, 12, 13, 14 can project digital holograms of the virtual data or virtual data into the view of the left eye using the view position and orientation of the left eye 26 and can project digital holograms of the virtual data or virtual data into the view of the right eye using the view position and orientation of the right eye 28 of each user, resulting in a shared digital holographic experience 30”).
Regarding claim 45, Lang in view of Ryan teaches all the limitations of claim 41 above.
Lang wherein the first tracking device comprises a first camera ([0103], [0104] disclose that the exemplary OHMD (Microsoft Hololens includes several cameras (two on each side)) configured to image at least the portion of the tool and the patient marker ([0032] states that “one or more cameras or image capture or video capture systems included in the optical head mounted display detect one or more optical markers including their coordinates (x,y,z) and at least one or more of a position, orientation, alignment, direction of movement or speed of movement of the one or more optical markers”).
Regarding claim 46, Lang in view of Ryan teaches all the limitations of claim 45 above.
Lang further teaches wherein the second tracking device camera ([0103], [0104] disclose that the exemplary OHMD (Microsoft Hololens includes several cameras (two on each side)) comprises a second camera configured to image at least the portion of the tool and the patient marker ([0032] states that “one or more cameras or image capture or video capture systems included in the optical head mounted display detect one or more optical markers including their coordinates (x,y,z) and at least one or more of a position, orientation, alignment, direction of movement or speed of movement of the one or more optical markers”).
Regarding claim 47, Lang in view of Ryan teaches all the limitations of claim 41 above.
Lang further teaches wherein generating the augmented reality image upon the first head-mounted display comprises:
determining a position of the tool with respect to the anatomy of the patient using data received from the first tracking device in combination with data received from the second tracking device ([0147], [0253] disclose a spatial mapping registration process, stating that “Live data, e.g. live data of the patient, the position and/or orientation of a physical instrument, the position and/or orientation of an implant component, the position and/or orientation of one or more OHMD's, can be acquired or registered”) ;
generating the virtual image of the tool and the anatomy of the patient upon the first head-mounted display, based upon the position of the tool with respect to the anatomy of the patient ([0253] then states “This process creates a three-dimensional mesh describing the surfaces of one or more objects or environmental structures using, for example and without limitation, a depth sensor, laser scanner, structured light sensor, time of flight sensor, infrared sensor, or tracked probe. These devices can generate 3D surface data by collecting, for example, 3D coordinate information or information on the distance from the sensor of one or more surface points on the one or more objects or environmental structures. The 3D surface points can then be connected to 3D surface meshes, resulting in a three-dimensional surface representation of the live data. The surface mesh can then be merged with the virtual data using any of the registration techniques described in the specification”);
determining a position of the body of the patient with respect to the first head-mounted device based upon data received from the first tracking device ([0146] states that “Once the virtual data and the live data of the patient and the OHMD are registered in the same coordinate system, e.g. using IMUs, optical markers, navigation markers including infrared markers, retroreflective markers, RF markers, and any other registration method described in the specification or known in the art, any change in position of any of the OHMD in relationship to the patient measured in this fashion can be used to move virtual data of the patient in relationship to live data of the patient, so that the visual image of the virtual data of the patient and the live data of the patient seen through the OHDM are always aligned, irrespective of movement of the OHMD and/or the operator's head and/or the operator wearing the OHMD”); and
overlaying the virtual image upon the body of the patient, based upon the position of the body of the patient with respect to the first head-mounted device ([0146] further states “The position, orientation, alignment, and change in position, orientation and alignment in relationship to the patient and/or the surgical site of each additional OHMD can be individually monitored thereby maintaining alignment and/or superimposition of corresponding structures in the live data of the patient and the virtual data of the patient for each additional OHMD irrespective of their position, orientation, and/or alignment in relationship to the patient and/or the surgical site”).
Regarding claim 48, Lang in view of Ryan teaches all the limitations of claim 47 above.
Lang further teaches wherein determining the position of the tool with respect to the patient's anatomy using data received from the first tracking device in combination with data received from the second tracking device comprises determining an average position of the tool with respect to the patient's anatomy ([1184] states that “the information from two or more cameras can be merged by averaging the 3D coordinates or detected surface points or other geometric structures such as planes or curved surfaces”).
Regarding claim 49, Lang in view of Ryan teaches all the limitations of claim 47 above.
Lang further teaches wherein determining the position of the body of the patient with respect to the first head-mounted device based upon data received from the first tracking device comprises analyzing one or more images of the patient marker created by a camera of the first tracking device ([0253] states that “Live data, e.g. live data of the patient, the position and/or orientation of a physical instrument, the position and/or orientation of an implant component, the position and/or orientation of one or more OHMD's, can be acquired or registered, for example, using a spatial mapping process. This process creates a three-dimensional mesh describing the surfaces of one or more objects or environmental structures using, for example and without limitation, a depth sensor, laser scanner, structured light sensor, time of flight sensor, infrared sensor, or tracked probe”, and [0270] states that “Image processing and/or pattern recognition of the live data of the patient can then be performed through the OHMD, e.g. using a built in image capture apparatus for capturing the live data of the patient or image and/or video capture systems attached to, integrated with or coupled to the OHMD”).
Regarding claim 50, Lang in view of Ryan teaches all the limitations of claim 49 above.
Lang further teaches wherein the one or more images of the patient marker comprise data that is sufficient to determine the position of the body of the patient with respect to the first head-mounted device without requiring use of triangulation techniques ([0270] describes image processing of the live data of the patient to detect the patient’s position and orientation ([0253]) does not involve triangulation).
Regarding claim 51, Lang teaches a system for use with a tool configured to be placed within a portion of a body of a patient ([0147] states that “Referring to FIG. 1, a system 10 for using multiple OHMD's 11, 12, 13, 14 for multiple viewer's, e.g. a primary surgeon, second surgeon, surgical assistant(s) and/or nurses(s) is shown”), the system comprising:
a first head-mounted device configured to be worn by a first person, the first head- mounted device comprising a first head-mounted display (optical head mounted display (OHMD) 1 worn by a primary surgeon, second surgeon, surgical assistant(s) and/or nurses(s) of [0147]), and a first tracking device ([0103], [0104] disclose that the exemplary OHMD (Microsoft Hololens includes several cameras (two on each side)) that is configured to track at least a portion of the tool ([0071] states “With surgical navigation, a first virtual instrument can be displayed on a computer monitor which is a representation of a physical instrument tracked with navigation markers, e.g. infrared or RF markers”) and the patient marker ([0032] states that “the one or more intraoperative measurements include detecting one or more optical markers attached to the patient's joint, the operating room table, fixed structures in the operating room or combinations thereof”) from a first line of sight ([0119] describes generation of the projection information onto the OHMD 1 along a viewpoint and view direction);
the patient marker configured to be placed upon the body of the patient ([0032] states that “the one or more intraoperative measurements include detecting one or more optical markers attached to the patient's joint, the operating room table, fixed structures in the operating room or combinations thereof”);
a second head-mounted device configured to be worn by a second person (OHMD 2 worn by a primary surgeon, second surgeon, surgical assistant(s) and/or nurses(s) of [0147]) , the second head-mounted device comprising a second tracking device ([0103], [0104] disclose that the exemplary OHMD (Microsoft Hololens includes several cameras (two on each side)) that is configured to track at least the portion of the tool ([0071] states “With surgical navigation, a first virtual instrument can be displayed on a computer monitor which is a representation of a physical instrument tracked with navigation markers, e.g. infrared or RF markers”) and the patient marker ([0032] states that “the one or more intraoperative measurements include detecting one or more optical markers attached to the patient's joint, the operating room table, fixed structures in the operating room or combinations thereof”) from a second line of sight ([0119] describes generation of the projection information onto the OHMD 1 along a viewpoint and view direction); and
at least one computer processor (processors of [0103] and computer graphics system of [0116],[0117]) configured to: track the location and the orientation of the first head-mounted device with respect to the patient marker based upon data received from the first tracking device ([0228] describes how optical, radiofrequency (RF) trackers, or other tracking devices known in the art can be applied to the OHMDs and the patient for tracking the movement, pose, location and orientation of both the OHMDs and the patient during the surgical procedure. This improves the accuracy of moving or re-orienting virtual data according to [0226]-[0227]); and
generate an augmented reality image upon the first head-mounted display ([0147] discloses generating shared digital holographic experience of the surgical scene on the OHMDs), based upon the tracking of the location and the orientation of the first head-mounted device with respect to the patient marker ([0227]-[0228] describe that the virtual data is reoriented to follow changes in the surgical scene including movements of the surgeon and the patient) in combination with data received from the second tracking device ([0116]-[0117] describe registering different objects to be displayed (display information) their individual coordinates to a common global coordinate system and [0119] describes projecting the display information onto the display plane using the different viewpoints and view directions and updating the projections with updated viewpoints and view directions. Meaning that the shared holographic experience 30 of [0147] includes a combination of the viewpoints and view directions of the OHMD 1 and OHMD 2. Of note, [0147] discloses additional OHMDs), the augmented reality image including (a) a virtual image of the tool and anatomy of the patient, overlaid upon (b) the body of the patient ([0004] states that “the virtual data are superimposed onto and aligned with the live data of the patient. Unlike virtual reality head systems that blend out live data, the OHMD allows the surgeon to see the live data of the patient, e.g. the surgical field, while at the same time observing virtual data of the patient and/or virtual surgical instruments or implants with a predetermined position and/or orientation using the display of the OHMD unit”).
Lang fails to teach wherein the patient marker comprises an array of elements configured to be visible to the first tracking device to enable determination of a location and an orientation of the first head-mounted device relative to the patient marker.
However, within the same field of endeavor, Ryan teaches a mixed reality surgical navigation system (10) comprising: a display device (104) comprising a processor unit (102), a display generator (204), a sensor suite (210) having at least one camera (206); and at least one marker (600) fixedly attached to a surgical tool (608); wherein the system (10) maps three-dimensional surfaces of partially exposed surfaces of an anatomical object of interest (604); tracks a six-degree of freedom post of the surgical tool (608), and provides a mixed reality user interface comprising stereoscopic virtual images of desired features of the surgical tool (608) and desired features of the anatomical object (604) in the user's (106) field of view (see abstract) wherein the patient marker comprises an array of elements configured to be visible to the first tracking device to enable determination of a location and an orientation of the first head-mounted device relative to the patient marker ([0103] FIG. 19 depicts an exemplary embodiment of a MXUI shown to the user 106 via the display device 104 of a patient 1900 at the beginning of a hip replacement procedure. A femur marker 1902, having a plurality of fiducials 1904 for tracking, is attached to the skin of the patient's 1900 thigh with adhesive tape such as Ioban. Alternatively, the femur marker 1902 could be fixated directly to the bone of the femur by use of pins and a clamp assembly like that depicted in FIG. 13B. The user 106 registers the anterior landmarks of the pelvis using the tip 1804 of the stylus 1800 to determine the location of the pelvis in the reference frame of the femur marker 1902 to establish a temporary pelvic reference frame. In another embodiment, this registration can be in the body reference frame defined by SLAM scanning of the visible surface of the patient. In another embodiment, the anterior landmarks of the pelvis can be registered by generating a surface map with SLAM and having the user 106 identify each point by positioning a virtual point 1910 on each landmark in turn by motion of his head. In another embodiment, a single fiduciary 1906 can be placed at the location to be registered. A virtual circle 1908 can be used to define a mask whose position is controlled by the gaze of the user 106. The machine vision algorithm only looks for a single fiduciary 1906 within the virtual circle 1908. Registration steps may be triggered with a voice command by the user 106 such as “register point”. The user 106 may also register a point representing the distal femur such as the center of the patella or the medial and lateral epicondyles. When each point is registered, a virtual marker, such as a small sphere, may be positioned and remain at the location of the tip at the time of registration and beyond to provide the user 106 a visual confirmation to the user 106 and check on the quality of the registration).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to configure Lang, wherein the patient marker comprises an array of elements configured to be visible to the first tracking device to enable determination of a location and an orientation of the first head-mounted device relative to the patient marker, as taught by Ryan, as such modification would provide a more accurate way to register the anatomy during surgery [0004] and positions of objects within the surgical scene [0080].
Regarding claim 52, Lang in view of Ryan teaches all the limitations of claim 51 above.
Lang further teaches wherein the tool includes a tool marker, and wherein the first tracking device is configured to track the portion of the tool by tracking the tool marker ([0071] states that “With surgical navigation, a first virtual instrument can be displayed on a computer monitor which is a representation of a physical instrument tracked with navigation markers, e.g. infrared or RF markers”).
Regarding claim 53, Lang in view of Ryan teaches all the limitations of claim 52 above.
Lang further teaches wherein the second tracking device is configured to track the portion of the tool by tracking the tool marker ([0071] states that “With surgical navigation, a first virtual instrument can be displayed on a computer monitor which is a representation of a physical instrument tracked with navigation markers, e.g. infrared or RF markers”).
Regarding claim 54, Lang in view of Ryan teaches all the limitations of claim 51 above.
Lang further teaches wherein the second head-mounted device comprises a second head-mounted display (OHMD 2 of [0147]), and wherein the at least one computer processor is further configured to generate a further augmented reality image upon the second head-mounted display ([0147] states that “The OHMD's 11, 12, 13, 14 can project digital holograms of the virtual data or virtual data into the view of the left eye using the view position and orientation of the left eye 26 and can project digital holograms of the virtual data or virtual data into the view of the right eye using the view position and orientation of the right eye 28 of each user, resulting in a shared digital holographic experience 30”).
Regarding claim 55, Lang in view of Ryan teaches all the limitations of claim 51 above.
Lang further teaches wherein the first tracking device comprises a first camera ([0103], [0104] disclose that the exemplary OHMD (Microsoft Hololens includes several cameras (two on each side)).
Regarding claim 56, Lang in view of Ryan teaches all the limitations of claim 55 above.
Lang further teaches wherein the second tracking device comprises a second camera ([0103], [0104] disclose that the exemplary OHMD (Microsoft Hololens includes several cameras (two on each side)).
Regarding claim 57, Lang in view of Ryan teaches all the limitations of claim 51 above.
Lang further teaches wherein the at least one computer processor (processors of [0103] and computer graphics system of [0116],[0117]) is configured to generate the augmented reality image upon the first head-mounted display ([0147] discloses generating shared digital holographic experience of the surgical scene on the OHMDs) by:
determining a position of the tool with respect to the anatomy of the patient using data received from the first tracking device in combination with data received from the second tracking device ([0147], [0253] disclose a spatial mapping registration process, stating that “Live data, e.g. live data of the patient, the position and/or orientation of a physical instrument, the position and/or orientation of an implant component, the position and/or orientation of one or more OHMD's, can be acquired or registered”) ;
generating the virtual image of the tool and the anatomy of the patient upon the first head-mounted display, based upon the position of the tool with respect to the anatomy of the patient ([0253] then states “This process creates a three-dimensional mesh describing the surfaces of one or more objects or environmental structures using, for example and without limitation, a depth sensor, laser scanner, structured light sensor, time of flight sensor, infrared sensor, or tracked probe. These devices can generate 3D surface data by collecting, for example, 3D coordinate information or information on the distance from the sensor of one or more surface points on the one or more objects or environmental structures. The 3D surface points can then be connected to 3D surface meshes, resulting in a three-dimensional surface representation of the live data. The surface mesh can then be merged with the virtual data using any of the registration techniques described in the specification”);
determining a position of the body of the patient with respect to the first head-mounted device based upon data received from the first tracking device ([0146] states that “Once the virtual data and the live data of the patient and the OHMD are registered in the same coordinate system, e.g. using IMUs, optical markers, navigation markers including infrared markers, retroreflective markers, RF markers, and any other registration method described in the specification or known in the art, any change in position of any of the OHMD in relationship to the patient measured in this fashion can be used to move virtual data of the patient in relationship to live data of the patient, so that the visual image of the virtual data of the patient and the live data of the patient seen through the OHDM are always aligned, irrespective of movement of the OHMD and/or the operator's head and/or the operator wearing the OHMD”); and
overlaying the virtual image upon the body of the patient, based upon the position of the body of the patient with respect to the first head-mounted device ([0146] further states “The position, orientation, alignment, and change in position, orientation and alignment in relationship to the patient and/or the surgical site of each additional OHMD can be individually monitored thereby maintaining alignment and/or superimposition of corresponding structures in the live data of the patient and the virtual data of the patient for each additional OHMD irrespective of their position, orientation, and/or alignment in relationship to the patient and/or the surgical site”).
Regarding claim 58, Lang in view of Ryan teaches all the limitations of claim 57 above.
Lang further teaches wherein determining the position of the tool with respect to the anatomy of the patient using data received from the first tracking device in combination with data received from the second tracking device comprises determining an average position of the tool with respect to the anatomy of the patient ([1184] states that “the information from two or more cameras can be merged by averaging the 3D coordinates or detected surface points or other geometric structures such as planes or curved surfaces”).
Regarding claim 59, Lang in view of Ryan teaches all the limitations of claim 57 above.
Lang further teaches wherein determining the position of the body of the patient with respect to the first head-mounted device based upon data received from the first tracking device comprises analyzing one or more images of the patient marker created by a camera of the first tracking device ([0253] states that “Live data, e.g. live data of the patient, the position and/or orientation of a physical instrument, the position and/or orientation of an implant component, the position and/or orientation of one or more OHMD's, can be acquired or registered, for example, using a spatial mapping process. This process creates a three-dimensional mesh describing the surfaces of one or more objects or environmental structures using, for example and without limitation, a depth sensor, laser scanner, structured light sensor, time of flight sensor, infrared sensor, or tracked probe”, and [0270] states that “Image processing and/or pattern recognition of the live data of the patient can then be performed through the OHMD, e.g. using a built in image capture apparatus for capturing the live data of the patient or image and/or video capture systems attached to, integrated with or coupled to the OHMD”).
Regarding claim 60, Lang in view of Ryan teaches all the limitations of claim 59 above.
Lang further teaches wherein the one or more images of the patient marker comprise data that is sufficient to determine the position of the body of the patient with respect to the first head-mounted device without requiring use of triangulation techniques ([0270] describes image processing of the live data of the patient to detect the patient’s position and orientation ([0253]) does not involve triangulation).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Farouk A Bruce whose telephone number is (408)918-7603. The examiner can normally be reached Mon-Fri 8-5pm PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Christopher Koharski can be reached at (571) 272-7230. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/FAROUK A BRUCE/ Examiner, Art Unit 3797
