Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
1. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
2. Claim(s) * is/are rejected under 35 U.S.C 102(a)(2) as being anticipated by Kostrzewski et al. (US 20210391058 A1, hereinafter referred as “Kostrzewski”).
Regarding claim 1, Kostrzewski discloses a method comprising: receiving position information for anatomical landmarks based on a position of a pointer having fiducials, the position information received at an augmented reality headset positioned to view the anatomical landmarks (Abstract, reflective fiducials (also called markers, para [0080], head-mounted display devices that can be worn by a surgeon - determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, and can be configured to determine a pose of the surgical saw based on determining a pose of tracking markers connected to at least one of the surgical saw and the passive end effector, para [0114]-[0118], para [0173]-[0180]);
generating a surgical plan based on the position information (reflective fiducials (also called markers, para [0080], para [0084]-[0094], para [0114]-[0118], FIG. 12, three stages of workflow are illustrated: pre-operative stage 1200; intra-operative stage 1202; and post-operative stage 1204: During the pre-operative stage 1200, a user (e.g., surgeon) generates a surgical plan (case) based on analyzed patient images with assistance from the surgical guidance system 1220, para [0160], para [0173]-[0180]); and
displaying, using the augmented reality headset, information to guide a device based on the surgical plan (ML guidance system 1220 (FIG. 12) provides guidance to a user during pre-operative planning and during intraoperative surgical execution of the surgical plan. The ML guidance system enables automatic or semi-automatic (involving human input) selection of implant(s) and generation of the surgical plan - augmented reality interaction via, e.g., a head-mounted display, para [0084]-[0094]).
Regarding claim 2, Kostrzewski discloses method of claim 1, further comprising displaying, using the augmented reality headset, indication of the position information (augmented reality interaction via, e.g., a head-mounted display, para [0084]-[0094], para [0114]-[0118], graphical representations of the poses of the resection planes displayed though the display device within an extended reality (XR) headset 920 (FIG. 9) as an overlay on the defined patient - control a sequence of movements of a surgical saw attached to an arm of the surgical robot so a cutting plane of the surgical saw (e.g., surgical saw 1040 in FIG. 10 or FIG. 11) becomes sequentially aligned with the poses of the resection planes, para [0173]-[0180]).
Regarding claim 3, Kostrzewski discloses method of claim 1, wherein the position information is further based on a fixed marker having fiducials, the augmented reality headset further positioned to view the fixed marker (reflective fiducials (also called markers, para [0080], head-mounted
display devices that can be worn by a surgeon - determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, and can be configured to determine a pose of the surgical saw based on determining a pose of tracking markers connected to at least one of the surgical saw and the passive
end effector, para [0114]-[0118], para [0173]-[01 BO], para [0246]-[0255]).
Regarding claim 4, Kostrzewski discloses method of claim 3, wherein the fixed marker is attached to anatomy (determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, para [0114]-[0118]).
Claim Rejections - 35 USC § 103
4. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
5. Claim(s) 5-32 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kostrzewski in view of Lang et al. (US 20210192759 A1, hereinafter referred as “Lang”).
Regarding claim 5, Kostrzewski discloses method comprising: detecting, by an augmented reality device, a marker affixed to anatomy (para [0084]-[0094], determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, para [0114]-[0118]);
creating a coordinate system based in part on fiducials attached to the marker ("pose" refers to the position and/or the rotational angle of one object (e.g., dynamic reference array, end effector, surgical tool, anatomical structure, etc.) relative to another object and/or to a defined coordinate system, para [0042], determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, and can be configured to determine a pose of the surgical saw based on determining a pose of tracking markers connected to at least one of the surgical saw and the passive end effector, para [0114]-[0118]);
detecting an identification of a landmark, by the augmented reality device, based on the position of the pointer, the identification including landmark position information of the landmark (reflective fiducials (also called markers, para [0080], head-mounted display devices that can be worn by a surgeon - determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, and can be configured to determine a pose of the surgical saw based on determining a pose of tracking markers connected to at least one of the surgical saw and the passive end effector, para [0114]-[0118], para [0173]-[0180], para [0246]-[0255]);
determining a position of the landmark in the coordinate system ("pose" refers to the position and/or the rotational angle of one object (e.g., dynamic reference array, end effector, surgical tool, anatomical structure, etc.) relative to another object and/or to a defined coordinate system, para [0042], determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, and can be configured to determine a pose of the surgical saw based on determining a pose of tracking markers connected to at least one of the surgical saw and the passive end effector, para [0114]-[0118]);
generating an intra-operative plan based on the position of the landmark (reflective fiducials (also called markers, para [0080], para [0084]-[0094], para [0114]-[0118], FIG. 12, three stages of workflow are illustrated: pre-operative stage 1200; intra-operative stage 1202; and post-operative stage 1204: During the pre-operative stage 1200, a user (e.g., surgeon) generates a surgical plan (case) based on analyzed patient images with assistance from the surgical guidance system 1220, para [0160], para [0173]-[0180]); and providing surgical guidance using the augmented reality device based on the intra-operative plan (ML guidance system 1220 (FIG. 12) provides guidance to a user during pre-operative planning and during intraoperative surgical execution of the surgical plan. The ML guidance system enables automatic or semi-automatic (involving human input) selection of implant(s) and generation of the surgical plan - augmented reality interaction via, e.g., a head-mounted display, para [0084]-[0094]).
Kostrzewski doesn’t disclose tracking a position of a pointer relative to the marker.
However, in the same field of endeavor, Lang in analogous art discloses tracking a position of a pointer relative to the marker (performing a surgical step or surgical procedure with visual guidance using an optical head mounted display, Abstract, para [0004]-[0014], optical imaging and/or 3D scanning probe used with an arthroscopic procedure can optionally be tracked by tracking the position, location, orientation, alignment and/or direction of movement using optical markers, e.g. with one or more geometric patterns, e.g. in 2D or 3D, or LED's using one or more camera or video systems integrated into, attached to, or separate from one or more OHMDs - use image processing and pattern recognition to recognize the known geometry of the one or more probes and their location within a coordinate system, e.g. in relationship to the patient, the surgical site and/or the OR table, para [0258), pointer with an attached optical marker or an attached navigation marker, para [0355]-[0358], dimensions of the pointer have been previously scanned and registered with the OHMD. The image and/or video capture system attached to, integrated with or coupled to the OHMD can recognize the pointer in the live data and can identify the tip of the pointer. When the tip of the pointer touches the live landmark on the patient that corresponds to the landmark in the virtual data, the surgeon can, for example, click to indicate successful cross-referencing. The two data points can then optionally be fused or superimposed in a common coordinate system, para [0380], para [1042], para [1062]-[1065]).
It would have been obvious to one of ordinary skill in the art before the effective filing date to add Lang's Intra-operative measurements to KOSTRZEWSKI's method/system/augmented reality device using one or more cameras, an image capture system, a video capture and/or 3D scanner separate from an OHMD can be advantageous when measurements are desired to be obtained from a view angle other than the surgeon or, when multiple OHMDs are used, from a view angle other than of a surgical assistant or second surgeon (see para [1062]-[1065]).
Regarding claim 6, Kostrzewski discloses method of claim 5, further comprising: selecting the landmark based on a particular surgical procedure (augmented reality interaction via, e.g., a head-mounted display, para [0084]-[0094], During the surgery, the surgical robot 4 moves the end effector coupler 22, and the passive end effector and surgical saw attached there, automatically to a position close to a knee or other anatomical structure, so that all bone to be cut is within the workspace of the passive end effector. This position depends on the cut to be made and the surgery planning and implant construction, para [0114]-[0121]).
Regarding claim 7, Kostrzewski discloses method of claim 5, further comprising: prompting, by the augmented reality device, the identification of the landmark at least in part by instructing to have the pointer indicate the landmark (para [0074], computer platform 910 can use two DRAs to tracking patient anatomy position: one on patient tibia and one on patient femur. The platform 900 may use standard navigated instruments for the registration and checks (e.g. a pointer similar to the one used in Kostrzewski ExcelsiusGPS system for spine surgery), para [0084]-[0094], surgeon will review registration precision by touching selected points on bone with a stylus and verifying that on the navigation screen pointer tip is placed on the bone, para (0249]-[0250]).
Regarding claim 8, Kostrzewski doesn’t explicitly discloses method of claim 5, wherein detecting the identification of the landmark is responsive to slowed or stopped motion of the pointer for a predetermined period.
However, in the same field of endeavor, Lang discloses wherein detecting the identification of the landmark is responsive to slowed or stopped motion of the pointer for a predetermined period (image and/or video capture system attached to, integrated with or coupled to the OHMD can recognize the pointer in the live data and can identify the tip of the pointer. When the tip of the pointer touches the live landmark on the patient that corresponds to the landmark in the virtual data, the surgeon can, for example, click to indicate successful cross-referencing. The two data points can then optionally be fused or superimposed in a common coordinate system, para (0380], Move pointer (Slow, Fast), TABLE 3).
It would have been obvious to one of ordinary skill in the art before the effective filing date so that the tracking system gets multiple frames to average or confirm the pointers position.
Regarding claim 9, Kostrzewski discloses method of claim 5, further comprising: prompting an additional identification of an additional landmark by the augmented reality device instructing to have the pointer indicate the landmark; detecting the additional identification of the additional landmark based on the position of the pointer by the augmented reality device following a surgical intervention based on the surgical guidance, the additional identification including additional landmark position information of the additional landmark; determining a position of the additional landmark in the coordinate system; modifying the intra-operative plan based in part on the position of the additional landmark; and providing the surgical guidance in the coordinate system based in part on the modified intra-operative plan (para (0084]-[0094], the system can operate to automatically propose initial implant size and placement based on image analysis. In the next steps, the user can modify the implant size and placement and observe results of such modifications by updated visual feedback displayed on the medical images. When the user indicates acceptability of the plan, the plan becomes validated as approved for surgery, para [0204]-[0210], For X-ray and imageless workflows, only condylar surfaces are measured in addition to previously collected natural landmarks. This is to complete the virtual model of the patient obtained intra-operatively called "stickman" which is composed of natural landmarks and knee joint surfaces, para (0245]-[0255]).
Regarding claim 10, Kostrzewski discloses method of claim 9, further comprising: generating an anatomical model of the anatomy based on the marker and the landmark; updating the anatomical model based on the additional landmark; and updating the surgical guidance based on the updated anatomical model (para [0084]-(0094], 3D model of the femur and tibial bone can be generated - user can modify the implant size and placement and observe results of such modifications by updated visual feedback displayed on the medical images. When the user indicates acceptability of the plan, the plan becomes validated as approved for surgery, para (0204]-[0210], For X-ray and imageless workflows, only condylar surfaces are measured in addition to previously collected natural landmarks. This is to complete the virtual model of the patient obtained intra-operatively called "stickman" which is composed of natural landmarks and knee joint surfaces, para [0245) [0255]).
Regarding claim 11, Kostrzewski discloses method of claim 10, wherein updating the surgical guidance is at least one of modifying a pre-operative plan or developing an additional portion of the pre-operative plan to update the intra-operative plan based on the updated anatomical model (para [0009]-[0010], para (0084)-(0094], 3D model of the femur 'and tibial bone can be generated - use'r can modify the implant size and placement and observe results of such modifications by updated visual feedback displayed on the medical images. When the user indicates acceptability of the plan, the plan becomes validated as approved for surgery, para [0204]-[0210], para [0245)-(0255]).
Regarding claim 12, Kostrzewski discloses method of claim 5, wherein providing the surgical guidance comprises: tracking a current position of at least one of a cut plane, a guide, a surgical instrument, or a tool on a display; and displaying the current position and a target position of the at least one of the cut plane, the guide, the surgical instrument, or the tool on the display (computer platform 910 can generate a graphical representation of a patient's anatomical structure, surgical tool, user's hand, etc. with a displayed size, shape, color, and/or pose that is controlled based on the determined pose(s) of one or more the tracked DRAs, and which the graphical representation that is displayed can be dynamically modified to track changes in the determined poses over time, para (0082)).
Regarding claim 13, Kostrzewski discloses method of claim 12, wherein tracking the current position comprises: detecting fiducials arranged in a predetermined pattern and attached to the at least one of the guide, the surgical instrument, or the tool by a depth camera of the augmented reality device; identifying positions of the fiducials arranged in the predetermined pattern; and determining position information of the at least one of the guide, the surgical instrument, or the tool based on the positions of the fiducials (reference arrays enable tracking by reflecting light in known patterns, which are decoded to determine their respective poses by the tracking subsystem of the surgical robot 4. If the line-of-sight between the patient reference array 602 and the tracking cameras 46 in the auxiliary tracking bar is blocked (for example, by a medical personnel, instrument, etc.), further navigation of the surgical instrument may not be able to be performed and a responsive notification may temporarily halt further movement of the robot arm 20 and surgical robot 4, display a warning on the display 34, and/or provide an audible warning to medical personnel, para [0065]).
Regarding claim 14, Kostrzewski discloses method of claim 12, wherein providing the surgical guidance further comprises displaying at least one of a distance between the current position and the target position, or a direction to move the guide, the surgical instrument, or the tool (system indicates to surgeon which navigated jig to use and shows target. position. Surgeon fixes the jig to bone in location indicated by the system (e.g. system indicates target position in green if navigated jigs is within certain maximal distance from it) and cuts manually using sagittal saw through the jig, para [0276]-[0278]).
Regarding claim 15, Kostrzewski doesn’t explicitly discloses method of claim 5, further comprising: obtaining marker position information of the marker and the landmark position information using data from a depth camera of the augmented reality device.
However, in the same field of endeavor, Lang discloses obtaining marker position information of the marker and the landmark position information using data from a depth camera of the augmented reality device (depth camera with a 120 degrees x 120 degrees angle of view, para (0179], para (0258], pointer with an attached optical marker or an attached navigation marker, para [0355]-[0358], para [0380], para [1042]. Para. [1062]- [1065]).
It would have been obvious to one of ordinary skill in the art before the effective filing date in order so that the system can precisely measure how far objects are from the camera and make tracking more robust and reduce errors due to flat images.
Regarding claim 16, Kostrzewski discloses method of claim 15, wherein the fiducials are first fiducials, the method further comprising: detecting the first fiducials arranged in a predetermined pattern by the depth camera; identifying positions of the first fiducials; determining the marker position information of the marker based on the positions of the first fiducials; detecting second fiducials arranged in a predetermined pattern and attached to the pointer by the depth camera; identifying positions of the second fiducials; and determining a location of the pointer relative to each marker based on the positions of the second fiducials (reference arrays enable tracking by reflecting light in known patterns, which are decoded to determine their respective poses by the tracking subsystem of the surgical robot 4. If the line-of-sight between the patient reference array 602 and the tracking cameras 46 in the auxiliary tracking bar is blocked (for example, by a medical personnel, instrument, etc.), further navigation of the surgical instrument may not be able to be performed and a responsive notification may temporarily halt further movement of the robot arm 20 and surgical robot 4, display a warning on the display 34, and/or provide an audible warning to medical personnel, para [0065]. para [0084]-[0094]).
Regarding claim 17, Kostrzewski discloses method of claim 5, further comprising: calculating at least one axis or plane based on the determined position of the landmark; and providing the surgical guidance on the augmented reality device based on the at least one axis or plane (controller is configured to obtain the surgical plan from the surgical guidance system, determine a pose of a target plane based on the surgical plan defining where the anatomical structure is to be cut and based on the pose of the anatomical structure, and generate steering information based on comparison of the pose of the target plane and the pose of the surgical saw, wherein the steering information indicates where the surgical saw needs to be moved to position a cutting plane of the surgical saw to become aligned with the target plane, para [0009]-[0010]).
Regarding claim 18, Kostrzewski discloses method of claim 17, further comprising displaying the at least one axis or plane superimposed on the anatomy in augmented reality by the augmented reality device (controller is configured to obtain the surgical plan from the surgical guidance system, determine a pose of a target plane based on the surgical plan defining where the anatomical structure is to be cut and based on the pose of the anatomical structure, and generate steering information based on comparison of the pose of the target plane and the pose of the surgical saw, wherein the steering information indicates where the surgical saw needs to be moved to position a cutting plane of the surgical saw to become aligned with the target plane, para [0009]-[0010], graphical representations of the poses of the resection planes displayed though the display device within an extended reality (XR) headset 920 (FIG. 9) as an overlay on the defined patient, para [0173]-[0180], deviation of actual cutting plane from planned; deviation of predicted gap from actual gap, para [0187]-[0191], para [0246]-[0255]).
Regarding claim 19, Kostrzewski discloses system comprising: a marker configured to be affixed to anatomy (determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, para [0114]-[0118]); a pointer configured to point a landmark (surgeon will review registration precision by touching selected points on bone with a stylus and verifying that on the navigation screen pointer tip is placed on the bone, para [0249]-[0250]); and an augmented reality device (augmented reality interaction via, e.g., a head-mounted display, para [0084]-[0094]) configured to: detect the marker (para [0084]-[0094], determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, para [0114]-[0118]); create a coordinate system based in part on fiducials attached to the marker ("pose" refers to the position and/or the rotational angle of one object (e.g., dynamic reference array, end effector, surgical tool, anatomical structure, etc.) relative to another object and/or to a defined coordinate system, para [0042], determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, and can be configured to determine a pose of the surgical saw based on determining a pose of tracking markers connected to at least one of the surgical saw and the passive end effector, para [0114]-[0118]); prompt an identification of the landmark by instructing to have the pointer indicate the landmark (Medical personnel may be prompted by surgical robot 4 to select a function, mode, and/or asses the condition of surgical system 2. Depressing secondary button a single time may activate certain functions, modes, and/or acknowledge information communicated to medical personnel through the XR headset(s) 920, display 34 and/or light indicator 28, para [0074], surgeon will review registration precision by touching selected points on bone with a stylus and verifying that on the navigation screen pointer tip is placed on the bone, para [0249]-[0250]); detect the identification of the landmark responsive to an indication of the landmark (para [0114]-[0118], graphical representations of the poses of the resection planes displayed though the display device within an extended reality (XR) headset 920 (FIG. 9) as an overlay on the defined patient - control a sequence of movements of a surgical saw attached to an arm of the surgical robot so a cutting plane of the surgical saw (e.g., surgical saw 1040 in FIG. 10 or FIG. 11) becomes sequentially aligned with the poses of the resection planes, para [0173]-[0180]); determine a position of the landmark in the coordinate system ("pose" refers to the position and/or the rotational angle of one object (e.g., dynamic reference array, end effector, surgical tool, anatomical structure, etc.) relative to another object and/or to a defined coordinate system, para [0042], determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, and can be configured to determine a pose of the surgical saw based on determining a pose of tracking markers connected to at least one of the surgical saw and the passive end effector, para [0114] [01181); generate an intra-operative plan based on the position of the landmark (reflective fiducials (also called markers, para [0080], para [0084]-[0094], para [0114]-[0118], FIG. 12, three stages of workflow are illustrated: pre-operative stage 1200; intra-operative stage 1202; and post-operative stage 1204: During the pre-operative stage 1200, a user (e.g., surgeon) generates a surgical plan (case) based on analyzed patient images with assistance from the surgical guidance system 1220, para [0160], para [0173]-[0180]); and provide surgical guidance using the augmented reality device based on the intra-operative plan (ML guidance system 1220 (FIG. 12) provides guidance to a user during pre-operative planning and during intraoperative surgical execution of the surgical plan. The ML guidance system enables automatic or semi-automatic (involving human input) selection of implant(s) and generation of the surgical plan - augmented reality interaction via, e.g., a head-mounted display, para [0084]-[0094]).
Regarding claim 20, Kostrzewski discloses system of claim 19, wherein the pointer comprises a stylus configured to point the landmark (surgeon will review registration precision by touching selected points on bone with a stylus and verifying that on the navigation screen pointer tip is placed on the bone, para [0245]-[0255]).
Regarding claim(s) 21, this/these system claim(s) has/have similar limitations as method claim(s) 9, and therefore rejected on similar grounds.
Regarding claim 22, Kostrzewski doesn’t explicitly discloses system of claim 19, further comprising fiducials attached to the pointer, wherein the augmented reality device is configured to determine the position of the landmark based in part on relative positions of the fiducials attached to the pointer and the position of the marker.
However, in the same field of endeavor, Lang discloses fiducials attached to the pointer, wherein the augmented reality device is configured to determine the position of the landmark based in part on relative positions of the fiducials attached to the pointer and the position of the marker (Abstract, para [0004]-[0014]. para [0258], pointer with an attached optical marker or an attached navigation marker, para [0355]-[0358], para [0380], para [1042], para [1062]-[1065]).
It would have been obvious to one of ordinary skill in the art before the effective filing date in order to enhanced positional accuracy for the augmented reality system.
Regarding claim 23, Kostrzewski doesn’t explicitly discloses system of claim 19, wherein the augmented reality device comprises a depth camera configured to receive a depth image that includes a representation of fiducials relative to the device, and wherein the augmented reality device is configured to determine the position of the landmark based on data received from the depth camera.
However, in the same field of endeavor, Lang discloses wherein the augmented reality device comprises a depth camera configured to receive a depth image that includes a representation of fiducials relative to the device, and wherein the augmented reality device is configured to determine the position of the landmark based on data received from the depth camera (depth camera with a 120 degrees x120 degrees angle of view, para [0179]. use image processing and pattern recognition to recognize the known geometry of the one or more probes and their location within a coordinate system, e.g. in relationship to the patient, the surgical site and/or the OR table, para [0258], pointer with an attached optical marker or an attached navigation marker, para (0355]-[0358], dimensions of the pointer have been previously scanned and registered with the OHMD. The image and/or video capture system attached to, integrated with or coupled to the OHMD can recognize the pointer in the live data and can identify the tip of the pointer. When the tip of the pointer touches the live landmark on the patient that corresponds to the landmark in the virtual data, the surgeon can, for example, click to indicate successful cross-referencing. The two data points can then optionally be fused or superimposed in a common coordinate system, para [0380], para [1042], para [1062] [1065]).
It would have been obvious to one of ordinary skill in the art before the effective filing date in order so that the system can precisely measure how far objects are from the camera and make tracking more robust and reduce errors due to flat images.
Regarding claim 24, Kostrzewski discloses system of claim 19, wherein the augmented reality device is further configured to: calculate at least one axis or plane based on the determined position of the landmark; and provide the surgical guidance on the augmented reality device based on the at least one axis or plane (controller is configured to obtain the surgical plan from the surgical guidance system, determine a pose of a target plane based on the surgical plan defining where the anatomical structure is to be cut and based on the pose of the anatomical structure, and generate steering information based on comparison of the pose of the target plane and the pose of the surgical saw, wherein the steering information indicates where the surgical saw needs to be moved to position a cutting plane of the surgical saw to become aligned with the target plane, para [0009]-[0010]).
Regarding claim 25, Kostrzewski discloses system of claim 24, wherein the augmented reality device is configured to display the at least one axis or plane by superimposing the at least one axis or plane on the anatomy (controller is configured to obtain the surgical plan from the surgical guidance system, determine a pose of a target plane based on the surgical plan defining where the anatomical structure is to be cut and based on the pose of the anatomical structure, and generate steering information based on comparison of the pose of the target plane and the pose of the surgical saw, wherein the steering information indicates where the surgical saw needs to be moved to position a cutting plane of the surgical saw to become aligned with the target plane, para [0009]-[0010], graphical representations of the poses of the resection planes displayed though the display device within an extended reality (XR) headset 920 (FIG. 9) as an overlay on the defined patient, para [0173]-[0180], deviation of actual cutting plane from planned; deviation of predicted gap from actual gap, para [0187]-[0191], para [0246]-[0255]).
Regarding claim 26, Kostrzewski discloses augmented reality device comprising: a processor; and a non-transitory computer readable medium comprising instructions that, when executed, cause the processor to perform operations comprising (surgical guidance system is disclosed for computer assisted navigation during surgery - obtain pre-operative data from one of the distributed network computers characterizing a defined patient, and generate a surgical plan for the defined patient based on processing the pre operative data through the machine learning model, Abstract, para [0009]-[0010], para [0080]-[0094], head-mounted display devices that can be worn by a surgeon - determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, para [0114]-[0118], graphical representations of the poses of the resection planes displayed though the display device within an extended reality (XR) headset 920 (FIG. 9) as an overlay on the defined patient, para [0173]-[0180], deviation of actual cutting plane from planned; deviation of predicted gap from actual gap, para [0187] [0191], para [0246]-[0255]):
detecting a marker affixed to anatomy (para [0084]-[0094], determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, para [0114]-[0118]);
creating a coordinate system based in part on fiducials attached to the marker ("pose" refers to the position and/or the rotational angle of one object (e.g., dynamic reference array, end effector, surgical tool, anatomical structure, etc.) relative to another object and/or to a defined coordinate system, para [0042], determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, and can be configured to determine a pose of the surgical saw based on determining a pose of tracking markers connected to at least one of the surgical saw and the passive end effector, para [0114]-[0118]);
prompting an identification of a landmark by an augmented reality device instructing to have a pointer indicate the landmark (Medical personnel may be prompted by surgical robot 4 to select a function, mode, and/or asses the condition of surgical system 2. Depressing secondary button a single time may activate certain functions, modes, and/or acknowledge information communicated to medical personnel through the XR headset(s) 920, display 34 and/or light indicator 28, para [0074], surgeon will review registration precision by touching selected points on bone with a stylus and verifying that on the navigation screen pointer tip is placed on the bone, para [0249]-[0250]);
detecting the identification of the landmark based on a position of the pointer, the identification including landmark position information of the landmark (para [0074], computer platform 910 can use two DRAs to tracking patient anatomy position: one on patient tibia and one on patient femur. The platform 900 may use standard navigated instruments for the registration and checks (e.g. a pointer similar to the one used in Kostrzewski ExcelsiusGPS system for spine surgery), para [0084]-[0094], surgeon will review registration precision by touching selected points on bone with a stylus and verifying that on the navigation screen pointer tip is placed on the bone, para (0249]-[0250]);
determining a position of the landmark in the coordinate system ("pose" refers to the position and/or the rotational angle of one object (e.g., dynamic reference array, end effector, surgical tool, anatomical structure, etc.) relative to another object and/or to a defined coordinate system, para [0042], determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, and can be configured to determine a pose of the surgical saw based on determining a pose of tracking markers connected to at least one of the surgical saw and the passive end effector, para [0114] [01181);
generating an intra-operative plan based on the position of the landmark (reflective fiducials (also called markers, para [0080], para [0084]-[0094], para [0114]-[0118], FIG. 12, three stages of workflow are illustrated: pre-operative stage 1200; intra-operative stage 1202; and post-operative stage 1204: During the pre-operative stage 1200, a user (e.g., surgeon) generates a surgical plan (case) based on analyzed patient images with assistance from the surgical guidance system 1220, para [0160], para [0173]-[0180]); and providing surgical guidance based on the intra-operative plan (ML guidance system 1220 (FIG. 12) provides guidance to a user during pre-operative planning and during intra operative surgical execution of the surgical plan. The ML guidance system enables automatic or semi-automatic (involving human input) selection of implant(s) and generation of the surgical plan - augmented reality interaction via, e.g., a head-mounted display, para [0084] [0094]).
Regarding claim 27, Kostrzewski discloses augmented reality device of claim 26, wherein the operations further comprise: prompting an additional identification of an additional landmark by the augmented reality device instructing to have the pointer indicate the landmark; detecting the additional identification of the additional landmark based on the position of the pointer following a surgical intervention based on the surgical guidance, the additional identification including additional landmark position information of the additional landmark; determining a position of the additional landmark in the coordinate system; modifying the intra-operative plan based in part on the position of the additional landmark; and providing the surgical guidance in the coordinate system based in part on the modified intra-operative plan (para (0084]-[0094], the system can operate to automatically propose initial implant size and placement based on image analysis. In the next steps, the user can modify the implant size and placement and observe results of such modifications by updated visual feedback displayed on the medical images. When the user indicates acceptability of the plan, the plan becomes validated as approved for surgery, para [0204]-[0210], For X-ray and imageless workflows, only condylar surfaces are measured in addition to previously collected natural landmarks. This is to complete the virtual model of the patient obtained intra-operatively called "stickman" which is composed of natural landmarks and knee joint surfaces, para (0245]-[0255]).
Regarding claim 28, Kostrzewski discloses augmented reality device of claim 26, wherein the operations further comprise selecting the landmark based on a surgical procedure (augmented reality interaction via, e.g., a head-mounted display, para [0084]-[0094], During the surgery, the surgical robot 4 moves the end effector coupler 22, and the passive end effector and surgical saw attached there, automatically to a position close to a knee or other anatomical structure, so that all bone to be cut is within the workspace of the passive end effector. This position depends on the cut to be made and the surgery planning and implant construction, para [0114]-[0121]).
Regarding claim 29, Kostrzewski doesn’t explicitly discloses augmented reality device of claim 26, further comprising a depth camera, wherein the operations further comprise obtaining marker position information of the marker and the landmark position information using data from the depth camera
However, in the same field of endeavor, Lang discloses obtaining marker position information of the marker and the landmark position information using data from a depth camera of the augmented reality device (depth camera with a 120 degrees x 120 degrees angle of view, para (0179], para (0258], pointer with an attached optical marker or an attached navigation marker, para [0355]-[0358], para [0380], para [1042]. Para. [1062]- [1065]).
It would have been obvious to one of ordinary skill in the art before the effective filing date in order so that the system can precisely measure how far objects are from the camera and make tracking more robust and reduce errors due to flat images.
Regarding claim 30, Kostrzewski discloses augmented reality device of claim 29, wherein the fiducials are first fiducials, and wherein the operations further comprise: detecting the first fiducials arranged in a predetermined pattern by the depth camera; identifying positions of the first fiducials; determining the marker position information of the marker based on the positions of the first fiducials; detecting second fiducials arranged in a predetermined pattern and attached to the pointer by the depth camera; identifying positions of the second fiducials; and determining a position of the pointer relative to each marker based on the positions of the second fiducials (reference arrays enable tracking by reflecting light in known patterns, which are decoded to determine their respective poses by the tracking subsystem of the surgical robot 4. If the line-of-sight between the patient reference array 602 and the tracking cameras 46 in the auxiliary tracking bar is blocked (for example, by a medical personnel, instrument, etc.), further navigation of the surgical instrument may not be able to be performed and a responsive notification may temporarily halt further movement of the robot arm 20 and surgical robot 4, display a warning on the display 34, and/or provide an audible warning to medical personnel, para [0065]. para [0084]-[0094]).
Regarding claim 31, Kostrzewski discloses augmented reality device of claim 26, wherein the operations further comprise: calculating at least one axis or plane based on the determined position of the landmark; and providing the surgical guidance on the augmented reality device based on the at least one axis or plane (controller is configured to obtain the surgical plan from the surgical guidance system, determine a pose of a target plane based on the surgical plan defining where the anatomical structure is to be cut and based on the pose of the anatomical structure, and generate steering information based on comparison of the pose of the target plane and the pose of the surgical saw, wherein the steering information indicates where the surgical saw needs to be moved to position a cutting plane of the surgical saw to become aligned with the target plane, para [0009]-[0010]).
Regarding claim 32, Kostrzewski discloses augmented reality device of claim 31, wherein the operations further comprise displaying the at least one axis or plane superimposed on the anatomy in augmented reality by the augmented reality device (controller is configured to obtain the surgical plan from the surgical guidance system, determine a pose of a target plane based on the surgical plan defining where the anatomical structure is to be cut and based on the pose of the anatomical structure, and generate steering information based on comparison of the pose of the target plane and the pose of the surgical saw, wherein the steering information indicates where the surgical saw needs to be moved to position a cutting plane of the surgical saw to become aligned with the target plane, para [0009]-[0010], graphical representations of the poses of the resection planes displayed though the display device within an extended reality (XR) headset 920 (FIG. 9) as an overlay on the defined patient, para [0173]-[0180], deviation of actual cutting plane from planned; deviation of predicted gap from actual gap, para [0187]-[0191], para [0246]-[0255]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PRIYANK J SHAH whose telephone number is (571)270-3732. The examiner can normally be reached on 10:00 - 6:00 M-F.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, LunYi Lao can be reached on 5712727671. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PRIYANK J SHAH/Primary Examiner, Art Unit 2621