DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
2. Acknowledgement is made of the response filed on 04/03/2026, in which claims 33-36 are added, rejection of claims 1-32 are traversed, and claims 1-36 now pending.
Claim Rejections - 35 USC § 102
3. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
4. Claim(s) 1-4, and 33-34 is/are rejected under 35 U.S.C 102(a)(2) as being anticipated by Kostrzewski et al. (US 20210391058 A1, hereinafter referred as “Kostrzewski”).
Regarding claim 1, Kostrzewski discloses a method comprising: receiving position information for anatomical landmarks based on a position of a pointer having fiducials, the position information received at an augmented reality headset positioned to view the anatomical landmarks (Abstract, reflective fiducials (also called markers, para [0080], head-mounted display devices that can be worn by a surgeon - determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, and can be configured to determine a pose of the surgical saw based on determining a pose of tracking markers connected to at least one of the surgical saw and the passive end effector, para [0114]-[0118], para [0173]-[0180]);
generating a surgical plan based on the position information (reflective fiducials (also called markers, para [0080], para [0084]-[0094], para [0114]-[0118], FIG. 12, three stages of workflow are illustrated: pre-operative stage 1200; intra-operative stage 1202; and post-operative stage 1204: During the pre-operative stage 1200, a user (e.g., surgeon) generates a surgical plan (case) based on analyzed patient images with assistance from the surgical guidance system 1220, para [0160], para [0173]-[0180]); and
displaying, using the augmented reality headset, information to guide a device based on the surgical plan (ML guidance system 1220 (FIG. 12) provides guidance to a user during pre-operative planning and during intraoperative surgical execution of the surgical plan. The ML guidance system enables automatic or semi-automatic (involving human input) selection of implant(s) and generation of the surgical plan - augmented reality interaction via, e.g., a head-mounted display, para [0084]-[0094]).
Regarding claim 2, Kostrzewski discloses method of claim 1, further comprising displaying, using the augmented reality headset, indication of the position information (augmented reality interaction via, e.g., a head-mounted display, para [0084]-[0094], para [0114]-[0118], graphical representations of the poses of the resection planes displayed though the display device within an extended reality (XR) headset 920 (FIG. 9) as an overlay on the defined patient - control a sequence of movements of a surgical saw attached to an arm of the surgical robot so a cutting plane of the surgical saw (e.g., surgical saw 1040 in FIG. 10 or FIG. 11) becomes sequentially aligned with the poses of the resection planes, para [0173]-[0180]).
Regarding claim 3, Kostrzewski discloses method of claim 1, wherein the position information is further based on a fixed marker having fiducials, the augmented reality headset further positioned to view the fixed marker (reflective fiducials (also called markers, para [0080], head-mounted
display devices that can be worn by a surgeon - determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, and can be configured to determine a pose of the surgical saw based on determining a pose of tracking markers connected to at least one of the surgical saw and the passive
end effector, para [0114]-[0118], para [0173]-[01 BO], para [0246]-[0255]).
Regarding claim 4, Kostrzewski discloses method of claim 3, wherein the fixed marker is attached to anatomy (determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, para [0114]-[0118]).
Regarding claim 33, Kostrzewski discloses the method of claim 1, further comprising receiving position information for multiple anatomical landmarks based on successive positions of the pointer (computer platform 910 can generate a graphical representation of a patient's anatomical structure with a displayed pose that is controlled based on the determined pose(s) of one or more the tracked DRAs, and which the graphical representation that is displayed can be dynamically modified to track changes in the determined poses over time, para [0082]; DRAs inclusive of markers or indicia formed on surgical tools, para [0054]) having fiducials (DRA can include fiducials, para [0054]).
Regarding claim 34, Kostrzewski discloses the method of claim 33, wherein the augmented reality headset determines the anatomical landmark positions from the position of the pointer (receiving pose information for an anatomical structure based on a pose of tracked DRAs which are markers attached or formed on the surgical saw, para [0082]) while the respective anatomical landmarks are in a field of view of the augmented reality headset (the computer platform 910 of the surgical system 900 (FIG. 9) which may include the camera tracking system component 6, para [0041]; and the computer platform 910 includes a display 912… the display 912 may be part of an XR headset 920, para. [0085]).
Claim Rejections - 35 USC § 103
5. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
6. Claim(s) 5-32 and 35-36 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kostrzewski in view of Lang et al. (US 20210192759 A1, hereinafter referred as “Lang”).
Regarding claim 5, Kostrzewski discloses method comprising: detecting, by an augmented reality device, a marker affixed to anatomy (para [0084]-[0094], determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, para [0114]-[0118]);
creating a coordinate system based in part on fiducials attached to the marker ("pose" refers to the position and/or the rotational angle of one object (e.g., dynamic reference array, end effector, surgical tool, anatomical structure, etc.) relative to another object and/or to a defined coordinate system, para [0042], determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, and can be configured to determine a pose of the surgical saw based on determining a pose of tracking markers connected to at least one of the surgical saw and the passive end effector, para [0114]-[0118]);
detecting an identification of a landmark, by the augmented reality device, based on the position of the pointer, the identification including landmark position information of the landmark (reflective fiducials (also called markers, para [0080], head-mounted display devices that can be worn by a surgeon - determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, and can be configured to determine a pose of the surgical saw based on determining a pose of tracking markers connected to at least one of the surgical saw and the passive end effector, para [0114]-[0118], para [0173]-[0180], para [0246]-[0255]);
determining a position of the landmark in the coordinate system ("pose" refers to the position and/or the rotational angle of one object (e.g., dynamic reference array, end effector, surgical tool, anatomical structure, etc.) relative to another object and/or to a defined coordinate system, para [0042], determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, and can be configured to determine a pose of the surgical saw based on determining a pose of tracking markers connected to at least one of the surgical saw and the passive end effector, para [0114]-[0118]);
generating an intra-operative plan based on the position of the landmark (reflective fiducials (also called markers, para [0080], para [0084]-[0094], para [0114]-[0118], FIG. 12, three stages of workflow are illustrated: pre-operative stage 1200; intra-operative stage 1202; and post-operative stage 1204: During the pre-operative stage 1200, a user (e.g., surgeon) generates a surgical plan (case) based on analyzed patient images with assistance from the surgical guidance system 1220, para [0160], para [0173]-[0180]); and providing surgical guidance using the augmented reality device based on the intra-operative plan (ML guidance system 1220 (FIG. 12) provides guidance to a user during pre-operative planning and during intraoperative surgical execution of the surgical plan. The ML guidance system enables automatic or semi-automatic (involving human input) selection of implant(s) and generation of the surgical plan - augmented reality interaction via, e.g., a head-mounted display, para [0084]-[0094]).
Kostrzewski doesn’t disclose tracking a position of a pointer relative to the marker.
However, in the same field of endeavor, Lang in analogous art discloses tracking a position of a pointer relative to the marker (performing a surgical step or surgical procedure with visual guidance using an optical head mounted display, Abstract, para [0004]-[0014], optical imaging and/or 3D scanning probe used with an arthroscopic procedure can optionally be tracked by tracking the position, location, orientation, alignment and/or direction of movement using optical markers, e.g. with one or more geometric patterns, e.g. in 2D or 3D, or LED's using one or more camera or video systems integrated into, attached to, or separate from one or more OHMDs - use image processing and pattern recognition to recognize the known geometry of the one or more probes and their location within a coordinate system, e.g. in relationship to the patient, the surgical site and/or the OR table, para [0258), pointer with an attached optical marker or an attached navigation marker, para [0355]-[0358], dimensions of the pointer have been previously scanned and registered with the OHMD. The image and/or video capture system attached to, integrated with or coupled to the OHMD can recognize the pointer in the live data and can identify the tip of the pointer. When the tip of the pointer touches the live landmark on the patient that corresponds to the landmark in the virtual data, the surgeon can, for example, click to indicate successful cross-referencing. The two data points can then optionally be fused or superimposed in a common coordinate system, para [0380], para [1042], para [1062]-[1065]).
It would have been obvious to one of ordinary skill in the art before the effective filing date to add Lang's Intra-operative measurements to KOSTRZEWSKI's method/system/augmented reality device using one or more cameras, an image capture system, a video capture and/or 3D scanner separate from an OHMD can be advantageous when measurements are desired to be obtained from a view angle other than the surgeon or, when multiple OHMDs are used, from a view angle other than of a surgical assistant or second surgeon (see para [1062]-[1065]).
Regarding claim 6, Kostrzewski discloses method of claim 5, further comprising: selecting the landmark based on a particular surgical procedure (augmented reality interaction via, e.g., a head-mounted display, para [0084]-[0094], During the surgery, the surgical robot 4 moves the end effector coupler 22, and the passive end effector and surgical saw attached there, automatically to a position close to a knee or other anatomical structure, so that all bone to be cut is within the workspace of the passive end effector. This position depends on the cut to be made and the surgery planning and implant construction, para [0114]-[0121]).
Regarding claim 7, Kostrzewski discloses method of claim 5, further comprising: prompting, by the augmented reality device, the identification of the landmark at least in part by instructing to have the pointer indicate the landmark (para [0074], computer platform 910 can use two DRAs to tracking patient anatomy position: one on patient tibia and one on patient femur. The platform 900 may use standard navigated instruments for the registration and checks (e.g. a pointer similar to the one used in Kostrzewski ExcelsiusGPS system for spine surgery), para [0084]-[0094], surgeon will review registration precision by touching selected points on bone with a stylus and verifying that on the navigation screen pointer tip is placed on the bone, para (0249]-[0250]).
Regarding claim 8, Kostrzewski doesn’t explicitly discloses method of claim 5, wherein detecting the identification of the landmark is responsive to slowed or stopped motion of the pointer for a predetermined period.
However, in the same field of endeavor, Lang discloses wherein detecting the identification of the landmark is responsive to slowed or stopped motion of the pointer for a predetermined period (image and/or video capture system attached to, integrated with or coupled to the OHMD can recognize the pointer in the live data and can identify the tip of the pointer. When the tip of the pointer touches the live landmark on the patient that corresponds to the landmark in the virtual data, the surgeon can, for example, click to indicate successful cross-referencing. The two data points can then optionally be fused or superimposed in a common coordinate system, para (0380], Move pointer (Slow, Fast), TABLE 3).
It would have been obvious to one of ordinary skill in the art before the effective filing date so that the tracking system gets multiple frames to average or confirm the pointers position.
Regarding claim 9, Kostrzewski discloses method of claim 5, further comprising: prompting an additional identification of an additional landmark by the augmented reality device instructing to have the pointer indicate the landmark; detecting the additional identification of the additional landmark based on the position of the pointer by the augmented reality device following a surgical intervention based on the surgical guidance, the additional identification including additional landmark position information of the additional landmark; determining a position of the additional landmark in the coordinate system; modifying the intra-operative plan based in part on the position of the additional landmark; and providing the surgical guidance in the coordinate system based in part on the modified intra-operative plan (para (0084]-[0094], the system can operate to automatically propose initial implant size and placement based on image analysis. In the next steps, the user can modify the implant size and placement and observe results of such modifications by updated visual feedback displayed on the medical images. When the user indicates acceptability of the plan, the plan becomes validated as approved for surgery, para [0204]-[0210], For X-ray and imageless workflows, only condylar surfaces are measured in addition to previously collected natural landmarks. This is to complete the virtual model of the patient obtained intra-operatively called "stickman" which is composed of natural landmarks and knee joint surfaces, para (0245]-[0255]).
Regarding claim 10, Kostrzewski discloses method of claim 9, further comprising: generating an anatomical model of the anatomy based on the marker and the landmark; updating the anatomical model based on the additional landmark; and updating the surgical guidance based on the updated anatomical model (para [0084]-(0094], 3D model of the femur and tibial bone can be generated - user can modify the implant size and placement and observe results of such modifications by updated visual feedback displayed on the medical images. When the user indicates acceptability of the plan, the plan becomes validated as approved for surgery, para (0204]-[0210], For X-ray and imageless workflows, only condylar surfaces are measured in addition to previously collected natural landmarks. This is to complete the virtual model of the patient obtained intra-operatively called "stickman" which is composed of natural landmarks and knee joint surfaces, para [0245) [0255]).
Regarding claim 11, Kostrzewski discloses method of claim 10, wherein updating the surgical guidance is at least one of modifying a pre-operative plan or developing an additional portion of the pre-operative plan to update the intra-operative plan based on the updated anatomical model (para [0009]-[0010], para (0084)-(0094], 3D model of the femur 'and tibial bone can be generated - use'r can modify the implant size and placement and observe results of such modifications by updated visual feedback displayed on the medical images. When the user indicates acceptability of the plan, the plan becomes validated as approved for surgery, para [0204]-[0210], para [0245)-(0255]).
Regarding claim 12, Kostrzewski discloses method of claim 5, wherein providing the surgical guidance comprises: tracking a current position of at least one of a cut plane, a guide, a surgical instrument, or a tool on a display; and displaying the current position and a target position of the at least one of the cut plane, the guide, the surgical instrument, or the tool on the display (computer platform 910 can generate a graphical representation of a patient's anatomical structure, surgical tool, user's hand, etc. with a displayed size, shape, color, and/or pose that is controlled based on the determined pose(s) of one or more the tracked DRAs, and which the graphical representation that is displayed can be dynamically modified to track changes in the determined poses over time, para (0082)).
Regarding claim 13, Kostrzewski discloses method of claim 12, wherein tracking the current position comprises: detecting fiducials arranged in a predetermined pattern and attached to the at least one of the guide, the surgical instrument, or the tool by a depth camera of the augmented reality device; identifying positions of the fiducials arranged in the predetermined pattern; and determining position information of the at least one of the guide, the surgical instrument, or the tool based on the positions of the fiducials (reference arrays enable tracking by reflecting light in known patterns, which are decoded to determine their respective poses by the tracking subsystem of the surgical robot 4. If the line-of-sight between the patient reference array 602 and the tracking cameras 46 in the auxiliary tracking bar is blocked (for example, by a medical personnel, instrument, etc.), further navigation of the surgical instrument may not be able to be performed and a responsive notification may temporarily halt further movement of the robot arm 20 and surgical robot 4, display a warning on the display 34, and/or provide an audible warning to medical personnel, para [0065]).
Regarding claim 14, Kostrzewski discloses method of claim 12, wherein providing the surgical guidance further comprises displaying at least one of a distance between the current position and the target position, or a direction to move the guide, the surgical instrument, or the tool (system indicates to surgeon which navigated jig to use and shows target. position. Surgeon fixes the jig to bone in location indicated by the system (e.g. system indicates target position in green if navigated jigs is within certain maximal distance from it) and cuts manually using sagittal saw through the jig, para [0276]-[0278]).
Regarding claim 15, Kostrzewski doesn’t explicitly discloses method of claim 5, further comprising: obtaining marker position information of the marker and the landmark position information using data from a depth camera of the augmented reality device.
However, in the same field of endeavor, Lang discloses obtaining marker position information of the marker and the landmark position information using data from a depth camera of the augmented reality device (depth camera with a 120 degrees x 120 degrees angle of view, para (0179], para (0258], pointer with an attached optical marker or an attached navigation marker, para [0355]-[0358], para [0380], para [1042]. Para. [1062]- [1065]).
It would have been obvious to one of ordinary skill in the art before the effective filing date in order so that the system can precisely measure how far objects are from the camera and make tracking more robust and reduce errors due to flat images.
Regarding claim 16, Kostrzewski discloses method of claim 15, wherein the fiducials are first fiducials, the method further comprising: detecting the first fiducials arranged in a predetermined pattern by the depth camera; identifying positions of the first fiducials; determining the marker position information of the marker based on the positions of the first fiducials; detecting second fiducials arranged in a predetermined pattern and attached to the pointer by the depth camera; identifying positions of the second fiducials; and determining a location of the pointer relative to each marker based on the positions of the second fiducials (reference arrays enable tracking by reflecting light in known patterns, which are decoded to determine their respective poses by the tracking subsystem of the surgical robot 4. If the line-of-sight between the patient reference array 602 and the tracking cameras 46 in the auxiliary tracking bar is blocked (for example, by a medical personnel, instrument, etc.), further navigation of the surgical instrument may not be able to be performed and a responsive notification may temporarily halt further movement of the robot arm 20 and surgical robot 4, display a warning on the display 34, and/or provide an audible warning to medical personnel, para [0065]. para [0084]-[0094]).
Regarding claim 17, Kostrzewski discloses method of claim 5, further comprising: calculating at least one axis or plane based on the determined position of the landmark; and providing the surgical guidance on the augmented reality device based on the at least one axis or plane (controller is configured to obtain the surgical plan from the surgical guidance system, determine a pose of a target plane based on the surgical plan defining where the anatomical structure is to be cut and based on the pose of the anatomical structure, and generate steering information based on comparison of the pose of the target plane and the pose of the surgical saw, wherein the steering information indicates where the surgical saw needs to be moved to position a cutting plane of the surgical saw to become aligned with the target plane, para [0009]-[0010]).
Regarding claim 18, Kostrzewski discloses method of claim 17, further comprising displaying the at least one axis or plane superimposed on the anatomy in augmented reality by the augmented reality device (controller is configured to obtain the surgical plan from the surgical guidance system, determine a pose of a target plane based on the surgical plan defining where the anatomical structure is to be cut and based on the pose of the anatomical structure, and generate steering information based on comparison of the pose of the target plane and the pose of the surgical saw, wherein the steering information indicates where the surgical saw needs to be moved to position a cutting plane of the surgical saw to become aligned with the target plane, para [0009]-[0010], graphical representations of the poses of the resection planes displayed though the display device within an extended reality (XR) headset 920 (FIG. 9) as an overlay on the defined patient, para [0173]-[0180], deviation of actual cutting plane from planned; deviation of predicted gap from actual gap, para [0187]-[0191], para [0246]-[0255]).
Regarding claim 19, Kostrzewski discloses system comprising: a marker configured to be affixed to anatomy (determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, para [0114]-[0118]); a pointer configured to point a landmark (surgeon will review registration precision by touching selected points on bone with a stylus and verifying that on the navigation screen pointer tip is placed on the bone, para [0249]-[0250]); and an augmented reality device (augmented reality interaction via, e.g., a head-mounted display, para [0084]-[0094]) configured to: detect the marker (para [0084]-[0094], determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, para [0114]-[0118]); create a coordinate system based in part on fiducials attached to the marker ("pose" refers to the position and/or the rotational angle of one object (e.g., dynamic reference array, end effector, surgical tool, anatomical structure, etc.) relative to another object and/or to a defined coordinate system, para [0042], determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, and can be configured to determine a pose of the surgical saw based on determining a pose of tracking markers connected to at least one of the surgical saw and the passive end effector, para [0114]-[0118]); prompt an identification of the landmark by instructing to have the pointer indicate the landmark (Medical personnel may be prompted by surgical robot 4 to select a function, mode, and/or asses the condition of surgical system 2. Depressing secondary button a single time may activate certain functions, modes, and/or acknowledge information communicated to medical personnel through the XR headset(s) 920, display 34 and/or light indicator 28, para [0074], surgeon will review registration precision by touching selected points on bone with a stylus and verifying that on the navigation screen pointer tip is placed on the bone, para [0249]-[0250]); detect the identification of the landmark responsive to an indication of the landmark (para [0114]-[0118], graphical representations of the poses of the resection planes displayed though the display device within an extended reality (XR) headset 920 (FIG. 9) as an overlay on the defined patient - control a sequence of movements of a surgical saw attached to an arm of the surgical robot so a cutting plane of the surgical saw (e.g., surgical saw 1040 in FIG. 10 or FIG. 11) becomes sequentially aligned with the poses of the resection planes, para [0173]-[0180]); determine a position of the landmark in the coordinate system ("pose" refers to the position and/or the rotational angle of one object (e.g., dynamic reference array, end effector, surgical tool, anatomical structure, etc.) relative to another object and/or to a defined coordinate system, para [0042], determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, and can be configured to determine a pose of the surgical saw based on determining a pose of tracking markers connected to at least one of the surgical saw and the passive end effector, para [0114] [01181); generate an intra-operative plan based on the position of the landmark (reflective fiducials (also called markers, para [0080], para [0084]-[0094], para [0114]-[0118], FIG. 12, three stages of workflow are illustrated: pre-operative stage 1200; intra-operative stage 1202; and post-operative stage 1204: During the pre-operative stage 1200, a user (e.g., surgeon) generates a surgical plan (case) based on analyzed patient images with assistance from the surgical guidance system 1220, para [0160], para [0173]-[0180]); and provide surgical guidance using the augmented reality device based on the intra-operative plan (ML guidance system 1220 (FIG. 12) provides guidance to a user during pre-operative planning and during intraoperative surgical execution of the surgical plan. The ML guidance system enables automatic or semi-automatic (involving human input) selection of implant(s) and generation of the surgical plan - augmented reality interaction via, e.g., a head-mounted display, para [0084]-[0094]).
Regarding claim 20, Kostrzewski discloses system of claim 19, wherein the pointer comprises a stylus configured to point the landmark (surgeon will review registration precision by touching selected points on bone with a stylus and verifying that on the navigation screen pointer tip is placed on the bone, para [0245]-[0255]).
Regarding claim(s) 21, this/these system claim(s) has/have similar limitations as method claim(s) 9, and therefore rejected on similar grounds.
Regarding claim 22, Kostrzewski doesn’t explicitly discloses system of claim 19, further comprising fiducials attached to the pointer, wherein the augmented reality device is configured to determine the position of the landmark based in part on relative positions of the fiducials attached to the pointer and the position of the marker.
However, in the same field of endeavor, Lang discloses fiducials attached to the pointer, wherein the augmented reality device is configured to determine the position of the landmark based in part on relative positions of the fiducials attached to the pointer and the position of the marker (Abstract, para [0004]-[0014]. para [0258], pointer with an attached optical marker or an attached navigation marker, para [0355]-[0358], para [0380], para [1042], para [1062]-[1065]).
It would have been obvious to one of ordinary skill in the art before the effective filing date in order to enhanced positional accuracy for the augmented reality system.
Regarding claim 23, Kostrzewski doesn’t explicitly discloses system of claim 19, wherein the augmented reality device comprises a depth camera configured to receive a depth image that includes a representation of fiducials relative to the device, and wherein the augmented reality device is configured to determine the position of the landmark based on data received from the depth camera.
However, in the same field of endeavor, Lang discloses wherein the augmented reality device comprises a depth camera configured to receive a depth image that includes a representation of fiducials relative to the device, and wherein the augmented reality device is configured to determine the position of the landmark based on data received from the depth camera (depth camera with a 120 degrees x120 degrees angle of view, para [0179]. use image processing and pattern recognition to recognize the known geometry of the one or more probes and their location within a coordinate system, e.g. in relationship to the patient, the surgical site and/or the OR table, para [0258], pointer with an attached optical marker or an attached navigation marker, para (0355]-[0358], dimensions of the pointer have been previously scanned and registered with the OHMD. The image and/or video capture system attached to, integrated with or coupled to the OHMD can recognize the pointer in the live data and can identify the tip of the pointer. When the tip of the pointer touches the live landmark on the patient that corresponds to the landmark in the virtual data, the surgeon can, for example, click to indicate successful cross-referencing. The two data points can then optionally be fused or superimposed in a common coordinate system, para [0380], para [1042], para [1062] [1065]).
It would have been obvious to one of ordinary skill in the art before the effective filing date in order so that the system can precisely measure how far objects are from the camera and make tracking more robust and reduce errors due to flat images.
Regarding claim 24, Kostrzewski discloses system of claim 19, wherein the augmented reality device is further configured to: calculate at least one axis or plane based on the determined position of the landmark; and provide the surgical guidance on the augmented reality device based on the at least one axis or plane (controller is configured to obtain the surgical plan from the surgical guidance system, determine a pose of a target plane based on the surgical plan defining where the anatomical structure is to be cut and based on the pose of the anatomical structure, and generate steering information based on comparison of the pose of the target plane and the pose of the surgical saw, wherein the steering information indicates where the surgical saw needs to be moved to position a cutting plane of the surgical saw to become aligned with the target plane, para [0009]-[0010]).
Regarding claim 25, Kostrzewski discloses system of claim 24, wherein the augmented reality device is configured to display the at least one axis or plane by superimposing the at least one axis or plane on the anatomy (controller is configured to obtain the surgical plan from the surgical guidance system, determine a pose of a target plane based on the surgical plan defining where the anatomical structure is to be cut and based on the pose of the anatomical structure, and generate steering information based on comparison of the pose of the target plane and the pose of the surgical saw, wherein the steering information indicates where the surgical saw needs to be moved to position a cutting plane of the surgical saw to become aligned with the target plane, para [0009]-[0010], graphical representations of the poses of the resection planes displayed though the display device within an extended reality (XR) headset 920 (FIG. 9) as an overlay on the defined patient, para [0173]-[0180], deviation of actual cutting plane from planned; deviation of predicted gap from actual gap, para [0187]-[0191], para [0246]-[0255]).
Regarding claim 26, Kostrzewski discloses augmented reality device comprising: a processor; and a non-transitory computer readable medium comprising instructions that, when executed, cause the processor to perform operations comprising (surgical guidance system is disclosed for computer assisted navigation during surgery - obtain pre-operative data from one of the distributed network computers characterizing a defined patient, and generate a surgical plan for the defined patient based on processing the pre operative data through the machine learning model, Abstract, para [0009]-[0010], para [0080]-[0094], head-mounted display devices that can be worn by a surgeon - determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, para [0114]-[0118], graphical representations of the poses of the resection planes displayed though the display device within an extended reality (XR) headset 920 (FIG. 9) as an overlay on the defined patient, para [0173]-[0180], deviation of actual cutting plane from planned; deviation of predicted gap from actual gap, para [0187] [0191], para [0246]-[0255]):
detecting a marker affixed to anatomy (para [0084]-[0094], determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, para [0114]-[0118]);
creating a coordinate system based in part on fiducials attached to the marker ("pose" refers to the position and/or the rotational angle of one object (e.g., dynamic reference array, end effector, surgical tool, anatomical structure, etc.) relative to another object and/or to a defined coordinate system, para [0042], determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, and can be configured to determine a pose of the surgical saw based on determining a pose of tracking markers connected to at least one of the surgical saw and the passive end effector, para [0114]-[0118]);
prompting an identification of a landmark by an augmented reality device instructing to have a pointer indicate the landmark (Medical personnel may be prompted by surgical robot 4 to select a function, mode, and/or asses the condition of surgical system 2. Depressing secondary button a single time may activate certain functions, modes, and/or acknowledge information communicated to medical personnel through the XR headset(s) 920, display 34 and/or light indicator 28, para [0074], surgeon will review registration precision by touching selected points on bone with a stylus and verifying that on the navigation screen pointer tip is placed on the bone, para [0249]-[0250]);
detecting the identification of the landmark based on a position of the pointer, the identification including landmark position information of the landmark (para [0074], computer platform 910 can use two DRAs to tracking patient anatomy position: one on patient tibia and one on patient femur. The platform 900 may use standard navigated instruments for the registration and checks (e.g. a pointer similar to the one used in Kostrzewski ExcelsiusGPS system for spine surgery), para [0084]-[0094], surgeon will review registration precision by touching selected points on bone with a stylus and verifying that on the navigation screen pointer tip is placed on the bone, para (0249]-[0250]);
determining a position of the landmark in the coordinate system ("pose" refers to the position and/or the rotational angle of one object (e.g., dynamic reference array, end effector, surgical tool, anatomical structure, etc.) relative to another object and/or to a defined coordinate system, para [0042], determine the pose of the anatomical structure that is to be cut by the saw blade based on determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure, and can be configured to determine a pose of the surgical saw based on determining a pose of tracking markers connected to at least one of the surgical saw and the passive end effector, para [0114] [01181);
generating an intra-operative plan based on the position of the landmark (reflective fiducials (also called markers, para [0080], para [0084]-[0094], para [0114]-[0118], FIG. 12, three stages of workflow are illustrated: pre-operative stage 1200; intra-operative stage 1202; and post-operative stage 1204: During the pre-operative stage 1200, a user (e.g., surgeon) generates a surgical plan (case) based on analyzed patient images with assistance from the surgical guidance system 1220, para [0160], para [0173]-[0180]); and providing surgical guidance based on the intra-operative plan (ML guidance system 1220 (FIG. 12) provides guidance to a user during pre-operative planning and during intra operative surgical execution of the surgical plan. The ML guidance system enables automatic or semi-automatic (involving human input) selection of implant(s) and generation of the surgical plan - augmented reality interaction via, e.g., a head-mounted display, para [0084] [0094]).
Regarding claim 27, Kostrzewski discloses augmented reality device of claim 26, wherein the operations further comprise: prompting an additional identification of an additional landmark by the augmented reality device instructing to have the pointer indicate the landmark; detecting the additional identification of the additional landmark based on the position of the pointer following a surgical intervention based on the surgical guidance, the additional identification including additional landmark position information of the additional landmark; determining a position of the additional landmark in the coordinate system; modifying the intra-operative plan based in part on the position of the additional landmark; and providing the surgical guidance in the coordinate system based in part on the modified intra-operative plan (para (0084]-[0094], the system can operate to automatically propose initial implant size and placement based on image analysis. In the next steps, the user can modify the implant size and placement and observe results of such modifications by updated visual feedback displayed on the medical images. When the user indicates acceptability of the plan, the plan becomes validated as approved for surgery, para [0204]-[0210], For X-ray and imageless workflows, only condylar surfaces are measured in addition to previously collected natural landmarks. This is to complete the virtual model of the patient obtained intra-operatively called "stickman" which is composed of natural landmarks and knee joint surfaces, para (0245]-[0255]).
Regarding claim 28, Kostrzewski discloses augmented reality device of claim 26, wherein the operations further comprise selecting the landmark based on a surgical procedure (augmented reality interaction via, e.g., a head-mounted display, para [0084]-[0094], During the surgery, the surgical robot 4 moves the end effector coupler 22, and the passive end effector and surgical saw attached there, automatically to a position close to a knee or other anatomical structure, so that all bone to be cut is within the workspace of the passive end effector. This position depends on the cut to be made and the surgery planning and implant construction, para [0114]-[0121]).
Regarding claim 29, Kostrzewski doesn’t explicitly discloses augmented reality device of claim 26, further comprising a depth camera, wherein the operations further comprise obtaining marker position information of the marker and the landmark position information using data from the depth camera
However, in the same field of endeavor, Lang discloses obtaining marker position information of the marker and the landmark position information using data from a depth camera of the augmented reality device (depth camera with a 120 degrees x 120 degrees angle of view, para (0179], para (0258], pointer with an attached optical marker or an attached navigation marker, para [0355]-[0358], para [0380], para [1042]. Para. [1062]- [1065]).
It would have been obvious to one of ordinary skill in the art before the effective filing date in order so that the system can precisely measure how far objects are from the camera and make tracking more robust and reduce errors due to flat images.
Regarding claim 30, Kostrzewski discloses augmented reality device of claim 29, wherein the fiducials are first fiducials, and wherein the operations further comprise: detecting the first fiducials arranged in a predetermined pattern by the depth camera; identifying positions of the first fiducials; determining the marker position information of the marker based on the positions of the first fiducials; detecting second fiducials arranged in a predetermined pattern and attached to the pointer by the depth camera; identifying positions of the second fiducials; and determining a position of the pointer relative to each marker based on the positions of the second fiducials (reference arrays enable tracking by reflecting light in known patterns, which are decoded to determine their respective poses by the tracking subsystem of the surgical robot 4. If the line-of-sight between the patient reference array 602 and the tracking cameras 46 in the auxiliary tracking bar is blocked (for example, by a medical personnel, instrument, etc.), further navigation of the surgical instrument may not be able to be performed and a responsive notification may temporarily halt further movement of the robot arm 20 and surgical robot 4, display a warning on the display 34, and/or provide an audible warning to medical personnel, para [0065]. para [0084]-[0094]).
Regarding claim 31, Kostrzewski discloses augmented reality device of claim 26, wherein the operations further comprise: calculating at least one axis or plane based on the determined position of the landmark; and providing the surgical guidance on the augmented reality device based on the at least one axis or plane (controller is configured to obtain the surgical plan from the surgical guidance system, determine a pose of a target plane based on the surgical plan defining where the anatomical structure is to be cut and based on the pose of the anatomical structure, and generate steering information based on comparison of the pose of the target plane and the pose of the surgical saw, wherein the steering information indicates where the surgical saw needs to be moved to position a cutting plane of the surgical saw to become aligned with the target plane, para [0009]-[0010]).
Regarding claim 32, Kostrzewski discloses augmented reality device of claim 31, wherein the operations further comprise displaying the at least one axis or plane superimposed on the anatomy in augmented reality by the augmented reality device (controller is configured to obtain the surgical plan from the surgical guidance system, determine a pose of a target plane based on the surgical plan defining where the anatomical structure is to be cut and based on the pose of the anatomical structure, and generate steering information based on comparison of the pose of the target plane and the pose of the surgical saw, wherein the steering information indicates where the surgical saw needs to be moved to position a cutting plane of the surgical saw to become aligned with the target plane, para [0009]-[0010], graphical representations of the poses of the resection planes displayed though the display device within an extended reality (XR) headset 920 (FIG. 9) as an overlay on the defined patient, para [0173]-[0180], deviation of actual cutting plane from planned; deviation of predicted gap from actual gap, para [0187]-[0191], para [0246]-[0255]).
Regarding claim 35, Kostrzewski doesn’t disclose the method of claim 1, wherein said generating the surgical plan comprises generating the surgical plan intra-operatively from the anatomical landmark positions obtained intra-operatively.
However, in the same field of endeavor, Lang discloses wherein said generating the surgical plan comprises generating the surgical plan intra-operatively from the anatomical landmark positions obtained intra-operatively (obtaining one or more intra-operative measurements from the patient's physical joint to determine one or more intra-operative coordinates, registering the one or more intra-operative coordinates from the patient's physical joint in the coordinate system, generating a virtual surgical guide, determining a predetermined position and/or orientation of the virtual surgical guide based on the one or more intra-operative measurements, displaying and superimposing the virtual surgical guide, using the one or more optical head mounted displays, para [0109]).
It would have been obvious to one of ordinary skill in the art before the effective filing date in order to match the plan to the actual anatomy at the time of surgery.
Regarding claim 36, Kostrzewski doesn’t disclose the method of claim 5, wherein the intra-operative plan is generated from the landmark position in the coordinate system during the surgical procedure.
However, in the same field of endeavor, wherein the intra-operative plan is generated from the landmark position in the coordinate system during the surgical procedure (obtaining one or more intra-operative measurements from the patient's physical joint to determine one or more intra-operative coordinates, registering the one or more intra-operative coordinates from the patient's physical joint in the coordinate system, generating a virtual surgical guide, determining a predetermined position and/or orientation of the virtual surgical guide based on the one or more intra-operative measurements, displaying and superimposing the virtual surgical guide, using the one or more optical head mounted displays, para [0109]).
It would have been obvious to one of ordinary skill in the art before the effective filing date in order to match the plan to the actual anatomy at the time of surgery.
Response to Arguments
7. Applicant's arguments filed 04/03/2026 have been fully considered but they are not persuasive.
I. With regards to arguments for independent claim 1, applicant argues Kostrzewski fails to disclose a “pointer” as recited in combination of applicant’s claim 1. Notably, applicant argues “Kostrzewski refers in [0117] to determining a power of an anatomical structure by ‘determining a pose of tracking markers, e.g., DRAs, that are attached to the anatomical structure.’” However, examiner respectfully disagrees.
Examiner interprets the surgical tool such as a surgical saw as a “pointer” of the recited claims. Kostrzewski discloses DRAs can be markers or other indicia which is attached or formed on a surgical tool and can include fiducials. See [0054, in part, below.
[0054] …In some embodiments, a dynamic reference array 52 is attached to the passive end effector 1000, e.g., to the tool attachment mechanism, and/or is attached to the surgical saw 1040. Dynamic reference arrays, also referred to as “DRAs” and “reference arrays” herein, can be rigid bodies, markers, or other indicia which may be attached or formed on one or more XR headsets being worn by personnel in the operating room, the end effector, the surgical robot, a surgical tool in a navigated surgical procedure, and an anatomical structure (e.g., bone) of a patient. The computer platform 910 in combination with the camera tracking system component 6 or other 3D localization system are configured to track in real-time the pose (e.g., positions and rotational orientations) of the DRA. The DRA can include fiducials, such as the illustrated arrangement of balls. This tracking of 3D coordinates of the DRA can allow the surgical system 2 to determine the pose of the DRA in any multidimensional space in relation to the target anatomical structure of the patient 50 in FIG. 5. Emphasis added.
Kosterzewski further discloses receiving pose information for an anatomical structure based on a pose of tracked DRAs which are markers attached or formed on the surgical saw. See [0082 in part below.
[0082] The computer platform 910 can also include a navigation controller that is configured to use the determined poses to provide navigation information to users that guides their movement of tracked tools relative to position-registered patient images and/or tracked anatomical structures during a planned surgical procedure… For example, as will be explained below the computer platform 910 can generate a graphical representation of a patient's anatomical structure, surgical tool, user's hand, etc. with a displayed size, shape, color, and/or pose that is controlled based on the determined pose(s) of one or more the tracked DRAs, and which the graphical representation that is displayed can be dynamically modified to track changes in the determined poses over time. Emphasis added.
Thus, Kostrzewwski establishes “receiving position information for anatomical landmarks based on a position of a pointer having fiducials,” as recited in claim 1.
II. With regards to arguments for independent claim 1, applicant further argues Kostrzewski fails to disclose “generating a surgical plan based on the position information,” as recited in claim 1. Specifically, applicant argues “Kostrzewski instead, at [0174] states that a surgical plan may be generated based on processing pre-operative data. The pre-operative data of Kostrzewski may include anatomical landmark locations of the defined patient. See [0179]. However, this pre-operative anatomical landmark locations would not be ‘position information’ that was ‘based on a position of a pointer having fiducials’ as recited in the combination of claim 1.” However, examiner respectfully disagrees.
As established above, the position information of anatomical landmarks is based on the position of a tracked DRAs having fiducials. See [0054 and [0082 above. Therefore, Kostrzewwski does disclose processing pre-operative data to generate the surgical plan is based on the position information of anatomical landmarks, as argued.
III. With regards to arguments for independent claim 5, applicant further argues Kostrzewski in view of Lang fails to disclose “tracking the position of a pointer relative to the marker” as recited in claim 5. However, examiner respectfully disagrees.
[0342 and [0380 of Lang discloses cross-referencing the location of the tip of a pointer relative to an anatomical landmark. Specifically, [0380 of Lang discloses “when the tip of the pointer touches the live landmark on the patient that corresponds to the landmark in the virtual data, the surgeon can, for example, click to indicate successful cross-referencing.” [0342 of Lang discloses “surgeon can then fixate optical markers to the virtual point and the underlying or corresponding anatomic landmark, for example using a screw or pin.” Thus, Lang as modified by Kostrzewski discloses “tracking the position of a pointer relative to the marker”, as recited in claim 5.
Moreover, even [0125 of Kostrzewski discloses when the position and orientation of robot arm 22 distal end position and orientation with respect to the bone is known, the position and orientation of the saw blade 1042 with respect to the bone can be computed and displayed as feedback to the surgeon. The position with respect to the bone corresponding to the marker, as disclosed in [0054 of Kostrzewski.
IV. With regards to arguments for independent claim 5, applicant further argues Lang as modified by Kostrzewski fails to disclose “generating an intra-operative plan based on the position of the landmark,” as recited in claim 5. However, examiner respectfully disagrees.
As established above, the position information of anatomical landmarks is based on the position of a tracked DRAs having fiducials. See [0054 and [0082 above. Therefore, Kostrzewwski does disclose processing pre-operative data to generate the surgical plan is based on the position information of anatomical landmarks, as argued.
V. With regards to arguments for independent claims 19 and 26, applicant further argues Kostrzewski fails to disclose “prompt an identification of the landmark by instructing to have the pointer indicate the landmark; detect the identification of the landmark responsive to an indication of the landmark” as recited in claim 19 and similarly in claim 26. However, examiner respectfully disagrees.
Kosterzewski discloses generating a graphical representation of an anatomical structure with a displayed size, shape, color that is controlled based on the determined pose of one or more tracked DRAs inclusive of the surgical saw. See [0082 in part below.
[0082] The computer platform 910 can also include a navigation controller that is configured to use the determined poses to provide navigation information to users that guides their movement of tracked tools relative to position-registered patient images and/or tracked anatomical structures during a planned surgical procedure… For example, as will be explained below the computer platform 910 can generate a graphical representation of a patient's anatomical structure, surgical tool, user's hand, etc. with a displayed size, shape, color, and/or pose that is controlled based on the determined pose(s) of one or more the tracked DRAs, and which the graphical representation that is displayed can be dynamically modified to track changes in the determined poses over time. Emphasis added.
Thus, Kostrzewwski establishes “prompt an identification of the landmark by instructing to have the pointer indicate the landmark; detect the identification of the landmark responsive to an indication of the landmark” as recited in claim 19.
VI. With regards to arguments for dependent claims examiner maintains Kostrzewski discloses each and every element of independent claims 1-4 and Kostrzewski in view of Lang discloses each and every element of independent claims 5-36, as elaborated above. Accordingly, the dependent claims are also rejected, at least, by the virtue their dependency on respectively rejected base claims.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PRIYANK J SHAH whose telephone number is (571)270-3732. The examiner can normally be reached on 10:00 - 6:00 M-F.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ghebretinsae, Temesghen can be reached on (571) 272-3017. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PRIYANK J SHAH/Primary Examiner, Art Unit 2626
Prosecution Insights