DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Healy et al. (US 2022/0313386 A1) in view of Ballagas et al. (US 2021/0042979 A1) and Hubert et al. (US 2018/0308248 A1).
As to claim 1, Healy teaches a surgical system comprising:
a head-mounted device (HMD) comprising: a controller, and an interior-facing display configured to face the eyes of a wearer of the HMD and to present a virtual object in conjunction with a real-world view (see at least [0035]: “An extended reality (XR) headset is operatively connected to the surgical system… The XR headset may be configured to augment a real-world scene with computer generated XR images. The XR headset may be configured to provide an augmented reality (AR) viewing environment by displaying the computer generated XR images on a see-through display screen that allows light from the real-world scene to pass therethrough for combined viewing by the user.”; [0099]: “The display screen 1302 is semi-transparent and semi-reflective allowing the user to see reflected virtual content superimposed on the user’s view of a real-world scene.”; [0100]: “the surgical system includes an XR headset 920 and an XR headset controller… The XR headset controller… is further configured to generate the XR image… for display on the see-through display screen 1302.”); and
a tracking system comprising a camera being spatially separated from the HMD (see at least [0038]: “surgical system 2 includes a surgical robot 4 and a camera tracking system component 6.”; [0039]: “the surgical system 2 may be physically separated into two separate and distinct systems, the surgical robot 4 and the camera tracking system component 6.”),
the tracking system being configured to localize the HMD in a coordinate system of the camera (see at least [0098]: “The DRA on the headset enables the tracking cameras on the auxiliary tracking bar to track pose of the headset 920.”; [0130]: “Using both visible and near infrared tracking coordinate systems can enable… naturally track an XR headset in the same coordinate system as the navigated surgical instruments.”).
Healy does not directly teach an exterior-facing display configured to present a trackable graphic that is computer-generated; and the tracking system being configured to detect the trackable graphic with the camera.
Ballagas teaches an exterior-facing display configured to present a trackable graphic that is computer-generated (see at least [0012]: “the HMD may include a second display that displays information outward, away from a user of the HMD.”; [0013]: “the HMD 100 may include… a first display 106… a second display 108…”; [0017]: “the first display 106 may be an inward facing display.”; [0026]: “The processor 102 may generate an animation, or a graphical representation.”; [0031]: “The animation… may then be displayed on the second display 108. The second display 108 may be an outward facing display…”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Healy’s XR headset to include the outward-facing second display taught by Ballagas so that information generated by the HMD controller could be presented externally. Doing so would have allowed the headset to communicate computer-generated visual information to persons and imaging systems external to the wearer while merely incorporating one known display arrangement into another known HMD, yielding the predictable result of providing both inward- and outward-facing displays without changing the fundamental operation of Healy’s surgical navigation system.
Ballagas does not directly teach the tracking system being configured to detect the trackable graphic with the camera.
Hubert teaches the tracking system being configured to detect the trackable graphic with the camera (see at least [0097]: “The marker manager device 3 manages at least one fiducial marker and the display thereof on the display device 2.”; [0100]: “The tracking module detects the displayed fiducial marker… and deduces the pose…”; [0102]: “dynamic displayed marker… associated with at least one modifiable attribute.”; [0112]-[0116]: dynamic modification of marker attributes including position, size, color, luminance, contrast, orientation; [0128]: “computes pose information representative of a pose of the portable device with regard to the displayed fiducial marker in the real world.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify Healy’s surgical XR headset, as modified by Ballagas, to display Hubert’s computer-generated fiducial marker on the outward-facing display so that the marker could be detected by Healy’s spatially separated tracking cameras. The motivation would have been to replace or supplement Healy’s attached physical DRA fiducials with a programmable visual tracking target that can be dynamically adapted to improve tracking performance. One of ordinary skill in the art would therefore have recognized that incorporating Hubert’s dynamically displayed fiducial marker into Ballagas’s outward-facing display on Healy’s HMD would have predictably improved visibility, detection reliability, and tracking robustness while allowing the existing camera tracking system to localize the HMD in the camera coordinate system.
As to claim 19, Healy teaches a head-mounted device (HMD) for use with a tracking system that includes a camera and is spatially separated from the HMD (see at least [0035]: “An extended reality (XR) headset is operatively connected to the surgical system… The XR headset may be configured to augment a real-world scene with computer generated XR images. The XR headset may be configured to provide an augmented reality (AR) viewing environment by displaying the computer generated XR images on a see-through display screen that allows light from the real-world scene to pass therethrough for combined viewing by the user.”; [0038]: “surgical system 2 includes a surgical robot 4 and a camera tracking system component 6.”; [0039]: “the surgical system 2 may be physically separated into two separate and distinct systems, the surgical robot 4 and the camera tracking system component 6.”), the HMD comprising:
a controller; an interior-facing display configured to face the eyes of a wearer of the HMD (see at least [0035]: “An extended reality (XR) headset is operatively connected to the surgical system… The XR headset may be configured to augment a real-world scene with computer generated XR images. The XR headset may be configured to provide an augmented reality (AR) viewing environment by displaying the computer generated XR images on a see-through display screen that allows light from the real-world scene to pass therethrough for combined viewing by the user.”; [0099]: “The display screen 1302 is semi-transparent and semi-reflective allowing the user to see reflected virtual content superimposed on the user’s view of a real-world scene.”; [0100]: “the surgical system includes an XR headset 920 and an XR headset controller… The XR headset controller… is further configured to generate the XR image… for display on the see-through display screen 1302.”).
Healy does not directly teach an exterior-facing display; wherein the controller is configured to: obtain or generate a trackable graphic that is computer-generated; and present the trackable graphic on the exterior-facing display for detection by the camera of the tracking system.
Ballagas teaches an exterior-facing display; wherein the controller is configured to: obtain or generate a trackable graphic that is computer-generated (see at least [0012]: “the HMD may include a second display that displays information outward, away from a user of the HMD.”; [0013]: “the HMD 100 may include… a first display 106… a second display 108…”; [0017]: “the first display 106 may be an inward facing display.”; [0026]: “The processor 102 may generate an animation, or a graphical representation.”; [0031]: “The animation… may then be displayed on the second display 108. The second display 108 may be an outward facing display…”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Healy’s XR headset to include the outward-facing second display taught by Ballagas so that information generated by the HMD controller could be presented externally. Doing so would have allowed the headset to communicate computer-generated visual information to persons and imaging systems external to the wearer while merely incorporating one known display arrangement into another known HMD, yielding the predictable result of providing both inward- and outward-facing displays without changing the fundamental operation of Healy’s surgical navigation system.
Ballagas does not directly teach wherein the controller is configured to: present the trackable graphic on the exterior-facing display for detection by the camera of the tracking system.
Hubert teaches wherein the controller is configured to: present the trackable graphic on the exterior-facing display for detection by the camera of the tracking system (see at least [0097]: “The marker manager device 3 manages at least one fiducial marker and the display thereof on the display device 2.”; [0100]: “The tracking module detects the displayed fiducial marker… and deduces the pose…”; [0102]: “dynamic displayed marker… associated with at least one modifiable attribute.”; [0112]-[0116]: dynamic modification of marker attributes including position, size, color, luminance, contrast, orientation; [0128]: “computes pose information representative of a pose of the portable device with regard to the displayed fiducial marker in the real world.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify Healy’s surgical XR headset, as modified by Ballagas, to display Hubert’s computer-generated fiducial marker on the outward-facing display so that the marker could be detected by Healy’s spatially separated tracking cameras. The motivation would have been to replace or supplement Healy’s attached physical DRA fiducials with a programmable visual tracking target that can be dynamically adapted to improve tracking performance. One of ordinary skill in the art would therefore have recognized that incorporating Hubert’s dynamically displayed fiducial marker into Ballagas’s outward-facing display on Healy’s HMD would have predictably improved visibility, detection reliability, and tracking robustness while allowing the existing camera tracking system to localize the HMD in the camera coordinate system.
As to claim 20, Healy teaches a method of operating a surgical system including a head-mounted device (HMD) with a controller, an interior-facing display configured to face the eyes of a wearer of the HMD (see at least [0035]: “An extended reality (XR) headset is operatively connected to the surgical system… The XR headset may be configured to augment a real-world scene with computer generated XR images. The XR headset may be configured to provide an augmented reality (AR) viewing environment by displaying the computer generated XR images on a see-through display screen that allows light from the real-world scene to pass therethrough for combined viewing by the user.”; [0099]: “The display screen 1302 is semi-transparent and semi-reflective allowing the user to see reflected virtual content superimposed on the user’s view of a real-world scene.”; [0100]: “the surgical system includes an XR headset 920 and an XR headset controller… The XR headset controller… is further configured to generate the XR image… for display on the see-through display screen 1302.”); and
a tracking system including a camera being spatially separated from the HMD (see at least [0038]: “surgical system 2 includes a surgical robot 4 and a camera tracking system component 6.”; [0039]: “the surgical system 2 may be physically separated into two separate and distinct systems, the surgical robot 4 and the camera tracking system component 6.”),
the method comprising:
localizing, with the tracking system, the HMD in a coordinate system of the camera (see at least [0098]: “The DRA on the headset enables the tracking cameras on the auxiliary tracking bar to track pose of the headset 920.”; [0130]: “Using both visible and near infrared tracking coordinate systems can enable… naturally track an XR headset in the same coordinate system as the navigated surgical instruments.”).
Healy does not directly teach an exterior-facing display; and the method comprising: obtaining or generating, with the controller of the HMD, a trackable graphic that is computer-generated; presenting, with the controller of the HMD, the trackable graphic on the exterior-facing display; detecting the trackable graphic with the camera of the tracking system.
Ballagas teaches an exterior-facing display; and the method comprising: obtaining or generating, with the controller of the HMD, a trackable graphic that is computer-generated; presenting, with the controller of the HMD, the trackable graphic on the exterior-facing display (see at least [0012]: “the HMD may include a second display that displays information outward, away from a user of the HMD.”; [0013]: “the HMD 100 may include… a first display 106… a second display 108…”; [0017]: “the first display 106 may be an inward facing display.”; [0026]: “The processor 102 may generate an animation, or a graphical representation.”; [0031]: “The animation… may then be displayed on the second display 108. The second display 108 may be an outward facing display…”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Healy’s XR headset to include the outward-facing second display taught by Ballagas so that information generated by the HMD controller could be presented externally. Doing so would have allowed the headset to communicate computer-generated visual information to persons and imaging systems external to the wearer while merely incorporating one known display arrangement into another known HMD, yielding the predictable result of providing both inward- and outward-facing displays without changing the fundamental operation of Healy’s surgical navigation system.
Ballagas does not directly teach detecting the trackable graphic with the camera of the tracking system.
Hubert teaches detecting the trackable graphic with the camera of the tracking system (see at least [0097]: “The marker manager device 3 manages at least one fiducial marker and the display thereof on the display device 2.”; [0100]: “The tracking module detects the displayed fiducial marker… and deduces the pose…”; [0102]: “dynamic displayed marker… associated with at least one modifiable attribute.”; [0112]-[0116]: dynamic modification of marker attributes including position, size, color, luminance, contrast, orientation; [0128]: “computes pose information representative of a pose of the portable device with regard to the displayed fiducial marker in the real world.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify Healy’s surgical XR headset, as modified by Ballagas, to display Hubert’s computer-generated fiducial marker on the outward-facing display so that the marker could be detected by Healy’s spatially separated tracking cameras. The motivation would have been to replace or supplement Healy’s attached physical DRA fiducials with a programmable visual tracking target that can be dynamically adapted to improve tracking performance. One of ordinary skill in the art would therefore have recognized that incorporating Hubert’s dynamically displayed fiducial marker into Ballagas’s outward-facing display on Healy’s HMD would have predictably improved visibility, detection reliability, and tracking robustness while allowing the existing camera tracking system to localize the HMD in the camera coordinate system.
As to claim 2, the combination of Healy, Ballagas and Hubert teach the surgical system of claim 1 (see above rejection), wherein to present the trackable graphic, the controller of the HMD is configured to present the trackable graphic with: a specified pose on the exterior-facing display; a specified shape on the exterior-facing display; a specified graphic type on the exterior-facing display; and/or specified display parameters on the exterior-facing display (see Ballagas at least [0012]: “the HMD may include a second display that displays information outward, away from a user of the HMD.”; [0026]: “The processor 102 may generate an animation, or a graphical representation.”; [0031]: “The animation… may then be displayed on the second display 108. The second display 108 may be an outward facing display…”; and Hubert at least [0097]: “The marker manager device 3 manages at least one fiducial marker and the display thereof on the display device 2.”; [0100]: “The tracking module detects the displayed fiducial marker… and deduces the pose…”; [0102]: “dynamic displayed marker… associated with at least one modifiable attribute.”; [0112]-[0116]: dynamic modification of marker attributes including position, size, color, luminance, contrast, orientation; [0128]: “computes pose information representative of a pose of the portable device with regard to the displayed fiducial marker in the real world.”).
As to claim 3, the combination of Healy, Ballagas and Hubert teach the surgical system of claim 1 (see above rejection), wherein the controller of the HMD is configured to dynamically change parameters of the trackable graphic on the exterior-facing display (see Ballagas at least [0012]: “the HMD may include a second display that displays information outward, away from a user of the HMD.”; [0026]: “The processor 102 may generate an animation, or a graphical representation.”; [0031]: “The animation… may then be displayed on the second display 108. The second display 108 may be an outward facing display…”; and Hubert at least [0097]: “The marker manager device 3 manages at least one fiducial marker and the display thereof on the display device 2.”; [0100]: “The tracking module detects the displayed fiducial marker… and deduces the pose…”; [0102]: “dynamic displayed marker… associated with at least one modifiable attribute.”; [0112]-[0116]: dynamic modification of marker attributes including position, size, color, luminance, contrast, orientation; [0128]: “computes pose information representative of a pose of the portable device with regard to the displayed fiducial marker in the real world.”; [0141] “In case of adaptive update (i.e. during the marker tracking), the marker manager device 3 might adjust the marker attribute(s) according to the movements of the portable device 1 relative to the display device 2.”).
As to claim 4, the combination of Healy, Ballagas and Hubert teach the surgical system of claim 3 (see above rejection), wherein parameters of the trackable graphic include one or more of: a pose of the trackable graphic; a shape of the trackable graphic; a graphic type of the trackable graphic; and/or display parameters of the trackable graphic (see Hubert at least [0103] “Depending on the implementations, one or several attributes of the dynamic marker is (are) modifiable. A non-exhaustive list of modifiable attributes comprises: [0104] a position of the displayed fiducial marker in the displayed image (i.e. the image displayed on the display device 2); [0105] an orientation of the displayed fiducial marker in the displayed image; [0106] a size of the displayed fiducial marker in the displayed image; [0107] a color of the displayed fiducial marker in the displayed image; [0108] a luminance of the displayed fiducial marker in the displayed image; [0109] a contrast of the displayed fiducial marker in the displayed image; [0110] a pattern of the displayed fiducial marker in the displayed image”).
As to claim 5, the combination of Healy, Ballagas and Hubert teach the surgical system of claim 3 (see above rejection), wherein the controller of the HMD is configured to dynamically change parameters of the trackable graphic on the exterior-facing display in response to changes in a relative spatial pose detected between the exterior-facing display and the camera (see Ballagas at least [0012]: “the HMD may include a second display that displays information outward, away from a user of the HMD.”; [0026]: “The processor 102 may generate an animation, or a graphical representation.”; [0031]: “The animation… may then be displayed on the second display 108. The second display 108 may be an outward facing display…”; and Hubert at least [0097]: “The marker manager device 3 manages at least one fiducial marker and the display thereof on the display device 2.”; [0100]: “The tracking module detects the displayed fiducial marker… and deduces the pose…”; [0102]: “dynamic displayed marker… associated with at least one modifiable attribute.”; [0112]-[0116]: dynamic modification of marker attributes including position, size, color, luminance, contrast, orientation; [0128]: “computes pose information representative of a pose of the portable device with regard to the displayed fiducial marker in the real world.”; [0141] “In case of adaptive update (i.e. during the marker tracking), the marker manager device 3 might adjust the marker attribute(s) according to the movements of the portable device 1 relative to the display device 2.”).
As to claim 6, the combination of Healy, Ballagas and Hubert teach the surgical system of claim 3 (see above rejection), wherein the controller of the HMD is configured to dynamically change parameters of the trackable graphic on the exterior-facing display in response to detection of presence or absence of line-of-sight between the camera of the tracking system and the HMD (see Healy at least [0100]: “the surgical system includes an XR headset 920 and an XR headset controller”; Ballagas at least [0012]: “the HMD may include a second display that displays information outward, away from a user of the HMD.”; Hubert at least [0122]-[0126] “If the displayed fiducial marker is not found in step 22 (i.e. in case of negative answer at test step 23), the portable device 1 carries out: a step 210, in which it transmits failure information to allow the marker manager device to change the current value (initialisation value in the present case) of one, several or all of the modifiable attributes to a new value and obtain new attribute information representative of the new value(s)”; [0139]-[0140] “a corrective update of the marker attribute(s) after a failure (see steps 210-211 of FIG. 2A and steps 223-225 of FIG. 2B after receiving the failure information), or an adaptive (and pro-active) update of the marker attribute(s) to avoid—or at least diminish—detection fail rate (see steps 26-29 of FIG. 2A and steps 223-225 of FIG. 2B after receiving the computed pose information)”).
As to claim 7, the combination of Healy, Ballagas and Hubert teach the surgical system of claim 3 (see above rejection), wherein the controller of the HMD is configured to dynamically change parameters of the trackable graphic on the exterior-facing display in response to detected movement of the HMD (see Healy at least [0100]: “the surgical system includes an XR headset 920 and an XR headset controller”; Ballagas at least [0012]: “the HMD may include a second display that displays information outward, away from a user of the HMD.”; Hubert at least [0141] “In case of adaptive update (i.e. during the marker tracking), the marker manager device 3 might adjust the marker attribute(s) according to the movements of the portable device 1 relative to the display device 2.”).
As to claim 8, the combination of Healy, Ballagas and Hubert teach the surgical system of claim 1 (see above rejection), wherein the controller of the HMD is configured to receive surgical information, and generate the trackable graphic based on the surgical information (see Healy at least [0035]: “An extended reality (XR) headset is operatively connected to the surgical system”; Hubert at least [0097]: “The marker manager device 3 manages at least one fiducial marker and the display thereof on the display device 2.”; [0100]: “The tracking module detects the displayed fiducial marker… and deduces the pose…”).
As to claim 9, the combination of Healy, Ballagas and Hubert teach the surgical system of claim 1 (see above rejection), wherein the controller of the HMD is configured to encode the trackable graphic with time stamps to facilitate synchronization with the camera of the tracking system (see Hubert at least [0097]: “The marker manager device 3 manages at least one fiducial marker and the display thereof on the display device 2.”; [0100]: “The tracking module detects the displayed fiducial marker… and deduces the pose…” – Note Healy, Ballagas and Hubert do not directly teach that the controller encodes the trackable graphic with time stamps to facilitate synchronization with the camera. Official Notice is taken that encoding displayed visual data with timing information (e.g., timestamps, frame identifiers, synchronization data, or sequence information) to facilitate synchronization between a display device and an image sensor is well known in the computer vision, imaging, and display arts. Such synchronization techniques are known to enable accurate correlation between displayed frames and captured images, particularly where latency or frame timing may affect pose estimation or tracking.). It would have been obvious to one of ordinary skill in the art to incorporate well-known synchronization information into Hubert’s displayed fiducial markers in order to improve synchronization between the displayed marker and the tracking camera, thereby improving pose estimation accuracy and reducing errors caused by display or camera latency.
As to claim 10, the combination of Healy, Ballagas and Hubert teach the surgical system of claim 1 (see above rejection), wherein the controller of the HMD is configured to: detect, or receive input signifying, an identify or a type of the tracking system; and generate the trackable graphic based on the identity or type of tracking system (see Hubert at least [0097]: “The marker manager device 3 manages at least one fiducial marker and the display thereof on the display device 2.”; [0100]: “The tracking module detects the displayed fiducial marker… and deduces the pose…” – Note generating trackable graphics based upon an identity or type of tracking system. However, Official Notice is taken that computer systems commonly determine or receive the identity, capabilities, or type of a connected peripheral device and adapt generated data, communication protocols, display formats, or output parameters accordingly. Such adaptive configuration based upon the receiving system are known design practices in the computer and display arts). Therefore, it would have been obvious to one of ordinary skill in the art to generate a trackable graphic appropriate for the particular tracking system being used so as to ensure compatibility and maximize detection reliability.
As to claim 11, the combination of Healy, Ballagas and Hubert teach the surgical system of claim 1 (see above rejection), wherein the trackable graphic is a first trackable graphic and the controller of the HMD is configured to present a second trackable graphic on the exterior-facing display, wherein the second trackable graphic is for a second tracking system and of a different configuration than the first trackable graphic (see Hubert at least [0161] “The marker manager device 3 manages the display of several fiducial markers”; [0162] “When carrying out the step of changing the current value of one, several or all of the modifiable attributes to a new value (see step 223 of FIG. 2B), for a given of the displayed fiducial markers (e.g. 5.sub.1), the marker manager device 3 handles one or several cases of conflict between the given displayed fiducial marker (e.g. 5.sub.1) and other ones of the displayed fiducial markers (e.g. 5.sub.2, 5.sub.3 . . . ). For example, the following cases of conflict are handled: collision between at least two displayed fiducial markers; and occlusion between at least two displayed fiducial markers.”).
As to claim 12, the combination of Healy, Ballagas and Hubert teach the surgical system of claim 1 (see above rejection), wherein: the HMD is a first HMD and the trackable graphic is a first trackable graphic; the surgical system includes a second HMD comprising a controller and a second exterior- facing display configured to present a second trackable graphic; and the tracking system is configured to: transmit a command to the controller of the first HMD that specifies a configuration of the first trackable graphic; and transmit a command to the controller of the second HMD that specifies a configuration of the second trackable graphic, wherein the configuration of the second trackable graphic is different from the configuration of the first trackable graphic; and detect the first and second trackable graphics with the camera to localize the first and second HMDs in the coordinate system of the camera. (see Healy at least [0098]: “The DRA on the headset enables the tracking cameras on the auxiliary tracking bar to track pose of the headset 920.”; [0130]: “Using both visible and near infrared tracking coordinate systems can enable… naturally track an XR headset in the same coordinate system as the navigated surgical instruments.”; Ballagas at least [0012]: “the HMD may include a second display that displays information outward, away from a user of the HMD.”; [0013]: “the HMD 100 may include… a first display 106… a second display 108…”; [0017]: “the first display 106 may be an inward facing display.”; [0026]: “The processor 102 may generate an animation, or a graphical representation.”; and Hubert at least [0161] “The marker manager device 3 manages the display of several fiducial markers”; [0162] “When carrying out the step of changing the current value of one, several or all of the modifiable attributes to a new value (see step 223 of FIG. 2B), for a given of the displayed fiducial markers (e.g. 5.sub.1), the marker manager device 3 handles one or several cases of conflict between the given displayed fiducial marker (e.g. 5.sub.1) and other ones of the displayed fiducial markers (e.g. 5.sub.2, 5.sub.3 . . . ). For example, the following cases of conflict are handled: collision between at least two displayed fiducial markers; and occlusion between at least two displayed fiducial markers.”).
As to claim 13, the combination of Healy, Ballagas and Hubert teach the surgical system of claim 1 (see above rejection), wherein the controller of the HMD is configured to implement a graphical user interface for presentation on one or both of the interior-facing display or the exterior-facing display, wherein the graphical user interface is configured to enable a user to provide input to the controller of the HMD to modify settings or operation of the trackable graphic (see Ballagas at least [0027] “the HMD 100 may include a graphical user interface that allows the user 110 to design, or customize, his or her own animation, or graphical representation, of the eyes 112 and the eyebrows 114 of the user 110.”).
As to claim 14, the combination of Healy, Ballagas and Hubert teach the surgical system of claim 1 (see above rejection), wherein the tracking system is configured to transmit the virtual object to the controller of the HMD in response to the camera detecting the trackable graphic, and wherein the virtual object comprises surgical information related to surgery (see Healy at least [0035]: “An extended reality (XR) headset is operatively connected to the surgical system”; and Hubert at least [0097]: “The marker manager device 3 manages at least one fiducial marker and the display thereof on the display device 2.”; [0100]: “The tracking module detects the displayed fiducial marker… and deduces the pose…”; [0102]: “dynamic displayed marker… associated with at least one modifiable attribute.”).
As to claim 15, the combination of Healy, Ballagas and Hubert teach the surgical system of claim 1 (see above rejection), wherein the trackable graphic is one or more of a QR code; a dynamic QR code; and a geometric array of at least three digital fiducials (see Hubert at least [0097]: “The marker manager device 3 manages at least one fiducial marker and the display thereof on the display device 2.”; [0100]: “The tracking module detects the displayed fiducial marker… and deduces the pose…” - Official Notice is taken that machine-readable fiducial patterns including QR codes, dynamic QR codes, and geometric arrays of multiple fiducial markers are known forms of optical fiducials used for camera-based localization, tracking, and pose estimation.). Such marker types represent known alternatives for implementing Hubert’s fiducial marker and would have been obvious substitutions because each performs the same function of enabling reliable optical detection and pose determination.
As to claim 16, the combination of Healy, Ballagas and Hubert teach the surgical system of claim 1 (see above rejection), wherein the exterior-facing display is an LED or OLED digital display screen with a resolution of at least 200 pixels by 200 pixels and a viewing range from 180-degrees to 360-degrees (note Official Notice is taken that LED and OLED display panels having resolutions exceeding 200×200 pixels and wide viewing angles exceeding 180-degrees are well known in the display arts. It is further well known that multiple display panels or curved display arrangements may collectively provide viewing coverage approaching 360 degrees around a device.). It would have been obvious to one of ordinary skill in the art to combine the known resolution and viewing ranges with the surgical system of Healy, Ballagas and Hubert because selection of display technology, display resolution, and viewing angle constitute routine engineering design choices made to improve marker visibility and detection without changing the fundamental operation of the tracking system.
As to claim 17, the combination of Healy, Ballagas and Hubert teach the surgical system of claim 1 (see above rejection), wherein the exterior-facing display is further configured to present one or more of: human-readable information; surgical graphics; surgical imagery; surgical video; and a video stream obtained by the camera of the tracking system (see Healy at least [0035]: “An extended reality (XR) headset is operatively connected to the surgical system”; and Ballagas [0030] “In one example, the animation may be a direct live feed of the images of the eyes 112 and the eyebrows 114 of the user 110 that are captured by the camera 104. In other words, the animation may simply be the actual video footage of the eyes 112 and the eyebrows 114 of the user 110.”).
As to claim 18, the combination of Healy, Ballagas and Hubert teach the surgical system of claim 1 (see above rejection), wherein the tracking system comprises one or more of: another HMD, a surgical navigation system including a localizer, or a tool comprising a camera (see Healy at least [0023] “a surgical system that includes a pair of XR headsets”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Chinoy et al. (USPN 9,984,354 B1): Col. 2 lines 40-44 “Time synchronization (synchronization) may be maintained between one or more of the cameras or at a device configured to process the images produced by the cameras. An optical timestamp is generated that includes one or more duplicates of a machine-readable code encoded time data.”
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JENNIFER L ZUBAJLO whose telephone number is (571)270-1551. The examiner can normally be reached Monday - Thursday 10 am - 8 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, KE XIAO can be reached at 571-272-7776. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JENNIFER L ZUBAJLO/ Examiner, Art Unit 2627 7/8/2026
/KE XIAO/ Supervisory Patent Examiner, Art Unit 2627
Prosecution Insights