DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-2, 7-9, 11-12 and, 15-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Johnson et al. (US 2019/0254753).
Regarding claim 1, Johnson teaches an extended reality system for use in a surgical procedure (abstract; Figs. 10-11), comprising:
a head-mounted device (HMD) (Fig. 9: HMD 600; Figs. 10-11: HMD 100) comprising an HMD display (Fig. 9: display screen 608; Figs. 10-11: display screen of HMD 100) positionable in front of a user's eyes and a sensing system (Fig. 9: gesture sensor 602, motion sensor 604, detector 610; [0110]: “Gesture sensor 602 may be motion sensor 604, detector 610, or a distinct sensor for detecting and processing gestures”; [0113]-[0115]; [0118]-[0119]) configured to sense control inputs of the user ([0119]: exemplary types of input gestures; [0122]: “a trajectory can be planned by orienting and/or positioning a trajectory selection guidance augmentation graphic (e.g., through movement of a surgeon's head or movement of the augmentation graphic) and providing a user input that captures a position and/or orientation indicated by the trajectory selection guidance augmentation graphic”; [0134]-[0137]: control inputs of surgeon lead to generation and display of augmentation graphics; [0142]: “The representation of at least a portion of a surgical tool and/or trajectory that is displayed on a display screen in exemplary method 1200 may be selected using input from a surgeon, for example, using a gesture, motion, or signal input as described herein above”; [0137]: inputs from surgeon to control graphical representation 922, i.e., augmentation graphics 922, as exemplary control inputs; [0146]: “In step 1310, a user input signal is received by the computer subsystem, wherein the user input signal is generated due to an action (e.g., gesture) by a surgeon using the augmented reality navigation system and the action is made when the trajectory selection guidance augmentation graphic is in a desired position and/or orientation. For example, any user input mechanism described herein above may be used”; [0153]: “The trajectory may be determined in step 1410 in response to a user input signal received by the computer subsystem, wherein the user input signal is generated due to an action (e.g., gesture) by a surgeon using the augmented reality navigation system and the action is made when the pointer tool is in a desired position and/or orientation. For example, any user input mechanism described herein above may be used”); and
one or more controllers (Fig. 9: General processor 626, Graphics processing unit (GPU) 638) coupled to the HMD and being configured to:
receive control inputs ([0137]: inputs from surgeon to control graphical representation 922, i.e., augmentation graphics 922, as exemplary control inputs) from the sensing system to establish a pose of a view coordinate system (Fig. 11: a view coordinate system is inherently associated with a virtual object, e.g., augmentation graphics of leg bone 922, and a pose of the view coordinate system is associated with a pose of the virtual object, e.g., augmentation graphics of leg bone 922; [0137]) in which to present a virtual object (Fig. 11: augmentation graphics of leg bone 922 as an exemplary virtual object) related to the surgical procedure;
define the view coordinate system relative to a world coordinate system after the pose of the view coordinate system is established (Fig. 11 and [0132]: relative positioning module 916 for defining direct and/or indirect relationships between different coordinate systems including a pose of a view coordinate system of a virtual object and a world coordinate system);
recognize surgical information ([0117]: “when a surgeon has maintained a relatively constant head position while viewing a surgical site of a patient for a threshold time, the directional reference (e.g., pitch or yaw) of the head motion signal during that dwell time can be used as a basis for compensating for drift error and/or setting as a reference origin for display of virtual display panels (e.g., as illustrated in FIG. 8) and/or other augmentation graphics (e.g., that appear overlaid over a physical object, such as patient anatomy)”; [0122]-[0128]; [0130]; [0134]-[0138]; [0141]; Examiner’s Note: surgical information is recognized and used for generating patient-specific augmentation graphics); and
in response to recognition of the surgical information, automatically present the virtual object on the HMD display combined with a real-world view and at a predetermined position and orientation within the view coordinate system (Fig. 11; [0137]).
Regarding claim 2, Johnson further teaches the extended reality system of claim 1, wherein the surgical procedure involves a target site (Fig. 11: site of patient’s leg), and wherein the one or more controllers are configured to:
receive control inputs ([0137]: inputs from surgeon to control graphical representation 922, i.e., augmentation graphics 922, as exemplary control inputs) from the sensing system to establish a position of the view coordinate system to be located directly above the target site ([0137]: “The surgeon may, for example, select to have the graphical representation 922 displayed with a defined offset distance above or below the leg”).
Regarding claim 7, Johnson further teaches the extended reality system of claim 1, wherein the virtual object is related to the surgical information (Fig. 11: augmentation graphics of leg bone 922 is related to surgical information).
Regarding claim 8, Johnson further teaches the extended reality system of claim 1, wherein the one or more controllers are configured to:
receive, from a surgical navigation system (Fig. 10), a video stream ([0128]: “Computer subsystem 820 may similarly receive other data and video streams from a patient database and other electronic equipment, which can be selectively displayed on one or more display screens of an HMD 100 using augmentation graphics. As used herein, a video stream can include any type of information that can be provided to a display device for display […] Such information can be displayed on a display screen using augmentation graphics and, moreover, can appear to be displayed on a virtual display screen and/or overlaid over an object in a surgical environment (e.g., patient anatomy or surgical equipment”) of a clinical application ([0124]-[0128]: operations of computer subsystem 820 result from implementing a clinical application) that is presented on a display of the surgical navigation system;
recognize the surgical information from the video stream of the clinical application ([0127]: “The computer subsystem 820 may display on the display screen 110 an indicia (e.g., a crosshair or color marker) aligned with an identified object within the video from the HMD camera to assist the surgeon with identifying the corresponding location on the patient”; [0128]: “Computer subsystem 820 can retrieve patient health information relating to a patient's medical history and data obtained by real-time monitoring of a patient, including, for example, hemodynamic, respiratory, and electrophysiological signals”); and
in response to recognition of the surgical information ([0128] “Computer subsystem 820 can retrieve patient health information relating to a patient's medical history and data obtained by real-time monitoring of a patient, including, for example, hemodynamic, respiratory, and electrophysiological signals”), automatically present the virtual object on the HMD display ([0128]: “Such information can be displayed on a display screen using augmentation graphics and, moreover, can appear to be displayed on a virtual display screen and/or overlaid over an object in a surgical environment (e.g., patient anatomy or surgical equipment)”).
Regarding claim 9, Johnson further teaches the extended reality system of claim 8, wherein the one or more controllers recognize the surgical information from the video stream of the clinical application by being configured to automatically identify text and/or imagery presented by the clinical application ([0128]).
Regarding claim 11, Johnson further teaches the extended reality system of claim 8, wherein:
the surgical information comprises a step of the surgical procedure ([0127]: “The computer subsystem 820 may display on the display screen 110 an indicia (e.g., a crosshair or color marker) aligned with an identified object within the video from the HMD camera to assist the surgeon with identifying the corresponding location on the patient” indicates a step of the surgical procedure); and
the step of the surgical procedure comprises one of: ([0127]: “The computer subsystem 820 may display on the display screen 110 an indicia (e.g., a crosshair or color marker) aligned with an identified object within the video from the HMD camera to assist the surgeon with identifying the corresponding location on the patient”),
Regarding claim 12, Johnson further teaches the extended reality system of claim 8, wherein the virtual object comprises a virtual information panel ([0128]: “Such information can be displayed on a display screen using augmentation graphics and, moreover, can appear to be displayed on a virtual display screen and/or overlaid over an object in a surgical environment (e.g., patient anatomy or surgical equipment).”) that is configured to display information related to the surgical information.
Regarding claim 15, Johnson further teaches the extended reality system of claim 1, wherein the virtual object comprises a 3D surgical object ([0123]: “computer subsystem 820 uses patient data from imaging equipment 830 to generate a two dimensional (2D) or three dimensional (3D) model”) including one or more of: a 3D model of a bone (Fig. 11: augmentation graphics of the leg bone 922), a 3D model of an implant ([0140]: “It is understood that such an exemplary method can also be adapted to display augmentation graphics representing at least a portion of a surgical apparatus (e.g., implant) that is, for example, attached directly or indirectly to a robotic surgical system”), and a 3D surgical plan (Fig. 12: step 1206; [0141]: “In step 1206, a representation of at least a portion of a surgical tool connected to and/or inserted into a robotic surgical system and/or a trajectory (e.g., actual or planned trajectory) of the surgical tool is generated and/or accessed”).
Regarding claim 16, Johnson further teaches the extended reality system of claim 1, wherein:
the HMD comprises a camera configured to produce a live video stream of the real-world view ([0126]: “Computer subsystem 820 may compare patterns of objects in a detector input signal (e.g., video stream) from a detector (e.g., camera) on the HMD 100 to […] For example, real-time video captured by an HMD-mounted camera during surgery of a patient may be processed by computer subsystem 820 and compared to video captured by one or more other sources”; [0127]: “The computer subsystem 820 may display on the display screen 110 an indicia (e.g., a crosshair or color marker) aligned with an identified object within the video from the HMD camera to assist the surgeon with identifying the corresponding location on the patient”); and
the one or more controllers combine the virtual object with the real-world view by combining the virtual object into the live video stream ([0127]: “the computer subsystem 820 displays a graphical indicia on the display screen 110 aligned with one of the anatomical objects displayed on the display screen 110 from the rotated and scaled three dimensional anatomical model responsive to identifying a threshold level of correspondence between a pattern of the one of the anatomical objects and a pattern of one of the anatomical objects in the video stream from the video camera”).
Regarding claim 17, Johnson further teaches the extended reality system of claim 16, wherein the one or more controllers recognize the surgical information from the camera of the HMD ([0127]: “the computer subsystem 820 displays a graphical indicia on the display screen 110 aligned with one of the anatomical objects displayed on the display screen 110 from the rotated and scaled three dimensional anatomical model responsive to identifying a threshold level of correspondence between a pattern of the one of the anatomical objects and a pattern of one of the anatomical objects in the video stream from the video camera”).
Regarding claim 18, Johnson further teaches the extended reality system of claim 1, further comprising a surgical device (Fig. 11: t least one detector 902, a surgery tool and/or surgical apparatus (e.g., implant) and/or robotic surgical system 908 collectively form a surgical device) coupled to the one or more controllers and comprising a camera source ([0130]: “he at least one detector 902 includes a plurality of cameras that are spaced apart at defined locations within an operating room and each having a field of view that can observe objects to be tracked”), and wherein the one or more controllers recognize the surgical information from the camera source of the surgical device ([0134]-[0135]; [0141]).
Regarding claim 19, Johnson teaches a head-mounted device (HMD) (Fig. 9: HMD 600; Figs. 10-11: HMD 100) for use in a surgical procedure, the HMD comprising:
an HMD display (Fig. 9: display screen 608; Figs. 10-11: display screen of HMD 100) positionable in front of a user's eyes;
a sensing system (Fig. 9: gesture sensor 602, motion sensor 604, detector 610; [0110]: “Gesture sensor 602 may be motion sensor 604, detector 610, or a distinct sensor for detecting and processing gestures”; [0113]-[0115]; [0118]-[0119]) configured to sense control inputs of the user ([0119]: exemplary types of input gestures; [0122]: “a trajectory can be planned by orienting and/or positioning a trajectory selection guidance augmentation graphic (e.g., through movement of a surgeon's head or movement of the augmentation graphic) and providing a user input that captures a position and/or orientation indicated by the trajectory selection guidance augmentation graphic”; [0134]-[0137]: control inputs of surgeon lead to generation and display of augmentation graphics; [0142]: “The representation of at least a portion of a surgical tool and/or trajectory that is displayed on a display screen in exemplary method 1200 may be selected using input from a surgeon, for example, using a gesture, motion, or signal input as described herein above”; [0137]: inputs from surgeon to control graphical representation 922, i.e., augmentation graphics 922, as exemplary control inputs; [0146]: “In step 1310, a user input signal is received by the computer subsystem, wherein the user input signal is generated due to an action (e.g., gesture) by a surgeon using the augmented reality navigation system and the action is made when the trajectory selection guidance augmentation graphic is in a desired position and/or orientation. For example, any user input mechanism described herein above may be used”; [0153]: “The trajectory may be determined in step 1410 in response to a user input signal received by the computer subsystem, wherein the user input signal is generated due to an action (e.g., gesture) by a surgeon using the augmented reality navigation system and the action is made when the pointer tool is in a desired position and/or orientation. For example, any user input mechanism described herein above may be used”); and
one or more controllers (Fig. 9: General processor 626, Graphics processing unit (GPU) 638) coupled to the HMD display and being configured to:
receive control inputs ([0137]: inputs from surgeon to control graphical representation 922, i.e., augmentation graphics 922, as exemplary control inputs) from the sensing system to establish a pose of a view coordinate system (Fig. 11: a view coordinate system is inherently associated with a virtual object, e.g., augmentation graphics of leg bone 922, and a pose of the view coordinate system is associated with a pose of the virtual object, e.g., augmentation graphics of leg bone 922; [0137]) in which to present a virtual object (Fig. 11: augmentation graphics of leg bone 922 as an exemplary virtual object) related to the surgical procedure;
define the view coordinate system relative to a world coordinate system after the pose of the view coordinate system is established (Fig. 11 and [0132]: relative positioning module 916 for defining direct and/or indirect relationships between different coordinate systems including a pose of a view coordinate system of a virtual object and a world coordinate system);
recognize surgical information ([0117]: “when a surgeon has maintained a relatively constant head position while viewing a surgical site of a patient for a threshold time, the directional reference (e.g., pitch or yaw) of the head motion signal during that dwell time can be used as a basis for compensating for drift error and/or setting as a reference origin for display of virtual display panels (e.g., as illustrated in FIG. 8) and/or other augmentation graphics (e.g., that appear overlaid over a physical object, such as patient anatomy)”; [0122]-[0128]; [0130]; [0134]-[0138]; [0141]; Examiner’s Note: surgical information is recognized and used for generating patient-specific augmentation graphics); and
in response to recognition of the surgical information, automatically present the virtual object on the HMD display combined with a real-world view and at a predetermined position and orientation within the view coordinate system (Fig. 11; [0137]).
Claim 20 is rejected for substantially the same rationale as applied to claim 1.
Allowable Subject Matter
Claims 3-6, 10 and 13-14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US 2015/0265362 by Anderson teaches in Fig. 10 a three-dimensional position adjustment object/guide 811a, 811b is associated with each 3D representation of the virtual implant component 810a, 810b. However, the three-dimensional position adjustment object/guide 811a, 811b are not combined with the real-world view (as what is recited in claim 3 of this application).
US 2022/0233251 by Bowling et al. teaches in Fig. 16A guide constraints are employed in six degrees of freedom to guide the user toward the target state, i.e., three position constraints along the x, y, z axes of the target coordinate system TF to guide the origin of the guided coordinate system GF to the origin of the target coordinate system TF and three orientation constraints about the x, y, z axes of the target coordinate system TF to guide the x, y, z axes of the guided coordinate system GF to align with the x, y, z axes of the target coordinate system TF. However, the guide is not combined with the real-world view (as what is recited in claim 3 of this application).
US 2019/0239850 by Dalvin et al. teaches use of guiding element not limited to “text instructions, guiding graphics such as one or more arrows, targets, circles, color-changing elements, progress bars, transparent elements, angles, ‘ghost’ outlines of real-world objects, projections of real-time or stored imaging data, overlaid virtual organs or other virtual item”.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XUEMEI ZHENG whose telephone number is (571)272-1434. The examiner can normally be reached Monday-Friday: 9:30 pm-6:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Benjamin Lee can be reached at 571-272-2963. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XUEMEI ZHENG/Primary Examiner, Art Unit 2629
Prosecution Insights