Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Use of indicates a limitation is not explicitly disclosed by the reference alone.
Claim(s) 1-5, 14, 17, 18, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lang (US 2022/0079675)
Claim 1
Lang discloses a method of performing an operation on a body part of a patient using a mixed reality (MR) device worn by a user (Lang, ¶ 4: “In some embodiments, the optical head mounted display can be a see through optical head mounted display, e.g. an optical see through head mounted display, for example for augmented reality applications.”) performing the operation (Lang, ¶ 4: “performing a surgical step or surgical procedure with visual guidance using an optical head mounted display”), comprising:
- providing a visual representation of an operating scene including an anatomical feature of the body part of the patient, wherein the visual representation comprises a virtual anatomical feature representing the anatomical feature of the patient (Lang, ¶¶ 367, 789: “An optical head mounted display can project or display a digital hologram of virtual data or virtual data of the patient 55…3D models 260, 261, 262 can be generated, e.g. one for subsurface anatomic or pathologic structures of the patient, one for virtual surgical tools or instruments and one for virtual surgical implant components. These can be registered, e.g. in a common coordinate system or multiple coordinate systems using coordinate transfers, also with the OHMD 263.”),
- performing a first adjusting of a position and/or orientation of the visual representation relative to the anatomical feature of the patient, wherein the virtual anatomical feature is in superposition with the anatomical feature of the patient (Lang, ¶ 789: “multiple 3D models 260, 261, 262 can be generated, e.g. one for subsurface anatomic or pathologic structures of the patient, one for virtual surgical tools or instruments and one for virtual surgical implant components. These can be registered, e.g. in a common coordinate system or multiple coordinate systems using coordinate transfers, also with the OHMD 263.”),
- displaying the visual representation by the MR device worn by the user, wherein the visual representation is displayed in superposition with the anatomical feature of the patient in a field of vision of the user (Lang, ¶¶ 192-93: “OHMDs 11, 12, 13, 14 to project digital holograms of the next surgical step 34 superimposed onto and aligned with the surgical site in a predetermined position and/or orientation…. Virtual data of the patient can be projected superimposed onto live data of the patient for each individual viewer by each individual OHMD for their respective view angle or perspective by registering live data of the patient, e.g. the surgical field, and virtual data of the patient as well as each OHMD in a common, shared coordinate system. Thus, virtual data of the patient including aspects of a virtual surgical plan can remain superimposed and/or aligned with live data of the patient irrespective of the view angle or perspective of the viewer and alignment and/or superimposition can be maintained as the viewer moves his or her head or body.”),
- performing repeated or continuous adjusting the position and/or orientation of the visual representation, wherein, when the MR device worn by the user moves relative to the anatomical feature of the patient, the virtual anatomical feature of the visual representation remains in superposition with the anatomical feature of the patient (Lang, ¶¶ 169, 191: “Changes in the position of the markers relative to the sensors or cameras are used to continuously determine the position and orientation of the HMD… The position, orientation, alignment, and change in position, orientation and alignment in relationship to the patient and/or the surgical site of each additional HMD can be individually monitored thereby maintaining alignment and/or superimposition of corresponding structures in the live data of the patient and the virtual data of the patient for each additional HMD irrespective of their position, orientation, and/or alignment in relationship to the patient and/or the surgical site.”).
Lang discloses the above features in different embodiments.
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to combine the different features claimed.
One of ordinary skill in the art would have motivation to combine visualization of one or more of anatomy, surgical tools and the like and include dynamic tracking and adjustment in order to register different virtual augmentations on a live scene (See ¶¶ 789 and Table 9).
Claim 2
Lang discloses comprising:
- acquiring successive images of the anatomical feature by a perception device of the MR device worn by the user (Lang, ¶ 83: “one or more cameras or image capture or video capture systems and/or a 3D scanner included in the optical head mounted display can detect one or more optical markers including their coordinates (x, y, z) and at least one or more of a position, orientation, alignment, direction of movement or speed of movement of the one or more optical markers.”), and
- adjusting the position and/or orientation of the visual representation by a display device of the MR device, as a function of the images acquired by the perception device (Lang, ¶ 162: “As the viewpoint and view direction change, for example due to head movement, the view projections are updated so that the computer-generated display follows the new view.”).
Claim 3
Lang discloses wherein the first adjusting comprises providing a first reference marker having a set position and/or orientation relative to the anatomical feature of the patient, and performing the first adjusting relative to the first reference marker (Lang, ¶ 5: “wherein the one or more predetermined coordinates of the virtual surgical guide are referenced to or based on the marker, wherein the processor is configured to adjust at least one focal plane, focal point, or combination thereof of the display of the 3D stereoscopic view based on the determined distance.”).
Claim 4
Lang discloses wherein
- the visual representation comprises indications of a target position and/or orientation of a surgical tool intended to be used by the user (Lang, ¶ 54: “displaying virtual representations of one or more of a virtual surgical tool, virtual surgical instrument including a virtual surgical guide…”), and the method further comprises:
- adjusting a position and/or orientation of the surgical tool as a function of the indications of the target position and/or orientation of the surgical tool (Lang, ¶ 165: “One or more optical markers can be applied to a physical tool, physical instrument, physical implant or physical device tracking the coordinates and/or the position and/or orientation of the physical tool, physical instrument, physical implant or physical device. One or more optical markers can be applied to the surgeon.”)
Claim 5
Lang discloses further comprising:
- providing a second reference marker on the surgical tool, the second marker having a set position and/or orientation relative to the surgical tool (Lang, ¶ 165: “One or more optical markers can be applied to a physical tool, physical instrument, physical implant or physical device tracking the coordinates and/or the position and/or orientation of the physical tool, physical instrument, physical implant or physical device. One or more optical markers can be applied to the surgeon.”),
- displaying in the virtual surgical scene an indication of a position and/or orientation of the surgical tool relative to the position and/or orientation of the virtual surgical tool, as a function of the position and/or orientation of the second marker (Lang, ¶ 54, 119: “displaying virtual representations of one or more of a virtual surgical tool, virtual surgical instrument including a virtual surgical guide…With guidance in mixed reality environment, a virtual surgical guide, tool, instrument or implant can be superimposed onto the physical joint, spine or surgical site. Further, the physical guide, tool, instrument or implant can be aligned with the virtual surgical guide, tool, instrument or implant displayed or projected by the OHMD”).
Claim 14
Lang discloses comprising, before adjusting the visual representation:
- registering the user (Lang, ¶ 192: “The multiple OHMDs can be registered in a common coordinate system 15 using anatomic structures, anatomic landmarks, calibration phantoms, reference phantoms, optical markers, navigation markers, and/or spatial anchors”), and
- receiving an input of information on the patient (Lang, ¶ 192, 652: “The pre-operative data 16 or live data 18 including intra-operative measurements or combinations thereof can be used to develop, generate or modify a virtual surgical plan 24. The virtual surgical plan 24 can be registered in the common coordinate system 15….The modified virtual surgical plan 162 can be further modified based on visual or optical feedback or input 161…”).
Claim 17
Lang discloses wherein the displaying comprises projecting a 3D visual representation of the visual representation in the field of vision of the user wearing the MR device (Lang, ¶ 103: “The one or more optical head mounted displays are registered with the physical surgical site, using, for example, one or more markers, e.g. attached to the surgical site or attached near the surgical site…The one or more OHMDs can display live images of the physical surgical site, one or more of a pre- or intra-operative imaging study, 2D or 3D images of the patient, graphical representations of one or more medical devices, and/or CAD files of one or more medical devices.”)
Claim 18
The same teachings and rationales in claim 1 are applicable to claim 18, with Lang disclosing a mixed reality navigation system for assisting an operation on a body part of a patient performed by a user of the system, the system comprising a mixed reality device worn by the user performing the operation, the mixed reality device comprising (Lang, ¶ 128: “A head mounted display can include a first display unit for the left eye and a second display unit for the right eye. The first and second display units can be transparent, semi-transparent or non-transparent. The system, comprising, for example, the head mounted display, one or more computer processors and/or an optional marker attached to the patient”):
Claim 20
Examiner’s Interpretation:
Machine readable media can encompass forms of signal transmission media that falls outside of the four statutory categories of invention. MPEP 2106; citing In re Nuijten, 500 F.3d 1346, 84 USPQ2d 1495 (Fed. Cir. 2007). A claim whose BRI covers both statutory and non-statutory embodiments embraces subject matter that is not eligible for patent protection and therefore is directed to non-statutory subject matter. MPEP 2106.
Applicant’s specification does not explicitly define the scope of non-transitory storage medium.
Claim 20 as drafted recites A non-transitory storage medium comprising a computer program, wherein the computer program is configured…
The claim term non-transitory unless defined otherwise, excludes signals and other forms of transmission media. Therefore the broadest reasonable interpretation of the claimed medium in view of Applicant’s specification covers only eligible subject matter.
Claim Mapping:
The same teachings and rationales in claim 1 are appliable to claim 20.
Claim(s) 6, 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lang (US 2022/0079675) in view of Fronk (US 2013/0267838)
Claim 6
Lang does not explicitly disclose, but Fronk discloses wherein the first reference marker is a plane marker and/or a monochrome marker (Fronk, ¶ 23: “This means of optically determining the position and orientation of a structure using a marker is well known, and can be accomplished for example using ARToolKit or ArUco, which are computer tracking software for creating augmented reality applications that overlay virtual imagery on the real world. See "ARToolKit," and "ArUco: a minimal library for Augmented Reality applications based on OpenCv”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to substitute an ArUco marker.
The functions of monochrome markers were well known in the art, and one of ordinary skill in the art would have recognized that “determining the position and orientation of a structure using a marker is well known, and can be accomplished for example using ARToolKit or ArUco” (Fronk, ¶ 23), which may be substituted for other types of markers. One of ordinary skill in the art could have made the substitution and the results would have been predictable because Lang utilizes markers which could be substituted.
Claim 8
Lang does not disclose, but Fronk discloses wherein the second reference marker is a plane marker and/or a monochrome marker (Fronk, ¶ 23: “This means of optically determining the position and orientation of a structure zusing a marker is well known, and can be accomplished for example using ARToolKit or ArUco, which are computer tracking software for creating augmented reality applications that overlay virtual imagery on the real world. See "ARToolKit," and "ArUco: a minimal library for Augmented Reality applications based on OpenCv”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to substitute an ArUco marker.
The functions of monochrome markers were well known in the art, and one of ordinary skill in the art would have recognized that “determining the position and orientation of a structure using a marker is well known, and can be accomplished for example using ARToolKit or ArUco” (Fronk, ¶ 23), which may be substituted for other types of markers. One of ordinary skill in the art could have made the substitution and the results would have been predictable because Lang utilizes markers which could be substituted.
Claim(s) 7, 15, 16, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lang (US 2022/0079675) in view of Slagmolen (US 2022/0125519)
Claim 7
Lang discloses adjusting predetermined locations and/or orientations of the visual representation to be superposed with corresponding predetermined positions and/or orientations of the anatomical feature, based on the position and/or orientation of the first reference marker (Lang, ¶¶ 169, 191: “Changes in the position of the markers relative to the sensors or cameras are used to continuously determine the position and orientation of the HMD… The position, orientation, alignment, and change in position, orientation and alignment in relationship to the patient and/or the surgical site of each additional HMD can be individually monitored thereby maintaining alignment and/or superimposition of corresponding structures in the live data of the patient and the virtual data of the patient for each additional HMD irrespective of their position, orientation, and/or alignment in relationship to the patient and/or the surgical site.”).
Lang does not explicitly disclose, but Slagmolen discloses wherein the first adjusting comprises:
- attaching the first reference marker to the shoulder blade (¶ 74: “Markers may be attached to multiple structures (e.g. the shoulder skin, shoulder blade and glenoid bone surface, surrounding soft tissue, muscles, ligaments, etc.) and indirectly related to the anatomy to be registered by using computational models.”), and
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to attach the marker to the shoulder blade.
One of ordinary skill in the art would have motivation to track relative positions with respect to anatomy and Lang considers applications to shoulder procedures. One of ordinary skill in the art would have had a reasonable expectation of success because Lang further suggest application to shoulder procedures.
Claim 15
Lang does not disclose, but Slagmolen discloses wherein the first adjusting of the visual representation comprises:
- by the user, identifying a plurality of reference points on the anatomical feature of the patient (Slagmolen, ¶ 75: “A marking device may also be used (e.g., a tracked pen) via I/O module 122 to register the anatomy of individual bones in the augmented environment. This may include the glenoid surface. The coracoid surface may also be used as a reference point as it is located outside of the surgical window. The shoulder blade may also serve as an indirect marker. The combination of multiple markers, in combination with vision data (e.g., from cameras of I/O module 122), may also be used to capture the anatomy of interest's position.”) , and
- adjusting the virtual anatomical feature of the patient in the visual representation to respective positions and/or orientations of the plurality of reference points of the anatomical feature of the patient (e.g. positioning content based on various user driving tracking (pointing, user identifying landmarks, etc.); ¶ 106: “In some embodiments, the surgical scene 230 may also be augmented with information that represents a mechanical, physical, anatomical or other 3D feature (which may be mapped to a surface) and which is derived from a calculation or a direct measurement 232 based on medical imaging or intra-operative data acquisitions.”).
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to mark reference points.
One of ordinary skill in the art would have motivation to track relative positions with respect to anatomy and Lang considers applications to shoulder procedures. One of ordinary skill in the art would have had a reasonable expectation of success because Lang further suggest application to shoulder procedures.
Claim 16
Lang discloses further comprising:
- receiving an input of information about a surgical tool (e.g. configuration data, marker setup, etc.),
- detecting a position and orientation of the surgical tool (Lang, ¶ 164: “One or more infrared and/or RF markers, active and/or passive markers can be applied to a physical tool, physical instrument, physical implant or physical device tracking the coordinates and/or the position and/or orientation of the physical tool, physical instrument, physical implant or physical device. One or more infrared and/or RF markers, active and/or passive markers can be applied to the surgeon.”).
Claim 19
Lang does not explicitly disclose, but Slagmolen discloses wherein the system is configured to perform the first adjusting of the visual representation, wherein the first adjusting comprises:
- receiving respective positions and/or orientation information on a plurality of reference points on the anatomical feature of the patient (Slagmolen, ¶ 75: “A marking device may also be used (e.g., a tracked pen) via I/O module 122 to register the anatomy of individual bones in the augmented environment. This may include the glenoid surface. The coracoid surface may also be used as a reference point as it is located outside of the surgical window. The shoulder blade may also serve as an indirect marker. The combination of multiple markers, in combination with vision data (e.g., from cameras of I/O module 122), may also be used to capture the anatomy of interest's position.”), and
- adjusting the virtual anatomical feature of the patient in the visual representation to correspond to the position and/or orientation of the anatomical feature of the patient in the surgical scene (e.g. positioning content based on various user driving tracking (pointing, user identifying landmarks, etc.); ¶ 106: “In some embodiments, the surgical scene 230 may also be augmented with information that represents a mechanical, physical, anatomical or other 3D feature (which may be mapped to a surface) and which is derived from a calculation or a direct measurement 232 based on medical imaging or intra-operative data acquisitions.”).
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to mark reference points.
One of ordinary skill in the art would have motivation to track relative positions with respect to anatomy and Lang considers applications to shoulder procedures. One of ordinary skill in the art would have had a reasonable expectation of success because Lang further suggest application to shoulder procedures.
Claim(s) 9-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lang (US 2022/0079675) in view of Nikou (US 2022/0151705)
Claim 9
Lang discloses wherein the body part of the patient is a shoulder (Lang, ¶ 76: “In some embodiments, the virtual surgical guide is configured to guide a bone cut in a knee replacement, hip replacement, shoulder joint replacement or ankle joint replacement.”), and
Lang does not disclose, but Nikou discloses the visual representation includes a display of a piercing axis in a glenoid of a scapula of the shoulder (Nikou, ¶ 122, 158, : “f tracking specific portions of a patient's skeletal structure during surgery using an surgical navigation system enhanced by an augmented reality system…Shoulder parameters may include, without limitation, humeral resection depth/angle, humeral stem version, humeral offset, glenoid version and inclination, as well as reverse shoulder parameters such as humeral resection depth/angle, humeral stem version, Glenoid tilt/version, glenosphere orientation, glenosphere offset and offset direction…. In a revision surgery, the surgeon may paint certain areas that contain anatomical defects to allow for better visualization and navigation of implant insertion. These defects can be identified based on analysis of the pre-operative images.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to provide a visual representation of a piercing axis such as for implant insertion.
One of ordinary skill in the art would have motivation to allow the surgeon to properly orient tools and implants. One of ordinary skill in the art would have had a reasonable expectation of success because Lang considers the use of visual guides for alignment of surgical tools.
Claim 10
Lang discloses wherein the body part of the patient is a shoulder (Lang, ¶ 76: “In some embodiments, the virtual surgical guide is configured to guide a bone cut in a knee replacement, hip replacement, shoulder joint replacement or ankle joint replacement.”)
Lang does not disclose, but Nikou discloses the visual representation includes a display of a piercing axis in a glenoid of a scapula of the shoulder, and the surgical tool is a piercing tool adapted to pierce the glenoid and/or a piercing guide adapted to guide piercing of the glenoid for installation of a shoulder prosthesis (Nikou, ¶ 122, 158, : “f tracking specific portions of a patient's skeletal structure during surgery using an surgical navigation system enhanced by an augmented reality system…Shoulder parameters may include, without limitation, humeral resection depth/angle, humeral stem version, humeral offset, glenoid version and inclination, as well as reverse shoulder parameters such as humeral resection depth/angle, humeral stem version, Glenoid tilt/version, glenosphere orientation, glenosphere offset and offset direction…. In a revision surgery, the surgeon may paint certain areas that contain anatomical defects to allow for better visualization and navigation of implant insertion. These defects can be identified based on analysis of the pre-operative images.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to provide a visual representation of a piercing axis such as for implant insertion.
One of ordinary skill in the art would have motivation to allow the surgeon to properly orient tools and implants. One of ordinary skill in the art would have had a reasonable expectation of success because Lang considers the use of visual guides for alignment of surgical tools.
Claim 11
Lang does not disclose, but Nikou discloses wherein the display of the piercing axis includes a display of an indication of inclination angle and an indication of anteversion angle (Nikou, ¶¶ 7, 65: “Once determined, the preferred location and angle are displayed on a mixed reality display using one or more associated visual indicators….The CASS-developed surgical plan can include proposed details on offset values, inclination and anteversion values”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to provide a visual representation of the claimed angles.
One of ordinary skill in the art would have motivation to allow the surgeon to properly orient tools and implants. One of ordinary skill in the art would have had a reasonable expectation of success because Lang considers the use of visual guides for alignment of surgical tools.
Claim(s) 12, 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lang (US 2022/0079675) in view of Nikou (US 2022/0151705) and Dorman (US 2024/0374313)
Claim 12
Lang as modified by Nikou does not disclose, but Dorman discloses wherein a precision of angular representation of an adequation of orientation of the surgical tool to the piercing axis is equal or less than 1 degree (Dorman, ¶ 86: “In some embodiments, once the angle of the gear shift probe 652 is about 20 degrees within the selected trajectory, the gear shift probe 652 will turn yellow, at 5 degrees, it will turn green, and when the alignment is within 1 degree of the target angle, a green line 654 will extend outward and the pedicle screw will disappear to signify that the apparatus 300 is properly aligned.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to provide a visual representation of the claimed angles.
One of ordinary skill in the art would have motivation to allow the surgeon to properly orient tools and implants. One of ordinary skill in the art would have had a reasonable expectation of success because Lang considers the use of visual guides for alignment of surgical tools.
Claim 13
Lang as modified by Nikou does not disclose, but Dorman discloses wherein the adequation is indicated by color coding, including different colors for angles of more than 1 degree and for angles equal or less than 1 degree, respectively (Dorman, ¶ 86: “In some embodiments, once the angle of the gear shift probe 652 is about 20 degrees within the selected trajectory, the gear shift probe 652 will turn yellow, at 5 degrees, it will turn green, and when the alignment is within 1 degree of the target angle, a green line 654 will extend outward and the pedicle screw will disappear to signify that the apparatus 300 is properly aligned.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to provide a visual representation of the claimed angles.
One of ordinary skill in the art would have motivation to allow the surgeon to properly orient tools and implants. One of ordinary skill in the art would have had a reasonable expectation of success because Lang considers the use of visual guides for alignment of surgical tools.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN M GRAY whose telephone number is (571)272-4582. The examiner can normally be reached on Monday through Friday, 9:00am-5:30pm (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached on (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN M GRAY/Primary Examiner, Art Unit 2611