Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This is in response to applicant’s amendment/response filed on 07/27/2026, which has
been entered and made of record. Claims 1 and 11 are amended. Claims 1-18 are pending in the application.
Response to Arguments
Applicant arguments regarding claim rejections under 103 are considered, but are not persuasive.
Applicant argues:
PNG
media_image1.png
106
650
media_image1.png
Greyscale
Examiner disagrees: As shown in FIG. 1 and 2 and corresponding paragraphs ([0023], [0033]) in Mahmood, the sensor 208 detects the fiducial marker 108/212 to tracking the movement of the HMD in real time. Based on the location of the 108/212, the system decides the movement of the HMD and generates and displays the 2D images on the left and right eyes and produces the 3D effects.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-3, 5, 8, 10, 11, 13, 16, 18 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1, 9, 1, 7, 4,8, 1, 7, 4, 8 of U.S. Patent No. 11510750. Although the claims at issue are not identical, they are not patentably distinct from each other because they are obvious variants of each other.
Claim 4, 6, 7, 9, 12, 14, 15, 17 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. 11510750 in view of Mahmood.
Regarding claim 4, U.S. Patent No. 11510750 teaches:
The method of claim 1,
Mahmood teaches:
wherein generating a first 2D image includes projecting the 3D volume on a plane at an orientation defined by the pose of the XR headset. (Mahmood [0033], “Based on the 3D coordinates of the fiducial markers 206, 212 and the determined orientation of the HMD 210, the surgical imaging system 200 can display an augmented reality within the field of view of the surgeon 202. For instance, the HMD 210 can display the captured 3D image data of one or more internal features of the patient 204 to the surgeon 202 via a display of the HMD 210.” [0020], “In other examples, the 3D information can be displayed using stereoscopy in which different 2D images are displayed to the left and right eye in order to give the perception of 3D depth. Other methods of displaying the 3D image information can be used as well.” [0019], “In examples where the display 102 includes transparent lenses, the wearer can observe the real-world view through the transparent lenses, and a projection device (not shown) can project a virtual image onto the display 102 such that the virtual image appears superimposed over the real-world view of the wearer.”)
Regarding claim 6, U.S. Patent No. 11510750 teaches:
The method of claim 1,
Mahmood teaches: further comprising combining by the XR headset the generated XR image with a real-world scene for augmented reality viewing by the user. (Mahmood [0019], “In examples where the display 102 includes transparent lenses, the wearer can observe the real-world view through the transparent lenses, and a projection device (not shown) can project a virtual image onto the display 102 such that the virtual image appears superimposed over the real-world view of the wearer.”)
Regarding claim 7, U.S. Patent No. 11510750 in view of Mahmood teaches:
The method of claim 6,
Mahmood teaches: wherein the XR headset includes an at least partially transparent screen to combine the generated XR image with the real-world scene. (Mahmood [0019], “In examples where the display 102 includes transparent lenses, the wearer can observe the real-world view through the transparent lenses, and a projection device (not shown) can project a virtual image onto the display 102 such that the virtual image appears superimposed over the real-world view of the wearer.”)
Regarding claim 9, U.S. Patent No. 11510750 teaches:
The method of claim 1,
Mahmood teaches: wherein the XR headset includes a set of cameras, the method further comprising when a dynamic reference array of a surgical tool comes within a field of view of the cameras, displaying a graphical representation of the surgical tool on in relation to the generated XR image.( Mahmood [0057], “Further, based on a detected location of a hand of the surgeon 202, the surgical imaging system 200 can detect that the surgeon 202 is interacting with a 3D model of internal features of the patient 204 and responsively cause the robotic device 216 to perform one or more corresponding surgical procedures on the patient 204. For instance, the HMD 210 can include motion capture cameras to detect a location of the hand of the surgeon 202 within the 3D coordinate system. Other positional tracking sensors can be used as well, such as one or more IMUs included in the external device 504 or otherwise attached to the hand of the surgeon 202. Based on the determined location of the hand of the surgeon 202, the surgical imaging system 200 can determine the location of the hand of the surgeon 202 relative to the 3D model of internal features of the patient 204. The relative location of the hand of the surgeon 202 to the 3D model can then be used to detect an interaction between the surgeon 202 and the 3D model. For instance, the surgical imaging system 200 can detect the surgeon 202 performing a pinching gesture on one or more features of the 3D model, and the robotic surgical device 216 can responsively perform a corresponding pinching action (e.g., using forceps, needle drivers, clamps, pliers, etc.) on the corresponding actual internal feature of the patient 204.”)
For the above claims 4, 6, 7, 9, It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the teachings of U.S. Patent No. 11510750 with the teachings of Mahmood to effectively and accurately generate an XR image to help surgeons perform operations.
Claims 12, 14, 15, 17 recite similar limitations of claim 4, 6-7, 9 respectively, thus are rejected accordingly.
Claims 1-2, 5, 8, 10, 11, 13, 16, 18 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim1, 9, 7, 4,8, 1, 7, 4, 8 of U.S. Patent No. 12115028. Although the claims at issue are not identical, they are not patentably distinct from each other because they are obvious variants of each other.
Claim 3, 4, 6, 7, 9, 12, 14, 15, 17 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. 12115028 in view of Mahmood.
Regarding claim 3, U.S. Patent No. 12115028teaches:
The method of claim 1,
Mahmood teaches: wherein providing a 3D volume of an anatomical structure of a 3D patient scan includes providing a plurality of 2D slices of a DICOM scan. (Mahmood [0002] “Medical imaging techniques allow for three-dimensional (3D) representations of various parts of the human body. For example, an X-ray computed tomography scan (CT scan) combines multiple X-ray images to produce cross-sectional images of a scanned object. Digital geometry processing can then be applied to the X-ray images to generate a 3D representation of the scanned object. Similarly, magnetic resonance imaging (MRI) can generate 3D representations by measuring a spatial distribution of water in the scanned object. Other medical imaging techniques can be used to generate 3D representations, such as ultrasound, positron emission tomography (PET), fluoroscopy, tractography, diffused tensor imaging (DTI), and nuclear magnetic resonance (NMR) spectroscopy, to name a few.”)
Regarding claim 4, U.S. Patent No. 12115028teaches:
The method of claim 1,
Mahmood teaches:
wherein generating a first 2D image includes projecting the 3D volume on a plane at an orientation defined by the pose of the XR headset. (Mahmood [0033], “Based on the 3D coordinates of the fiducial markers 206, 212 and the determined orientation of the HMD 210, the surgical imaging system 200 can display an augmented reality within the field of view of the surgeon 202. For instance, the HMD 210 can display the captured 3D image data of one or more internal features of the patient 204 to the surgeon 202 via a display of the HMD 210.” [0020], “In other examples, the 3D information can be displayed using stereoscopy in which different 2D images are displayed to the left and right eye in order to give the perception of 3D depth. Other methods of displaying the 3D image information can be used as well.” [0019], “In examples where the display 102 includes transparent lenses, the wearer can observe the real-world view through the transparent lenses, and a projection device (not shown) can project a virtual image onto the display 102 such that the virtual image appears superimposed over the real-world view of the wearer.”)
Regarding claim 6, U.S. Patent No. 12115028teaches:
The method of claim 1,
Mahmood teaches: further comprising combining by the XR headset the generated XR image with a real-world scene for augmented reality viewing by the user. (Mahmood [0019], “In examples where the display 102 includes transparent lenses, the wearer can observe the real-world view through the transparent lenses, and a projection device (not shown) can project a virtual image onto the display 102 such that the virtual image appears superimposed over the real-world view of the wearer.”)
Regarding claim 7, U.S. Patent No. 12115028in view of Mahmood teaches:
The method of claim 6,
Mahmood teaches: wherein the XR headset includes an at least partially transparent screen to combine the generated XR image with the real-world scene. (Mahmood [0019], “In examples where the display 102 includes transparent lenses, the wearer can observe the real-world view through the transparent lenses, and a projection device (not shown) can project a virtual image onto the display 102 such that the virtual image appears superimposed over the real-world view of the wearer.”)
Regarding claim 9, U.S. Patent No. 12115028teaches:
The method of claim 1,
Mahmood teaches: wherein the XR headset includes a set of cameras, the method further comprising when a dynamic reference array of a surgical tool comes within a field of view of the cameras, displaying a graphical representation of the surgical tool on in relation to the generated XR image.( Mahmood [0057], “Further, based on a detected location of a hand of the surgeon 202, the surgical imaging system 200 can detect that the surgeon 202 is interacting with a 3D model of internal features of the patient 204 and responsively cause the robotic device 216 to perform one or more corresponding surgical procedures on the patient 204. For instance, the HMD 210 can include motion capture cameras to detect a location of the hand of the surgeon 202 within the 3D coordinate system. Other positional tracking sensors can be used as well, such as one or more IMUs included in the external device 504 or otherwise attached to the hand of the surgeon 202. Based on the determined location of the hand of the surgeon 202, the surgical imaging system 200 can determine the location of the hand of the surgeon 202 relative to the 3D model of internal features of the patient 204. The relative location of the hand of the surgeon 202 to the 3D model can then be used to detect an interaction between the surgeon 202 and the 3D model. For instance, the surgical imaging system 200 can detect the surgeon 202 performing a pinching gesture on one or more features of the 3D model, and the robotic surgical device 216 can responsively perform a corresponding pinching action (e.g., using forceps, needle drivers, clamps, pliers, etc.) on the corresponding actual internal feature of the patient 204.”)
For the above claims 4, 6, 7, 9, It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the teachings of U.S. Patent No. 12115028with the teachings of Mahmood to effectively and accurately generate an XR image to help surgeons perform operations.
Claims 12, 14, 15, 17 recite similar limitations of claim 4, 6-7, 9 respectively, thus are rejected accordingly.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mahmood et al. (US 2017/0296292 A1) in view of May et al. (US 2018/0256256 A1).
Regarding claim 1, Mahmood teaches:
A method of operating an XR headset in a surgical system, (FIG. 2) the method comprising:
providing a 3D volume of an anatomical structure of a 3D patient scan from a medical imaging device, ([0002] “Medical imaging techniques allow for three-dimensional (3D) representations of various parts of the human body. For example, an X-ray computed tomography scan (CT scan) combines multiple X-ray images to produce cross-sectional images of a scanned object. Digital geometry processing can then be applied to the X-ray images to generate a 3D representation of the scanned object. Similarly, magnetic resonance imaging (MRI) can generate 3D representations by measuring a spatial distribution of water in the scanned object. Other medical imaging techniques can be used to generate 3D representations, such as ultrasound, positron emission tomography (PET), fluoroscopy, tractography, diffused tensor imaging (DTI), and nuclear magnetic resonance (NMR) spectroscopy, to name a few.”)
generating, by an XR headset controller, a first 2D image representing an MIP (maximal intensity projection) view from a perspective of a left eye from the provided 3D volume based on a pose of the XR headset; ([0033], “Based on the 3D coordinates of the fiducial markers 206, 212 and the determined orientation of the HMD 210, the surgical imaging system 200 can display an augmented reality within the field of view of the surgeon 202. For instance, the HMD 210 can display the captured 3D image data of one or more internal features of the patient 204 to the surgeon 202 via a display of the HMD 210.”[0020], “In other examples, the 3D information can be displayed using stereoscopy in which different 2D images are displayed to the left and right eye in order to give the perception of 3D depth. Other methods of displaying the 3D image information can be used as well.” [0019], “In examples where the display 102 includes transparent lenses, the wearer can observe the real-world view through the transparent lenses, and a projection device (not shown) can project a virtual image onto the display 102 such that the virtual image appears superimposed over the real-world view of the wearer.”)
generating, by the XR headset controller, a second 2D image representing an MIP view from a perspective of a right eye from the provided 3D volume based on a pose of the XR headset; ([0033], “Based on the 3D coordinates of the fiducial markers 206, 212 and the determined orientation of the HMD 210, the surgical imaging system 200 can display an augmented reality within the field of view of the surgeon 202. For instance, the HMD 210 can display the captured 3D image data of one or more internal features of the patient 204 to the surgeon 202 via a display of the HMD 210.” [0020], “In other examples, the 3D information can be displayed using stereoscopy in which different 2D images are displayed to the left and right eye in order to give the perception of 3D depth. Other methods of displaying the 3D image information can be used as well.” [0019], “In examples where the display 102 includes transparent lenses, the wearer can observe the real-world view through the transparent lenses, and a projection device (not shown) can project a virtual image onto the display 102 such that the virtual image appears superimposed over the real-world view of the wearer.”)
generating, by the XR headset controller, a XR image by displaying the first 2D image in a field of view of the left eye of the user and displaying the second 2D image in a field of view of the right eye of the user such that the displayed first and second 2D images provide a depth perception. ([0020], “In other examples, the 3D information can be displayed using stereoscopy in which different 2D images are displayed to the left and right eye in order to give the perception of 3D depth. Other methods of displaying the 3D image information can be used as well.”) and
responsive to movement of the XR headset, regenerating the first and second 2D images representing the MIP views such that the 2D images behave like a 3D object. (The sensor 208 detects the fiducial marker 108/212 to tracking the movement of the HMD. Based on the location of the 108/212, the system generates and displays the 2D images on the left and right eyes and produces the 3D effects: [0023], “The fiducial marker 108 can be any feature capable of being detected by one or more sensors remote from the HMD 100 to determine a position of the HMD 100. For instance, the fiducial marker 108 can be retroreflective such that the marker reflects incoming light back towards a light source. Such retroreflective markers can be tracked using optical tracking systems, such as a laser tracker or a motion capture system, among others. By measuring the manner in which light is reflected off the fiducial marker 108, an optical tracking system can determine with high precision a three-dimensional location of the fiducial marker 108 relative to the optical tracking system.”[0033], “Based on the 3D coordinates of the fiducial markers 206, 212 and the determined orientation of the HMD 210, the surgical imaging system 200 can display an augmented reality within the field of view of the surgeon 202. For instance, the HMD 210 can display the captured 3D image data of one or more internal features of the patient 204 to the surgeon 202 via a display of the HMD 210.”)
However, Mahmood does not, but May teaches:
the 3D volume containing both bone anatomy and non-bone anatomy surrounding the bone anatomy and which has not been transformed into a 3D mesh; ([0060], “e technique 700 may include determining a relative physical position and orientation of the physical feature to the anatomical aspect, such as preoperatively using an x-ray or an MRI. Presenting the virtual feature may include presenting a virtual saw blade corresponding to a saw blade, and wherein the anatomical aspect is a bone of the patient. The AR display may display a virtual indication of a cut completion location for the saw blade. The technique 700 may include displaying, using the AR display, a warning that the hidden physical feature is approaching or has crossed a threshold safety zone. An audible warning may be played indicating that the hidden feature is approaching or has crossed a threshold safety zone. In some examples, displaying the virtual feature may include calculating an approximate location of the virtual feature in reference to portion of adjacent anatomy, such as approximating positions of soft tissue based on known positions of adjacent bones. In these examples, pre-operative imaging allows for creation of starting 3D models of anatomy surrounding a joint, such as the knee, with approximations generated based on movement of adjacent bones from known locations in the starting 3D model.”)
Mahmood teaches a method of XR headset in a surgical system, where the brain is given as an example. May teaches a bone with surrounding tissue can be as a surgical example.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have applied the surgical method of Mahmood to the bone structure to help surgeons to operate on a bone structure.
Regarding claim 2, Mahmood in view of May teaches:
The method of claim 1, wherein providing a 3D volume of an anatomical structure of a 3D patient scan includes providing a plurality of 2D image data as a plurality of 2D scans of the anatomical structure. (Mahmood [0002] “Medical imaging techniques allow for three-dimensional (3D) representations of various parts of the human body. For example, an X-ray computed tomography scan (CT scan) combines multiple X-ray images to produce cross-sectional images of a scanned object. Digital geometry processing can then be applied to the X-ray images to generate a 3D representation of the scanned object.”)
Regarding claim 3, Mahmood in view of May teaches:
The method of claim 1, wherein providing a 3D volume of an anatomical structure of a 3D patient scan includes providing a plurality of 2D slices of a DICOM scan. (Mahmood [0002] “Medical imaging techniques allow for three-dimensional (3D) representations of various parts of the human body. For example, an X-ray computed tomography scan (CT scan) combines multiple X-ray images to produce cross-sectional images of a scanned object. Digital geometry processing can then be applied to the X-ray images to generate a 3D representation of the scanned object. Similarly, magnetic resonance imaging (MRI) can generate 3D representations by measuring a spatial distribution of water in the scanned object. Other medical imaging techniques can be used to generate 3D representations, such as ultrasound, positron emission tomography (PET), fluoroscopy, tractography, diffused tensor imaging (DTI), and nuclear magnetic resonance (NMR) spectroscopy, to name a few.”)
Regarding claim 4, Mahmood in view of May teaches:
The method of claim 1, wherein generating a first 2D image includes projecting the 3D volume on a plane at an orientation defined by the pose of the XR headset. (Mahmood [0033], “Based on the 3D coordinates of the fiducial markers 206, 212 and the determined orientation of the HMD 210, the surgical imaging system 200 can display an augmented reality within the field of view of the surgeon 202. For instance, the HMD 210 can display the captured 3D image data of one or more internal features of the patient 204 to the surgeon 202 via a display of the HMD 210.” [0020], “In other examples, the 3D information can be displayed using stereoscopy in which different 2D images are displayed to the left and right eye in order to give the perception of 3D depth. Other methods of displaying the 3D image information can be used as well.” [0019], “In examples where the display 102 includes transparent lenses, the wearer can observe the real-world view through the transparent lenses, and a projection device (not shown) can project a virtual image onto the display 102 such that the virtual image appears superimposed over the real-world view of the wearer.”)
Regarding claim 5, Mahmood in view of May teaches:
The method of claim 1, further comprising determining the pose of the XR headset based on an array of tracking markers on the XR headset as viewed by a camera tracking system. (Mahmood [0032]-[0033], “The tracking sensors 208 can determine a 3D position of the fiducial markers 206, 212 relative to a 3D position of the tracking sensors 208, for instance by measuring the manner in which light reflects off of the fiducial markers 206, 212. By assigning a reference point or origin, a 3D coordinate system can be established within the surgical imaging system 200. For instance, the tracking sensors 208 can be located in fixed positions in the surgical imaging system 200. The location of one of the tracking sensors 208 can be treated as the origin of the 3D coordinate system. 3D coordinates (e.g., Cartesian or polar coordinates) can then be associated with each of the fiducial markers 206, 212 based on the measured position of the fiducial markers 206, 212 relative to the tracking sensors 208. Based on the 3D coordinates of the fiducial markers 206, 212 and the determined orientation of the HMD 210, the surgical imaging system 200 can display an augmented reality within the field of view of the surgeon 202. For instance, the HMD 210 can display the captured 3D image data of one or more internal features of the patient 204 to the surgeon 202 via a display of the HMD 210. Based on the orientation of the HMD 210 and the determined relative positions of the fiducial markers 206 to the HMD 210, the HMD 210 can display the 3D image data so that the internal features of the patient 204 appears superimposed on at least a portion of the patient 204 within the field of view of the surgeon 202. Such an augmented reality scenario 300 is illustrated in FIG. 3.”)
Regarding claim 6, Mahmood in view of May teaches:
The method of claim 1, further comprising combining by the XR headset the generated XR image with a real-world scene for augmented reality viewing by the user. (Mahmood [0019], “In examples where the display 102 includes transparent lenses, the wearer can observe the real-world view through the transparent lenses, and a projection device (not shown) can project a virtual image onto the display 102 such that the virtual image appears superimposed over the real-world view of the wearer.”)
Regarding claim 7, Mahmood in view of May teaches:
The method of claim 6, wherein the XR headset includes an at least partially transparent screen to combine the generated XR image with the real-world scene. (Mahmood [0019], “In examples where the display 102 includes transparent lenses, the wearer can observe the real-world view through the transparent lenses, and a projection device (not shown) can project a virtual image onto the display 102 such that the virtual image appears superimposed over the real-world view of the wearer.”)
Regarding claim 8, Mahmood in view of May teaches:
The method of claim 1, further comprising: responsive to generating the first 2D image, adjusting a window or level of the first 2D image. (Mahmood [0048], the projected 2D image (left or right 2D image to compose a 3D image) is adjusted to positionally offset: “FIG. 5 illustrates an augmented reality scenario 500 in which the HMD 210 displays a 3D model 502 that is away (e.g., positionally offset) from the body of the patient 204.”.)
Regarding claim 9, Mahmood in view of May teaches:
The method of claim 1, wherein the XR headset includes a set of cameras, the method further comprising when a dynamic reference array of a surgical tool comes within a field of view of the cameras, displaying a graphical representation of the surgical tool on in relation to the generated XR image.( Mahmood [0057], “Further, based on a detected location of a hand of the surgeon 202, the surgical imaging system 200 can detect that the surgeon 202 is interacting with a 3D model of internal features of the patient 204 and responsively cause the robotic device 216 to perform one or more corresponding surgical procedures on the patient 204. For instance, the HMD 210 can include motion capture cameras to detect a location of the hand of the surgeon 202 within the 3D coordinate system. Other positional tracking sensors can be used as well, such as one or more IMUs included in the external device 504 or otherwise attached to the hand of the surgeon 202. Based on the determined location of the hand of the surgeon 202, the surgical imaging system 200 can determine the location of the hand of the surgeon 202 relative to the 3D model of internal features of the patient 204. The relative location of the hand of the surgeon 202 to the 3D model can then be used to detect an interaction between the surgeon 202 and the 3D model. For instance, the surgical imaging system 200 can detect the surgeon 202 performing a pinching gesture on one or more features of the 3D model, and the robotic surgical device 216 can responsively perform a corresponding pinching action (e.g., using forceps, needle drivers, clamps, pliers, etc.) on the corresponding actual internal feature of the patient 204.”)
Regarding claim 10, Mahmood in view of May teaches:
The method of claim 5, wherein the camera tracking system is further configured to track a pose of a surgical tool, wherein the method of generating the XR image includes a graphical representation of the surgical tool based on the pose of the surgical tool relative to the anatomical structure. (Mahmood [0057], “Further, based on a detected location of a hand of the surgeon 202, the surgical imaging system 200 can detect that the surgeon 202 is interacting with a 3D model of internal features of the patient 204 and responsively cause the robotic device 216 to perform one or more corresponding surgical procedures on the patient 204. For instance, the HMD 210 can include motion capture cameras to detect a location of the hand of the surgeon 202 within the 3D coordinate system. Other positional tracking sensors can be used as well, such as one or more IMUs included in the external device 504 or otherwise attached to the hand of the surgeon 202. Based on the determined location of the hand of the surgeon 202, the surgical imaging system 200 can determine the location of the hand of the surgeon 202 relative to the 3D model of internal features of the patient 204. The relative location of the hand of the surgeon 202 to the 3D model can then be used to detect an interaction between the surgeon 202 and the 3D model. For instance, the surgical imaging system 200 can detect the surgeon 202 performing a pinching gesture on one or more features of the 3D model, and the robotic surgical device 216 can responsively perform a corresponding pinching action (e.g., using forceps, needle drivers, clamps, pliers, etc.) on the corresponding actual internal feature of the patient 204.”)
Regarding claim 11, Mahmood teaches:
A method of operating an XR headset in a surgical system, (Fig. 2) the method comprising:
providing a 3D volume of an anatomical structure of a 3D patient scan from a medical imaging device, the 3D volume including a plurality of 2D DICOM files defining a plurality of 2D scans at various depths; ([0002] “Medical imaging techniques allow for three-dimensional (3D) representations of various parts of the human body. For example, an X-ray computed tomography scan (CT scan) combines multiple X-ray images to produce cross-sectional images of a scanned object. Digital geometry processing can then be applied to the X-ray images to generate a 3D representation of the scanned object. Similarly, magnetic resonance imaging (MRI) can generate 3D representations by measuring a spatial distribution of water in the scanned object. Other medical imaging techniques can be used to generate 3D representations, such as ultrasound, positron emission tomography (PET), fluoroscopy, tractography, diffused tensor imaging (DTI), and nuclear magnetic resonance (NMR) spectroscopy, to name a few.” X-ray and CT generate 2D images with various depth.)
generating, by an XR headset controller, a first 2D image representing an MIP (maximal intensity projection) view from a perspective of a left eye from the provided 3D volume based on a pose of the XR headset; ([0033], “Based on the 3D coordinates of the fiducial markers 206, 212 and the determined orientation of the HMD 210, the surgical imaging system 200 can display an augmented reality within the field of view of the surgeon 202. For instance, the HMD 210 can display the captured 3D image data of one or more internal features of the patient 204 to the surgeon 202 via a display of the HMD 210.”[0020], “In other examples, the 3D information can be displayed using stereoscopy in which different 2D images are displayed to the left and right eye in order to give the perception of 3D depth. Other methods of displaying the 3D image information can be used as well.” [0019], “In examples where the display 102 includes transparent lenses, the wearer can observe the real-world view through the transparent lenses, and a projection device (not shown) can project a virtual image onto the display 102 such that the virtual image appears superimposed over the real-world view of the wearer.”)
generating, by the XR headset controller, a second 2D image representing an MIP view from a perspective of a right eye from the provided 3D volume based on a pose of the XR headset; ([0033], “Based on the 3D coordinates of the fiducial markers 206, 212 and the determined orientation of the HMD 210, the surgical imaging system 200 can display an augmented reality within the field of view of the surgeon 202. For instance, the HMD 210 can display the captured 3D image data of one or more internal features of the patient 204 to the surgeon 202 via a display of the HMD 210.”[0020], “In other examples, the 3D information can be displayed using stereoscopy in which different 2D images are displayed to the left and right eye in order to give the perception of 3D depth. Other methods of displaying the 3D image information can be used as well.” [0019], “In examples where the display 102 includes transparent lenses, the wearer can observe the real-world view through the transparent lenses, and a projection device (not shown) can project a virtual image onto the display 102 such that the virtual image appears superimposed over the real-world view of the wearer.”)
generating, by the XR headset controller, a XR image by displaying the first 2D image in a field of view of the left eye of the user and displaying the second 2D image in a field of view of the right eye of the user such that the displayed first and second 2D images provide a depth perception. ([0020], “In other examples, the 3D information can be displayed using stereoscopy in which different 2D images are displayed to the left and right eye in order to give the perception of 3D depth. Other methods of displaying the 3D image information can be used as well.”)
However, Mahmood does not, but May teaches:
the 3D volume containing both bone anatomy and non-bone anatomy surrounding the bone anatomy and which has not been transformed into a 3D mesh, ([0060], “e technique 700 may include determining a relative physical position and orientation of the physical feature to the anatomical aspect, such as preoperatively using an x-ray or an MRI. Presenting the virtual feature may include presenting a virtual saw blade corresponding to a saw blade, and wherein the anatomical aspect is a bone of the patient. The AR display may display a virtual indication of a cut completion location for the saw blade. The technique 700 may include displaying, using the AR display, a warning that the hidden physical feature is approaching or has crossed a threshold safety zone. An audible warning may be played indicating that the hidden feature is approaching or has crossed a threshold safety zone. In some examples, displaying the virtual feature may include calculating an approximate location of the virtual feature in reference to portion of adjacent anatomy, such as approximating positions of soft tissue based on known positions of adjacent bones. In these examples, pre-operative imaging allows for creation of starting 3D models of anatomy surrounding a joint, such as the knee, with approximations generated based on movement of adjacent bones from known locations in the starting 3D model.”)
Mahmood teaches a method of XR headset in a surgical system, where the brain is given as an example. May teaches a bone with surrounding tissue can be as a surgical example.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have applied the surgical method of Mahmood to the bone structure to help surgeons to operate on a bone structure.
Claims 12-18 recite similar limitations of claim 4-10 respectively, thus are rejected accordingly.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YANNA WU whose telephone number is (571)270-0725. The examiner can normally be reached Monday-Thursday 8:00-5:30 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at 5712722330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YANNA WU/Primary Examiner, Art Unit 2615
Prosecution Insights