DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 10010379. Although the claims at issue are not identical, they are not patentably distinct from each other because
19031208 (claim 1)
US 10010379 (claim 1)
A method for augmenting medical imaging for use in a medical procedure, comprising:
A method for augmenting medical imaging of a patient during a medical procedure, the medical imaging displayed using a augmented reality headset worn by a medical professional during the medical procedure, the method comprising:
receiving an image of patient anatomy captured by a visual image camera during the medical procedure;
receiving a visual image of patient anatomy captured by a visual image camera during the medical procedure, the visual image comprising a viewable portion of the patient anatomy obtained during the medical procedure;
retrieving an acquired medical image associated with the patient anatomy;
retrieving an acquired medical image associated with the patient anatomy from data storage, the acquired medical image comprising imaging acquired of one or more anatomical structures at a plurality of anatomical layers of the patient anatomy;
associating the acquired medical image to the patient anatomy;
associating the acquired medical image to align with the viewable portion of the patient anatomy captured by the visual image camera, wherein the one or more anatomical structures of the medical imaging at the plurality of layers are aligned with the visual image of the patient anatomy;
retrieving an augmentation tag associated with a location in one layer of the acquired medical image; and
retrieving an augmentation tag from data storage, the augmentation tag associated with a location in one layer of the acquired medical image, the augmentation tag identifying at least one anatomical structure of the acquired medical image found at the location, the augmentation tag further comprising a shape of the anatomical structure; and
projecting the acquired medical image and the augmentation tag using an augmented
reality headset to form a single graphical view as an overlay to the patient anatomy.
projecting, during the medical procedure, the acquired medical image and the augmentation tag using the augmented reality headset to form a single graphical view as an overlay to the patient anatomy viewable through a lens of the augmented reality headset.
US 10010379 (claim 1)’s limitations anticipate this application 19031208 (claim 1)’s respective limitations while US 10010379 (claim 1)’s differences of the limitations just have more specific details.
19031208 (claim 8)
US 10010379 (claim 8)
A non-transitory machine readable storage medium having instructions embodied thereon, the instructions when executed cause a processor to augment medical imaging for a medical procedure using an AR headset, comprising:
A non-transitory machine readable storage medium having instructions embodied thereon, the instructions when executed cause a processor to augment medical imaging of a patient during [[for]] a medical procedure using an augmented reality headset worn by a medical professional during the medical procedure, comprising:
receiving an image of patient anatomy using a live image camera during the medical procedure;
receiving a visual image of patient anatomy captured by a visual image camera during the medical procedure, the visual image comprising a viewable portion of the patient anatomy obtained during the medical procedure;
identifying a patient marker in the image of the patient anatomy and the patient
marker includes information identifying the patient, anatomy to be operated upon, a patient orientation marker or an image inversion prevention tag;
identifying a patient marker in the visual image of the patient anatomy, the patient marker includes comprising information identifying the patient, information identifying patient anatomy that is the subject of the medical procedure to be operated upon, a patient orientation marker, or an image inversion prevention tag;
retrieving an acquired medical image associated with the patient anatomy based in part on the patient marker;
retrieving an acquired medical image associated with the patient anatomy from a data store based in part on the patient marker, the acquired medical image comprising imaging acquired of one or more anatomical structures at a plurality of anatomical layers of the patient anatomy;
anchoring the acquired medical image to the patient anatomy based in part on the
patient orientation marker;
anchoring the acquired medical image to the patient anatomy based in part on the patient orientation marker;
retrieving an augmentation tag associated with the patient marker and a location in the acquired medical image; and
retrieving an augmentation tag from data storage, the augmentation tag associated with the patient marker and a location in the acquired medical image, the augmentation tag identifying at least one anatomical structure of the acquired medical image found at the location, the augmentation tag further comprising a shape of the anatomical structure; and
projecting the acquired medical image and the augmentation tag onto lenses in an augmented reality headset to form a single graphical view which is overlaid on the patient anatomy being viewed.
projecting, during the medical procedure, the acquired medical image and the augmentation tag onto lenses in an augmented reality headset to form a single graphical view which is overlaid on the patient anatomy viewable by the medical professional through the lenses of the augmented reality headset.
US 10010379 (claim 8)’s limitations anticipate this application 19031208 (claim 8)’s respective limitations while US 10010379 (claim 8)’s differences of the limitations just have more specific details.
19031208 (claim 13)
US 10010379 (claim 13)
A system for augmenting a view of a medical procedure, comprising:
A system for augmenting a view of patent anatomy during a medical procedure for a medical professional using an augmented reality headset, comprising:
a camera configured to obtain images of patient anatomy during the medical
procedure;
a camera configured to obtain images of the patient anatomy during the medical procedure, the images comprising the view of the patient anatomy;
an augmentation processor configured to:
an augmentation processor in communication with the camera and configured to:
capture morphometric measurements of patient anatomy from the images;
capture morphometric measurements of the patient anatomy from the images captured by the camera during the medical procedure;
identify a patient marker in the image of the patient anatomy and the patient
marker includes information identifying the patient in order to retrieve premeasured morphometric measurements;
identify a patient marker in the images captured by the camera of the patient anatomy, the patient marker comprising information identifying the patient in order to retrieve pre-measured morphometric measurements;
retrieving pre-measured morphometric measurements associated with the patient anatomy from data storage using the patient marker identified in the images captured by the camera;
determine whether the morphometric measurements from the images match the pre-measured morphometric measurements associated with the patient anatomy as retrieved using the patient marker;
determine whether the morphometric measurements of the patent anatomy captured from the image[[s]] match the pre-measured morphometric measurements associated with the patient anatomy as retrieved using the patient marker;
retrieving an acquired medical image associated with the patient anatomy as
defined by the patient marker and matched morphometric measurements;
retrieving an acquired medical image associated with the patient anatomy as defined by the patient marker and matched morphometric measurements from data storage, the acquired medical image comprising imaging acquired of one or more anatomical structures at a plurality of anatomical layers of the patient anatomy;
aligning the acquired medical image with the patient anatomy using the
morphometric measurements;
aligning the acquired medical image with the view provided by the augmented reality headset during the medical procedure of the patient anatomy using the morphometric measurements; and
form a single graphical view with the acquired medical image and an augmentation tag, the augmentation tag identifying at least one anatomical structure of the acquired medical image found at the location, the augmentation tag further comprising a shape of the anatomical structure; and
an augmented reality headset configured to:
an augmented reality headset and in communication with the augmentation processor and configured to:
project the acquired medical image and an augmentation tag onto lenses in an augmented reality headset to form a single graphical view which is overlaid on the patient anatomy being viewed; and
project the single graphical view formed from the acquired medical image and the augmentation tag onto lenses in an augmented reality headset to form a single graphical view which is overlaid on the view of the patient anatomy during the medical procedure being viewed; and
provide a notification the acquired medical image matches the patient anatomy.
provide a notification the acquired medical image matches the patient anatomy.
US 10010379 (claim 13)’s limitations anticipate this application 19031208 (claim 13)’s respective limitations while US 10010379 (claim 13)’s differences of the limitations just have more specific details.
The following table provides the matching for other claims.
19031208
2-7
9-12
14-20
US 10010379
2-7
9-12
14-20
Drawings
The drawings are objected to because: in FIG. 5 “Data Storage” and the patient marker 530 share the same label 530. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities:
In page 5, line 29, “122 (FIG. 1B)” should read “122 (FIG. 1C)”.
In page 8, line 15, “FIG. 3D illustrates” should read “FIG. 2D illustrates”.
In page 12, line 13, “An acquired medical image 524” should read “An acquired medical image 522”.
In page 12, line 15, “The acquired medical image 540” should read “The acquired medical image 522”.
Appropriate correction is required.
Claim Objections
Claim 13 is objected to because of the following informalities:
In claim 13, line 12, “retrieving an” should read “retrieve an”.
In claim 13, line 14, “aligning the” should read “align the”.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3 and 7-12 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (20140275760 A1, hereinafter “Lee”) in view of Schoepp (US 20120078236 A1, hereinafter “Schoepp”).
Regarding claim 1, Lee discloses A method for augmenting medical imaging for use in a medical procedure, comprising: (para. [0105], “The apparatus and methods for controlling a configuration or operation mode of the surgical robot and augmented reality image display system according to the above described example embodiments may use one or more processors”).
receiving an image of patient anatomy captured by a visual image camera during the medical procedure; (para. [0086], “a "real image" may refer to an image of the real world captured by the camera 410 and may be an image of the patient P lying on the operating table or the human body model P' in the illustrated embodiment. The real image may include the markers attached to the patient P or the human body model P'”).
retrieving an acquired medical image associated with the patient anatomy; (para. [0071], “The imaging system 300 may include an image storage unit 310 to store a 30 image generated using a medical image of the patient P before surgery, a virtual image obtained by projecting the 30 image onto an image acquired by the endoscope 220, and the like”). Note that: an acquired medical image can be retrieved from the storage unit 310.
associating the acquired medical image to the patient anatomy; (para. [0085], “a "virtual image" may include a 3D image generated using a medical image of the patient P before surgery and an image obtained by projecting the 3D image onto an image acquired by the endoscope 220, as described above. That is, a "virtual image" may be an image of the inside of the patient P's body.”). Note that: an image acquired by the endoscope 220 reflects the patient anatomy, and projecting the 3D image onto an image is an associating process.
projecting the acquired medical image and (para. [0075], “an augmented reality image generator 430 that detects the plurality of markers in the real image acquired by the camera 410, estimates position and gaze direction of the camera 410 using the detected markers, and generates an augmented reality image by overlaying a virtual image of a corresponding region over the real image, and a display 420 that displays the augmented reality image, as illustrated in FIG. 2”).
However, Lee fails to disclose, but in the same aft of computer graphics, Schoepp discloses
retrieving an augmentation tag associated with a location in one layer of the acquired medical image; … the augmentation tag … and (Schoepp, para. [0026], “the camera assembly 16 includes a plurality, preferably three or more, fiducial markers 34, such as metal balls, targets, or other known types of fiducial markers that can be identified in visual light, CT scan, MRI, X-ray, and/or other types of imaging modalities … As shown in FIG. 1, the fiducial markers 34 comprise metal balls, which are visible in most applicable imaging modalities, disposed directly on the top side of the base plate 38 in positions selected to maximize the ability and accuracy of triangulation calculations that will be based thereon, such as nearly an equilateral triangle or a right triangle as shown in FIG. 1.”). Note that: (1) fiducial markers 34 can be presented in one layer of the acquired medical image (e.g., an MRI image layer or slice); and (2) the presented fiducial markers 34 in the acquired medical image as indicators can be regarded as augmentation tags associated with locations.
Lee and Schoepp are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply retrieving augmentation tags associated with locations in acquired medical image, as taught by Schoepp into Lee. The motivation would have been “the camera assembly 16 includes a plurality, preferably three or more, fiducial markers 34, such as metal balls, targets, or other known types of fiducial markers that can be identified in visual light, CT scan, MRI, X-ray, and/or other types of imaging modalities” (Schoepp, para. [0026]). The suggestion for doing so would allow to retrieve augmentation tags associated with locations in at least one layer of the acquired medical image . Therefore, it would have been obvious to combine Lee and Schoepp.
Regarding claim 2, Lee in view of Schoepp discloses The method as in claim 1, further wherein augmentation tag conforms to a three dimensional (3D) structure in the acquired medical image to identify an anatomical structure associated with the medical procedure to be performed. (Lee, para. [0073], “The controller 320 may generate a virtual image obtained by projecting the 30 image onto the received real image by the virtual image generator 323”; para. [0092], “detects the markers, estimates that the camera 410 faces the abdomen of the patient Pat the left side of the patient P in a state of being spaced apart from the patient P using position information of each marker in the real image, and overlays a virtual image of the corresponding region received from the imaging system 300 over the real image”). Note that: augmentation tag is used to conform or overlay a virtual image of the corresponding region received from the imaging system 300 over the real image.
Regarding claim 3, Lee in view of Schoepp discloses The method as in claim 1, wherein a plurality of augmentation tags are provided in a plurality of layers of the acquired medical image and the plurality of augmentation tags are associated to form a tag group to guide a surgeon. (Schoepp, para. [0041], “a surgeon manipulates the surgical tool 12 through the nasal cavities of the patient 20, the location of the tool tip 28 inside the patient's head is shown on the display monitor 24 in correlated relation to the scan image 48, thereby effectively allowing the surgeon to see on the display device where the surgical tool 12 is in relation to the target location 62 inside the patient 20”; para. [0026], “the camera assembly 16 includes a plurality, preferably three or more, fiducial markers 34, such as metal balls, targets, or other known types of fiducial markers that can be identified in visual light, CT scan, MRI, X-ray, and/or other types of imaging modalities). Note that: since the presented fiducial markers (a plurality of augmentation tags large in size) in the acquired medical image usually occupy a plurality of layers (slices), augmentation tags are associated to the presented fiducial markers to form a tag group to guide a surgeon for the layers (slices).
The motivation to combine Lee and Schoepp given in claim 1 is incorporated here.
Regarding claim 7, Lee in view of Schoepp discloses The method as in claim 1, further comprising capturing the acquired medical image using a MRI, CT scan, X-ray, ultrasound, or photographic images internal to a human body. (Schoepp, para. [0041], “The scan image 48 may be any image useful to the surgeon to plan and/or execute the surgical procedure, and will generally include such modalities as a CT, ultra-sound, Xray, MRI, infra-red, visible spectrum, and any other imaging modality suitable for use with surgical procedures. Often, the scan image 48 includes image data relative to subcutaneous structures in the patient's body, such as organs, bones, tumors, cancers, blood clots, and/or other items of interest inside the patient's body”).
The motivation to combine Lee and Schoepp given in claim 1 is incorporated here.
Regarding claim 8, Lee discloses A non-transitory machine readable storage medium having instructions embodied thereon, the instructions when executed cause a processor to augment medical imaging for a medical procedure using an AR headset, comprising: (para. [0108], “non-transitory computer-readable media including program instructions to implement various operations embodied by a computer.”)
receiving an image of patient anatomy using a live image camera during the medical procedure; (FIG. 11: “IMAGE CAPRURED BY ENDOSCOPE” during a medical procedure). Note that: an endoscope has a live camera.
identifying a patient marker in the image of the patient anatomy and (para. [0075], “augmented reality image generator 430 that detects the plurality of markers in the real image acquired by the camera 410, estimates position and gaze direction of the camera 410 using the detected markers.").
retrieving an acquired medical image associated with the patient anatomy based in part on the patient marker; (para. [0085], “a virtual image may include a 3D image generated using a medical image of the patient P before surgery and an image obtained by projecting the 3D image onto an image acquired by the endoscope 220, as described above. That is, a “ virtual image” may be an image of the inside of the patient P’s body.”).
anchoring the acquired medical image to the patient anatomy (para. [0085], “a "virtual image" may include a 3D image generated using a medical image of the patient P before surgery and an image obtained by projecting the 3D image onto an image acquired by the endoscope 220, as described above. That is, a "virtual image" may be an image of the inside of the patient P's body.”). Note that: an image acquired by the endoscope 220 reflects the patient anatomy, and projecting the 3D image onto an image is an anchoring process.
projecting the acquired medical image and the augmentation tag onto lenses in an augmented reality headset to form a single graphical view which is overlaid on the patient anatomy being viewed. (para. [0075], “augmented reality image generator 430 that detects the plurality of markers in the real image acquired by the camera 410, estimates position and gaze direction of the camera 410 using the detected markers, and generates an augmented reality image by overlaying a virtual image of a corresponding region over the real image, and a display 420 that displays the augmented reality image, as illustrated in FIG. 2.”).
However, Lee fails to disclose, but in the same art of computer graphics, Schoepp discloses
… patient marker includes information identifying the patient, anatomy to be operated upon, a patient orientation marker or an image inversion prevention tag; (Schoepp, para. [0006], “A target patch with optical navigation markers, printed thereon, is placed on the patient's body at or near a desired point of entry into the body. The patch is registered with the body, and the camera views the patch as the surgical tool is inserted into the patient's body.”…”The camera is connected to a computer processor, which is programmed to process images of the optical navigation markers from the camera to determine a position of the surgical tool in reference to the patch and, thereby, the patient's body.”).
retrieving an augmentation tag associated with the patient marker and a location in the acquired medical image; and (Schoepp, para. [0026], “the camera assembly 16 includes a plurality, preferably three or more, fiducial markers 34, such as metal balls, targets, or other known types of fiducial markers that can be identified in visual light, CT scan, MRI, X-ray, and/or other types of imaging modalities … As shown in FIG. 1, the fiducial markers 34 comprise metal balls, which are visible in most applicable imaging modalities, disposed directly on the top side of the base plate 38 in positions selected to maximize the ability and accuracy of triangulation calculations that will be based thereon, such as nearly an equilateral triangle or a right triangle as shown in FIG. 1.”). Note that: (1) fiducial markers 34 can be presented in one layer of the acquired medical image (e.g., an MRI image layer or slice); and (2) the presented fiducial markers 34 in the acquired medical image as indicators can be regarded as augmentation tags associated with locations.
… anchoring the acquired medical image to the patient anatomy based in part on the patient orientation marker; (Schoepp, para. [0021], “However, in other applications, the tracking element 14 may include active markers, such as LEDs and/or other types of navigation markers that may be sensed by the camera 18. The surgical tool 12 and tracking element 14 are pre-calibrated, whereby the position of the tracking element 14 with respect to the camera assembly 16 may also be automatically identified based on the size and orientation of the optical navigation markers 30 in images obtained from the camera 18.”). Note that: the patient orientation marker and other navigation markers can be used to track and match orientation of the acquired medical image with the patient orientation marker on the patient anatomy during performing anchoring and tracking the acquired medical image to the patient anatomy.
The motivation to combine Lee and Schoepp given in claim 1 is incorporated here.
Regarding claim 9, Lee in view of Schoepp discloses The non-transitory machine readable storage medium as in claim 8, further comprising matching the patient orientation marker with an image orientation marker in the acquired medical image to enable matching orientation of the acquired medical image with the patient orientation marker on the patient anatomy. (Schoepp, para. [0021], “However, in other applications, the tracking element 14 may include active markers, such as LEDs and/or other types of navigation markers that may be sensed by the camera 18. The surgical tool 12 and tracking element 14 are pre-calibrated, whereby the position of the tracking element 14 with respect to the camera assembly 16 may also be automatically identified based on the size and orientation of the optical navigation markers 30 in images obtained from the camera 18.”)
The motivation to combine Lee and Schoepp given in claim 8 is incorporated here.
Regarding claim 10, Lee in view of Schoepp discloses The non-transitory machine readable storage medium as in claim 8, further comprising using a visually scannable symbol attached to a patient to retrieve an identity of a patient and to enable retrieval of the acquired medical image associated with the visually scannable symbol. (Schoepp, para. [0006], “A target patch with optical navigation markers, printed thereon, is placed on the patient's body at or near a desired point of entry into the body. The patch is registered with the body, and the camera views the patch as the surgical tool is inserted into the patient's body”).
The motivation to combine Lee and Schoepp given in claim 8 is incorporated here.
Regarding claim 11, Lee in view of Schoepp discloses The non-transitory machine readable storage medium as in claim 8, further comprising using the image inversion prevention tag to ensure the acquired medical image is not inverted with respect to the patient anatomy. (Schoepp, para. [0061], “In a point-to-point registration procedure, physical landmarks located at block 126 are matched with corresponding structures in the scan image 48. Thereafter, a point-to-point registration procedure is performed to register the camera space with the scan image space by, for example, a point-to-point matching procedure and appropriate coordinate transformations known in the art.”).
The motivation to combine Lee and Schoepp given in claim 8 is incorporated here.
Regarding claim 12, Lee in view of Schoepp discloses The non-transitory machine readable storage medium as in claim 8, further comprising:
combining an endoscopic video feed with the single graphical view; and (Lee, para. [0019], “virtual image generator to generate a virtual image by projecting the converted 3D image onto the image acquired by the endoscope.”)
using the augmentation tag to identify a size and shape of 3D structures in the endoscopic video feed and (Schoepp, para. [0055], “set of optical navigation markers 30, i.e., having different shapes, colors, and/or spacing,”) to match where an endoscope is estimated to be located in the patient anatomy using the augmentation tag. (Schoepp, para. [0061], “In a point-to-point registration procedure, physical landmarks located at block 126 are matched with corresponding structures in the scan image 48”).
The motivation to combine Lee and Schoepp given in claim 8 is incorporated here.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Schoepp and Hasan et al. (US 20160180441 A1, hereinafter “Hasan”).
Regarding claim 4, Lee in view of Schoepp discloses The method as in claim 1, further comprising:
However, Lee in view of Schoepp fails to disclose, but in the same art of computer graphics, Hasan discloses
creating a plurality of two dimensional (2D) augmentation tags each located on one of a plurality of layers of the acquired medical image; and
joining the plurality of 2D augmentation tags to fort a three dimensional (3D) augmentation tag that extends through multiple layers of the acquired medical image. (Hasan, para. [0046], “The 2D to 3D converter 362 may include a system capable of transforming a 2D image into a 3D image. To convert an image from a 2D image to a 3D image, the 2D to 3D converter 362 may apply one or more transformations to a portion of an image that includes a depiction of an item. In some cases, the 2D to 3D converter 362 may convert a 2D image of an item to a 3D image by extruding the 2D image. In some embodiments, the 2D to 3D converter 360 may determine how much to extrude and/or what other transformations to apply to a 2D image based on a comparison between the 2D image and a portion of the received image that includes the reference marker, such as by determining a perspective angle of the scene captured in the image based on skew and other characteristics identified from the reference marker depicted in the image.”). Note that: (1) for the presented augmentation tags each tag is presented as 2D tag and is located on one of a plurality of layers of the acquired medical image; and (2) each 2D can be joined or combined to form a three dimensional (3D) augmentation tag that extends through multiple layers of the acquired medical image.
Lee in view of Schoepp, and Hasan, are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply creating a plurality of 2D augmentation tags and forming a 3D augmentation tags by joining the plurality of 2D augmentation tags, as taught by Hasan into Lee in view of Schoepp. The motivation would have been “the 2D to 3D converter 362 may convert a 2D image of an item to a 3D image by extruding the 2D image” (Hasan, para. [0046]). The suggestion for doing so would allow to form three dimensional (3D) augmentation tag that extends through multiple layers of the acquired medical image. Therefore, it would have been obvious to combine Lee, Schoepp, and Hasan.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Schoepp and Rios et al. (US 20170178540 A1 hereinafter “Rios”).
Regarding claim 5, Lee in view of Schoepp discloses The method as in claim 1, wherein
However, Lee in view of Schoepp fails to disclose, but in the same art of computer graphics, Rios discloses
retrieving an augmentation tag further comprises retrieving a plurality of augmentation tags that are linked together to represent an anatomical structure in independent layers of the acquired medical image. (Rios, para. [0040], “In all FIGS. 2A-E, markers 260 are shown on various landmarks on the face of the simulated head 210, 220, 230, 240, 250. In some embodiments, these markers 260 represent injection targets. In some embodiments, different layers of the anatomical structure can be shown in different colors to distinguish the layers and/or tissue structures.”). Note that: markers 260 can be regarded as a plurality of augmentation tags that are linked together to represent an anatomical structure (head) in independent layers of the acquired medical image.
Lee in view of Schoepp, and Rios, are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply retrieving a plurality of augmentation tags that are linked together to represent an anatomical structure, as taught by Rios into Lee in view of Schoepp. The motivation would have been “markers 260 are shown on various landmarks on the face of the simulated head 210, 220, 230, 240, 250. In some embodiments, these markers 260 represent injection targets. In some embodiments, different layers of the anatomical structure can be shown in different colors to distinguish the layers and/or tissue structures” (Rios, para. [0040]). The suggestion for doing so would allow to retrieve a plurality of augmentation tags that are linked together to represent an anatomical structure in independent layers of the acquired medical image. Therefore, it would have been obvious to combine Lee, Schoepp, and Rios.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Schoepp and Slivka et al. (US 20160220393 A1, hereinafter “Slivka”).
Regarding claim 6, Lee in view of Schoepp discloses The method as in claim 1, wherein
However, Lee in view of Schoepp fails to disclose, but in the same art of computer graphics, Slivka discloses
retrieving an augmentation tag further comprises retrieving a plurality of augmentation tags that are linked together to show a surgical path through the patient anatomy. (Slivka, para. [0060], “The type, mode, and duration of applying the energy can be selected based on the tissue and the type of markers that are to be created to define the desired surgical access path”). Note that: the markers show or define a desired surgical path.
Lee in view of Schoepp, and Slivka, are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply links a plurality of augmentation tags to show or define a desired path, as taught by Slivka into Lee in view of Schoepp. The motivation would have been “The type, mode, and duration of applying the energy can be selected based on the tissue and the type of markers that are to be created to define the desired surgical access path” (Slivka, para. [0060]). The suggestion for doing so would allow to retrieve a plurality of augmentation tags that are linked together to show a surgical path through the patient anatomy. Therefore, it would have been obvious to combine Lee, Schoepp, and Slivka.
Claims 13-20 are rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Pearson (US 20140355840, hereinafter “Pearson”), Schoepp, and Kubota (US 20100049548, hereinafter “Kubota”).
Regarding claim 13, Lee discloses A system for augmenting a view of a medical procedure, comprising: (para. [0105], “The apparatus and methods for controlling a configuration or operation mode of the surgical robot and augmented reality image display system according to the above described example embodiments may use one or more processors”).
a camera configured to obtain images of patient anatomy during the medical procedure; (para. [0084], “a “real image” may refer to an image of the real world captured by the camera 410 and may be an image of the patient P lying on the operating table or the human body model P’ in the illustrated embodiment.”).
identify a patient marker in the image of the patient anatomy and (para. [0075],” augmented reality image generator 430 that detects the plurality of markers in the real image acquired by the camera 410, estimates position and gaze direction of the camera 410 using the detected markers").
retrieving an acquired medical image associated with the patient anatomy as defined by the patient marker and
an augmented reality headset configured to:
project the acquired medical image and an augmentation tag onto lenses in an augmented reality headset to form a single graphical view which is overlaid on the patient anatomy being viewed; and (para, [0075], “augmented reality image generator 430 that detects the plurality of markers in the real image acquired by the camera 410, estimates position and gaze direction of the camera 410 using the detected markers, and generates an augmented reality image by overlaying a virtual image of a corresponding region over the real image, and a display 420 that displays the augmented reality image, as illustrated in FIG. 2”).
However, Lee fails to disclose, but in the same art of Medical Applications, Pearson discloses
an augmentation processor configured to:
capture morphometric measurements of patient anatomy from the images; (Pearson, para. [0029], “In one example, breast landmarks may include nipples, pectoralis muscle, skin lines for breast size, shape or contour of the breast, for example, with calculation of measurements of posterior nipple line depth and estimated width and height of breast.”). Note that: morphometric measurements can be estimated from the images.
determine whether the morphometric measurements from the images match the pre-measured morphometric measurements associated with the patient anatomy as retrieved using (Pearson, prara. [0028], “current mammogram images are compared with stored prior mammogram images of patients queried under name and birthdate. A probabilistic match is calculated and a reasonable "match" is determined to provide a desirable level of secure release of a patient's medical records and mammogram images. In one example, once there is verification that prior mammogram images belong to the same patient, then access to these prior mammogram images may be given to the requesting medical personnel”; para. [0045], “An example process of matching mammogram images may include matching one or more of the following: breast landmarks, breast tissue densities, tissue patterns, breast findings and other distinguishing features. For example, matching breast landmarks may include matching nipples, pectoralis muscle, skin lines for breast size, shape or contour of the breast. The matching may include calculating the posterior nipple line depth and estimating the width and the height of breast”). Note that: the morphometric measurements from the prior breast images retrieved from data storage using patient ID and current breast images can be compared to determine if they match.
Lee and Pearson are in the same field of endeavor, namely Medical Applications. Before the effective filing date of the claimed invention, it would have been obvious to apply estimating or capturing morphometric measurements and determining if the morphometric measurements from prior images match that from current images, as taught by Pearson into Lee. The motivation would have been “with calculation of measurements of posterior nipple line depth and estimated width and height of breast” (Pearson, para. [0029]). The suggestion for doing so would allow to determine if the morphometric measurements from prior images match that from current images. Therefore, it would have been obvious to combine Lee and Pearson.
However, Lee in view of Pearson fails to disclose, but in the same art of computer graphics, Schoepp discloses
… the patient marker …
… the patient marker includes information identifying the patient in order to retrieve pre-measured morphometric measurements; (Schoepp, para. [0006], “A target patch with optical navigation markers, printed thereon, is placed on the patient's body at or near a desired point of entry into the body. The patch is registered with the body, and the camera views the patch as the surgical tool is inserted into the patient's body.”…”The camera is connected to a computer processor, which is programmed to process images of the optical navigation markers from the camera to determine a position of the surgical tool in reference to the patch and, thereby, the patient's body.”).
matched morphometric measurements; (Schoepp, para. [0061], “At block 128, the camera 18 is registered with the scan image 48 by matching points located on the body at block 126 to corresponding portions of the scan image 48 of the patient.”)
aligning the acquired medical image with the patient anatomy using the morphometric measurements; (Schoepp, para. [0061],”In a point-to-point registration procedure, physical landmarks located at block 126 are matched with corresponding structures in the scan image 48.”). Note that: after the registration, the acquired medical image is aligned with the patient anatomy.
Lee in view of Pearson, and Schoepp, are in the same field of endeavor, namely Medical Applications. Before the effective filing date of the claimed invention, it would have been obvious to apply aligning the acquired image with the patient anatomy, as taught by Schoepp into Lee in view of Pearson. The motivation would have been “In a point-to-point registration procedure, physical landmarks located at block 126 are matched with corresponding structures in the scan image 48 of the patient.” (Schoepp, para. [0061]). The suggestion for doing so would allow to align the acquired image with the patient anatomy using markers attached to patient body. Therefore, it would have been obvious to combine Lee, Pearson, and Schoepp.
However, the combination of Lee, Pearson, and Schoepp fails disclose, in the same art of Medical Applications, Kubota discloses
provide a notification the acquired medical image matches the patient anatomy. (Kubuta, para. [0080], “If the order information does not match to the additional information (step S6; mismatch), the control section 41 determines that the delivered medical image is not a requested patient image, and causes the display section 43 to display an error message notifying the determination result (step S7).”).
The combination of Lee, Pearson, and Schoepp, and Kubota, are in the same field of endeavor, namely Medical Applications. Before the effective filing date of the claimed invention, it would have been obvious to apply providing a notification the acquired medical image matches the patient anatomy, as taught by Kubota into combination of Lee, Pearson, and Schoepp. The motivation would have been ““If the order information does not match to the additional information (step S6; mismatch), the control section 41 determines that the delivered medical image is not a requested patient image, and causes the display section 43 to display an error message notifying the determination result (step S7).” (Kubota, para. [0080]). The suggestion for doing so would allow to provide a notification the acquired medical image matches the patient anatomy. Therefore, it would have been obvious to combine Lee, Pearson, Schoepp, and Kubota.
Regarding claim 14, the combination of Lee, Pearson, Schoepp, and Kubota discloses The system as in claim 13, wherein the morphometric measurements are shape, width, height, depth and contour of an appendage or patient anatomy. (Pearson, para. [0045], “An example process of matching mammogram images may include matching one or more of the following: breast landmarks, breast tissue densities, tissue patterns, breast findings and other distinguishing features. For example, matching breast landmarks may include matching nipples, pectoralis muscle, skin lines for breast size, shape or contour of the breast. The matching may include calculating the posterior nipple line depth and estimating the width and the height of breast.”).
Regarding claim 15, the combination of Lee, Pearson, Schoepp, and Kubota discloses The system as in claim 13, wherein the acquired medical image is a MRI, CT scan, X-ray, ultrasound, or photographic images internal to a human body. (Schoepp, para. [0041], “The scan image 48 may be any image useful to the surgeon to plan and/or execute the surgical procedure, and will generally include such modalities as a CT, ultra-sound, X-ray, MRI, infra-red, visible spectrum, and any other imaging modality suitable for use with surgical procedures. Often, the scan image 48 includes image data relative to subcutaneous structures in the patient's body, such as organs, bones, tumors, cancers, blood clots, and/or other items of interest inside the patient's body”).
Regarding claim 16, the combination of Lee, Pearson, Schoepp, and Kubota discloses The system as in claim 13, wherein the camera provides a live video feed of viewable patient anatomy. (Lee, para. [0088], “In this regard, the camera 410 may capture an image of a region viewed by the user in real-time, and the augmented reality image generator 430 may receive the real image captured by the camera 410 in real-time so as to generate an augmented reality image in accordance with movement of the camera 410.”)
Regarding claim 17, the combination of Lee, Pearson, Schoepp, and Kubota discloses The system as in claim 13, wherein the acquired medical image is displayed using the augmented reality headset with the acquired medical image projected on a semi- transparent optical imaging area. (Lee, para. [0075], “augmented reality image generator 430 that detects the plurality of markers in the real image acquired by the camera 410, estimates position and gaze direction of the camera 410 using the detected markers, and generates an augmented reality image by overlaying a virtual image of a corresponding region over the real image, and a display 420 that displays the augmented reality image, as illustrated in FIG. 2”).
Regarding claim 18, the combination of Lee discloses The system as in claim 13, further comprising matching a patient orientation marker with an image orientation tag in the acquired medical image to enable correct orientation of the acquired medical image with the patient anatomy. (Schoepp, para. [0021], “However, in other applications, the tracking element 14 may include active markers, such as LEDs and/or other types of navigation markers that may be sensed by the camera 18. The surgical tool 12 and tracking element 14 are pre-calibrated, whereby the position of the tracking element 14 with respect to the camera assembly 16 may also be automatically identified based on the size and orientation of the optical navigation markers 30 in images obtained from the camera 18.”)
Regarding claim 19, the combination of Lee, Pearson, Schoepp, and Kubota discloses The system as in claim 13, wherein the patient marker is a scannable symbol including at least one of: a 1D (one dimensional) bar code, a 2D (two dimensional) bar code, a picture, a custom generated geometric shape, or a RFID (radio frequency ID) used to retrieve the identity of the patient and to retrieve the acquired medical image. (Schoepp, para. [0006], “A target patch with optical navigation markers, printed thereon, is placed on the patient's body at or near a desired point of entry into the body. The patch is registered with the body, and the camera views the patch as the surgical tool is inserted into the patient's body.”; para. [0021], “The optical navigation markers 30 on the tracking element 14 comprise a set of shapes and/or colors”).
Regarding claim 20, the combination of Lee, Pearson, Schoepp, and Kubota discloses The system as in claim 13, wherein the augmentation processor performs:
identifying an image inversion prevention tag; and
aligning the acquired medical image and patient anatomy using an image inversion prevention tag to ensure the acquired medical image is not inverted with respect to the patient anatomy. (Schoepp, para. [0061], “In a point-to-point registration procedure, physical landmarks located at block 126 are matched with corresponding structures in the scan image 48. Thereafter, a point-to-point registration procedure is performed to register the camera space with the scan image space by, for example, a point-to-point matching procedure and appropriate coordinate transformations known in the art.”)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BIAO CHEN whose telephone number is (703)756-1199. The examiner can normally be reached M-F 8am-5pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee M Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Biao Chen/
Patent Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights