DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 17-18 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 17 recites the limitation "the apparatus comprising". There is insufficient antecedent basis for this limitation in the claim.
Claim 18 recites the limitation "The apparatus according to claim 17". There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Black (US 20220117674 A1), and in view of Liu (US 20200260066 A1), and further in view of Gao (US 20190290365 A1).
Regarding Claim 1, Black An apparatus for use during a surgical procedure that is performed on a portion of a body of a patient using a surgical instrument, an imaging system, and an output device (ABST reciting “A holographic augmented reality visualization and guidance system for performing a medical procedure includes an augmented reality system for displaying operating information to a user.”. Fig. 1 showing tracked instrument 104, image acquisition systems, 108, 110, and augmented reality system 102. Fig. 3 showing the AR system 102 including an HMD 103), the apparatus comprising:
at least one computer processor configured: (Fig. 1 showing processor 126)
to receive preoperative imaging data of the portion of the patient's body (¶93 reciting “The image detector 178 receives the X-rays 180 that pass through the target volume 184 and generates preoperative CT scan images based on the X-rays 180 received. The CT scan images are used to generate a CBCT holographic representation (preoperative data 158) of the patients target anatomy.”); and during the surgical procedure:
to receive intraoperative images of the portion of the patient's body from the imaging system; (¶94 reciting “The automatic segmentation and registration process 200 can further obtain one or more intraoperative images 160 from the ultrasound imaging 156”)
to perform 3D reconstruction of the portion of the patient's body based on at least some of the intraoperative images; (¶95 reciting “the preoperative data 158 and intraoperative data 160 can be merged together generating fused data 162 that provide the baseline for reconstructing the 3D representation 192,” Fig. 4C showing 3D representation 192 corresponds to a patient’s head.)
to coregister the portion of the patient's body to a common coordinate system with the portion of the patient's body as it appears within the preoperative imaging data using a non-rigid coregistration algorithm; (¶94 reciting “The automatic segmentation and registration process 200 can further obtain one or more intraoperative images 160 from the ultrasound imaging 156 being emitted from the ultrasound probe 188 and can co-register the intraoperative images 160 with the segments 190 into the preoperative data 158”; and ¶98 reciting “Referring to FIG. 6, in operation, the deformation engine 150 merges the coordinate systems of the preoperative data 158 and the intraoperative data 160. The deformation engine 150 also identifies and registers key anatomical structures 196 in the preoperative data 158 to be used as reference points 194. This can include identifying echogenic profiles of the anatomical structures manually or through AI. For example, the specific voxels associated with a tissue type can be identified and registered using AI and machine learning. The deformation engine 150 then identifies these reference points 194 in the intraoperative data 160 and compares the differences with the preoperative data 158. The differences between the intraoperative data 160 and the preoperative data 158 can include changes in the deformation and the spatial positioning of anatomical structures, represented as arrows in FIG. 6.” Further, ¶99 disclosing a non-rigid coregistration algorithm) and
to drive the output device to display an output, based upon the coregistering. (¶101 reciting “the augmented reality system 102 selectively displays the fused data 162”)
However, Black does not explicitly disclose to identify the portion of the patient's body within the intraoperative images; to segment the portion of the patient's body within the intraoperative images.
Liu teaches “a system and method for utilizing 3D scanning in combination with biomedical imaging, such as optical imaging and nuclear imaging modalities, to perform diagnostic functions.” (¶2). ¶17 recites “It is another aspect of the optical imaging system of the present invention to be configured, whereby fluorescence imaging (such as 5-Aminolevulinic acid or Indocyanine green) is used in conjunction with preoperative surgical navigation (e.g. MRI, CT, PET, SPECT . . . ) for guiding interventions or surgeries.” More specifically, ¶241 recites “Image segmentation can be performed to isolate the organ of interest. Features and landmarks can be extracted from both the preoperative image model (based on tomographic imaging data) and the intraoperative image model (3D topography captured by our invention). A first image registration is performed based on the features and landmarks identified.” The features and landmarks from the intraoperative image model read on identified and segmented portion of the patient’s body within the intraoperative images.
It would have been obvious to one with ordinary skill, before the effective filing date of the claimed invention, to modify the apparatus (taught by Black) to identify and segment the portion of the patient’s body within the intraoperative images (taught by Liu). The suggestions/motivations would have been that “If the soft tissues have different deformation between preoperative imaging and intraoperative imaging, registration errors between the images will occur, if deformation is not properly addressed.” (¶241), and to apply a known technique to a known device (method, or product) ready for improvement to yield predictable results.
However, Black in view of Liu does not explicitly disclose to receive intraoperative images of the surgical instrument from the imaging system.
It is well known in the art in an image-guided surgery, the users receive intraoperative images of the surgical instrument from an imaging system. In addition, Gao teaches “Image guided surgery systems use cameras or electromagnetic fields to capture and relay the patient's anatomy and the surgeon's precise movements of the instrument in relation to the patient, to computer monitor in the operating room.” (¶7).
It would have been obvious to one with ordinary skill, before the effective filing date of the claimed invention, to combine the teachings from Black in view of Liu and Gao to receive intraoperative images of the surgical instrument from the imaging system. The suggestions/motivations would have been “in order to indirectly guide the procedure” (¶7), and to apply a known technique to a known device (method, or product) ready for improvement to yield predictable results.
Regarding Claim 2, Black in view of Liu and Gao discloses The apparatus according to claim 1, wherein the computer processor is configured to coregister the portion of the patient's body to a common coordinate system with the portion of the patient's body as it appears within the preoperative imaging data by deforming the portion of the patient's body within the preoperative imaging data, using the non-rigid coregistration algorithm. (Black, ¶99 reciting “Once differences have been identified, the deformation engine 150 can compensate for deformations by altering, deforming, or shifting the preoperative data 158 to align the preoperative data 158 to the corresponding anatomical structure 196.”)
Regarding Claim 3, Black in view of Liu and Gao discloses The apparatus according to claim 1, wherein in response to detecting that the portion of the patient's body has undergone a change in shape since the preoperative imaging data were acquired, the computer processor is configured to update a shape of the preoperative imaging data to reflect the change in shape that the portion of the patient's body has undergone. (Black, ¶98 reciting “The differences between the intraoperative data 160 and the preoperative data 158 can include changes in the deformation and the spatial positioning of anatomical structures, represented as arrows in FIG. 6.”; and ¶99 reciting ““Once differences have been identified, the deformation engine 150 can compensate for deformations by altering, deforming, or shifting the preoperative data 158 to align the preoperative data 158 to the corresponding anatomical structure 196.”)
Regarding Claim 4, Black in view of Liu and Gao discloses The apparatus according to claim 1, wherein the computer processor is configured to drive the output device to display a current location of the surgical instrument with respect to the preoperative imaging data, based upon the coregistering. (Black, ¶83 reciting “The augmented reality system 102 can retrieve a preoperative CT image (preoperative data 158) of the hepatic vein of the liver and superimpose the preoperative image 158 over the poor-quality live ultrasound image stream (intraoperative data 160) and display a holographic visualization within the field of view of the user. The holographic visualization includes a holographic representation of a tracked instrument 104 (e.g., interventional device) congruent with the registered holographic projection of the live ultrasound image stream (intraoperative data 160) and the holographic image of the hepatic vein derived from CT (preoperative data 158). This holographic visualization is used to navigate and guide the interventional device or tracked instrument 104 to the target anatomical structure.”)
Regarding Claim 5, Black in view of Liu and Gao discloses The apparatus according to claim 4, wherein the computer processor is configured to drive the output device to display the current location of the surgical instrument with respect to the preoperative imaging data, based upon the coregistering, without requiring use of instrument fiducial markers disposed on the surgical instrument. (Black, Fig. 5, ¶83)
Regarding Claim 6, Black in view of Liu and Gao discloses The apparatus according to claim 1, further comprising the imaging system, wherein the imaging system comprises one or more infrared cameras that are configured to acquire images of the veins within the portion of the patient's body. (Liu, ¶232 reciting “The system 100 may also be configured to facilitate imaging of a location where a human or animal vein target object 324 is located to facilitate intravenous placement of a needle or injection. . . the imaging configuration 2000A includes transmission imaging geometry, whereby the infrared light source 450 and the image detection module 320 are positioned on different or opposite sides of the tissues or target object 324, whereby the portion 2020 of the emitted IR light 2030 that light passes through the tissue is imaged by the image detection module 320.”)
Regarding Claim 7, Black in view of Liu and Gao discloses The apparatus according to claim 1, further comprising the imaging system, wherein the imaging system comprises one or more depth cameras. (Liu, ¶227 reciting “the image detection module 320 is configured as a depth camera” The suggestions/motivations would have been the same as that of Claim 1 rejections.)
Regarding Claim 8, Black in view of Liu and Gao discloses The apparatus according to claim 1, wherein the computer processor is configured to coregister the portion of the patient's body to the common coordinate system with the portion of the patient's body as it appears within the preoperative imaging data, by performing surface-matching registration between a surface of the portion of the patient's body as it appears within the preoperative imaging data and a current shape of the surface of portion of the patient's body. (Liu, ¶257 reciting ”the imaging system utilizes a surface 3D profile to track surface deformation, and to provide surface-based registration.” The suggestions/motivations would have been the same as that of Claim 1 rejections.)
Regarding Claim 9, Black in view of Liu and Gao discloses The apparatus according to claim 8, wherein the computer processor is further configured to coregister the portion of the patient's body to the common coordinate system with the portion of the patient's body as it appears within the preoperative imaging data by modelling changes between the shapes of internal portions of the portion of the patient's body as the internal portions of the portion of the patient's body appear in preoperative imaging data and current shapes of internal portions of the portion of the patient's body appear, based upon the surface-matching registration and tissue that is present within the internal portions of the portion of the patient's body. (Liu, ¶142 reciting “ICG fluorescence can image vascularity of tissues to provide useful information in addition to the preoperative MRI. . . The registration algorithm comprises surface-based registration, . . . In one aspect, the deformation of the brain can be calculated with biomechanical modeling using finite element method (FEM) to facilitate the registration process.” The suggestions/motivations would have been the same as that of Claim 1 rejections.)
Regarding Claim 11, Black in view of Liu and Gao discloses The apparatus according to claim 1, further comprising a light source, wherein the computer processor is configured to perform 3D reconstruction of the portion of the patient's body based on at least some of the intraoperative images by:
driving the light source to direct light toward the portion of the patient's body; and
detecting light that is reflected from the portion of the patient's body within the intraoperative images.
(Liu, ¶98 reciting “The image detection module 320 includes any suitable light detection device configured to detect the light 326 reflected or emitted by the target object 324 in response to being illuminated by the emitted light rays 322 output by the light emitter module 310.” The suggestions/motivations would have been the same as that of Claim 1 rejections.)
Regarding Claim 10, Black in view of Liu and Gao discloses The apparatus according to claim 8, wherein the computer processor is further configured to coregister the portion of the patient's body to the common coordinate system with the portion of the patient's body as it appears within the preoperative imaging data by determining that the portion of the patient's body has been cut, and modifying the preoperative imaging data to create an accurate representation of the cut organ. (Liu, ¶241 reciting “the present invention allows 3D topography that is captured by the system 100 to be used to update the preoperative images and account for the tissue deformation.”)
Regarding Claim 11, Black in view of Liu and Gao discloses The apparatus according to claim 1, wherein the computer processor is configured to receive preoperative planning that is performed with respect to the preoperative imaging data of the portion of the patient's body and the computer processor is configured to drive the output device to display a current location of the surgical instrument with respect to the preoperative planning, based upon the coregistering.
Regarding Claim 13, Black in view of Liu and Gao discloses The apparatus according to claim 1, further comprising the imaging system, wherein the imaging system comprises a hyperspectral camera (¶92 reciting “The optical imaging module 120 includes . . ., hyperspectral imaging,”), and the computer processor is configured to coregister the portion of the patient's body to a common coordinate system with the portion of the patient's body as it appears within the preoperative imaging data using imaging data acquired using the hyperspectral camera. (¶162 reciting “A process 1000 executed by the controller 130 for co-registration of the intraoperative optical data captured by the imaging system 100 of the present invention” In addition, ¶90 teaching the imaging system 100 includes a 3D scanning module 110 and an optical imaging module 120; and ¶92 reciting “The optical imaging module 120 includes . . ., hyperspectral imaging”. The suggestions/motivations would have been the same as that of Claim 1 rejections.)
Regarding Claim 14, Black in view of Liu and Gao discloses The apparatus according to claim 1, wherein the computer processor is configured for use with a surgical navigation system that is configured to coregister anatomy of the patient with the preoperative imaging data of the portion of the patient's body such that the patient's anatomy and the preoperative imaging data are registered with each other within a navigation system common coordinate system, wherein the computer processor is configured to coregister images acquired by the imaging system within the navigation system common coordinate system, and wherein in response to detecting that the portion of the patient's body has undergone a change in shape since the preoperative imaging data were acquired, the computer processor is configured to update a shape of the preoperative imaging data and registration of the preoperative imaging data within the navigation system common coordinate system to reflect the change in shape that the portion of the patient's body has undergone. (Black, ¶87 reciting “Referring to FIG. 7, the intraoperative data 160 can be super-imposed over the preoperative data 158 to identify changes that can occur between the preoperative imaging and the imaging occurring during the medical procedure. In addition, the intraoperative data 160 can be displayed when the intraoperative data 160 is better suited to be the navigational guide.” Further, ¶98 reciting “The differences between the intraoperative data 160 and the preoperative data 158 can include changes in the deformation and the spatial positioning of anatomical structures, represented as arrows in FIG. 6.”; and ¶99 reciting ““Once differences have been identified, the deformation engine 150 can compensate for deformations by altering, deforming, or shifting the preoperative data 158 to align the preoperative data 158 to the corresponding anatomical structure 196.”)
Regarding Claim 15, Black in view of Liu and Gao discloses The apparatus according to claim 1, wherein the computer processor is configured to coregister the portion of the patient's body to a common coordinate system with the portion of the patient's body as it appears within the preoperative imaging data, subsequent to the portion of the patient's body having undergone movement, deformation and/or resection since the preoperative imaging data were acquired. (Liu, ¶154 reciting “the present invention uses a 3D surface profile to track tissue movements of the target object 324 during an operating procedure to reduce motion artifacts in the projected fluorescence image.” The suggestions/motivations would have been the same as that of Claim 1 rejections.)
Regarding Claim 16, Black in view of Liu and Gao discloses The apparatus according to claim 1, wherein the computer processor is configured to coregister the portion of the patient's body to a common coordinate system with the portion of the patient's body as it appears within the preoperative imaging data, while the portion of the patient's body undergoes intraprocedural movement, deformation and/or resection. (Liu, ¶154 reciting “the present invention uses a 3D surface profile to track tissue movements of the target object 324 during an operating procedure to reduce motion artifacts in the projected fluorescence image.” The suggestions/motivations would have been the same as that of Claim 1 rejections.)
Claim 17, has similar limitations as of Claim(s) 1, therefore it is rejected under the same rationale as Claim(s) 1.
Regarding Claim 18, Black in view of Liu and Gao discloses The apparatus according to claim 17, wherein displaying the output upon the output device comprises display a current location of the surgical instrument with respect to the preoperative imaging data on the output device, based upon the coregistering. (Black, ¶83 reciting “The augmented reality system 102 can retrieve a preoperative CT image (preoperative data 158) of the hepatic vein of the liver and superimpose the preoperative image 158 over the poor-quality live ultrasound image stream (intraoperative data 160) and display a holographic visualization within the field of view of the user. The holographic visualization includes a holographic representation of a tracked instrument 104 (e.g., interventional device) congruent with the registered holographic projection of the live ultrasound image stream (intraoperative data 160) and the holographic image of the hepatic vein derived from CT (preoperative data 158). This holographic visualization is used to navigate and guide the interventional device or tracked instrument 104 to the target anatomical structure.”)
Regarding Claim 19, Black in view of Liu and Gao discloses An apparatus for use during a surgical procedure that is performed on a portion of a body of a patient using a surgical instrument, an imaging system, an output device (Black, ABST reciting “A holographic augmented reality visualization and guidance system for performing a medical procedure includes an augmented reality system for displaying operating information to a user.”. Fig. 1 showing tracked instrument 104, image acquisition systems, 108, 110, and augmented reality system 102. Fig. 3 showing the AR system 102 including a HMD 103), fiducial markers placed upon the patient's body (Black, ¶60 reciting “as one or more sensors located on fiducial markers or other imaging targets.” ¶81 reciting “one or more markers or flagged points or portions of the anatomical site of the patient.”), and a surgical navigation system that is configured to coregister anatomy of the patient with the preoperative imaging data of the portion of the patient's body such that the patient's anatomy and the preoperative imaging data are registered with each other within a navigation system common coordinate system, by identifying the fiducial markers within images of the patient's body, the apparatus comprising: (See Claim 14 rejections for detailed analysis.)
at least one computer processor configured:
to receive preoperative imaging data of the portion of the patient's body and the surgical instrument; (¶93 reciting “The image detector 178 receives the X-rays 180 that pass through the target volume 184 and generates preoperative CT scan images based on the X-rays 180 received. The CT scan images are used to generate a CBCT holographic representation (preoperative data 158) of the patients target anatomy.”)
segment the preoperative imaging data of the portion of the subject's body into substructures; (Black, ¶94 reciting “The preoperative scans are automatically split into segments 190. Next, each segment is registered into the preoperative data 158.”) and
during the surgical procedure:
to coregister the segmented substructure to the patient's body, such that the patient's body and the segmented substructures within the preoperative imaging data are registered within a navigation system common coordinate system; (Black, ¶94 reciting “The automatic segmentation and registration process 200 can further obtain one or more intraoperative images 160 from the ultrasound imaging 156 being emitted from the ultrasound probe 188 and can co-register the intraoperative images 160 with the segments 190 into the preoperative data 158”)
to coregister images acquired by the imaging system within the navigation system common coordinate system;
to receive intraoperative images of the portion of the patient's body from the imaging system;
to identify the portion of the patient's body within the intraoperative images;
to segment the portion of the patient's body within the intraoperative images;
to perform 3D reconstruction of the portion of the patient's body based on at least some of the intraoperative images;
to coregister the portion of the patient's body to the preoperative imaging data within the navigation system common coordinate system, the coregistering comprising updating the preoperative imaging data by moving at least one of the segmented substructures relative to others of the segmented substructures; and
drive the output device to display an output, based upon the coregistering.
(See Claim 1 and Claim 14 rejections for detailed analysis)
Regarding Claim 20, Black in view of Liu and Gao discloses The apparatus according to claim 19, wherein the computer processor is configured to coregister the portion of the patient's body to the preoperative imaging data within the navigation system common coordinate system at least partially by deforming the portion of the patient's body within the preoperative imaging data, using a non-rigid coregistration algorithm. (Black, ¶99 reciting “Once differences have been identified, the deformation engine 150 can compensate for deformations by altering, deforming, or shifting the preoperative data 158 to align the preoperative data 158 to the corresponding anatomical structure 196.”)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YI WANG whose telephone number is (571)272-6022. The examiner can normally be reached 9am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at (571)272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YI WANG/ Primary Examiner, Art Unit 2619
