DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “a machine learning based segmentation module being adapted to” in claims 14 and 20, “a model matching module configured to” in claims 14, 16 and 18, “a user interface module adapted to” in claim 18.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 3-11, 13-16 and 18-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Jaramaz et al (2023/0363831).
Regarding claim 1, Jaramaz et al disclose a method for real-time surgical navigation, comprising:
capturing an intraoperative image stream from a distal portion of an endoscopic instrument inserted into a patient’s body (the endoscope further comprises an image sensor, wherein the computer system is further configured to cause the display of a video feed obtained via the image sensor – [0012]);
processing the intraoperative image stream with a machine learning-based segmentation algorithm configured to identify anatomical structures (the trained neural network can thus be configured to segment intraoperatively received depth images to obtain the surface geometry of the anatomical structure located therein – [0195]);
matching segmented images to a patient-specific three-dimensional (3D) model of the anatomy (3D model is developed – [0179]; registering the segmented anatomical structure surface geometry with the preoperative model – [0197]); and
outputting navigational data indicating the position of the endoscopic instrument relative to the anatomical structures (the computer system can intraoperatively track the anatomical structure by continuously or periodically generating a depth image of the anatomical structure, segmenting the depth image and registering the segmented anatomical structure surface geometry with the preoperative model – [0197]).
Regarding claim 3, Jaramaz et al disclose further comprising acquiring stereoscopic image data from two cameras located at the distal portion of the endoscopic instrument, and using the stereoscopic image data to estimate a depth map of the operative field (depth image – abstract; the image sensor assembly can include or be part of an assembly of multiple cameras (e.g., binocular or stereo cameras) - [0188]).
Regarding claim 4, Jaramaz et al disclose further comprising combining the depth map with the patient-specific three-dimensional (3D) model to measure: a distance between the endoscopic instrument and a selected anatomical landmark (an endoscopic visualization system could calculate the distance between the landmark and the light source – [0006];[0195]) or an angle or orientation of the endoscopic instrument relative to the anatomy ([0006]).
Regarding claim 5, Jaramaz et al disclose further comprising aggregating multiple consecutive segmented frames to refine the matching of the intraoperative images against the patient-specific three-dimensional (3D) model (during each phase, data collected or generated that can be used to analyze the episode of care in order to understand various features of the procedure and identify patterns that may be used, for example, in training models to make decisions with minimal human intervention. the data collected over the episode of care may be stored – [0110]).
Regarding claim 6, Jaramaz et al disclose further comprising calibrating the location of the endoscope instrument relative to the patient’s anatomy by referencing one or more known anatomical landmarks ([0024]).
Regarding claim 7, Jaramaz et al disclose further comprising controlling, via a robotic manipulator, the position or orientation of the endoscopic instrument in response to the navigational data, thereby enabling semi-autonomous or autonomous navigation with the patient’s anatomy (guiding surgical instruments – [0061]; fig.1; the endoscopic visualization system can be embodied as a robotic surgical system – [0200]; Examiner notes “thereby enabling…” is merely a result of the controlling, via a robotic manipulator and does not provide any further patentable weight).
Regarding claim 8, Jaramaz et al disclose further comprising storing segmented frames and corresponding location data for post-operative review, wherein the stored data are used to retrain the machine learning-based segmentation algorithm and improve its accuracy over subsequent procedures (during each phase, data collected or generated that can be used to analyze the episode of care in order to understand various features of the procedure and identify patterns that may be used, for example, in training models to make decisions with minimal human intervention. the data collected over the episode of care may be stored – [0110]).
Regarding claim 9, Jaramaz et al disclose wherein the patient-specific three-dimensional (3D) model is derived from at least one of a computed tomography scan or a magnetic resonance imaging scan (the preoperative model of the anatomical structure can be received from preoperative medical image data (e.g., sliced CT or MRI images) – [0193]), and is segmented to distinguish bony structures, nerves and other soft tissues relevant to a surgical target (bone, cartilage, muscle, nervous, and/or vascular tissues – [0081]).
Regarding claim 10, Jaramaz et al disclose further comprising displaying a color-coded overlay of identified anatomical structures on a monitor or head-mounted display to provide real-time visual feedback to a surgeon or operator during navigation (the display can depict the progress of the bone being resected as compared to the surgical plan using different colors – [0088];[0124]).
Regarding claim 11, Jaramaz et al disclose wherein the endoscopic instrument comprises a surgical tool comprising a dissector ([0097];[0199];[0205]).
Regarding claim 13, Jaramaz et al disclose further comprising using data from the patient-specific three-dimensional (3D) model to improve segmentation accuracy of the machine learning-based segmentation algorithm (during each phase, data collected or generated that can be used to analyze the episode of care in order to understand various features of the procedure and identify patterns that may be used, for example, in training models to make decisions with minimal human intervention. the data collected over the episode of care may be stored – [0110]).
Regarding claim 14, Jaramaz et al disclose a system for real-time surgical navigation, comprising:
an endoscopic instrument having a display portion configured to capture an intraoperative image stream via an imaging module (the endoscope further comprises an image sensor, wherein the computer system is further configured to cause the display of a video feed obtained via the image sensor – [0012]);
a computer device including one or more processors, memory and optionally a graphic processing unit ([0083]);
a machine learning-based segmentation module stored in the memory, the machine learning-based segmentation module being adapted to identify anatomical structures in the intraoperative image stream (the trained neural network can thus be configured to segment intraoperatively received depth images to obtain the surface geometry of the anatomical structure located therein – [0195]);
a model matching module configured to align segmented images from the machine learning-based segmentation module with a patient-specific three-dimensional (3D) model of the anatomy (3D model is developed – [0179]; registering the segmented anatomical structure surface geometry with the preoperative model – [0197]); and
an output interface that provides navigational data indicating the position of the endoscopic instrument relative to the anatomical structures (the computer system can intraoperatively track the anatomical structure by continuously or periodically generating a depth image of the anatomical structure, segmenting the depth image and registering the segmented anatomical structure surface geometry with the preoperative model – [0197]).
Regarding claim 15, Jaramaz et al disclose wherein the instrument comprises two cameras at the distal portion for stereoscopic imaging, enabling the computing device to estimate a depth map of the operative field (depth image – abstract; the image sensor assembly can include or be part of an assembly of multiple cameras (e.g., binocular or stereo cameras) - [0188]).
Regarding claim 16, Jaramaz et al disclose further comprising a robotic manipulator communicatively coupled to the computing device, whrein the model matching module provides location and orientation data to the robotic manipulator to enable semi-autonomous or autonomous navigation with the patient’s anatomy (guiding surgical instruments – [0061]; fig.1; the endoscopic visualization system can be embodied as a robotic surgical system – [0200]; Examiner notes “to enabling…” is merely a result of the controlling, via a robotic manipulator and does not provide any further patentable weight).
Regarding claim 18, Jaramaz et al disclose further comprising a user interface module adapted to display a real-time overlay of segmented anatomical structures on a visual module (display the registered surface geometry data in association with the video feed – [0013]) and to output navigational prompts or warnings based on data from the model matching module (the CASS can provide visual or audible prompts to the surgeon to warn the surgeon – [0088]).
Regarding claim 19, Jaramaz et al disclose wherein the patient-specific three-dimensional (3D) model is obtained from at least one of a computed tomography scan or a magnetic resonance imaging scan, is stored in a data or model storage, and is segmented to distinguish anatomical features (the preoperative model of the anatomical structure can be received from preoperative medical image data (e.g., sliced CT or MRI images) – [0193]).
Regarding claim 20, Jaramaz et al disclose further comprising a training and update engine configured to store intraoperative image data and segmentation result, and to retrain or fine-tune the machine learning-based segmentation model based on post-operative analysis (during each phase, data collected or generated that can be used to analyze the episode of care in order to understand various features of the procedure and identify patterns that may be used, for example, in training models to make decisions with minimal human intervention. the data collected over the episode of care may be stored – [0110).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 2 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jaramaz et al (2023/0363831) in view of Gao et al (2020/0349697).
Regarding claims 2 and 17, Jaramaz et al disclose the invention substantially as claimed, but fail to explicitly disclose wherein the machine learning-based segmentation algorithm is an encoder-decoder neural network that comprises an encoder portion for extracting feature maps from each image frame and a decoder portion for reconstructing a segmented output identifying specific anatomical structures.
However, Gao et al teach in the same medical field of endeavor, wherein a machine learning-based segmentation algorithm is an encoder-decoder neural network that comprises an encoder portion for extracting feature maps from each image frame and a decoder portion for reconstructing a segmented output identifying specific anatomical structures (extract feature maps from each image slice using and encoder…segment each image slice using the decoder to obtain an ICH region based on the extracted feature maps of the image slice - abstract).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the machine learning-based segmentation algorithm of Jaramaz et al with an encoder-decoder neural network of Gao et al as it would provide detection of specific anatomical features of interest as set forth in Gao et al (Abstract).
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jaramaz et al (2023/0363831) in view of Zhou et al (2019/0205606).
Regarding claim 12, Jaramaz et al disclose the invention substantially as claimed, but fail to explicitly disclose wherein the machine learning-based segmentation algorithm is trained on a dataset combining real surgical images and synthetic images obtained from physical or virtual three-dimensional printed anatomical models.
However, Wang et al teach in an analogous field of endeavor, wherein a machine learning-based segmentation algorithm is trained on a dataset combining real images and synthetic images obtained from physical three-dimensional printed models (image capture unit that acquires the vectorized three-dimensional model of a target object; image rendering unit that generate a large volume of the synthetic images of various states…machine learning unit that learn from plurality and diversified synthetic images – [0007]; the learning system can also use real images – [0052]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the machine learning-based segmentation algorithm that is trained on a dataset of real surgical images and anatomical models of Jaramaz et al with combining real images and synthetic images obtained from physical printed models of Wang et al as it would provide optimization of the training algorithm based on a real images and synthetic images of various proposed states.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROCHELLE DEANNA TURCHEN whose telephone number is (571)270-7104. The examiner can normally be reached Mon - Fri 6:30-2:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Christopher Koharski can be reached at (571)272-7230. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROCHELLE D TURCHEN/ Primary Examiner, Art Unit 3797
Prosecution Insights