DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
New 101 and 112 rejections necessitated by amendment.
Regarding prior art
Applicant’s arguments with respect to claims 1 and 22 have been considered but are moot in view of the new grounds of rejection necessitated by amendment.
Regarding applicant’s arguments on pg. 7 that “Liao segments separately for each type of image. Thus, Liao does not have one machine-learned model that output s the location in response to input of both pre-operative and real-time images”, examiner respectfully disagrees for the same reasons listed previously. Specifically, it is noted that separate segmentations are not precluded by the current claim language which merely recites indicating a location in response to input of the images and does not specify the nature of the processing of the images which would preclude any such separate segmentations. Furthermore, Liao explicitly discloses the output of a spatial position of a third target structure during the surgery which is in response to the input of both the pre-operative image and the real-time image. Applicant’s arguments against the teachings of Liao are not found persuasive for at least these reasons.
Claim Objections
Claim 22 is objected to because of the following informalities: the “s” after image appears to be a typographical error and should be removed. Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 22 is rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception in the form of an abstract idea without significantly more.
In a test for patent subject matter eligibility, the claims pass Step 1 (see 2019 Revised Patent Subject Matter Eligibility), as they are related to a process, machine, manufacture, or composition of matter.
When assessed under Step2A, Prong I, Independent claim 22 is found to recite a judicial exception (i.e. abstract idea). In this instance, claim 22 recites the limitation “indicating a location”, “outputting the location in response to input of information”, “receives both the pre-operative image and the real-time image as an input and outputs the location in response to the input of both the pre-operative image and the real-time”, “the location as output is generated by learned values applied to both the pre-operative image and the real-time image’. The cited limitation(s), under their broadest reasonable interpretation, encompass a mental process (i.e. abstract idea) of receiving information, applying learned values, and outputting/indicating a location which can be performed in the mind or by a human using a pen and a paper (e.g. observation, evaluation, judgment, opinion). In other words, a person could reasonably evaluated a real-time image and pre-operative image, apply learned values/information to the images and output/indicate a location via thought in response thereof. Examiner notes that with the exception of generic computer-implemented steps (e.g. a machine-learned model recited in claim 22), there is nothing in the claims that preclude the limitation from being performed by a human, mentally or with pen and paper, thus the cited limitation(s) recites a judicial exception (MPEP 2106.04(a)) and the claim must be reviewed under Step 2A, Prong II to determine patent eligibility.
Step 2A, Prong II determines whether any claim recites an additional element that integrates the judicial exception into a practical application. Independent claims recites the following additional element(s):
Training a machine-learned model using a plurality of ground truth pre-operative images with labeled mass locations and a plurality of ground truth intra-operative images with labeled mass locations
The additional element(s) in the cited independent claim(s) are not found to integrate the judicial exception into a practical application. In this case, training the machine learned model is recited with such high generality that it amounts to merely insignificant extra-solution activity of training a generic computer. Examiner further notes that the output of location the machine learning model alternatively constitutes an additional element which merely requires a generic output using the machine learned model and does not impose any specific limits on how the data is output (other than in response to input and applied learned values). The additional elements are seen as adding insignificant extra-solution activity to the judicial exception. They do no more than link the judicial exception to a particular technological environment or field of use. Therefore, under step 2A Prong II the Judicial exception is not integrated into a practical application by additional elements of independent claim 22 and the claims must be reviewed under Step 2B to determine patent eligibility.
Step 2B determines where a claim amounts to significantly more.
The additional element(s) listed above do not amount to significantly more than the judicial exception. In this instance, as noted above the additional elements amount to merely insignificant extra-solution activity. Additionally there is no improvement in the functioning of the computer or technological field, and there is no transformation of subject matter into a different state. Therefore, under Step 2B in a test for patent subject matter eligibility, the judicial exception of the independent claim(s) do not amount to significantly more and the independent claim(s) remain patent ineligible.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-5, 7-10, 12, and 21-22 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1 and 22 recite the limitation “training a machine-learned model…with labeled mass locations” and “indicating a location of the object, the location indicated by the machine-learned model”. It is unclear how the training ties to the indicating of the location and if the object is a mass or another object. If the object is not a mass, it is unclear how the machine learned model is trained with the mass locations and indicates a location of an object other than a mass. For examination purposes, it has been interpreted that the object is a mass, however, clarification is required.
Claims 1 and 22 recite the limitation “the information comprising a pre-operative image of a patient”. It is unclear if the pre-operative image is the same as or is included in the ground truth pre-operative images or if this is a different pre-operative image. For examination purposes, it has been interpreted that it may be the same or different, however, clarification is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim 22 is rejected under 35 U.S.C. 102(a)(1) as being anticipated by Meral et al. (US 20210204914 A1), hereinafter Meral.
Regarding claim 22,
Meral discloses a method for intraoperative guidance to an object in a patient for a surgical system, the method comprising:
Training a machine-learned model using a plurality of ground truth pre-operative images with labeled mass locations and a plurality of ground truth intra-operative images with labeled mass locations ([0046] which discloses the disclosure includes a machine learning framework using deep learning with convolutional neural networks (CNN) to train on segmented (i.e. contoured) pre-operative MRI, pre-operative US, segmented intra-operative MRI and intra-operative US, to understand how different tumors look like in MRI vs US, what are their similar or dissimilar features. During the training, MRI's provide ground truth by manually performed segmentations, which are overlaid on the US images, which serves as the training data, via tracked acquisition. [0038] which discloses At 104, the at least one electronic processor 20 is programmed to provide the GUI 26 via which the acquired preoperative images are labeled with contours of the tumor and the surrounding blood vessels in the ROI and [0043] which discloses the at least one electronic processor 20 is programmed to tune the NN 30 using the labeled preoperative images (from 104). In another embodiment, the patient-tuned trained neural network 30 is updated at least partially with contours drawn in intraoperative MR images and [0048] which discloses tracked pre-operative and intra-operative US images (intra-operative US can only be used if there is a matching intra-op segmented MR image to pair it with) are used to create a volume, which will be called as a training sample. See also [0025] which discloses if an intra-operative MRI scanner is available in the surgical suite then updated MRI images may be occasionally acquired and segmented to produce updated MRI contours and [0040] which discloses at a later stage of the neurosurgery if the neurosurgeon suspects that the contours superimposed on the live imaging have become out of position due to gradual brain shift, then additional patient-specific training data can be generated by manual contouring of intraoperative images and/or MR images acquired by the in-suite MRI 32 (if available) and the operation 108 repeated with this additional patient-specific data to update the patient-specific tuning of the NN 30)
Indicating a location of an object during surgery, the location indicated by the machine-learned model, the machine-learned model outputting the location in response to input of information to the machine-learned model, the information comprising a pre-operative image of a patient and a real-time image of the patient in the surgery, the preoperative image representing anatomy and the real-time image representing the anatomy distorted relative to the pre-operative image (at least fig. 2 (112) and corresponding disclosure in at least [0042]), wherein the machine-learned model receives both the pre-operative image and the real-time image as an input and outputs the location in response to the input of both the pre-operative image and the real-time image such that the location as output by the machine-learned model is generated by learned values of the machine-learned model applied to both the pre-operative image and the real-time image ([0027] which discloses the input data to the neural network is a sequence of 2D ultrasound images (that is, the intraoperative or “live” US images used to provide visual guidance to the surgeon during the surgery), or one or a sequence of 3D ultrasound images which can be sliced and for processing through the neural network, alternatively a network can be trained to work with 3D US volumetric images as input. The output of the neural network is one or more tumor and/or blood vessel contours. the output of the neural network is one or more tumor and/or blood vessel contours and [0042] which discloses at 112, the at least one electronic processor 20 is programmed to input the live images to the patient-tuned trained NN to output live contours of the tumor and the surrounding blood vessels. Examiner notes that the training data including the pre-operative MRI images includes a pre-operative image that is input to the machine learned model as information)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 7, and 9-10 are rejected under 35 U.S.C. 103 as being unpatentable over Ye et al. (US 20210196398 A1), hereinafter Ye in view of Liao et al. (US 20250131556 A1), hereinafter Liao or in the alternative Ye in view of Liao and Meral.
Regarding claim 1,
Ye discloses a method (at least fig. 6 (600) and corresponding disclosure in at least [0103]) for intraoperative guidance to an object in a patient for a surgical system (at least fig. 2 (11) and corresponding disclosure in at least [0058]) ([0100] which discloses methods for guiding and/or automating endoscope and/or percutaneous access instruments based at least in part on a static position marker in view of certain target-identification and/or target tracking image-processing techniques), the method comprising:
Training a machine-learned model using a plurality of ground truth images with labeled mass locations (at least fig. 9 (900) and corresponding disclosure in at least [0140]. See also [0011] which discloses the artificial neural network can be pretrained based on known image and label data)
Indicating a location of the object during surgery (at least fig. 6 (621/623 and/or 637) and corresponding disclosure in at least [0105] and [0116]), the location indicated by a machine-learning model, the machine-learned model (at least fig. 8 (800) and corresponding disclosure in at least [0133] or at least fig .7 (720) and corresponding disclosure in at least [0126]) outputting the location in response to input of information to the machine-learned model ([0107] which discloses image-based tagging processes, at block 623, the process 600 involves determining the position of the target anatomical feature using image data input and certain neural network functionality and [0132] which discloses In some implementations, once the neural network 720 has been trained, operational input to the neural network framework can comprise a plurality (e.g., two) of images, wherein the output 735 comprises data indicating the locations of one or more targets across the images (e.g., over time)), the information comprising a real-time image of a patient in the surgery ([0041] which discloses real-time endoscopic images captured therewith, to assist the physician 5 in navigating the medical instrument), the real-time image representing anatomy, where the machine-learned model receives the real-time image as an input and outputs the location based on the real-time image ([0041] which discloses real-time endoscopic images captured therewith, to assist the physician 5 in navigating the medical instrument); and
guiding the surgical system to the location based on the indication (at least fig. 6 (630) and corresponding disclosure in at least [0109] The process 600 proceeds to subprocesses 630, which may involve tracking the target anatomical feature over an operative period of time while advancing percutaneous-access instrument, such as a needle or the like, in the direction of the target anatomical feature. The real-time tracking of the dynamic relative position of the target anatomical feature can advantageously facilitate accurate direction of the percutaneous-access instrument to the treatment site and [0039] which discloses direct a percutaneous-access instrument and/or to guide robotic instrumentation such as by adjusting endoscope position and/or alignment automatically in response to such real-time target tracking information and [0101] which discloses systems, devices, and methods of the present disclosure may provide for identification of target anatomical features in real-time endoscope images, wherein identification of a target anatomical features in an image may prompt certain responsive action. For example, control circuitry communicatively coupled to robotic endoscopy and/or percutaneous-access device(s) may be configured to track movements of a target feature and take action, such as articulating one or more portions of the endoscope (e.g., distal end portion), or adjusting target position data. For example, the control circuitry may be configured to cause the endoscope to articulate so as to center the target position/points at or near a center of the field of view of an interface and/or image field of the endoscope camera, where articulation to center the target position is additionally/alternatively considered to mean guiding the surgical system to the location), the guiding comprising controlling a surgical robotic arm to guide a surgical instrument to the location ([0052] which discloses the robotic system can include one or more robotic arms to control the scope 32 to perform a procedure and [0152] which discloses one or more robotic arms may be configured to respond by adjusting the arm and/or an endoscope position controlled by the robotic arms and [0072] which discloses For example, the control circuitry 60 can be configured to provide control signals to the robotic arms 12 of the robotic system 11 to manipulate the relevant instrument to position the same at a target location, position, and/or orientation/alignment)
While Ye teaches in [0062] pre-operative plans, navigation and mapping data derived from pre-operative computerized tomography (CT) scans and further teaches in [0173] depth information can be determined from CT or can be derived from other information, for example, depth and/or pixel coordinate information can be outputs from the neural network, it is unclear if the CT of [0173] is the same as the pre-operative CT image disclosed in [0062], therefore, Ye fails to explicitly teach that the information comprises a pre-operative image of the patient. Ye further fails to explicitly teach training the machine-learned model using a plurality of ground truth pre-operative images with labeled mass locations and a plurality of ground truth intra-operative images with labeled mass locations; the pre-operative image representing anatomy and the real-time image representing the anatomy distorted relative to the pre-operative image, where the machine-learned model receives both the pre-operative image and the real-image image as an input and outputs the location in response to the input of both the pre-operative image and the real-time image such that the location as output by the machine-learned model is generated by learned values of the machine-learned model applied to both the pre-operative image and the real-time image.
Liao, in a similar field of endeavor involving image guided surgery, teaches
Training a machine-learned model using a plurality of ground truth images with labeled mass locations ([0112] which discloses an initial machine learning model may be trained using the big data and/or the historical data as training samples to obtain the machine learning model and [0164] which discloses historical medical images preliminarily obtained before the coarse segmentation may be used as training data, and the deep convolutional neural network model may be trained with historical precise segmentation result data. Examiner notes that such precise segmentation result data constitutes ground truth images with labeled mass locations)
indicating a location of an object during surgery ([0170] which discloses In 230B, a spatial position of a third target structure set during the surgery may be determined by registering a first segmentation image with a second segmentation image. In some embodiments, the operation 230B may be performed by the result determination module 2330), the location indicated by the machine-learned model (at least fig. 23 (2300 and/or 2320) and corresponding disclosure in at least [0071] which discloses that the modules of fig. 23 may be different modules in a system or a module may implement the functions of two or more of the above modules, thus the system 2300 and/or module 2320 alone considered a machine-learned model in its broadest reasonable interpretation may implement the functions of the obtaining unit, the segmentation module, and determination module and [0170] which discloses the operation 230B may be performed by the result determination module 2330 (i.e. by any of the modules including 2320 and the system 2300)), the machine-learned model (2300/2320) outputting the location in response to input of information to the machine-learned model ([0264] which discloses for example, the obtaining module 2310 may be configured to obtain a pre-operative image with contrasts and an intra-operative image without contrast) the information comprising a pre-operative image of a patient and a real-time image of the patient in surgery ([0264] which discloses for example, the obtaining module 2310 may be configured to obtain a pre-operative image with contrasts and an intra-operative image without contrast. [0005] which discloses in some embodiments, the medical image may include a pre-operative image with contrast and an intra-operative image without contrast and [0078] which discloses the intra-operative image without contrast may be a real-time scan image), the pre-operative image representing anatomy and the real-time image representing the anatomy distorted relative to the pre-operative image ([0176] which discloses the registration deformation field may reflect a spatial position change between the first segmentation image and the second segmentation image. Thus the anatomy is distorted in the intra-operative image relative to the pre-operative image), wherein the machine-learned model receives both the pre-operative image and the real-time image as an input ([0264] which discloses for example, the obtaining module 2310 may be configured to obtain a pre-operative image with contrasts and an intra-operative image without contrast. The segmentation module 2320 may be configured to segment a target structure set from the medical image. For example, the segmentation module 2320 may be configured to segment a first target structure set from the pre-operative image with contrast and a second target structure set from the intra-operative image without contrast, thus the segmentation module receives both the pre-operative image and the intra-operative image) and outputs the location in response to the input of both the pre-operative image and the real-time image ([0066] which discloses the processing device 140 may obtain data obtained by the medical scanning device 110, generate a medical image (e.g., a pre-operative image with contrast and an intra-operative image without contrast) using the data, and generate segmentation result data (e.g., a first segmentation image, a second segmentation image, spatial positions of blood vessels and lesions during the surgery, a registration map, a fat-free mask, etc.) by segmenting the medical image. And [0170] which discloses in 230B, a spatial position of a third target structure set during the surgery may be determined by registering a first segmentation image with a second segmentation image. In some embodiments, the operation 230B may be performed by the result determination module 2330), such that the location as output by the machine-learned model is generated by learned values of the machine-learned model applied to both the pre-operative image and the real-time image ([0084] which discloses the target structure set may be segmented from the medical image using a manner based on a deep learning convolutional network. In some embodiments, the manner based on the deep learning convolutional network may include a segmentation manner based on a fully convolutional network, such as U-net, etc. Examiner thus notes that the output of the location (i.e. the spatial position of the third target structure during the surgery) is generated by learned values applied to both the pre-operative image and the real-time image in order to generate the first segmentation and second segmentation from which the spatial position is output following registration).
It would have been obvious to a person having ordinary skill in the art before the effective filing date to have modified Ye to include a pre-operative image and an intra-operative image of the patient during surgery where the anatomy of the intra-operative image is distorted relative to the pre-operative image and indicating the location accordingly as taught by Liao in order to provide a solution for image-assisted interventional surgery, which makes the pre-operative planning achieve a higher accuracy to better assist in the accurate implementation of the corresponding puncture path during operation, thereby obtaining the ideal surgical effect (Liao [0003]). Such a modification would allow for determination of a third target structure set during the surgery which may more comprehensively and accurately reflect a current condition of the scan subject, such that the path of the interventional surgery may be planned to effectively avoid one or more non-interventional regions and/or all important organs and successfully reach the target (Liao [0090]).
Examiner notes that Ye, as currently modified, fails to explicitly teach wherein the ground truth images with labeled mass locations include both pre-operative images and intra-operative images.
However, since the machine-learned model is trained to segment both pre-operative images and intra-operative images, it would have been obvious to a person having ordinary skill in the art before the effective filing date to have modified Ye, as currently modified, to include training the machine-learned model using a plurality of ground truth pre-operative images with labeled mass locations and a plurality of ground truth intra-operative images with labeled mass locations in order to enhance the accuracy of the segmentation of both the pre-operative images and intra-operative images accordingly.
Alternatively, Meral, in a similar field of endeavor involving intra-operative guidance to an object, teaches training a machine-learned module using a plurality of ground truth pre-operative images with labeled mass locations and a plurality of ground truth intra-operative images with labeled mass locations ([0046] which discloses the disclosure includes a machine learning framework using deep learning with convolutional neural networks (CNN) to train on segmented (i.e. contoured) pre-operative MRI, pre-operative US, segmented intra-operative MRI and intra-operative US, to understand how different tumors look like in MRI vs US, what are their similar or dissimilar features. During the training, MRI's provide ground truth by manually performed segmentations, which are overlaid on the US images, which serves as the training data, via tracked acquisition. [0038] which discloses At 104, the at least one electronic processor 20 is programmed to provide the GUI 26 via which the acquired preoperative images are labeled with contours of the tumor and the surrounding blood vessels in the ROI and [0043] which discloses the at least one electronic processor 20 is programmed to tune the NN 30 using the labeled preoperative images (from 104)).
It would have been obvious to a person having ordinary skill in the art before the effective filing date to have modified Ye, as currently modified, to include training the machine-learned model as taught by Meral in order to train a neural network to learn correlations between pre-operative tumor contours and intra-operative tumor contours (Meral [0012]). Furthermore, such a modification would allow for fine-tuning a patient-specific neural network in case anatomy shifts during the procedure (Meral [0040]).
Regarding claim 2,
Ye further teaches where the guiding comprises overlaying the location (see at least fig. 11 (112/111) and corresponding disclosure in at least [0149]) on an image (at least fig. 11 (121/122) and corresponding disclosure in at least [0149]) generated during surgery.
It is not explicitly clear if the location is overlaid on a displayed view generated during surgery.
Nonetheless, Liao further teaches overlaying the location on an image generated during surgery ([0027] In some embodiments, the device for the image-assisted interventional surgery may further include a display device. The display device may be configured to display a segmentation result of the method for the image-assisted interventional surgery implemented by the processor, such display of a segmentation result is necessarily considered to be overlaid on a displayed view during surgery (i.e. for the intraoperative image. See also figs 15 depicting a segmentation result in which the location contour is overlaid on a displayed view generated during surgery see at least organ contour B intra-operative image and [0180] which discloses an element contour may be a lesion contour).
It would have been obvious to a person having ordinary skill in the art before the effective filing date to have modified Ye, as currently modified, to include overlaying the location as taught by Liao, in order to allow for a user to visualize/recognize the location of elements which are segmented. Such a modification would ensure the user better understands the location of the object during the procedure, thereby enhancing the accuracy/safety of the procedure.
Regarding claim 3,
Ye further discloses wherein guiding comprises guiding based on detected location of a surgical instrument relative to the location of the object (at least fig. 6 (622) and corresponding disclosure in at least [0106] which discloses at block 622 the process 600 involves contacting the target and electromagnetic sensors/beacon may indicate a position of the distal end of the medical instrument and with the distal end of the medical instrument disposed against and/or adjacent to the target anatomical features, such position reading can be relied upon as indicating the position of the target anatomical feature and [0117] which discloses the subprocess 631 may include the contacting 624 and retracting 635 steps)
Regarding claim 4,
Ye further teaches wherein guiding comprises guiding by a controller (at least fig. 2 (60) and corresponding disclosure in at least [0065]) of the surgical system (10) using the detected location of the surgical instrument and the location of the object ([0117] which discloses the subprocess 631 may include the contacting 624 and retracting, where guiding uses subprocess 631 and therefore uses the detected location of the surgical instrument of step 624 and the location of the object (i.e. tracked location of the target anatomical feature))
Regarding claim 5,
Ye further teaches wherein indicating comprises indicating by the machine-learned model, the machine-learned model comprising a convolutional neural network ([0107], [0130], and [0133] disclosing the framework 720 and 800 as a convolutional neural network)
Liao, as applied to claim 1 above, further teaches the machine-learned model comprising a convolutional neural network, a U-net ([0084]), or an encoder-decoder ([0122]).
Regarding claim 7,
Ye, as modified, teaches the elements of claim 6 as previously stated.
Liao, as applied to claim 6 above, further teaches wherein the pre-operative image comprises a computed tomography image and the real-time image comprises a computed tomography image ([0066] which discloses the processing device 140 may obtain data obtained by the medical scanning device 110, generate a medical image (e.g., a pre-operative image with contrast and an intra-operative image without contrast) using the data, [0061] which discloses the medical scanning device 110 may include a computed tomography (CT) scanner and [0074] which discloses the medical image may be a CT image and [0078] which discloses the intra-operative image without contrast may be a real-time scan image).
Regarding claim 9,
Ye further discloses wherein the object comprises a mass ([0156] which discloses respect to bronchoscopy procedures, target anatomical feature identification and tracking concepts of the present disclosure may be utilized to identify tumors (i.e. a mass) and/or manage scope position for the purpose of collecting samples or the like).
Liao, as applied to claim 1 above, further teaches wherein the object comprises a mass ([0081]-[0082] disclosing that the first target structure set and second target structure set include the target organ and a lesion (i.e. a mass). See also [0096]-[0097])
Regarding claim 10,
Ye further teaches wherein guiding comprises guiding, in the patient, the surgical instrument, the surgical instrument comprising a bronchoscope of the surgical system ([0038] which discloses reference herein to scopes or endoscopes may refer to a bronchoscope and [0052])
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Ye and Liao, as applied to claim 1 above, and further in view of Wei (US 20180161102 A1), hereinafter Wei.
Regarding claim 8,
Ye, as modified, teaches the elements of claim 1 as previously stated. Ye, as currently modified, further teaches wherein the indicating comprises indicating where the information includes the pre-operative image where the anatomy comprises a lung (Ye [0123] which discloses the processes may be implemented with other procedures such as lung tumor biopsy or lung operations and Liao [0080] which discloses the target organ may include a lung. See also [0095] disclosing the pre-operative image may include the target organ and a lesion the target organ may be a lung), where the location is on the lung (examiner notes that any segmentation of the target organ that is a lung would include the location being on the lung), the location being a predicted intra-operative location (Liao [0097] which discloses the second segmentation refers to the second target structure set (e.g. the lesion) obtained by segmenting the intra-operative image without contrast thus the segmentation constitutes a predicted intra-operative location).
Ye, as modified, fails to explicitly teach wherein the location is on the lung with the lung is deflated relative to the lung in the pre-operative image.
Wei, in a similar field of endeavor involving lung procedures, teaches wherein a lung is deflated relative to a lung in a pre-operative image during video assisted thoracic surgery ([0004] which discloses estimating a deflated lung shape in video assisted thoracic surgery and [0008] which discloses A first volume of air inside a lung is obtained based on a first image, e.g., a CT image, of the lung captured prior to a surgical procedure. The lung has a first shape on the first image. A second volume of air deflated from the lung is determined based on a second image, e.g., a video image, of the lung captured during the surgical procedure and [0025] which discloses for estimating a deflated lung shape and for transforming a pre-surgical plan made on images of non-deflated lung).
It would have been obvious to a person having ordinary skill in the art before the effective filing date to have modified Ye, as currently modified, to include a location of the lung with the lung deflated relative to the lung in the pre-operative image as taught by Wei in order to perform the procedure of Ye in an instance where the lungs are deflated (i.e. during a bronchoscopic procedure/thoracic surgery). Such a modification would allow for additional uses of the process of Ye, as modified, by providing the predicted intra-operative location for lungs in any condition (including when they are deflated). Examiner notes that deflating the lungs for performing the Furthermore, such a modification amounts to merely a combination of prior art elements according to known methods yielding predictable results with respect to object localization thereby rendering the claim obvious (MPEP 2143).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Ye and Liao, as applied to claim 1 above, and further in view of Kaouk et al. (US 20220354597 A1), hereinafter Kaouk.
Regarding claim 12,
Ye further discloses wherein indicating comprises indicating with the information comprising patient clinical data ([0132] which discloses In some implementations, once the neural network 720 has been trained, operational input to the neural network framework can comprise a plurality (e.g., two) of images, wherein the output 735 comprises data indicating the locations of one or more targets across the images (e.g., over time) and may provide for identification of target anatomical features in real-time endoscope images. Examiner notes that the images (i.e. real-time endoscope images) input to the neural network are considered patient clinical data (i.e. data related to the patient) in its broadest reasonable interpretation) and further teaches wherien a value of the first preset parameter of the machine learned model may be adjusted according to patient information (e.g. gender, age, a physical condition, etc.), however, fails to explicitly teach such non-image patient clinical data is input,
Ye, as modified, fails to explicitly teach the information comprising non-image patient clinical data and/or breathing cycle data.
Kaouk, in a similar field of endeavor involving robotic surgery, teaches wherein information input to a machine-learned model comprises non-image patient clinical data ([0034] which discloses data from past simulated and executed surgical procedures is input into a machine learning system which is trained to output feedback to a surgeon and to the robotic system itself. The input data may include information about the surgical procedure (e.g. patient data, the type of surgery, and the like) and [0075] which discloses according to another example, the machine learning system may be input with information indicating that the patient is elderly and thus provide control outputs that are limit force and speed due to fragile tissues of the patient).
It would have been obvious to a person having ordinary skill in the art before the effective filing date to have modified Ye, as currently modified, to include information including non-image patient clinical data as taught by Kaouk in order to improve the robotic control of Ye. Such a modification would thereby allow for improved safety of the system by allowing for adjusted force control gains and speed control such that the robotic control can be made to slow down or use more/less force based on information about the patient such as age (Kaouk [0075]). Furthermore, such a modification would allow for inputting of patient information (e.g. gender, age, etc.) such that parameters may be adjusted to adapt to the clinical situation as desired by Liao ([0213])
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Ye and Liao, as applied to claim 1 above, and further in view of Meral et al. (US 20210204914 A1), hereinafter Meral.
Regarding claim 21,
Ye, as modified, teaches the elements of claim 1 as previously stated. Ye, as modied, fails to explicitly teach wherein information input into the machine learned model comprises a position of the object in the pre-operative image, the real-time image, and the position of the object in the pre-operative image, the indicated location comprising an intra-operative location.
Meral, in a similar field of endeavor involving medical procedure guidance, teaches wherein information input into a machine learned model comprises a position of an object in a pre-operative image ([0036] which discloses in this configuration, the electronic processor of the US scanner 14 operates the scanner to acquire US images of the patient and to provide the GUI via which a user contours select US images of the patient, and this patient-specific training data is then uploaded to the remote server which performs the patient-specific tuning by update-training the NN 30 using the supplied patient-specific training data and [0040] which discloses [0040] At 108, the at least one electronic processor 20 is programmed to tune the trained NN 30 for the patient using the labeled 2D preoperative images generated in the operations 102, 104 so as to generate a patient-tuned trained neural network) such that the information received by the machine learned model to indicate a location of an object comprises the pre-operative image (see at least fig. 2 (108) and corresponding disclosure in at least [0040] At 108, the at least one electronic processor 20 is programmed to tune the trained NN 30 for the patient using the labeled 2D preoperative images generated in the operations 102, 104 so as to generate a patient-tuned trained neural network [0036] which discloses this patient-specific training data is then uploaded to the remote server which performs the patient-specific tuning by update-training the NN 30 using the supplied patient-specific training data), a real-time image (see at least fig. 2 (112) and corresponding disclosure in at least [0042]), and the position of the object in the pre-operative image (see at least fig. 2 (108) and corresponding disclosure in at least [0040] At 108, the at least one electronic processor 20 is programmed to tune the trained NN 30 for the patient using the labeled 2D preoperative images generated in the operations 102, 104 so as to generate a patient-tuned trained neural network [0036] which discloses this patient-specific training data is then uploaded to the remote server which performs the patient-specific tuning by update-training the NN 30 using the supplied patient-specific training data), the indicated location comprising an intra-operative location (at least fig. 2 (114) and corresponding disclosure in at least [0045]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date to have modified the information of Ye, as currently modified, to include a position of the object in the pre-operative image as taught by Meral in order to tune a trained neural network for the patient using labeled preoperative images to generate a patient-tuned trained neural network (Meral [0008]). Such a modification would provide for determining object position during shifting of the organ and predicting a new position of the object due to shifting of the organ during a surgical procedure (Meral [0011]). Such a modification would thereby improve the contour estimation in the intra-operative images (Meral [0016]).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BROOKE L KLEIN whose telephone number is (571)270-5204. The examiner can normally be reached Mon-Fri 7:30-4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anne Kozak can be reached at 5712700552. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BROOKE LYN KLEIN/Primary Examiner, Art Unit 3797
Prosecution Insights