DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-12, 14-16, 25, 28, and 29 are rejected under 35 U.S.C. 103 as being unpatentable over Ohara, US 2022/0222840 A1 (Ohara).
Regarding claim 1, Ohara teaches a system (endoscope system) (Fig. 1; [0039]) comprising:
a memory storing instructions (storage section 330; including a program and various data) (Fig. 1; [0082-0083]); and
one or more processors (wherein the processing section 310 and the control circuit 320 may be implemented by a processor) (Fig. 1; [0081] and [0083]) communicatively coupled to the memory (storage section 330; including a program and various data) (Fig. 1; [0082-0083]) and configured to execute the instructions to perform a process (a processor that operates based on the information stored in the memory; wherein the process is implemented when the processor executes the instruction) ([0082-0083]) comprising:
accessing a first image sequence (capturing “n” surface images) (Fig. 15; [0132]) (sequential imaging) ([0058]) captured by an imaging device (captured by the image acquisition section 311 of the endoscope system) (Figs. 1 and 2; [0039] and [0132]) during a medical treatment (wherein the surgeon or physician is imaging for position and size of the target portion and performs treatment) ([0064-0065]), the first image sequence comprising first images (the first surface images) ([0132]) (sequential imaging) ([0058]), the first images based on illumination of a scene (living body part) ([0054]) associated with the medical treatment (wherein the surgeon or physician is imaging for position and size of the target portion and performs treatment) ([0064-0065]) using visible-spectrum light (acquiring the surface image(s) based on illuminating with white light) (Fig. 15; [0073] and [0132]);
accessing a second image sequence (capturing “n” luminance images) (Fig. 15; [0132]) (sequential imaging) ([0058]) captured by the imaging device (captured by the image acquisition section 311 of the endoscope system) (Figs. 1 and 2; [0039] and [0132]) during the medical treatment (wherein the surgeon or physician is imaging for position and size of the target portion and performs treatment) ([0064-0065]), the second image sequence comprising second images (the second luminance images) ([0132]), the second images comprising non-visible-spectrum images (acquiring luminescence image(s)) ([0073] and [0132]) based on illumination of the scene (living body part) ([0054]) using non-visible spectrum light (based on infrared (IR) light) ([0073] and [0132]);
detecting one or more features in the second images (such as detecting an observation target as the liver, and the target portion as a tumor in the liver) (Fig. 7; [0063]) (as well as detecting three-dimensional depth information) ([0133]);
applying one or more labels to the second images to generate labeled second images (wherein labels can be applied to the luminescence image) ([0122]), the one or more labels indicating the one or more features (wherein the label depicts the three-dimensional depth information) ([0133-0134]); and
processing the first images and the labeled second images using a machine learning module (processing the surface images and the luminescence images using machine learning; trained model) ([0134-0135]).
Although Ohara does not explicitly teach that images are taken during a medical “procedure”, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention that the images are taken to assist the user to make a diagnosis or perform treatment ([0151-0152]) including allowing the user to easily perform the treatment by performing the excision along the excision position indicated by the support information ([0153]); which is an obvious medical procedure.
Regarding claim 2, Ohara teaches wherein the second images (luminescence image(s)) ([0073]) are based on sensing of infrared light (based on sensing the second illumination which is infrared light) ([0063] and [0073]).
Regarding claim 3, Ohara teaches wherein the infrared light comprises light emitted by illuminated fluorophores (the IR light is used for fluorescence observation, using a fluorescent pigment of ICG which absorbs the infrared light and emits fluorescence) ([0041]).
Regarding claim 4, Ohara teaches wherein the machine learning module comprises a machine learning algorithm (the machine learning being a type of software) ([0126-0127]), and wherein the processing comprises training (training) ([0126-0127]), by the machine learning algorithm, a machine learning model (trained model is obtained by the machine learning) ([0133]) based on the first image sequence and the second image sequence (training a machine learning model based on the surface image(s) and the luminescence image(s)) ([0126]).
Regarding claim 5, Ohara teaches wherein the machine learning module comprises a trained machine learning model (trained machine learning module) ([0127], [0129], and [0133]), and wherein the processing comprises generating, by the trained machine learning model (trained machine learning module) ([0127], [0129], and [0133]), a prediction based on the first image sequence and the second image sequence (an inference is made based on the surface image(s) and the luminescence image(s)) ([0131-0134]).
Regarding claim 6, Ohara teaches wherein the prediction comprises one or more of: a predicted image, a predicted label indicative of features in one or more of the first image sequence or the second image sequence, an image segmentation, a predicted stage of a medical procedure, or a predicted geometry corresponding to the scene (the three-dimensional shape information based on the surface image and the luminescence image can be estimated based on the trained model) ([0133-0134]).
Regarding claim 7, Ohara teaches the process further comprising providing the prediction (based on the estimated three-dimensional shape information) ([0151-0152]) to a computer-assisted medical system that performs an operation based on the prediction (display; wherein the depth can be displayed to assist the user to make a diagnosis or perform treatment) ([0151-0153]).
Regarding claim 8, Ohara teaches wherein the processing comprises generating labels for the first image sequence based on the second image sequence (superimposing information about the target portion, such as a label for depth, included in the IR image on the white light image) (Fig. 8; [0064] and [0151]).
Regarding claim 9, Ohara teaches wherein the processing further comprises training a machine learning model based on the first image sequence and the labels (training a model based on machine learning using the surface image(s) and the correct labels) ([0133] and [0135]).
Regarding claim 10, Ohara teaches a system (endoscope system) (Fig. 1; [0039]) comprising:
a memory storing instructions (storage section 330; including a program and various data) (Fig. 1; [0082-0083]); and
one or more processors (wherein the processing section 310 and the control circuit 320 may be implemented by a processor) (Fig. 1; [0081] and [0083]) communicatively coupled to the memory (storage section 330; including a program and various data) (Fig. 1; [0082-0083]) and configured to execute the instructions to perform a process (a processor that operates based on the information stored in the memory; wherein the process is implemented when the processor executes the instruction) ([0082-0083]) comprising:
accessing a first image sequence (capturing “n” surface images) (Fig. 15; [0132]) (sequential imaging) ([0058]) captured by an imaging device (captured by the image acquisition section 311 of the endoscope system) (Figs. 1 and 2; [0039] and [0132]) during a medical treatment (wherein the surgeon or physician is imaging for position and size of the target portion and performs treatment) ([0064-0065]), the first image sequence comprising first images (the first surface images) ([0132]) (sequential imaging) ([0058]), the first images based on illumination of a scene (living body part) ([0054]) associated with the medical treatment (wherein the surgeon or physician is imaging for position and size of the target portion and performs treatment) ([0064-0065]) using visible-spectrum light (acquiring the surface image(s) based on illuminating with white light) (Fig. 15; [0073] and [0132]);
providing the first images (wherein the surface images are provided to the trained model) ([0127] and [0132]) to a trained machine learning model (trained model based on machine learning) ([0127]),
wherein the trained machine learning model has been trained using a second image sequence (wherein the trained model has been trained using luminescence images) ([0126-0127]) comprising second images (“n” luminance images) (Fig. 15; [0132]) based on illumination of the scene(living body part) ([0054]) using non-visible-spectrum light (based on infrared (IR) light) ([0073] and [0132]); and
performing, based on an output of the trained machine learning model (using the trained model to estimate the three-dimensional shape information) ([0134]), an operation with respect to the first image sequence (wherein the surface image can be displayed, superimposed with the luminescence image, can include depth information, and/or can include support information) (Figs. 16 and 17; [0151] and [0153]).
Although Ohara does not explicitly teach that images are taken during a medical “procedure”, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention that the images are taken to assist the user to make a diagnosis or perform treatment ([0151-0152]) including allowing the user to easily perform the treatment by performing the excision along the excision position indicated by the support information ([0153]); which is an obvious medical procedure.
Regarding claim 11, Ohara teaches the process further comprising generating, based on the output of the trained machine learning model, a prediction (using the trained model to estimate the three-dimensional shape information) ([0134]) for use with a computer-assisted medical system (display screen; wherein the depth can be displayed to assist the user to make a diagnosis or perform treatment) ([0151-0152]).
Regarding claim 12, Ohara teaches the process further comprising displaying (displaying) ([0151-0152]), based on the prediction (based on the estimated three-dimensional shape information) ([0151-0152]), a user interface by way of a display of the computer-assisted medical system (display screen; wherein the depth can be displayed to assist the user to make a diagnosis or perform treatment) ([0151-0153]).
Regarding claim 14, Ohara teaches wherein the output comprises a modified version of an image in the first images (wherein the surface image can be superimposed with the luminescence image, can include depth information, and/or can include support information) (Figs. 16 and 17; [0151] and [0153]), and wherein the operation comprises displaying the modified version of the image (displaying the modified image on the display) (Figs. 16 and 17; [0151] and [0153]).
Regarding claim 15, Ohara teaches wherein the modified version of the image comprises a segmentation of the image (segmenting the image by showing support information that shows an excision position by a broken line) (Fig. 17; [0153]).
Regarding claim 16, Ohara teaches wherein the operation comprises one or more of: segmenting an image in the first images (segmenting the image by showing support information that shows an excision position by a broken line) (Fig. 17; [0153]), labeling the image (labeling the image) ([0122]), categorizing the image, reconstructing a geometry or measure of the scene, or identifying a feature depicted in the image (such as detecting an observation target as the liver, and the target portion as a tumor in the liver; as well as a three-dimensional depth information) (Fig. 7; [0063] and [0133]).
Regarding claim 25, Ohara teaches a system (endoscope system) (Fig. 1; [0039]) comprising:
a memory storing instructions (storage section 330; including a program and various data) (Fig. 1; [0082-0083]); and
one or more processors (wherein the processing section 310 and the control circuit 320 may be implemented by a processor) (Fig. 1; [0081] and [0083]) communicatively coupled to the memory (storage section 330; including a program and various data) (Fig. 1; [0082-0083]) and configured to execute the instructions to perform a process (a processor that operates based on the information stored in the memory; wherein the process is implemented when the processor executes the instruction) ([0082-0083]) comprising:
accessing a first image sequence (capturing “n” surface images) (Fig. 15; [0132]) captured by an imaging device (captured by the image acquisition section 311 of the endoscope system) (Figs. 1 and 2; [0039] and [0132]) during a medical treatment (wherein the surgeon or physician is imaging for position and size of the target portion and performs treatment) ([0064-0065]), the first image sequence comprising first images (the first surface images) ([0132]) (sequential imaging) ([0058]), the first images based on illumination of a scene (living body part) ([0054]) associated with the medical treatment (wherein the surgeon or physician is imaging for position and size of the target portion and performs treatment) ([0064-0065]) using visible-spectrum light (acquiring the surface image(s) based on illuminating with white light) (Fig. 15; [0073] and [0132]);
providing the first images (wherein the surface images are provided to the trained model) ([0127] and [0132]) to a trained machine learning model (trained model based on machine learning) ([0127]),
wherein the trained machine learning model has been trained using a second image sequence (wherein the trained model has been trained using luminescence images) ([0126-0127]) (sequential imaging) ([0058]) comprising second images (“n” luminance images) (Fig. 15; [0132]) based on illumination of the scene(living body part) ([0054]) using non-visible-spectrum light (based on infrared (IR) light) ([0073] and [0132]); and
generating, based on an output of the trained machine learning model, a prediction (using the trained model to estimate the three-dimensional shape information) ([0134]).
Although Ohara does not explicitly teach that images are taken during a medical “procedure”, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention that the images are taken to assist the user to make a diagnosis or perform treatment ([0151-0152]) including allowing the user to easily perform the treatment by performing the excision along the excision position indicated by the support information ([0153]); which is an obvious medical procedure.
Regarding claim 28, Ohara teaches wherein the process further comprises performing, based on the prediction (using the trained model to estimate the three-dimensional shape information) ([0134]), an operation with respect to a computer-assisted medical system (display screen; wherein the depth can be displayed to assist the user to make a diagnosis or perform treatment) ([0151-0152]).
Regarding claim 29, Ohara teaches wherein the performing the operation comprises one or more of
displaying a graphical user interface by way of a display of the computer-assisted medical system (display screen; wherein the depth can be displayed to assist the user to make a diagnosis or perform treatment) ([0151-0152]),
displaying an image included in the first images by way of the display of the computer-assisted medical system (wherein the surface image can be displayed, superimposed with the luminescence image, can include depth information, and/or can include support information) (Figs. 16 and 17; [0151] and [0153]),
.
Claim(s) 13 is rejected under 35 U.S.C. 103 as being unpatentable over Ohara, US 2022/0222840 A1 (Ohara), and further in view of Ye et al., US 2021/0196398 A1 (Ye).
Regarding claim 13, Ohara teaches based on a prediction (based on the estimated three-dimensional shape information) ([0151-0152]) displaying the depth and the excision position (Figs 16 and 17; [0151-0153]).
However, Ohara does not explicitly teach “controlling a movement of a component of the computer-assisted medical system”.
Ye teaches systems, devices, and methods to facilitate the identification and tracking of various anatomical features based on images of such features obtained using a scope device or other medical instrument ([0004]); and wherein the process further comprising controlling, based on the updated target position, a movement of a component of the computer-assisted medical system (causing a medical instrument to be articulated in response to the updated target position data; wherein the method is performed by control circuitry of a medical system and the target position data and the first image are received from an endoscope of the medical system) ([0010-0011]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Ohara to include controlling a movement of the medical system since it allows for accurate, real-time target tracking which can enable relatively precise single-stick access to the treatment site (Ye; [0104]).
Claim(s) 17 is rejected under 35 U.S.C. 103 as being unpatentable over Ohara, US 2022/0222840 A1 (Ohara), and further in view of Lyman et al., US 2020/0160980 A1 (Lyman).
Regarding claim 17, Ohara teaches wherein the output comprises a label associated with the image (such as detecting an observation target as the liver, and the target portion as a tumor in the liver) (Fig. 7; [0063]) (as well as detecting/labeling three-dimensional depth information) ([0133]).
However, Ohara does not explicitly teach “wherein the label comprises an indication of at least one of a type of tissue, an identification of an organ, or an indication of a type of object”.
Lyman teaches captured images of physical, anatomical features of a patient, taken by a camera, can be similarly be processed by the captured image processing system to generate diagnosis data ([0328]); and wherein the label comprises an indication of at least one of a type of tissue, an identification of an organ, or an indication of a type of object (wherein a computer vision model is trained and can output labeling data indicating the anatomical regions, type of bodily tissue, type of bodily fluid, and/or other classifying information in the captured image) ([0329]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Ohara to include labeling a type of tissue since it can thus be used to more accurately determine or predict if a patient's condition is bettering or worsening, to more accurately determine or predict if a patient is responding well or poorly to treatment, and/or to otherwise aid in diagnosing a patient's condition (Lyman; [0356]).
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL J VANCHY JR whose telephone number is (571)270-1193. The examiner can normally be reached Monday - Friday 9am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL J VANCHY JR/Primary Examiner, Art Unit 2666 Michael.Vanchy@uspto.gov
Prosecution Insights