Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
DETAILED ACTION
Response to Arguments
The amendments address the 112b rejections and those rejections are withdrawn.
Claim Objections
Claims 26, 28, and 50 are objected to for the use of the word “filed.” It appears that applicant intended the word “field.”
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 26-37, 41-45, and 50 are rejected under 35 USC 103 as being unpatentable over US 2021/0307841, Buch et al. (hereafter Buch) in view of US 2022/0246307, Nakamura (hereafter Nakamura).
26. A non-transitory computer readable recording medium storing a computer program for causing a computer to execute processing of:
acquiring an operation field image obtained by shooting an operation field of a scopic surgery; and (Buch ¶5 obtains images during surgery)
distinctively recognizing blood vessels included in the acquired operation field image and a notable blood vessel among the blood vessels by using a first learning model trained to output information relevant to a blood vessel when the operation field image is input, and (Buch ¶5 outputting anatomical objects identified by a neural network; ¶21 blood vessel)
Buch does not explicitly disclose
a second learning model, generated separately from the first learning model, trained to output information relevant to the notable blood vessel among the blood vessels when the operation filed image is input.
However, Nakamura at ¶179-188 discloses using various trained models for specific purposes, including suture and bleeding sites as well as complications. Thus, it would have been obvious to modify the system of Buch to include the use of a second trained model, different from the first to output information relevant to the notable blood vessels of Bush for the purposes of utilizing a specialized trained model for the specific purpose of the model.
Claim 50 is similarly rejected.
27. The non-transitory computer readable recording medium according to claim 26, storing the computer program for causing the computer to execute processing of: displaying a blood vessel portion recognized from the operation field image and a notable blood vessel portion on the operation field image to be discriminable. (Buch ¶5 outputting anatomical objects identified by a neural network; ¶21 blood vessel; see also ¶39)
Buch does not disclose
28. The non-transitory computer readable recording medium according to claim 27, storing the computer program for causing the computer to execute processing of: switching between a display of the blood vessel portion recognized from the operation filed image and a display . (Nakamura ¶62 provides the ability to switch between images on the display)
29. The non-transitory computer readable recording medium according to claim 27 storing the computer program for causing the computer to execute processing of: displaying both of the blood vessel portions in different display modes. (Buch ¶5 outputting anatomical objects identified by a neural network; ¶21 blood vessel; see also ¶39; capable of displaying with labels and without)
30. The non-transitory computer readable recording medium according to claim 27, storing the computer program for causing the computer to execute processing of: periodically switching display and non-display of at least one recognized blood vessel portion. (Buch ¶5 outputting anatomical objects identified by a neural network; ¶21 blood vessel; see also ¶39; capable of displaying with labels and without)
31. The non-transitory computer readable recording medium according to claim 27, storing the computer program for causing the computer to execute processing of: applying a predetermined effect to the display of the at least one recognized blood vessel portion. (Buch ¶5 outputting anatomical objects identified by a neural network; ¶21 blood vessel; see also ¶39; capable of displaying with labels and without)
32. The non-transitory computer readable recording medium according to claim 27, storing the computer program for causing the computer to execute processing of: calculating a confidence of a recognition result of the first learning model and the second learning model; and displaying at least one blood vessel portion in a display mode according to the calculated confidence. (Buch ¶20 predict most likely object and then display the result; this would apply to both learning models as the system in claim 26 is modified by Nakamura)
33. The non-transitory computer readable recording medium according to claim 26, storing the computer program for causing the computer to execute processing of: displaying an estimated position of a blood vessel portion hidden behind other objects, with reference to a recognition result of the first learning model and second learning model. (Buch ¶23 hidden objects; this would apply to both learning models as the system in claim 26 is modified by Nakamura)
34. The non-transitory computer readable recording medium according to claim 26, storing the computer program for causing the computer to execute processing of: estimating a running pattern of the blood vessel by using the first learning model and the second learning model; and (Buch ¶23 positional relationships between objects; this would apply to both learning models as the system in claim 26 is modified by Nakamura)
displaying an estimated position of a blood vessel portion not appearing in the operation field image, on the basis of the estimated running pattern of the blood vessel. (Buch ¶24 output feedback)
35. The non-transitory computer readable recording medium according to claim 26, wherein the second learning model is trained to output information relevant to the blood vessel not existing in a central visual field of a surgeon, as a recognition result of the notable blood vessel. (Buch ¶23 positional relationships between objects; this would apply to both learning models as the system in claim 26 is modified by Nakamura)
36. The non-transitory computer readable recording medium according to claim 26, wherein the second learning model is trained to output information relevant to a blood vessel existing in the central visual field of the surgeon, as the recognition result of the notable blood vessel. (Buch ¶23 positional relationships between objects; this would apply to both learning models as the system in claim 26 is modified by Nakamura)
37. The non-transitory computer readable recording medium according to claim 26, wherein the second learning model is trained to output information relevant to a blood vessel in a state of tension, and the computer program causes the computer to further execute processing of recognizing a blood vessel portion in a state of tension as the notable blood vessel, on the basis of the information output from the learning model. (Buch ¶22 amount of manipulation that a part can handle (tension) and output warnings as necessary; this would apply to both learning models as the system in claim 26 is modified by Nakamura)
41. The non-transitory computer readable recording medium according to claim 26, storing the computer program for causing the computer to execute processing of:
acquiring a special light image obtained by shooting the operation field by emitting another illumination light different from illumination light for the operation field image;
generating a combined image of the operation field image and the special light image;
recognizing a blood vessel portion appearing in the combined image by using a learning model for a combined image trained to output information relevant to a blood vessel appearing in the combined image when the combined image is input; and
displaying the recognized blood vessel portion to be superimposed on the operation field image.
42. The non-transitory computer readable recording medium according to claim 26, storing the computer program for causing the computer to execute processing of: detecting bleeding, on the basis of the operation field image; and
outputting warning information when the bleeding is detected. (Buch ¶28 detect blood loss)
43. The non-transitory computer readable recording medium according to claim 26, storing the computer program for causing the computer to execute processing of: detecting approach of a surgical tool, on the basis of the operation field image; and (Buch ¶28 detect movement of surgical object)
displaying the notable blood vessel to be discriminable when the approach of the surgical tool is detected. (Buch ¶32 warning of approach; ¶29 indicates that blood vessels are part of the intended invention and ¶32 bone was used as an example)
44. The non-transitory computer readable recording medium according to claim 26, storing the computer program for causing the computer to execute processing of: enlarged displaying a blood vessel portion recognized as the notable blood vessel. (Buch ¶25 zoom)
45. The non-transitory computer readable recording medium according to claim 26, storing the computer program for causing the computer to execute processing of: outputting control information to a medical device, on the basis of the recognized blood vessel. (Buch ¶33 output to a device that provides guidance to a surgeon)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 38-40 are rejected under 35 USC 103 as being unpatentable over Buch in view of Nakamura and US 2018/0211385, Imai (hereafter Imai)
Buch does not disclose all of:
38. The non-transitory computer readable recording medium according to claim 26, storing the computer program for causing the computer to execute processing of: recognizing a blood flow flowing the blood vessel included in the operation field image by using a learning model for recognizing a blood flow trained to output information relevant to a blood flow, in accordance with the input of the operation field image; and
displaying a blood vessel recognized by using a learning model for recognizing a blood vessel in a display mode according to an amount of blood flow, with reference to a recognition result of the blood flow of the learning model.
Imai ¶55 discloses recognizing blood flow from an endoscope image. It would have been obvious to modify the system of Buch to include the use of blood flow as an additional feature in its machine learning and surgery assistance display system for the purposes of providing further details about the blood vessel as taught by Imai (¶10)
39. The non-transitory computer readable recording medium according to claim 26, storing the
displaying the recognized blood vessel portion to be superimposed on the operation field image. (Buch ¶39 output overlay)
Bush does not fully disclose:
computer program for causing the computer to execute processing of: acquiring a special light image obtained by shooting the operation field by emitting another illumination light different from illumination light for the operation field image;
recognizing a blood vessel portion appearing in the special light image by using a learning model for a special light image trained to output information relevant to a blood vessel appearing in the special light image when the special light image is input; and
Imai ¶4 discloses the use of narrow wavelength light for blood vessel observation. See also ¶41. Buch discloses a learning model as in claim 26. It would have been obvious to modify the system of Buch to include the endoscopic light system of Imai for the purposes of obtaining special observation images emphasizing features as taught by Imai (¶4)
40. The non-transitory computer readable recording medium according to claim 39, storing the computer program for causing the computer to execute processing of: displaying the blood vessel portion recognized from the operation field image and the blood vessel portion recognized from the special light image to be switchable. (Imai ¶41 switchable observation mode)
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ming Shui whose telephone number is (303)297-4247. The examiner can normally be reached on 7-5 Pacific Time, M-Th.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Greg Morse can be reached on 571-272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ming Shui/
Primary Examiner, Art Unit 2663