DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 17 is objected to because of the following informalities:
Claim 17 appears to be written in an independent form, yet also refers back to the other independent claim (claim 1). In one interpretation, claim 17 may be construed as an independent apparatus claim. In another interpretation, claim 17 may be construed as a dependent claim. In order to prevent any ambiguity, it is suggested to bring the entire claim 1 (i.e., the medical support device) into claim 17 to have claim 17 construed as a proper independent claim.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2, 3, and 9-11 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 2 recites “wherein the medical support device and the image processing device are separate bodies.” The only element of the medical support device is a processor. While there are no recited elements of the image processing device, it is described as being essentially a computer with a processor, memory, and storage (see, e.g., Figures 4 and 5). While Applicant’s disclosure supports the medical support device and the image processing devices being distinct elements, the term “bodies” is confusing, particularly because of the distributive natures of a processor and a processing device and the broad meaning of “body.”
Based on Applicant’s disclosure (see, e.g., Figures 4 and 5 and [0074]), Examiner is interpreting claim 2 as reciting “wherein the medical support device further comprises a housing, the image processing device having a housing that is physically separate from the housing of the medical support device.
Claim 3 recites “wherein the medical support device is connected to the image processing device by using a cable, and the processor is configured to acquire the first image information and the second image information from the image processing device via the cable.” Neither the cable nor the image processing device are recited as elements (i.e., features) of the claimed medical support device. This is consistent with Applicant’s disclosure. (see, e.g., Figures 2-4 and [0076]). Instead, claim 3 appears to be reciting that the medical support device is capable of (or configured for) acquiring the image information from the image processing device via the cable.
Accordingly, Examiner is interpreting claim 3 as reciting “wherein the medical support device is capable of connecting to the image processing device by using a cable, and the processor is configured to acquire the first image information and the second image information from the image processing device via the cable.”
Claims 9 and 10 recite “wherein the processor is configured to not output information, which is obtained from the optical-image-trained model by inputting the first optical image included in the first image information to the optical-image-trained model,….” It is unclear if the scope of claims 9 and 10 include (a) the first optical image being inputted into the optical-image-trained model but the processor suppressing information that would have been outputted or (b) the first optical image is not being inputted into the optical-image-trained model and, consequently, the processor does not output information.
Examiner is interpreting claims 9 and 10 to include the possibility of either (a) or (b). However, Applicant should amend the claims to more clearly set forth the scope.
Claim 11 recites “wherein the optical-image-trained model is a trained model obtained by performing machine learning for the second optical image.” It is unclear if (a) the trained model is being trained using the second optical image or (b) the trained model was trained using optical images that are similar to the second optical image.
Examiner is interpreting claim 11 to be consistent with (b). However, Applicant should amend the claims to more clearly set forth the scope.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-18 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claims recite:
acquire first image information and output a first optical image and an ultrasound image from the image information; (claims 1 and 18)
acquire second image information and output a second optical image and first recognition result, which is obtained by inputting the second optical image to an optical-image-trained model and causing the optical-image-trained model to recognize a first feature region shown in the second optical image; (claims 1 and 18)
the first trained model generates the first positional information based on the input second optical image; (claim 13)
output a second recognition result, which is obtained by inputting the ultrasound image included in the first image information to an ultrasound-image-trained model and causing the ultrasound-image-trained model to recognize a second feature region shown in the ultrasound image; (claim 14)
the second trained model generates the second positional information based on the input ultrasound image; (claim 16)
wherein the first image processing device generates the ultrasound image based on a reflected wave of ultrasound emitted from the ultrasound endoscope; (claim 17)
the second image processing device generates the first optical image based on a first image signal obtained by performing optical imaging via the ultrasound endoscope and generates the second optical image based on a second image signal obtained by performing optical imaging via the optical endoscope; (claim 17).
Claim limitation [a], as drafted and under its broadest reasonable interpretation, recites a mathematical concept. (MPEP 2106.04(a)(2)(I) (see, e.g., Digitech Image Techs., LLC v. Electronics for Imaging, Inc., 758 F.3d 1344, 1350, 111 USPQ2d 1717, 1721 (Fed. Cir. 2014) (although the claims did not recite a particular mathematical formula, the court held “[w]ithout additional limitations, a process that employs mathematical algorithms to manipulate existing information to generate additional information is not patent eligible.”)). The claim limitation is a mathematical concept because the claim limitation requires mathematical manipulation of image data to acquire and output an image.
Claim limitations [b] and [d], as drafted and under their broadest reasonable interpretations, recite a mathematical concept and/or mental process. (MPEP 2106.04(a)(2)(I) (see, e.g., Digitech Image Techs., LLC v. Electronics for Imaging, Inc., 758 F.3d 1344, 1350, 111 USPQ2d 1717, 1721 (Fed. Cir. 2014) (although the claims did not recite a particular mathematical formula, the court held “[w]ithout additional limitations, a process that employs mathematical algorithms to manipulate existing information to generate additional information is not patent eligible.”)). The claim limitation is a mathematical concept because the claim limitation requires mathematical manipulation of image data to acquire and output an image. The claim limitation also recites a mental process because it can be performed in a human mind. (see MPEP § 2106.04(a)(2)(III)). Examples of mental processes include “observations, evaluations, judgments, and opinions.” (Id). In this case, the first or second recognition results are an evaluation of image data, which is a human cognitive action that has been performed for decades. The fact that a “machine-learned model” is used does not negate the judicial exception. (see MPEP § 2106.04(a)(2)(III) (explaining that claims requiring a computer still may recite a mental process) (see also the recently decided Recentive Analytics, Inc. v. Fox Corp., No. 2023-2437, Federal Circuit, 18 April 2025: “[C]laims that do no more than apply established methods of machine learning to a new data environment” are not patent eligible).
Claim limitation [c] and [e], as drafted and under their broadest reasonable interpretations, recite a mental process that can be performed in a human mind. (see MPEP § 2106.04(a)(2)(III)). Examples of mental processes include “observations, evaluations, judgments, and opinions.” (Id). In this case, the positional information is based on an evaluation of image data, which is a human cognitive action that has been performed for decades. The fact that a “machine-learned model” is used does not negate the judicial exception. (see MPEP § 2106.04(a)(2)(III) (explaining that claims requiring a computer still may recite a mental process) (see also the recently decided Recentive Analytics, Inc. v. Fox Corp., No. 2023-2437, Federal Circuit, 18 April 2025: “[C]laims that do no more than apply established methods of machine learning to a new data environment” are not patent eligible).
Claim limitations [f] and [g], as drafted and under their broadest reasonable interpretation, recite a mathematical concept. (MPEP 2106.04(a)(2)(I) (see, e.g., Digitech Image Techs., LLC v. Electronics for Imaging, Inc., 758 F.3d 1344, 1350, 111 USPQ2d 1717, 1721 (Fed. Cir. 2014) (although the claims did not recite a particular mathematical formula, the court held “[w]ithout additional limitations, a process that employs mathematical algorithms to manipulate existing information to generate additional information is not patent eligible.”)). The claim limitation is a mathematical concept because the claim limitation requires mathematical manipulation of data (e.g., optical or ultrasonic) to generate an image.
It is also noted that claim limitations [a]-[g] collectively claim the abstract idea of “collecting information, analyzing it, and displaying certain results of the collection and analysis,” where the data analysis steps are recited at a high level of generality such that they could practically be performed in the human mind, Electric Power Group v. Alstom, S.A., 830 F.3d 1350, 1353-54, 119 USPQ2d 1739, 1741-42 (Fed. Cir. 2016).
This judicial exception is not integrated into a practical application. “A claim that integrates a judicial exception into a practical application will apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception.” (MPEP 2106.04(d)). Courts look to additional claimed elements/steps to determine whether the judicial exception is integrated into a practical application. In this case, the additional elements include:
a processor; (claims 1 and 18)
using an ultrasound endoscope; (claims 1 and 18)
using an optical endoscope; (claims 1 and 18)
using a display device; (claims 1 and 18)
using imaging processing devices, including first and second image processing device; (claims 1-9, 17, and 18)
using a cable to communicate image information; (claim 3)
the image processing device(s) and the medical support device being separate bodies; (claims 2 and 6);
details of the recognition results based on image analysis; (claims 12 and 15)
specification information regarding a type of endoscope and image; (claim 8).
These additional elements and/or steps do not meaningful limit the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception. For example, the additional elements and/or steps do not improve the functioning of a computer or other technology or technical field. Moreover, the additional elements and/or steps do not apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment (i.e., endoscopic procedures).
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. The additional elements and/or steps do not meaningfully limit the claim. For example, most of the additional elements and/or steps recite well-understood, routine, conventional activities previously known to the industry (e.g., processor or image processing devices, endoscopes, display device, cable). Moreover, identifying details within a medical image is a well-known feature to those having ordinary skill in the art. (see, e.g., discussion below in Section 103 rejection of claims 12 and 15).
Accordingly, each of claims 1-18, as a whole, does not amount to significantly more than the judicial exception. Claims 1-18 are rejected under 35 U.S.C. 101 for lacking patent-eligible subject matter.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6, 12, 13, 17, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Appl. Publ. No. 2024/0037808 A1 (hereinafter “FOUTS”) and U.S. Patent Appl. Publ. No. 2007/0265492 A1 (hereinafter “SONNENSCHEIN”).
PNG
media_image1.png
391
677
media_image1.png
Greyscale
FOUTS teaches system architectures for real-time processing and displaying of medical imaging data using an external processing device. (Abstract and Title). The external processing device receives surgical video data and analyzes the video data using a machine-learning model. The ML model generates overlay data to add to the video data, which is then displayed to the user. (Abstract and [0004]). “Post-capture processing of imaging data may improve how the imaging data is displayed to the surgeon. For example, the imaging data may be modified or augmented to improve the appearance and the interpretability of the imaging data. In particular, machine learning algorithms can be used to identify objects or artifacts in the imaging data, segment image data, and/or augment the imaging data.” ([0004]).
In Figure 4 of FOUTS, the central device 402 can include the external AI processing device. ([0133]: “In some examples, the central device 402 includes one or more components described herein for processing surgical video data, such as programmable circuit 202 of FIG. 2 or its subparts, processing system 222 of FIG. 2 or its subparts, or any combination thereof.”; and [0098]: “the system 200 can comprise a programmable circuit 202 and a processing system 222. As discussed above, in some examples, the two components may be enclosed in a single housing as an external AI processing device.”).
With respect to claim 1, FOUTS teaches a medical support device. Figure 4 of FOUTS illustrates an exemplary system 200 (i.e., medical support device), which can be external to and receive image data from an endoscope. “The system 200 may receive imaging data from a medical imaging device (e.g., endoscopic camera head 108 in FIG. 1 , or the camera control unit 112 in FIG. 1 ), process the received imaging data, and output the processed imaging data for display on a display (e.g., display 118 in FIG. 1 ).” ([0097]). The medical support device comprising:
a processor. Figure 4 of FOUTS illustrates an exemplary system for managing a surgical environment, each of which is connected to a central device 402. “[T]he central device 402 includes one or more components described herein for processing surgical video data, such as programmable circuit 202 of FIG. 2 or its subparts, processing system 222 of FIG. 2 or its subparts, or any combination thereof.”
wherein the processor is configured to:
acquire first image information including a first optical image, which are generated by performing imaging via an endoscope. Figure 1 of FOUTS shows an endoscope 102. “The endoscope 102 may extend from an endoscopic camera head 108 that includes one or more imaging sensors 110.” ([0094]). “The one or more imaging sensors 110 generate pixel data that can be transmitted to a camera control unit 112 that can be communicatively connected to the camera head 108. The camera control unit 112 generates a video feed from the pixel data that shows the tissue being viewed by the camera at any given moment in time.” ([0110]).
PNG
media_image2.png
857
1328
media_image2.png
Greyscale
acquire second image information including a second optical image, which is generated by performing imaging via a different endoscope. See above regarding endoscope 102. NOTE: The central device 402 can be connected to one or more surgical devices. ([0135]). The surgical devices can include “…an ultrasound detector, and imager, or any combination thereof.” ([0072] or [0135]). Moreover, it is suggested that multiple surgical devices can be connected at once. (See, e.g., Figure 4 illustrating two different CCUs, a pointer device, and a C-arm capture device).
output the first optical image, which are included in the first image information, to a display device in a case in which the ultrasound endoscope is connected to the image processing device. See Figure 2 shown here. The processing system 222 inputs the images to trained ML models and then outputs information. “The processor(s) on the processing system 222 can then execute the one or more trained machine-learning models 226 to generate output data such as overlay data or on-screen overlay display (OSD) data 228.” ([0099]). “The one or more machine-learning models 226 can generate various types of output data that may enhance the live surgical video data. In some examples, the one or more machine-learning models 226 can be configured to identify one or more objects of interest in an input frame and output one or more graphical overlays indicating the one or more objects of interest.” ([0100]).
output the second optical image, which is included in the second image information, and a first recognition result, which is obtained by inputting the second optical image to an optical-image-trained model and causing the optical-image-trained model to recognize a first feature region shown in the second optical image, to the display device in a case in which the optical endoscope is connected to the image processing device. The processing system 222 inputs the images to trained ML models and then outputs information. “The processor(s) on the processing system 222 can then execute the one or more trained machine-learning models 226 to generate output data such as overlay data or on-screen overlay display (OSD) data 228.” ([0099]). “The one or more machine-learning models 226 can generate various types of output data that may enhance the live surgical video data. In some examples, the one or more machine-learning models 226 can be configured to identify one or more objects of interest in an input frame and output one or more graphical overlays indicating the one or more objects of interest. For example, the machine-learning models may identify anatomical features of interest (e.g., a polyp or cyst) in an input frame and output graphical overlays (e.g., a bounding box) indicating the detected anatomical features.” ([0100]). As shown in Figure 2, the overlay data is mixed with the original image data to provide a composite image.
While FOUTS clearly teaches an optical endoscope that captures and communicates optical images and an external medical support device that acquires the optical images (e.g., [0094] and [0097]), FOUTS does not explicitly teach the external device acquiring a first optical image and an ultrasound image, which are generated by performing imaging via an ultrasound endoscope. Nonetheless, FOUTS teaches that the surgical devices can include endoscopes ([0094]) as well as “…an ultrasound detector, and imager, or any combination thereof.” ([0072] or [0135]). FOUTS also suggests receiving and processing image data from one imaging device or from multiple imaging devices. “In some examples, the system 200 may be associated with a single medical imaging device and be configured to process the imaging data (e.g., one or more medical images) captured by the single medical imaging device. In other examples, the system 200 may be associated with multiple medical imaging devices and configured to process imaging data captured by the multiple imaging devices (e.g., a frame with color information and a frame with fluorescence information).” ([0097]).
FOUTS also does not explicitly teach that an ultrasound endoscope and an optical endoscope that may be, separately, connected to the image processing device. Nonetheless, FOUTS teaches that the surgical devices can include endoscopes ([0094]) as well as “…an ultrasound detector, and imager, or any combination thereof.” ([0072] or [0135]). It is also strongly suggested that the FOUTS system is not limited to a single configuration for a single workflow or procedure. (see, e.g., Figure 4 and [0097] and [0134]). FOUTS also considers circumstances in which an ML model is not used. “While the processing system 222 depicted in FIG. 2 stores machine-learning models, it should be appreciated that the processing system 222 may store image processing algorithms that are not machine-learning-based or AI-based. Indeed, the techniques described herein can allow processing of any type of imaging data (e.g., obtained by any type of imaging systems) by any type of imaging processing algorithms without introducing significant latency between the collection of the imaging data and the display of such data.” ([0111]).
In the same field of endeavor, SONNENSCHEIN teaches a multipurpose endoscopy suite that is “adaptable for carrying out a plurality of endoscopy procedures and/or for supporting several endoscopes being used simultaneously in a given procedure.” (Abstract and Title). SONNENSCHEIN teaches that “[e]ndoscopy has advanced in recent years to the stage where many different procedures which are related to a specific body system or organ have become commonplace and are carried out in large numbers.” ([0003]). Moreover, “differing requirements of different [endoscopic] procedures have resulted in the development of a large variety of specialized endoscopes.” ([0005]). Notably, the advantage to having an endoscopy suite configured for a certain procedure “is in the simplicity of its operation and in its safety. This is because the operator does not have to be concerned with the settings of the different procedure-related parameters (e.g., light intensity, insufflation and suction pressures, etc) since these are determined in advance, set during the production stage of the endoscopy suite, and cannot be easily changed.” SONNENSCHEIN concludes that because “most endoscopy procedures require more or less the same peripheral equipment and resources, and the main differences between different endoscopic procedures is the range of parameters within which the equipment must work, it would be advantageous to supply a single endoscopy suite that is capable of being used with many, if not all, types of endoscopes.” ([0007]).
One object of SONNENSCHEIN’s invention is to provide an “endoscopes suite that can simultaneously support more than two different types of endoscopes used during a single procedure.”
PNG
media_image3.png
578
654
media_image3.png
Greyscale
SONNENSCHEIN teaches separate image processing devices, specifically an ultrasound module 206 and a video module 210. (see, e.g., Figure 2 and [0048] and [0049]). “Ultrasound module 206 comprises a signal generator, processing means, memory, and interfacing circuitries 205 required for emitting and acquiring ultrasound signals from the ultrasound transducer via the ultrasound connector 202…The processing means provides the digital signal processing (DSP) capabilities required for processing and analyzing the acquired ultrasound signals. The results of the signal analysis are displayed on display 203, which is linked to ultrasound module 206.” ([0048]).
With respect to the video module 210, “[t]he image data received form the camera is processed by the video module 210 and displayed on the video display 214. To enable it to display the images, the video module 210 should include means for acquiring the image signals from the camera and processing means capable of carrying out the DSP tasks involved in processing the acquired image data.” ([0049]). Notably, SONNENSCHEIN considers using both the ultrasound module and the video module together. “It should be noted that the display 203 of unit 250 is not necessarily required if the ultrasound module 206 and video module 210 are linked (broken line in FIG. 2). In that case modules 206 and 210 can be adapted to provide the information output produced by the ultrasound module 206 on the video display 214, which is directly linked to the video module 210.” ([0050]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the system for managing surgical devices in FOUTS to include an endoscopy suite as taught in SONNENSCHEIN such that the FOUTS system would apply to image data from different endoscopes, including image data from an optical endoscope and image data from an ultrasound endoscope. One of ordinary skill in the art would have been motivated to add the external AI processing device of FOUTS to the endoscopy suite of SONNENSCHEIN so that, in the case of the optical endoscope, the surgeon/technician could better visualize the patient’s anatomy in the optical images using overlay data from FOUTS that is provided without latency. In the case of an ultrasound endoscope, the external AI processing device would output the optical image and the ultrasound image to the display device. There would have been a reasonable expectation of success as FOUTS teaches that the external AI processing device can be applied to endoscopy image data.
With respect to claim 2 (and in light of the Section 112(b) rejection), FOUTS teaches wherein the medical support device (system 200 in Figure 2) and the image processing device (camera control unit 112 and/or image processing unit 116) are separate bodies. As taught in FOUTS, the external AI processing device can be separate from the imaging device. (see, e.g., [0071]: “…a housing; a programmable circuit enclosed in the housing configured to receive the surgical video data from a camera control unit;… and at least one galvanically-isolated USB connector configured to be connected to a surgical device for contact with a patient during the surgery” ([0072]).
With respect to claim 3 (and in light of the Section 112(b) rejection), FOUTS teaches wherein the medical support device is connected to the image processing device by using a cable (see [0071] describing USB connectors exposed on the housing), and the processor is configured to acquire the first image information and the second image information from the image processing device via the cable (the image information would necessarily be acquired through the cable). See also [0075]: “[T]he at least one galvanically-isolated USB connector is configured to provide a power line, a ground line, and one or more signal transmission lines with the surgical device.” SONNENSCHEIN also teaches a camera connector 211 for receiving image data through the connection. (see, e.g., [0049] and Figure 3).
With respect to claim 4, as discussed above with respect to claim 1, SONNENSCHEIN teaches wherein the image processing device includes a first image processing device (ultrasound module 206) and a second image processing device (video module 210), the first image processing device generates the ultrasound image based on a reflected wave of ultrasound emitted from the ultrasound endoscope (see, e.g., [0048] of SONNENSCHEIN), and the second image processing device generates the first optical image based on a first image signal obtained by performing optical imaging via the ultrasound endoscope and generates the second optical image based on a second image signal obtained by performing optical imaging via the optical endoscope. See, e.g., [0049] and [0050] describing instances in which an ultrasound image and an optical image are acquired together. Moreover, SONNENSCHEIN teaches ultrasound transducers being positioned on the endoscope. ([0022]). NOTE: Applicant’s disclosure describes examples in which the ultrasound endoscope includes integrated ultrasonic and optical features as well as separate scopes or probes that are coupled together. (see, e.g., [0059] of Applicant’s disclosure).
With respect to claim 5, as discussed above with respect to claim 1, SONNENSCHEIN teaches wherein the first image information is generated by the first image processing device or the second image processing device (ultrasound image is generated by the ultrasound module 206, [0048]), the second image information is generated by the first image processing device or the second image processing device (the second optical image is generated by the video module 210, ([0049]).
As discussed above with respect to claim 1, FOUTS teaches that the processor is configured to: acquire the first image information from the first image processing device or the second image processing device (the system 200 would receive the ultrasound image from the ultrasound module 206); and acquire the second image information from the first image processing device or the second image processing device (the system 200 would receive the optical image from the video module 210).
With respect to claim 6, FOUTS and SONNENSCHEIN teach wherein the first image processing device, the second image processing device, and the medical support device are separate bodies. As discussed above with respect to claim 2, the processing system 222 of FOUTS is in a separate, external housing. Moreover, SONNENSCHEIN teaches that the ultrasound module and the video module may be separate elements. “The endoscopy suite may comprise a communication interface linked to the processor and/or a communication interface linked to the ultrasound module and/or a communication interface linked to the video module….” ([0025]; see also Figure 2 showing modules 206 and 210 being separate, being separately powered by power supplies 200, 215, respectively, being separately connected to the endoscope through connectors 202, 211, respectively, and being connected to different displays 203, 214, respectively.
With respect to claim 12, FOUTS teaches wherein the first recognition result includes first positional information for specifying a position of the first feature region, first feature information for specifying a medical feature of the first feature region, and/or first part information for specifying a part inside a body as a target of an examination via the optical endoscope. More specifically, FOUTS teaches that the first recognition result includes first position information for specifying a position of the first feature region. “For example, if the machine-learning models 226 include a segmentation model configured to identify an object of interest and provide overlay data indicating the location of the object of interest, the system may track the movement of the object of interest across multiple frames in the video data. When the overlay data is generated based on a given frame to indicate the location of the object of interest, the system can adjust the location of the overlay because the overlay data is to be mixed with a frame that is captured later than the given frame.” ([0112]).
With respect to claim 13 (depending from claim 12), FOUTS teaches wherein the optical-image-trained model includes a first trained model to which the second optical image is input, and the first trained model generates the first positional information based on the input second optical image. “Optionally, the machine-learning model is configured to identify one or more objects of interest in an input frame….” ([0041]; see also [0106] and Figure 2 and the input frame).
With respect to claim 17 (depending from claim 12), as discussed above, FOUTS and SONNENSCHEIN teach an endoscope system (Figure 1 and Figure 4) comprising: the medical support device according to claim 1; and the image processing device including a first image processing device and a second image processing device (ultrasound module 206 and video module 210), wherein the first image processing device generates the ultrasound image based on a reflected wave of ultrasound emitted from the ultrasound endoscope (ultrasound module 206, [0048]), and the second image processing device generates the first optical image based on a first image signal obtained by performing optical imaging via the ultrasound endoscope (video module 210, [0049] and [0050]) and generates the second optical image based on a second image signal obtained by performing optical imaging via the optical endoscope (video module 210, [0049] and [0050]).
With respect to claim 18, FOUTS teaches a medical support method. Figure 4 of FOUTS illustrates an exemplary system 200 (i.e., medical support device), which can be external to and receive image data from an endoscope. “The system 200 may receive imaging data from a medical imaging device (e.g., endoscopic camera head 108 in FIG. 1 , or the camera control unit 112 in FIG. 1 ), process the received imaging data, and output the processed imaging data for display on a display (e.g., display 118 in FIG. 1 ).” ([0097]). The medical support device comprising:
acquiring first image information including a first optical image, which are generated by performing imaging via an endoscope. Figure 1 of FOUTS shows an endoscope 102. “The endoscope 102 may extend from an endoscopic camera head 108 that includes one or more imaging sensors 110.” ([0094]). “The one or more imaging sensors 110 generate pixel data that can be transmitted to a camera control unit 112 that can be communicatively connected to the camera head 108. The camera control unit 112 generates a video feed from the pixel data that shows the tissue being viewed by the camera at any given moment in time.” ([0110]).
acquiring second image information including a second optical image, which is generated by performing imaging via a different endoscope. See above regarding endoscope 102. NOTE: The central device 402 can be connected to one or more surgical devices. ([0135]). The surgical devices can include “…an ultrasound detector, and imager, or any combination thereof.” ([0072] or [0135]). Moreover, it is suggested that multiple surgical devices can be connected at once. (See, e.g., Figure 4 illustrating two different CCUs, a pointer device, and a C-arm capture device).
outputting the first optical image, which are included in the first image information, to a display device in a case in which the endoscope is connected to the image processing device. The processing system 222 inputs the images to trained ML models and then outputs information. “The processor(s) on the processing system 222 can then execute the one or more trained machine-learning models 226 to generate output data such as overlay data or on-screen overlay display (OSD) data 228.” ([0099]). “The one or more machine-learning models 226 can generate various types of output data that may enhance the live surgical video data. In some examples, the one or more machine-learning models 226 can be configured to identify one or more objects of interest in an input frame and output one or more graphical overlays indicating the one or more objects of interest.” ([0100]).
outputting the second optical image, which is included in the second image information, and a first recognition result, which is obtained by inputting the second optical image to an optical-image-trained model and causing the optical-image-trained model to recognize a first feature region shown in the second optical image, to the display device in a case in which the optical endoscope is connected to the image processing device. The processing system 222 inputs the images to trained ML models and then outputs information. “The processor(s) on the processing system 222 can then execute the one or more trained machine-learning models 226 to generate output data such as overlay data or on-screen overlay display (OSD) data 228.” ([0099]). “The one or more machine-learning models 226 can generate various types of output data that may enhance the live surgical video data. In some examples, the one or more machine-learning models 226 can be configured to identify one or more objects of interest in an input frame and output one or more graphical overlays indicating the one or more objects of interest. For example, the machine-learning models may identify anatomical features of interest (e.g., a polyp or cyst) in an input frame and output graphical overlays (e.g., a bounding box) indicating the detected anatomical features.” ([0100]). As shown in Figure 2, the overlay data is mixed with the original image data to provide a composite image.
While FOUTS clearly teaches an optical endoscope that captures and communicates optical images and an external medical support device that acquires the optical images (e.g., [0094] and [0097]), FOUTS does not explicitly teach the external device acquiring a first optical image and an ultrasound image, which are generated by performing imaging via an ultrasound endoscope. Nonetheless, FOUTS teaches that the surgical devices can include endoscopes ([0094]) as well as “…an ultrasound detector, and imager, or any combination thereof.” ([0072] or [0135]). FOUTS also suggests receiving and processing image data from one imaging device or from multiple imaging devices. “In some examples, the system 200 may be associated with a single medical imaging device and be configured to process the imaging data (e.g., one or more medical images) captured by the single medical imaging device. In other examples, the system 200 may be associated with multiple medical imaging devices and configured to process imaging data captured by the multiple imaging devices (e.g., a frame with color information and a frame with fluorescence information).” ([0097]).
FOUTS also does not explicitly teach that an ultrasound endoscope and an optical endoscope that may be, separately, connected to the image processing device. Nonetheless, FOUTS teaches that the surgical devices can include endoscopes ([0094]) as well as “…an ultrasound detector, and imager, or any combination thereof.” ([0072] or [0135]). It is also strongly suggested that the FOUTS system is not limited to a single configuration for a single workflow or procedure. (see, e.g., Figure 4 and [0097] and [0134]). FOUTS also considers circumstances in which an ML model is not used. “While the processing system 222 depicted in FIG. 2 stores machine-learning models, it should be appreciated that the processing system 222 may store image processing algorithms that are not machine-learning-based or AI-based. Indeed, the techniques described herein can allow processing of any type of imaging data (e.g., obtained by any type of imaging systems) by any type of imaging processing algorithms without introducing significant latency between the collection of the imaging data and the display of such data.” ([0111]).
In the same field of endeavor, SONNENSCHEIN teaches a multipurpose endoscopy suite that is “adaptable for carrying out a plurality of endoscopy procedures and/or for supporting several endoscopes being used simultaneously in a given procedure.” (Abstract and Title). SONNENSCHEIN teaches that “[e]ndoscopy has advanced in recent years to the stage where many different procedures which are related to a specific body system or organ have become commonplace and are carried out in large numbers.” ([0003]). Moreover, “differing requirements of different [endoscopic] procedures have resulted in the development of a large variety of specialized endoscopes.” ([0005]). Notably, the advantage to having an endoscopy suite configured for a certain procedure “is in the simplicity of its operation and in its safety. This is because the operator does not have to be concerned with the settings of the different procedure-related parameters (e.g., light intensity, insufflation and suction pressures, etc) since these are determined in advance, set during the production stage of the endoscopy suite, and cannot be easily changed.” SONNENSCHEIN concludes that because “most endoscopy procedures require more or less the same peripheral equipment and resources, and the main differences between different endoscopic procedures is the range of parameters within which the equipment must work, it would be advantageous to supply a single endoscopy suite that is capable of being used with many, if not all, types of endoscopes.” ([0007]).
One object of SONNENSCHEIN’s invention is to provide an “endoscopes suite that can simultaneously support more than two different types of endoscopes used during a single procedure.”
SONNENSCHEIN teaches separate image processing devices, specifically an ultrasound module 206 and a video module 210. (see, e.g., Figure 2 and [0048] and [0049]). “Ultrasound module 206 comprises a signal generator, processing means, memory, and interfacing circuitries 205 required for emitting and acquiring ultrasound signals from the ultrasound transducer via the ultrasound connector 202…The processing means provides the digital signal processing (DSP) capabilities required for processing and analyzing the acquired ultrasound signals. The results of the signal analysis are displayed on display 203, which is linked to ultrasound module 206.” ([0048]).
With respect to the video module 210, “[t]he image data received form the camera is processed by the video module 210 and displayed on the video display 214. To enable it to display the images, the video module 210 should include means for acquiring the image signals from the camera and processing means capable of carrying out the DSP tasks involved in processing the acquired image data.” ([0049]). Notably, SONNENSCHEIN considers using both the ultrasound module and the video module together. “It should be noted that the display 203 of unit 250 is not necessarily required if the ultrasound module 206 and video module 210 are linked (broken line in FIG. 2). In that case modules 206 and 210 can be adapted to provide the information output produced by the ultrasound module 206 on the video display 214, which is directly linked to the video module 210.” ([0050]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the system for managing surgical devices in FOUTS to include an endoscopy suite as taught in SONNENSCHEIN such that the FOUTS system would apply to image data from different endoscopes, including image data from an optical endoscope and image data from an ultrasound endoscope. One of ordinary skill in the art would have been motivated to add the external AI processing device of FOUTS to the endoscopy suite of SONNENSCHEIN so that, in the case of the optical endoscope, the surgeon/technician could better visualize the patient’s anatomy in the optical images using overlay data from FOUTS that is provided without latency. In the case of an ultrasound endoscope, the external AI processing device would output the optical image and the ultrasound image to the display device. There would have been a reasonable expectation of success as FOUTS teaches that the external AI processing device can be applied to endoscopy image data.
Claims 7-10 and 14-16 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Appl. Publ. No. 2024/0037808 A1 (hereinafter “FOUTS”) and U.S. Patent Appl. Publ. No. 2007/0265492 A1 (hereinafter “SONNENSCHEIN”) as applied to claim 1 above, and further in view of U.S. Patent Appl. Publ. No. 2005/0157168 A1 (hereinafter “KANEKO”) or, alternatively, in view of International Patent Publ. No. WO 2022/210508 A1 (hereinafter “SHIMIZU”), the disclosure of which is provided by counter-part U.S. Patent Appl. Publ. No. 2024/0013392 A1.
With respect to claim 7, FOUTS and SONNENSCHEIN do not explicitly teach the claim limitations. More specifically, FOUTS and SONNENSCHEIN do not explicitly teach wherein each of the first image information and the second image information includes first specification information and/or second specification information, the first specification information is information for specifying a type of an endoscope connected to the image processing device, the second specification information is information for specifying a type of an image included in the first image information or the second image information, the types of the endoscope are the ultrasound endoscope and the optical endoscope, the types of the image are the ultrasound image, the first optical image, and the second optical image, and the processor is configured to specify whether the ultrasound endoscope is connected to the image processing device or the optical endoscope is connected to the image processing device, based on the first specification information and/or the second specification information.
However, FOUTS suggests receiving and processing image data from one imaging device or from multiple imaging devices. “In some examples, the system 200 may be associated with a single medical imaging device and be configured to process the imaging data (e.g., one or more medical images) captured by the single medical imaging device. In other examples, the system 200 may be associated with multiple medical imaging devices and configured to process imaging data captured by the multiple imaging devices (e.g., a frame with color information and a frame with fluorescence information).” ([0097]). Likewise, SONNENSCHEIN contemplates using different types of optical and imaging systems ([0003]). SONNENSCHEIN also desires to make the endoscopy suite more user-friendly by reducing the number of steps performed by the user. (Abstract and [0006]).
In the same field of endeavor, KANEKO concerns endoscopic imaging systems and procedures in which the distinction between different endoscopies can cause changes in the performance of the system. (see, e.g., [0004]: “…if a different type of electronic endoscope is employed and, for example, a solid-state image pickup device incorporated in the endoscope offers a different number of pixels, it is hard to display an image using an appropriate domain within the number of pixels.”). As such, one “object” of KANEKO is to “provide an endoscopic image processing system that even when a different type of endoscope is employed….can display an image….” ([0006]).
To this end, KANEKO teaches “a detecting means that detects at least one of a type of electronic endoscope connected to the endoscopic image processing system on the basis of information inherent to the electronic endoscope and a type of image pickup device incorporated in the connected electronic endoscope….” ([0009]). The endoscope-inherent identification information may include “optical type information representing an angle of view or a degree of optical zoom, usage information (indicating use for the superior digestive tract or use for the inferior directive tract), and information representing the number of pixels to be offered by the incorporated CCD 28.” ([0050]). This identification information is detected using a detection circuit. Notably, the identification information “can be directly detected on the path of a signal sent from the CCD 28.” In other words, the identification information is part of the image information.
Accordingly, KANEKO teaches communicating “first specification information” within the “image information,” wherein “the first specification information is information for specifying a type of an endoscope connected to the image processing device” as recited in claim 7. Moreover, KANEKO teaches that the processor is configured to specify the type of endoscope connected to the image-processing device based on the specification information. (see, e.g., “CCD=TYPE 1?” in Figure 6).
It would have been obvious to one having ordinary skill in the art at the time of filing to determine the type of endoscope (i.e., ultrasound endoscope or optical endoscope) that is connected to the FOUTS-SONNENSCHEIN system, as taught in KANEKO. One of ordinary skill in the art would have been motivated to determine the type of endoscope connected to the system so that the correct ML model could be applied to the image data. (see, e.g., FOUTS teaching that different ML models may be used and that different image data may be processed). There would have been a reasonable expectation of success as KANEKO teaches that the type of endoscope can be determined using the image information.
Alternatively, in the same field of endeavor, SHIMIZU teaches a medical image processing system (e.g., endoscopy system) that generates images with information that identifies the type of image. (Abstract and Title). SHIMIZU teaches that systems may use “image enhanced endoscopy (IEE)” to assist in diagnosis. ([0003]). SHIMIZU also teaches that “there is a case where an appropriate diagnosis may be made by acquiring a plurality of types of endoscope images obtained using a plurality of types of illumination light or the like, and by comparing in detail or superimposing these endoscope images.” However, different images require different analysis. “Appropriate diagnosis support information can be obtained by performing different image processing between the endoscope image based on the normal image signal obtained using white light and the endoscope image based on the special image signal obtained using special light.” ([0005]). In other words, “it is preferable to… perform image processing on the endoscope image selected according to the type of the endoscope image.” ([0005]).
With this goal in mind, SHIMIZU teaches a system in which the processor device “generates an identification information-assigned medical image generated by assigning a part of data constituting a medical image as identification information indicating a type of the medical image….” (Abstract; see also [0086]). The system then “acquires the identification information-assigned medical image, identifies the type of the medical image, and performs image processing corresponding to the type of the medical image.” (Abstract). Notably, the identification information is part of the data constituting the image. “The data constituting the endoscope image means, for example, not data other than the image, such as a header portion of an information container storing the endoscope image, but data of the image itself.” ([0087]).
It would have been obvious to one having ordinary skill in the art at the time of filing to determine the type of image (i.e., image being communicated from an ultrasound endoscope or optical endoscope) that is connected to the FOUTS-SONNENSCHEIN system, as taught in SHIMIZU. One of ordinary skill in the art would have been motivated to determine the type of image being communicated so that the correct ML model could be applied to the image data. (see, e.g., FOUTS teaching that different ML models may be used and that different image data may be processed). There would have been a reasonable expectation of success as KANEKO teaches that the type of endoscope can be determined using the image information.
With respect to claim 8, FOUTS-SONNENSCHEIN, as modified by KANEKO or, alternatively, SHIMIZU, teaches wherein the processor is configured to specify whether the ultrasound endoscope is connected to the image processing device or the optical endoscope is connected to the image processing device, by analyzing an image included in the first image information or the second image information.
As discussed above, KANEKO teaches that image analysis can be used to determine whether the ultrasound endoscope is connected to the image processing device or the optical endoscope is connected to the image processing device. “[T]ype information included in the identification information, such as, the number of pixels to be offered by the image pickup device (CCD 28) incorporated in the electronic endoscope 2 can be directly detected on the path of a signal sent from the CCD 28.” ([0052]). For example, “a means that applies a CCD driving signal to the CCD, checking the number of peaks of the waveform of an output signal of the CCD, and thus detecting the number of pixels (number of horizontal dots, number of vertical dots) will do.” ([0054]).
Alternatively, SHIMIZU teaches that image analysis can be used to determine whether the ultrasound endoscope is connected to the image processing device or the optical endoscope is connected to the image processing device. SHIMIZU teaches a system in which the processor device “generates an identification information-assigned medical image generated by assigning a part of data constituting a medical image as identification information indicating a type of the medical image….” (Abstract; see also [0086]). To be clear, the identification information is part of the image itself. “The data constituting the endoscope image means, for example, not data other than the image, such as a header portion of an information container storing the endoscope image, but data of the image itself.” ([0087]).
With respect to claim 9, FOUTS does not explicitly teach wherein the processor is configured to not output information, which is obtained from the optical-image-trained model by inputting the first optical image included in the first image information to the optical-image-trained model, to the display device in a case in which the ultrasound endoscope is connected to the image processing device. However, FOUTS does teach that different ML models may be used and different image data may be analyzed. (see, e.g., [0097]).
SHIMIZU teaches that different types of images require different ML model analysis. “It is preferable that the image analysis section 96 performs the analysis image processing using a machine learning-based analysis model…Further, it is preferable that the analysis model is different for each type of the identification information-assigned endoscope image 82. This is because the content of the analysis image processing capable of outputting a good result is different for each type of the identification information-assigned endoscope image 82…Therefore, it is preferable that the image analysis section 96 comprises a plurality of the analysis models and uses an appropriate analysis model according to the type of the endoscope image.” ([0126]).
NOTE: In the case in which the ultrasound endoscope is connected to the image processing device, the type of image generated by the ultrasound endoscope would not be the same as the type of image for which the optical-image-trained model was designed.
It would have been obvious to one having ordinary skill in the art at the time of filing to configure the processor to not output information obtained from a trained model that is trained for a different type of image from a different endoscope. Instead, the processor would output information from the appropriate trained model, as taught in SHIMIZU. One of ordinary skill in the art would have been motivated to input the image data to the trained models that were trained for that image data. There would have been a reasonable expectation of success as SHIMIZU teaches that different ML models may be used for different image data.
With respect to claim 10, FOUTS does not explicitly teach wherein the processor is configured to not output information, which is obtained from the optical-image-trained model by inputting the first optical image included in the first image information to the optical-image-trained model, to the display device in a case in which the ultrasound image is displayed in a first display region of the display device. However, FOUTS does teach that different ML models may be used and different image data may be analyzed. (see, e.g., [0097]).
SHIMIZU teaches that different types of images require different ML model analysis. “It is preferable that the image analysis section 96 performs the analysis image processing using a machine learning-based analysis model…Further, it is preferable that the analysis model is different for each type of the identification information-assigned endoscope image 82. This is because the content of the analysis image processing capable of outputting a good result is different for each type of the identification information-assigned endoscope image 82…Therefore, it is preferable that the image analysis section 96 comprises a plurality of the analysis models and uses an appropriate analysis model according to the type of the endoscope image.” ([0126]).
NOTE: In the case in which the ultrasound image is displayed in a first display region of the display device, the type of image generated by the ultrasound endoscope would not be the same as the type of image for which the optical-image-trained model was designed.
It would have been obvious to one having ordinary skill in the art at the time of filing to configure the processor to not output information obtained from a trained model that is trained for a different type of image from a different endoscope. Instead, the processor would output information from the appropriately trained model, as taught in SHIMIZU. One of ordinary skill in the art would have been motivated to input the image data to the trained models that were trained for that image data. There would have been a reasonable expectation of success as SHIMIZU teaches that different ML models may be used for different image data.
With respect to claim 14, FOUTS does not explicitly teach wherein the processor is configured to output a second recognition result, which is obtained by inputting the ultrasound image included in the first image information to an ultrasound-image-trained model and causing the ultrasound-image-trained model to recognize a second feature region shown in the ultrasound image, to the display device in a case in which the ultrasound image is displayed in a first display region of the display device. However, FOUTS does teach outputting a first recognition result, obtained by inputting an image into a trained model and causing the model to recognize a feature region. For example, “[t]he processor(s) on the processing system 222 can then execute the one or more trained machine-learning models 226 to generate output data such as overlay data or on-screen overlay display (OSD) data 228.” ([0099]). “The one or more machine-learning models 226 can generate various types of output data that may enhance the live surgical video data. In some examples, the one or more machine-learning models 226 can be configured to identify one or more objects of interest in an input frame and output one or more graphical overlays indicating the one or more objects of interest. For example, the machine-learning models may identify anatomical features of interest (e.g., a polyp or cyst) in an input frame and output graphical overlays (e.g., a bounding box) indicating the detected anatomical features.” ([0100]; see also [0112]).
FOUTS does not explicitly teach the input image being an ultrasound image or the model being trained specifically for those images.
However, SHIMIZU teaches that different types of images require different ML model analysis. “It is preferable that the image analysis section 96 performs the analysis image processing using a machine learning-based analysis model…Further, it is preferable that the analysis model is different for each type of the identification information-assigned endoscope image 82. This is because the content of the analysis image processing capable of outputting a good result is different for each type of the identification information-assigned endoscope image 82…Therefore, it is preferable that the image analysis section 96 comprises a plurality of the analysis models and uses an appropriate analysis model according to the type of the endoscope image.” ([0126]).
It would have been obvious to one having ordinary skill in the art at the time of filing to configure the processor to use a differently trained model (i.e., trained with ultrasound images) when inputting the ultrasound images and have the model recognize features that are capable of being found using the model. One of ordinary skill in the art would have been motivated to input the image data into the appropriately trained model in order to assist the surgeon in identifying relevant features within the image (e.g., polyps). There would have been a reasonable expectation of success as SHIMIZU teaches that different ML models may be used for different image data.
With respect to claim 15, while FOUTS does not explicitly teach wherein the second recognition result includes second positional information for specifying a position of the second feature region, second feature information for specifying a medical feature of the second feature region, and/or second part information for specifying a part inside a body as a target of an examination via the ultrasound endoscope in the particular context of ultrasound images, FOUTS does teach the claim limitation with respect to images in general (see discussion above with respect to claim 12). More specifically, FOUTS teaches that the first recognition result includes first position information for specifying a position of the first feature region. “For example, if the machine-learning models 226 include a segmentation model configured to identify an object of interest and provide overlay data indicating the location of the object of interest, the system may track the movement of the object of interest across multiple frames in the video data. When the overlay data is generated based on a given frame to indicate the location of the object of interest, the system can adjust the location of the overlay because the overlay data is to be mixed with a frame that is captured later than the given frame.” ([0112]).
It would have been obvious to one having ordinary skill in the art at the time of filing to configure the processor to use a differently trained model (i.e., trained with ultrasound images) to identify the positional information of the feature region. One would have been motivated to configure the processor to do so in order to assist the surgeon in identifying relevant features within the image (e.g., polyps). There would have been a reasonable expectation of success as SHIMIZU teaches that different ML models may be used for different image data.
With respect to claim 16, while FOUTS does not explicitly teach wherein the ultrasound-image-trained model includes a second trained model to which the ultrasound image is input, and the second trained model generates the second positional information based on the input ultrasound image, it is well-known that different type of images should be applied to different AI models. This is explicitly taught in SHIMIZU. “It is preferable that the image analysis section 96 performs the analysis image processing using a machine learning-based analysis model…Further, it is preferable that the analysis model is different for each type of the identification information-assigned endoscope image 82. This is because the content of the analysis image processing capable of outputting a good result is different for each type of the identification information-assigned endoscope image 82…Therefore, it is preferable that the image analysis section 96 comprises a plurality of the analysis models and uses an appropriate analysis model according to the type of the endoscope image.” ([0126]).
It would have been obvious to one having ordinary skill in the art at the time of filing to configure the processor to use a differently trained model (i.e., trained with ultrasound images) when inputting the ultrasound images and have the model recognize features that are capable of being found using the model. One of ordinary skill in the art would have been motivated to input the image data into the appropriately trained model in order to assist the surgeon in identifying relevant features within the image (e.g., polyps). There would have been a reasonable expectation of success as SHIMIZU teaches that different ML models may be used for different image data.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Appl. Publ. No. 2024/0037808 A1 (hereinafter “FOUTS”), and U.S. Patent Appl. Publ. No. 2007/0265492 A1 (hereinafter “SONNENSCHEIN”), and U.S. Patent Appl. Publ. No. 2005/0157168 A1 (hereinafter “KANEKO”) or, alternatively, in view of International Patent Publ. No. WO 2022/210508 A1 (hereinafter “SHIMIZU”), the disclosure of which is provided by counter-part U.S. Patent Appl. Publ. No. 2024/0013392 A1, as applied to claim 9 above, and further in view of Park, Junseok, et al. “Reduced detection rate of artificial intelligence in images obtained from untrained endoscope models and improvement using domain adaptation algorithm.” Frontiers in Medicine 9 (2022): 1036974 (hereinafter “PARK”).
With respect to claim 11, the cited art does not explicitly describe training AI models with particular endoscope images. As such, the cited art does not explicitly teach wherein the optical-image-trained model is a trained model obtained by performing machine learning for the second optical image. However, it is well within the knowledge of one having ordinary skill in the art that, when analyzing medical images using a trained AI model, the AI model should be trained using similar types of images. This is supported by PARK.
PARK teaches that “[a] training dataset that is limited to a specific endoscope model can overfit artificial intelligence (AI) to its unique image characteristics. The performance of the AI may degrade in images of different endoscope model.” (Abstract). PARK noted that images from two different endoscopes can be distinguished using AI. “Between the two endoscope manufacturers, there was a difference in image characteristics that could be distinguished without error by AI.” (Abstract). Moreover, “[t]he accuracy of the AI in recognizing gastroesophageal junction was >0.979 in the same endoscope-examined validation dataset as the training dataset. However, they deteriorated in datasets from different endoscopes.”
It would have been obvious to one having ordinary skill in the art at the time of filing to train the optical-image-trained model by performing machine learning for the second optical image. One of ordinary skill in the art would have been motivated to train the model using the same type of images that would be subsequently input into the model for analysis. There would have been a reasonable expectation of success as taught in PARK.
Prior Art Made of Record
Prior art that is made of record but not relied upon in the above Section 103 rejections includes:
US-20180253839-A1, which describes different AI models that detect different features in endoscopic images.
US-20210192738-A1, which describes AI models that identify site information and lesion type information to generate a report to the user.
US-20230210603-A1, which describes an external AI model system that is configured to augment the visualization environment presented to the surgeon by merging, in real-time, video feed and ultrasound imaging; tracking anatomy and instruments; identifying critical structures; generating and displaying 3-dimensional models of relevant anatomy; providing actionable guidance to the user.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASON P GROSS whose telephone number is (571)272-1386. The examiner can normally be reached Monday-Friday 9:00-5:00CT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anne M. Kozak can be reached at (571) 270-5284. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASON P GROSS/Examiner, Art Unit 3797
/SERKAN AKAR/Primary Examiner, Art Unit 3797
Prosecution Insights