Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 29 August 2025 have been fully considered but they are not persuasive. Applicant’s amendments to independent claim 1 appear to clarify the recited subject matter of original claim 1, while specifying which of the models is used at which zoom level. Applicant’s arguments with respect to the original rejection under 35 U.S.C. § 103 are as follows:
“Karino does not disclose or suggest "when the magnification change information indicates the normal magnification...run the first trained model to perform the detection process on the received target image" and "when the magnification change information indicates the greater magnification...magnify the received target image according to the greater magnification and run the second trained model to perform the diagnosis process on the magnified received target image" .
“Therefore, Shiratani does not disclose or suggest "when the magnification change information indicates the normal magnification, output the received target image overlaid with a detection result from the first trained model" and "when the magnification change information indicates the greater magnification, magnify the received target image...and output the magnified received target image overlaid with a diagnosis result from the second trained model”.
Examiner respectfully disagrees.
Regarding Applicant’s first argument, Applicant asserts that that Karino does not disclose or suggest "when the magnification change information indicates the normal magnification...run the first trained model to perform the detection process on the received target image". However, the Examiner utilized a combination of both the disclosures of Karino and Shiratani in order to address this particular limitation of the instant application. The disclosure of Shiratani, specifically disclosing the first trained model and the first detection process being distinct from the second trained model and second (diagnosis) process was utilized by the Examiner to explain why the disclosure of the instant applicant would have been obvious over Karino in view of Shiratani according to the rationale utilized in the prior Office Action (see MPEP § 2143). Furthermore, Applicant asserts that Karino does not disclose or suggest "when the magnification change information indicates the greater magnification...magnify the received target image according to the greater magnification and run the second trained model to perform the diagnosis process on the magnified received target image". However, Karino does disclose wherein the magnification change information indicates the greater magnification, a magnification of the received target magnification, and the execution of a trained model to perform the diagnosis process on the magnified received target image (paras. 0066-0067, where the recognition unit is active when step 213’s response is “yes” to the enlarged observation mode, as a result of the identification unit 73’s mode query and magnification level observation, and the execution of the recognition unit 72). Karino does not disclose wherein the model is the second of two models, but that is taught by secondary reference Shiratani whose combination with Karino is obvious according to the rationale stated in the prior Office Action.
Regarding Applicant’s second argument, Applicant asserts that Shiratani does not disclose or suggest "when the magnification change information indicates the normal magnification, output the received target image overlaid with a detection result from the first trained model" and "when the magnification change information indicates the greater magnification, magnify the received target image...and output the magnified received target image overlaid with a diagnosis result from the second trained model”. However, Shiratani, like Karino, does disclose when the magnification change information indicates the normal magnification (paras. 0070-0072 of Shiratani disclose detection of a current magnification, thus allowing determination that the current magnification is that of a normal magnification), and outputting the received target image overlaid with a detection result from the first trained model (para. 0116, wherein the RPN’s output includes a rectangle bound polyp region (the detection overlay) within the original ground-truth masked image). Furthermore, the Examiner utilized Karino in view of Shiratani to disclose the limitation "when the magnification change information indicates the greater magnification, magnify the received target image...and output the magnified received target image overlaid with a diagnosis result from the second trained model”; specifically, Karino discloses when the magnification change information indicates the greater magnification, magnify the received target image...and output the magnified received target image overlaid with a diagnosis result from a trained model (paras. 0066-0067, where the recognition unit is active when step 213’s response is “yes” to the enlarged observation mode, as a result of the identification unit 73’s mode query and magnification level observation, and the execution of the recognition unit 72, and Fig. 6 for the overlay for diagnosis).
In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 8-11, 13-14, and 16-17 are rejected under 35 U.S.C. 103 as being obvious over Karino (WIPO PG Pub No. WO2020036121) in view of Shiratani (WIPO PG Pub No. WO 2018105062).
Regarding claim 1, Karino discloses an information processing system comprising a memory storing a trained model (para. 0067, wherein the trained model is the recognition unit 72 of fig. 3, whose architecture is that of a trained neural network model, such as the listed CNN) and a processor (para. 0103) configured to perform a detection process and a diagnosis process (paras. 0067-0072, wherein both the detection and diagnosis processes are performed by the recognition unit 72 as described in paras. 0067-0072, and whose accuracy and outcomes are disclosed), the detection process being a process of detecting a lesion from a target image through processing (paras. 0067-0072, wherein the detection of the lesion is performed through training a model to detect an image in normal operation), the target image being captured by an imaging device of an endoscope system and input to the processor (paras. 0045-0048, wherein the endoscope image was captured using the imager acquisition unit and fed to the system including image generation unit 71, which performs processing using a processor), the diagnosis process being a process of diagnosing a type of the lesion from the target image through processing based on the trained model (paras. 0068-0070, wherein the trained AI model is able to detect and discriminate between lesion types within the magnification observation mode), wherein the trained model is a trained model that has been trained, using a training image that has a greater magnification than the normal magnification, to perform the diagnosis process on the second training image (para. 0067, where the recognition unit is active when step S213’s response is “yes” to the enlarged observation mode) and the processor is further configured to: receive the target image from the imaging device of the endoscope system (paras. 0045-0047, disclosing the endoscope system and receiving the imaging device as capturing the target image);
receive magnification change information from an operation device of the endoscope system (para. 0066, the “mode switching operation unit 13b” as part of the endoscope control unit, which can change magnification modes), wherein the magnification change information indicates one of the normal magnification and the greater magnification (para. 0067, where the recognition unit is active when step S213’s response is “yes” to the enlarged observation mode, as a result of the identification unit 73’s mode query and observation); when the magnification change information indicates the greater magnification: magnify the received target image according to the greater magnification and run a trained model to perform the diagnosis process on the magnified received target image (paras. 0066-0067, the “mode switching operation unit 13b” as part of the endoscope control unit, dependent on the return from para. 0067’s “greater” magnification determination, and paras. 0068-0070, wherein the trained AI model is able to detect and discriminate between lesion types within the magnification observation mode); and output the magnified received target image overlaid with a diagnosis result from the diagnosis model (paras. 0082-0084 and Fig. 6).
Karino does not disclose distinct trained models, wherein the first trained model is a trained model that has been trained, using a first training image having a normal magnification, to perform the detection process on the first training image, the detection process being a process of detecting a lesion from a target image through processing is performed by a trained model, or when the magnification change information indicates the normal magnification: run the first trained model to perform the detection process on the received target image; and output the received target image overlaid with a detection result from the first trained model.
However, Shiratani discloses distinct trained models, wherein the first trained model is a trained model that has been trained, using a first training image having a normal magnification, to perform the detection process on the first training image, the detection process being a process of detecting a lesion from a target image through processing is performed by a trained model (paras. 0091-0097, wherein the first trained model is the Region Proposal Network, or RPN, for extracting rectangular areas containing lesions from normal magnification inputs); when the magnification change information indicates the normal magnification: run the first trained model to perform the detection process on the received target image (paras. 0070-0071 for the operation state presumption section/result, an informer of the observation magnification for the execution of the RPN, and 0091-0097, wherein the first trained model is the RPN for extracting rectangular areas containing lesions from normal magnification inputs as a detection mechanism); and output the received target image overlaid with a detection result from the first trained model (paras. 0070-0077, wherein the output comprises the detected frame with at least a portion of the polyp present).
Specifically, Shiratani discloses an image processing device and method for endoscopic observation wherein polyps or other lesions are detected through the use
of trained models. As a result, both Karino and Shiratani disclose systems of endoscopic
observation of regions of interest including (and determined by) lesions, specifically, polyps.
Thus, it would have been obvious for one having ordinary skill in the art prior to the effective
filing date of the claimed invention, to adapt the distinct, neural network-based RPN detection
model disclosed by Shiratani in the system of Karino as a simple substitution of two known
elements (Karino’s bounding exchanged for Shiratani’s RPN) with the same function in the prior
art to yield the predictable result of a more robust, boxed region of interest for easier diagnosis.
Regarding claim 17, Karino discloses a processor (para. 0103) configured to:
receive magnification change information from an operation device of the endoscope system (para. 0066, the “mode switching operation unit 13b” as part of the endoscope control unit, which can change magnification modes), determine whether a magnification of a target image is a normal magnification or a greater magnification than the normal magnification on the basis of the magnification change information (para. 0067, where the recognition unit is active when step S213’s response is “yes” to the enlarged observation mode, as a result of the identification unit 73’s mode query and observation); in response to a determination that the magnification of the target image is greater magnification than the normal magnification, diagnose a type of lesion from the magnified target information based on a trained model, wherein the trained model is trained using a trained image that has a greater magnification than normal (paras. 0066-0067, the “mode switching operation unit 13b” as part of the endoscope control unit, dependent on the return from para. 0067’s “greater” magnification determination, and paras. 0068-0070, wherein the trained AI model is able to detect and discriminate between lesion types within the magnification observation mode); and when the magnification change information indicates the greater magnification, output the received target image that is magnified according to the greater magnification overlaid with a diagnosis result from the second trained model (paras. 0082-0084 and Fig. 6).
Karino does not disclose distinct trained models, wherein, in response a determination that the magnification of the target image is normal magnification, detect a lesion from a target image based on a first trained model, wherein the first trained model is trained using a first training image having a normal magnification; or, when the magnification change information indicates the normal magnification, output the received target image overlaid with a detection result from the first trained model.
However, Shiratani discloses distinct trained models, wherein, in response a determination that the magnification of the target image is normal magnification, detect a lesion from a target image based on a first trained model, wherein the first trained model is trained using a first training image having a normal magnification (paras. 0091-0097, wherein the first trained model is the Region Proposal Network, or RPN, for extracting rectangular areas containing lesions from normal magnification inputs); and when the magnification change information indicates the normal magnification, output the received target image overlaid with a detection result from the first trained model (paras. 0070-0077, wherein the output comprises the detected frame with at least a portion of the polyp present).
Regarding claim 2, Karino and Shiratani disclose all limitations of claim 1. Karino further discloses wherein a normal image being the target image at the normal magnification is an image captured under a first condition, the first condition comprising a white light condition (para. 0025), a special image being the target image at the greater magnification is an image captured under a second condition, the second condition comprising a special light condition (para. 0025, specifically potentially capturing under different light filters to observe blood vessels), the special image being an enlarged part of a field of view of the normal image (para. 0024, in conjunction with the established special filtering disclosed above), and the second trained model is trained using the second training image captured under the second condition (para. 0075, “useful for when the recognition unit 72 has learned AI for detecting and / or discriminating a lesion or the like using an endoscopic image in which a lesion or the like has been enhanced.”).
Karino does not disclose wherein the first trained model is trained using the first training image captured under the first condition.
However, Shiratani discloses wherein the first trained model is trained using the first training image captured under the first condition (para. 0021 for the white light, normal observation mode, and para. 0106 for training the RPN using raw endoscope images with white light and normal magnification). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to include the training strategy of Shiratani within the method of Karino according to the rationale of claim 1.
Regarding claim 3, Karino and Shiratani disclose all limitations of claim 1. Karino further discloses wherein each of the target image at the normal magnification and the target image at a greater magnification is an image captured under special light condition and unstained condition (para. 0048).
Regarding claim 8, Karino and Shiratani disclose all limitations of claim 1. Karino further discloses wherein the target image is a normal image captured under white light condition (para. 0025) the first training image includes a first training normal image captured under the white light condition (para. 0025) and a first training special image captured under the special light condition (para. 0025, specifically potentially capturing under different light filters to observe blood vessels), the first trained model includes a normal-image first trained model that has been trained using the first training normal image (paras. 0068-0071) and a special-image first trained model that has been trained using the first training special image (paras. 0073-0075, wherein the special light conditions are the “emphasis” mode), and the processor detects, when the normal image is input, the lesion from the target image through processing based on the normal-image first trained model (paras. 0068-0071), and detect, when the special image is input, the lesion from the target image through processing based on the special-image first trained model (paras. 0073-0075).
Regarding claim 9, Karino and Shiratani disclose all limitations of claim 1. Shiratani further discloses wherein the target image is a normal image captured under white light condition (para. 0021, wherein the white light observation mode is one of many potential endoscope observation modes), the first training image includes a first training normal image captured under the white light condition and a first training special image captured under the special light condition (para. 0021, wherein the special light condition is narrow band light observation imaging mode), and the first trained model is trained using the first training normal image and the first training special image (paras. 0091-0097, wherein the first trained model is the Region Proposal Network, or RPN, for extracting rectangular areas containing lesions from normal magnification inputs, and para. 0133 for the database for training prior to the classification). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to employ the RPN architecture with the white-light and narrow-band endoscopic training image data of Shiratani in the system of Karino according to the rationale of claim 1.
Regarding claim 10, Karino and Shiratani disclose all limitations of claim 1. Karino further discloses wherein the target image is a normal image captured under white light condition (para. 0025) or a special image captured under a special light condition (para. 0025, specifically potentially capturing under different light filters to observe blood vessels), the second training image includes a second training normal image captured under the white light condition and a second training special image captured under the special light condition (paras. 0071-0076, wherein the different trained models can be trained and executed on any combination of white light and special image capture), the second trained model includes a normal-image second trained model that has been trained using the second training normal image and a special-image second trained model that has been trained using the second training special image (paras. 0075-0076), and the processor diagnoses, when the normal image is input, the type of the lesion from the target image through processing based on the normal-image second trained model (paras. 0075-0076), and diagnose, when the special image is input, the type of the lesion from the target image through processing based on the special-image second trained model (paras. 0075-0076).
Regarding claim 11, Karino and Shiratani disclose all limitations of claim 1. Karino further discloses wherein the target image is a normal image captured under white light condition (para. 0025) or a special image captured under a special light condition (para. 0025), the second training image includes a second training normal image captured under the white light condition (para. 0025) and a second training special image captured under a special light condition (para. 0025), and the second trained model is trained using the second training normal image and the second training special image (paras. 0073-0076, wherein the special light conditions are the “emphasis” mode, and the training data and set can consist of any combination of normal and emphasized images).
Regarding claim 13, Karino and Shiratani disclose all limitations of claim 1. Karino further discloses an imaging device (para. 0045-0047 and fig. 1 element endoscope system 10), and an operation device (para. 0045-0047, specifically the console 19 interfaced with the processor 16 as an input device and processor for operating the endoscope).
Regarding claim 14, Karino discloses an information processing method comprising inputting a target image captured by an imaging device of an endoscope system (paras. 0058-0059 and 0065-0066, wherein the endoscope image was captured using the imager acquisition unit and fed to the system including image generation unit 71, which performs processing using a processor) and magnification change information acquired by an operation device of the endoscope system that changes magnification of the target image, wherein the magnification change information indicates one of the normal magnification and the greater magnification (para. 0053-0055, “imaging optical system 30b” uses “zoom operation unit 30a” to enlarge or reduce subject image size, feedback gained through identification unit 73, and paras. 0067-0070, wherein para. 0067 discloses identification unit 73, which identifies magnification change information); performing a detection process when the magnification change information indicates the normal magnification, the detection process being a process of detecting a lesion from the target image through processing based on a trained model that has been trained, using a training image having the normal magnification, to detect the lesion from the training image (paras. 0067-0072, wherein both the detection and diagnosis processes are performed by the recognition unit 72 as described in paras. 0067-0072, and whose accuracy and outcomes are disclosed in paras 0068-0070; although this process is configured to both detect and diagnose, the separation of models into two distinct steps is described in the rationale of claim 1 for combination of the disclosures of Karino and Shiratani); and magnifying the received target image according to the greater magnification when the magnification change information indicates the greater magnification (para. 0066, the “mode switching operation unit 13b” as part of the endoscope control unit, which can change magnification modes); and performing a diagnosis process when the magnification change information indicates the greater magnification than the normal magnification, the diagnosis process being a process of diagnosing a type of the lesion from the target image through processing based on a second trained model that has been trained, using a second training image having the greater magnification, to diagnose a type of the lesion from the second training image (para. 0067, where the recognition unit is active when step S213’s response is “yes” to the enlarged observation mode, as a result of the identification unit 73’s mode query and observation).
Karino does not disclose distinct trained models, wherein the detection process of detecting a lesion from the target image through processing is based on a first trained model that has been trained, using a first training image having the normal magnification, to detect the lesion from the first training image.
However, Shiratani discloses distinct trained models, wherein the detection process of detecting a lesion from the target image through processing is based on a first trained model that has been trained, using a first training image having the normal magnification, to detect the lesion from the first training image (paras. 0091-0097, wherein the first trained model is the Region Proposal Network, or RPN, for extracting rectangular areas containing lesions from normal magnification inputs);.
Thus, it would have been obvious to the ordinarily skilled artisan to have combined the disclosures of Karino and Shiratani according to the rationale of claim 1.
Regarding claim 16, Karino and Shiratani discloses all limitations of claim 1. Karino further discloses wherein the processor determines whether the magnification of the target image is the normal magnification or the greater magnification on the basis of the magnification change information (para. 0047 discloses the actual enlargement change, and para. 0067 for the determination of magnification).
Claims 5, 15, and 18-20 are rejected under 35 U.S.C. 103 as being obvious over Karino in view of Shiratani and in further view of Sharma (“Magnification Endoscopy”, The American Society for Gastrointestinal Endoscopy, Vol. 61 No. 3, 2005).
Regarding claim 5, Karino and Shiratani disclose all limitations of claim 3. Karino and Shiratani do not disclose wherein the lesion is a polyp in a large-intestinal mucosa, and the type of the lesion is a type of the polyp classified by a pit pattern of micro vessels in the mucosa of the polyp.
However, Sharma discloses wherein the lesion is a polyp in a large-intestinal mucosa, and the type of the lesion is a type of the polyp classified by a pit pattern of micro vessels in the mucosa of the polyp (pgs. 437 para. 4– pg. 438 para. 1, wherein Sharma discloses that the lesion of interest is a colonic polyp classified using Kudo et al.’s pit-pattern classification for micro vessels in the mucosa).
Specifically, Sharma is a state-of-the-art review of magnification endoscopy, including techniques, practices, and applications of magnification endoscopy specifically directed to intestinal polyp detection and diagnosis.
Therefore, it would have been obvious for one having ordinary skill in the art prior to the effective filing date of the claimed invention to utilize the disclosed lesion information of Sharma within the system and method of Karino and Shiratani as a teaching or suggestion within the prior art which would have led an ordinarily skilled artisan to modify the method of Karino as modified by Shiratani to polyps within the large-intestinal mucosa.
Regarding claim 15, Karino and Shiratani disclose all limitations of claim 1.
Karino and Shiratani do not disclose wherein the greater magnification is 50 times or more and less than 100 times the normal magnification in terms of observation magnification on a display screen.
However, Sharma discloses wherein the greater magnification is 50 times or more and less than 100 times the normal magnification in terms of observation magnification on a display screen (table 2, wherein the stainless observations utilized magnifications of 80x, between 50x and 100x normal magnification, in order to observe and diagnose gastric mucosal lesions). Thus, it would have been obvious to the ordinarily skilled artisan to have utilized the particular zoom level disclosed by Sharma within the system and method of Karino as modified by Shiratani according to the rationale of claim 5.
Regarding claim 18, Karino and Shiratani disclose all limitations of claim 16.
Karino and Shiratani do not disclose wherein the greater magnification is 50 times or more the normal magnification in terms of observation magnification on a display screen.
However, Sharma discloses wherein the greater magnification is 50 times or more the normal magnification in terms of observation magnification on a display screen (table 2, wherein the stainless observations utilized magnifications of 80x, greater than 50x normal magnification, in order to observe and diagnose gastric mucosal lesions). Thus, it would have been obvious to the ordinarily skilled artisan to have utilized the particular zoom level disclosed by Sharma within the system and method of Karino as modified by Shiratani according to the rationale of claim 5.
Regarding claim 19, Karino and Shiratani disclose all limitations of claim 17.
Karino and Shiratani do not disclose wherein the greater magnification is less than 100 times the normal magnification in terms of observation magnification on a display screen.
However, Sharma discloses wherein the greater magnification is less than 100 times the normal magnification in terms of observation magnification on a display screen (table 2, wherein the stainless observations utilized magnifications of 80x, less than 100x normal magnification, in order to observe and diagnose gastric mucosal lesions). Thus, it would have been obvious to the ordinarily skilled artisan to have utilized the particular zoom level disclosed by Sharma within the system and method of Karino as modified by Shiratani according to the rationale of claim 5.
Regarding claim 20, Karino and Shiratani and Sharma disclose all limitations of claim 18. Karino further discloses wherein the processor determines whether the magnification of the target image is the normal magnification or the greater magnification on the basis of the magnification change information (para. 0047 discloses the actual enlargement change, and para. 0067 for the determination of magnification).
Claim 4 is rejected under 35 U.S.C. 103 as being obvious over Karino in view of Shiratani and Sharma, and in further view of Inoue et al. (“Magnification endoscopy in esophageal squamous cell carcinoma: a review of the intrapapillary capillary loop classification”, Ann Gastroenterol. 2015 Jan-Mar;28(1):41-48, hereafter referred to as Inoue).
Regarding claim 4, Karino and Shiratani and Sharma disclose all limitations of claim 3.
Karino and Shiratani and Sharma do not disclose wherein a spectrum of the special light has a peak belonging to a bandwidth of 390 nm to 445 nm and a peak belonging to a bandwidth of 530 nm to 550 nm.
However, Inoue discloses wherein a spectrum of the special light has a peak belonging to a bandwidth of 390 nm to 445 nm and a peak belonging to a bandwidth of 530 nm to 550 nm (pg. 1, Introduction section, paras. 2-3).
Specifically, Inoue discloses a review of the use of magnification in esophageal squamous cell carcinoma, specifically describing the development of endoscopic descriptions of intrapapillary capillary loops. As a result, both Karino and Shiratani and Sharma and Inoue disclose endoscopy-based methods and systems for detecting lesions and growths within the body.
Thus, it would have been obvious for one having ordinary skill in the art prior to the effective filing date of the claimed invention to utilize the wavelength ranges disclosed by Inoue within the system and method of Karino and Shiratani as the application of a teaching in the prior art regarding peak wavelengths and wavelength ranges for narrow band imaging enhancement in endoscopy for diagnosis to modify the endoscope system of Karino and Shiratani.
Claims 6 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Karino in view of Shiratani and in further view of Liang et al. (WIPO PG Pub No. WO 2017027475, hereafter referred to as Liang) and Tada et al. (WIPO PG Pub No. WO 2018225448, hereafter referred to as Tada)
Regarding claim 6, Karino and Shiratani disclose all limitations of claim 1. Karino further discloses wherein the processor detects the position of the lesion from the target image in the detection process, and wherein the processor diagnoses the type of the lesion from the target image in the diagnosis process.
Karino and Shiratani do not explicitly disclose wherein the first trained model is trained, using a first annotation indicating a position of the lesion in the first training image, to detect the position of the lesion indicated by the first annotation from the first training image, the second trained model is trained, using a second annotation indicating a type of the lesion in the second training image, to detect the type of the lesion indicated by the second annotation from the second training image.
However, Liang discloses wherein the first trained model is trained, using a first annotation indicating a position of the lesion in the first training image, to detect the position of the lesion indicated by the first annotation from the first training image (para. 102-103, figs. 12, 16, and 17, wherein the paragraphs describe the process for detection of polyps from endoscopic images, training the classifier using labeled patches with both present and absent polyps, and testing on test images post-training). Specifically, Liang discloses a method of colonoscopy video quality assurance and polyp detection. Thus, Karino and Shiratani and Liang all disclose methods of identifying intestinal polyps for diagnosis of diseases. Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to utilize the specific training workflow of Liang within the system of Karino as modified by Shiratani as the use of the known training method of Liang to improve the known method of Karino modified by Shiratani by creating a training paradigm reflective of realistic conditions.
Karino and Shiratani and Liang do not disclose wherein the second trained model is trained, using a second annotation indicating a type of the lesion in the second training image, to detect the type of the lesion indicated by the second annotation from the second training image.
However, Tada discloses wherein the second trained model is trained, using a second annotation indicating a type of the lesion in the second training image, to detect the type of the lesion indicated by the second annotation from the second training image (para. 0154 for specific mention of annotations, 0156 for trained CNN based diagnosis, 0197 for diagnosis using trained CNN based on input image and diagnosis label). Specifically, Tada discloses a method of disease diagnosis support within the digestive system, centered within the stomach and intestines. Thus, Karino and Shiratani and Liang and Tada all disclose methods of identifying intestinal polyps for diagnosis of diseases. Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to utilize the specific training workflow of Tada within the system of Karino as modified by Shiratani and further modified by Liang as the use of the known training method of Tada to improve the known method of Karino modified by Shiratani and Liang by creating a training paradigm reflective of realistic conditions.
Regarding claim 7, Karino and Shiratani and Liang and Tada disclose all limitations of claim 6. Karino further discloses wherein the processor performs processing of acquiring a diagnosis result together with reliability of the diagnosis result in the diagnosis process and displaying the diagnosis result and the reliability on a display device (paras. 0046-0047, wherein para. 0047 describes the specific accuracy/reliability threshold for sufficiently robust diagnosis and recognition results as outputs of the recognition unit 72 and para. 0046 discloses feeding the results of the operations of recognition unit 72 into the display control unit 66 to be displayed on display device 18).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Karino in view of Shiratani, Liang, and Tada, and in further view of Standley et al. (“Which Tasks Should Be Learned Together in Multi-task Learning?”, arXiv, International conference on machine learning (pp. 9120-9132), PMLR., hereafter referred to as Standley).
Regarding claim 12, Karino and Shiratani and Liang and Tada disclose all limitations of claim 6. Karino further discloses wherein:
the memory stores a trained model, wherein the trained model is a trained model that has been trained, using a training image having the normal magnification, to classify whether or not the training image is an image containing the lesion (paras. 0069 and 0081, wherein the presence or absence of the lesion within the frame at normal magnification is detected using the identification unit, a trained model, and is output on the display and serves as the trigger for the diagnosis process’s execution),
and the processor classifies, when the magnification change information indicates the normal magnification, whether or not the target image is the image containing the lesion through processing based on the trained model, and perform, when determining that the target image is the image containing the lesion, the detection process on the target image (paras. 0069 and 0081, wherein the presence or absence of the lesion within the frame at normal magnification is detected using the identification unit, a trained model, and is output on the display and serves as the trigger for the diagnosis process’s execution).
Karino and Shiratani and Liang and Tada do not disclose wherein the trained model is a separate, third model trained on a third input image for the detection of the lesion.
However, Standley discloses the utilization of separate, small trained models in order to determine task grouping for computational efficiency (Abstract and page 2 para. 3, wherein the decision of using multiple different models or a jointly trained model is based on associated tasks within a group; wherein the task set is the detection and diagnosis pipeline of Karino and Shiratani and Liang and Tada; wherein the first model and third model of the instant application both detect lesions based on observed image features and can thus be grouped together, and wherein a pair of small neural network models would make detection of lesions within a normally magnified image fast and efficient using the lesion annotation points and an object detection framework, respectively). Specifically, Standley’s teachings are directed towards the improvement of computer vision multitasking, aiming to increase workflow efficiency by deciding whether to group small neural network models to cover the entirety of the task set.1 Therefore, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the disclosure of Standley with respect to small trained models being grouped together to break down and execute multitasking directives in parallel as a teaching in the prior art which would have led to the predictable improvement of a modular, two-pronged (first and third models in the instant application, while splitting the multi-feature multi-tasking model disclosed by Karino in view of Shiratani and Liang and Tada) network workflow for efficient lesion detection using multiple features.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROHAN TEJAS MUKUNDHAN whose telephone number is (571)272-2368. The examiner can normally be reached Monday - Friday 9AM - 6PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at 5712723838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROHAN TEJAS MUKUNDHAN/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
1 Standley is cited largely for the proposition that there is an efficiency-based decision to either use multiple neural networks for multiple somewhat related decisions or to jointly train a single network for those decisions. Where this choice exists, it would have been obvious to do that analysis and decide to use multiple neural networks based on the specific implementation..
Prosecution Insights