DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed on 03/18/2026 has been entered. Claims 1-9 and 11-14 have been amended. New Claim 16 has been added. Claims 1-16 remain pending.
The previously raised objection for Claim 14 is withdrawn because the issue has been properly corrected.
The previously raised rejections under 35 U.S.C. 112(b) for Claims 1-15 are withdrawn because the issues have been properly corrected.
Response to Arguments
On Pages 12-14 of Remarks, Applicant argues that, regarding the previously issued 35 USC 101 rejection for Claim 1, the claim is directed to a concrete machine-implemented control process (Para 3, Page 13 of Remarks) that improves the reliability of the determination processing for a same evaluation target, therefore integrating alleged abstract idea into practical application (Step 2A, Prong 2), and defines a specific technical arrangement (Para 4, Page 13 of Remarks) so as to amount to significantly more (Step 2B). Examiner respectfully disagrees. First, the steps of image acquisition and image processing are recited in a very high level of generality. For example, image acquisition for lesions with light of different wavelength ranges is commonly used in clinical practice (as an example, Xu disclosed “all the images were collected during the clinical nasopharyngoscopy performed by the endoscopists” in Page 1006, Column 2, Para 1), and the claims do not provide meaningful limits so that the image acquisition step can be regarded as insignificant extra-solution activity. As another example, the learning models are merely recited with the most generic terms, such as “learning model” and “learning data”, which again provides no meaningful limit or inventive concept. Second, the determination of the claimed relationship (Claims 1, 14 and 15) is obviously a mathematical concept by which different prediction models are compared and selected in many fields of applications, and using such mathematical concept for the problem of this application does not provide significant inventive concept. In view of the above discussion, Examiner believes that the claims are directed to an abstract idea without significantly more.
On Pages 14-15 of Remarks, Applicant argues that reference Oosake does not disclose that “the first recognizer and the second recognizer respectively process two images of the same target”, but rather merely teaches “confidence-based switching between recognizers with respect to the same captured image”. Examiner respectfully disagrees. In Claim 1, especially before the amendment, the first light source image is input to the first learning model, but the limitation of the second light source image being input to the second learning model is NOT explicitly recited. Furthermore, the inputting of the first light source image to the first learning model results in the calculating of a determination certainty factor from a determination result, but not explicitly the calculating of the determination result itself. The limitation of “decide whether …” (Claim 1, Lines 18-21) is based on “the determination accuracy factor” (from the first learning model) and “the certainty-factor determination value stored in the first memory”, which again indicates that the second learning model or its determination result could never be used in processing a data; in other words, if based on the determination accuracy factor and the certainty-factory determination value, a decision is made to use determination result of the first learning model, the second learning model, its determination result and the second light source image are not used in processing. In view of this discussion, the Application is the same as reference Oosake by claiming “confidence-based switching between two recognizers”.
On Pages 15-16 of Remarks, Applicant argues that, regarding reference Xu, the two models (DCNN-1 and DCNN-2) “are distinguished by different image modalities used for training and testing”, i.e. WLI images and NBI images. Examiner respectfully disagrees. Xu discloses the images used for model training and testing in Page 1001, Column 1, Para 1, “A total of 4,783 nasopharyngoscopy images of the lesion were collected, including 2,898 WLI images and 1,885 NBI images. Each patient contains at least one WLI image and one NBI image.”, indicating that the images used for training and testing the two models are from a same image modality, and are only different in their wavelength range (white light or narrow band). In view of this, the disclosed two models in Xu properly modify the two recognizers of Oosake to match the two models of Application.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1 - 16 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
With regard to Claims 1-10:
Step 1: the claims are drawn to a system/apparatus, one of the four statutory categories.
Step 2A, Prong One:
The claims recite the limitations of “calculate a determination certainty factor” (Claim 1, Lines 16-17), “decide whether to use … or to use …” (Claim 1, Lines 18-21), “when the determination certainty factor is greater than or equal to …” and “… is smaller than …” (Claim 2, Lines 2-6), “when … is greater than or equal to …” and “… is smaller than …” (Claim 3, Lines 6-9), “when … is smaller than …” (Claim 4, Lines 2-3), “select whether to output … or to output …” (Claim 5, Lines 5-8), “calculates the determination certainty factor …” (Claim 8, Lines 4-5), and “calculates the determination certainty factor …” (Claim 9, Lines 4-5), which are, under their broadest reasonable interpretation, limitations that cover either performance of the limitation in the mind, or mathematical calculations. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind or mathematical calculations, then it falls within the “Mental Processes” or “Mathematical Concepts” grouping of abstract ideas. Accordingly, the claims recite an abstract idea.
Step 2A, Prong Two:
This judicial exception is not integrated into a practical application. In particular, the claims recite the additional elements: a first learning model and a second learning model learned by using a first learning data and a second learning data (Claim 1), a first and a second light source (Claim 1), a first processor (Claim 1), a first memory (Claim 1), a third memory (Claim 7), executing learning models (Claim 7), one or two second processors (Claim 7), and an endoscope (Claim 10), which are recited at a high-level of generality such that they amount no more than mere instructions to apply the exception as generic computer components, light source, endoscope, and/or learning models in performing generic function of acquiring and processing endoscopic images. The other additional elements, including photographing a target (Claim 1), acquiring images (Claims 1, 5 and 16), inputting images to learning model (Claims 1 and 4), and outputting determination outputs (Claims 2-3 and 8-9), are insignificant extra-solution activities. Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
Step 2B:
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements either apply generic computer components, light source, endoscope, and/or learning model in performing generic function of acquiring and processing endoscopic images, or are insignificant extra-solution activities, which cannot provide an inventive concept.
For the reasons set forth above, Claims 1-10 and 16 are not patent eligible.
With regard to Claims 11-15:
Step 1: the claims are drawn to a method/process, one of the four statutory categories.
Step 2A, Prong One:
The claims recite the limitations of “calculate a determination certainty factor” (Claim 11, Line 15), “decide whether to use … or to use …” (Claim 11, Lines 17-21), “when the determination certainty factor is greater than or equal to …” and “… is smaller than …” (Claim 12, Lines 4-8), “when … is greater than or equal to …” and “… is smaller than …” (Claim 13, Lines 8-11), “calculate a plurality of first determination certainty factors” (Claim 14, Lines 16-18), “calculate a plurality of first determination accuracies and a plurality of second determination accuracies …” (Claim 14, Lines 19-22), “calculate the certainty-factor determination value …” (Claim 14, Lines 25-30), and “linearly approximates …, and calculates …” (Claim 15, Lines 2-5), which are, under their broadest reasonable interpretation, limitations that cover either performance of the limitation in the mind, or mathematical calculations. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind or mathematical calculations, then it falls within the “Mental Processes” or “Mathematical Concepts” grouping of abstract ideas. Accordingly, the claims recite an abstract idea.
Step 2A, Prong Two:
This judicial exception is not integrated into a practical application. In particular, the claims recite the additional elements: a first learning model and a second learning model learned by using a first learning data and a second learning data (Claim 11), a first and a second light source (Claim 11), a first processor (Claim 11), a first memory (Claim 11), a third processor (Claim 14), a second memory (Claim 14), obtaining determination results from the evaluation data sets by the learning models (Claim 14), which are recited at a high-level of generality such that they amount no more than mere instructions to apply the exception as generic computer components, light source, endoscope, and/or learning model in performing generic function of acquiring and processing endoscopic images. The other additional elements, including photographing a target (Claim 11), acquiring images (Claims 11), inputting images to learning model (Claims 11 and 14), and outputting determination outputs (Claims 12-13), and storing the calculated certainty-factor determination value (Claim 14), are insignificant extra-solution activities. Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
Step 2B:
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements either apply generic computer components, light source, endoscope, and/or machining learning algorithm in performing generic function of acquiring and processing endoscopic images, or are insignificant extra-solution activities, which cannot provide an inventive concept.
For the reasons set forth above, Claims 11-15 are not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-8, 10-13 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Oosake (US 20200193236 A1; hereafter Oosake), in view of Xu et al (Laryngoscope. 2022 May;132(5):999-1007; hereafter Xu).
With regard to Claim 1, Oosake discloses an image processing apparatus (Oosake, Para 0048; “a medical image processing device 14”) comprising:
a first learning model (Oosake, Para 0082; “the first recognizer 41 includes a convolutional neural network (CNN)”);
a second learning model (Oosake, Para 0089; “ the second recognizer 42 has the same configuration as the first recognizer 41, and performs a recognition process of the captured image to obtain a second recognition result of a category classification or the like”);
a first processor (Oosake, Para 0075; “A control unit 44 corresponding to the processor (not shown) … generally controls a medical image acquisition unit 40, a first recognizer 41, a second recognizer 42, a recognition confidence level determination device 43 …”); and
a first memory (Oosake, Para 0075; “a recognition confidence level determination device 43”) storing a certainty-factor determination value (Oosake, Para 0094; “The result of determining whether the confidence level is “high” or “low” can be decided by whether the difference between the highest score and the second highest score is equal to or greater than a first reference value …”),
wherein the first processor is configured to acquire a first light source image (Oosake, Para 0065; “… a normal light image obtained by the above-described white light.”) obtained by photographing an evaluation target illuminated by the first light source (Oosake, Para 0057; “… light in various wavelength ranges is selected according to the observation purpose, such as white light …”) and a second light source image (Oosake, Para 0064; “in a case where the video 38 and the static image 39 are images obtained by light (special light) in the above-described specific wavelength range, the video 38 and the static image 39 are special light images.”) obtained by photographing the evaluation target illuminated by the second light source (Oosake, Para 0057; “… light in various wavelength ranges is selected according to the observation purpose, such as … or light in one or a plurality of specific wavelength ranges, or a combination thereof. The specific wavelength range is a range narrower than the white-light wavelength range”),
input the first light source image to the first learning model (Oosake, Para 0082; “The first recognizer 41 is a portion that performs recognition of the image (video 38 and static image 39) captured during the observation of the body cavity …”) and calculate a determination certainty factor (Oosake, Para 0094; “the difference between the highest score and the second highest score”) from a determination result (Oosake, Para 0093; “… the score (confidence level) of “neoplastic” is 80%, the score of “non-neoplastic” is 15%, and the score of “others” is 5% …”. Here the different scores correspond to the “determination result” of Application) of the first learning model, and
decide whether to use the determination result of the first learning model or to use a determination result of the second learning model (Oosake, Para 0096; “in a case where the recognition confidence level determination device 43 determines that the confidence level for the recognition result by the first recognizer 41 is “low”, the control unit 44 causes the second recognizer 42 to perform the recognition process of the medical image of which the confidence level is determined to be “low”.”) based on the determination certainty factor and the certainty-factor determination value stored in the first memory (Oosake, Para 0094; “The result of determining whether the confidence level is “high” or “low” can be decided by whether the difference between the highest score and the second highest score is equal to or greater than a first reference value …”. Here “the difference …” corresponds to the “determination certainty factor”, and “a first reference value” corresponds to the “certainty-factor determination value” of Application, respectively.).
Oosake does not clearly and explicitly disclose using two sets of learning data obtained by photographing a target illuminated by two different light sources to train the two learning models respectively.
Xu in the same field of endeavor discloses using two sets of learning data obtained by photographing a target illuminated by two different light sources (Xu, Page 1001, Column 1, Para 1; 2,898 WLI images and 1,881 NBI images) to train the two learning models respectively (Xu, Page 1002, Column 2, Para 2; “… we trained and tested two traditional DCNNs as shown in Figure 2: DCNN-1 - only using WLI images for training and testing, and DCNN-2 - only using NBI images for training and testing.”. Here DCNN-1 and DCNN-2 correspond to the two learning models). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Oosake, as suggested by Xu, in order to train the two learning models using white-light images and narrow-band light images respectively. One of ordinary skill in the art would have been motivated to make the modification for the benefit of improved diagnostic capability by utilizing white-light endoscopy’s high repeatability (Xu, Page 1001, Column 1, Para 2; “the intraobserver and interobserver agreement of malignancies is strong in WLI endoscopy”) and narrow-band light endoscopy’s high differential ability between benign and malignant lesions (Xu, Page 1000, Column 2, Para 1; “NBI endoscopy can significantly enhance the ability of differential diagnosis between NPC and nasopharyngeal benign hyperplasia with specificity up to 91.3%”).
With regard to Claim 2, Oosake and Xu disclose the image processing apparatus according to Claim 1.
Oosake further discloses wherein the first processor outputs the determination result of the first learning model (Oosake, Para 0092; “the recognition confidence level determination device 43 inputs the recognition result (in this example, three scores) of the first recognizer 41, and in a case where the difference between the highest score among the three scores and the other scores is large, the recognition confidence level determination device 43 classifies the image into a category having the highest score, and determines that the confidence level for the category classification is “high”.”)(Oosake, Para 0102; “… a control unit that causes the display unit 16 to display required information other than images, such as at least one of the first recognition result by the first recognizer 41 or the second recognition result by the second recognizer 42.”) when the determination certainty factor is greater than or equal to the certainty-factor determination value (Oosake, Para 0094; “The result of determining whether the confidence level is “high” or “low” can be decided by whether the difference between the highest score and the second highest score is equal to or greater than a first reference value”), and outputs the determination result of the second learning model when the determination certainty factor is smaller than the certainty-factor determination value (Oosake, Para 0096; “… in a case where the recognition confidence level determination device 43 determines that the confidence level for the recognition result by the first recognizer 41 is “low”, the control unit 44 causes the second recognizer 42 to perform the recognition process …”).
With regard to Claim 3, Oosake and Xu disclose the image processing apparatus according to Claim 1.
Oosake further discloses wherein the certainty-factor determination value stored in the first memory comprises a first certainty-factor determination value (the reference value used at the time of determining the confidence level for the recognition result obtained by the first recognizer 41) and a second certainty-factor determination value (the reference value used at the time of determining the confidence level for the recognition result obtained by the second recognizer 42) smaller than the first certainty-factor determination value (Oosake, Para 0095; “… the reference value used at the time of determining the confidence level for the recognition result obtained by the second recognizer 42 may be smaller than the reference value used at the time of determining the confidence level for the recognition result obtained by the first recognizer 41.”), and
wherein the first processor outputs the determination result of the first learning model when the determination certainty factor is greater than or equal to the first certainty-factor determination value (Oosake, Para 0092; “the recognition confidence level determination device 43 inputs the recognition result (in this example, three scores) of the first recognizer 41, and in a case where the difference between the highest score among the three scores and the other scores is large, the recognition confidence level determination device 43 classifies the image into a category having the highest score, and determines that the confidence level for the category classification is “high”.”), and outputs the determination result of the second learning model when the determination certainty factor is smaller than the second certainty-factor determination value (Oosake, Para 0096; “… in a case where the recognition confidence level determination device 43 determines that the confidence level for the recognition result by the first recognizer 41 is “low”, the control unit 44 causes the second recognizer 42 to perform the recognition process …”. Note that in Oosake, in a case where the confidence level of the result by the first recognizer 41 is lower than the second reference value, then the level is lower than the first reference value, and as a result, the second determination unit would be executed and its determination result be outputted, which is the same as the Application).
With regard to Claim 4, Oosake and Xu disclose the image processing apparatus according to Claim 1.
Oosake further discloses wherein, when the determination certainty factor is smaller than the certainty-factor determination value, the first processor causes the second learning model to output the determination result (Oosake, Para 0096; “… in a case where the recognition confidence level determination device 43 determines that the confidence level for the recognition result by the first recognizer 41 is “low”, the control unit 44 causes the second recognizer 42 to perform the recognition process …”).
Oosake and Xu do not clearly and explicitly disclose acquiring the second light source image, and causing the second learning model to receive the second light source image.
Xu further discloses acquiring the second light source image, and causing the second learning model to receive the second light source image (Xu, Page 1002, Column 2, Para 2; “… we trained and tested … DCNN-2 - only using NBI images for training and testing.”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Oosake and Xu, as further suggested by Xu, in order to acquire the second light source image and cause the second learning model to analyze the second light source image. One of ordinary skill in the art would have been motivated to make the modification for the benefit of improved diagnostic capability by analyzing images containing more disease-related characteristics (Xu, Page 1004, Column 1, Para 2; “… the NBI mode can better observe such slight microvessel changes”) and training the learning model by images of the same type.
With regard to Claim 6, Oosake and Xu disclose the image processing apparatus according to Claim 1.
Oosake further discloses wherein the determination result output by each of the first learning model and the second learning model indicates a presence or an absence of a lesion, a classification of the lesion, a degree of severity of the lesion, or remission or non-remission of an inflammatory disease (Oosake, Para 0022; “each of the first recognizer and the second recognizer may detect a position of a lesion candidate from the medical image”; Para 0023; “… classify the medical image into any category of a plurality of categories relating to a lesion”).
With regard to Claim 7, Oosake and Xu disclose the image processing apparatus according to Claim 1.
Oosake further discloses comprising:
a third memory storing the first learning model and the second learning model (Oosake, Para 0073; “a hardware structure which executes various controls of the medical image processing device 14”. The diagram of the medical image processing device 14 in Fig. 2 shows including of the two recognizers); and
one or two second processors (Oosake, Para 0075; “… A control unit 44 corresponding to the processor (not shown) of the medical image processing device 14 …”) configured to:
read the first learning model from the third memory, input the first light source image to the first learning model, and cause the first learning model to output the determination result (Oosake, Para 0097; “… the control unit 44 causes the first recognizer 41 to perform the image recognition process …”. Fig. 7 shows that steps of S10, S12 and S14 performs the claimed steps); or read the second learning model from the third memory, input the second light source image to the second learning model, and cause the second learning model to output the determination result.
With regard to Claim 8, Oosake and Xu disclose the image processing apparatus according to Claim 1.
Oosake further discloses wherein the first processor acquires a plurality of determination results (Oosake, Para 0153; “… the feature quantity of the pixels in the local region is calculated for each local region of the medical image, a lesion candidate having a specific color, shape, or the like is extracted …”) with respect to a plurality of regions (Oosake, Para 0153; “… the medical image is divided into a plurality of rectangular regions, and each divided rectangular region is set as a local region.”) of the first light source image from the first learning model, and calculates the determination certainty factor based on the plurality of determination results (Oosake, Para 0153; “… the category classification of the medical image is performed by collating the image of the extracted lesion candidate (feature quantity of the image) with a reference lesion image (feature quantity of the lesion image) for each of category classifications prepared in advance.”. This disclosure indicates that the classification of the whole medical image is based on the extracted lesion candidates from the plurality of local regions.).
With regard to Claim 10, Oosake and Xu disclose the image processing apparatus according to Claim 1.
Oosake further discloses wherein illumination light from one of the first light source and the second light source is special light in a narrow band (Oosake, Para 0057; “… light in various wavelength ranges is selected according to the observation purpose, such as … or light in one or a plurality of specific wavelength ranges, or a combination thereof. The specific wavelength range is a range narrower than the white-light wavelength range”), and illumination light from the other light source is white light (Oosake, Para 0057; “… light in various wavelength ranges is selected according to the observation purpose, such as white light …”), and
wherein each of the first light source image and the second light source image is an endoscopic image captured by an endoscope (Oosake, Para 0078; “The medical image acquisition unit 40 acquires the medical image (in this example, video 38 captured by the endoscope 10) …”).
With regard to Claim 11, Oosake discloses an actuation method (Oosake, Para 0141; “… a medical image processing method according to an embodiment of the invention …”) of an image processing apparatus comprising a first learning model (Oosake, Para 0075; “a first recognizer 41”), a second learning model (Oosake, Para 0075; “a second recognizer 42”), a first processor (Oosake, Para 0075; “A control unit 44 corresponding to the processor (not shown) … generally controls a medical image acquisition unit 40, a first recognizer 41, a second recognizer 42, a recognition confidence level determination device 43 …”), and a first memory (Oosake, Para 0075; “a recognition confidence level determination device 43”) storing a certainty-factor determination value (Oosake, Para 0094; “The result of determining whether the confidence level is “high” or “low” can be decided by whether the difference between the highest score and the second highest score is equal to or greater than a first reference value …”), the actuation method of the image processing apparatus comprising:
a step for causing the first processor to acquire a first light source image (Oosake, Para 0065; “… a normal light image obtained by the above-described white light.”) obtained by photographing an evaluation target illuminated by the first light source (Oosake, Para 0057; “… light in various wavelength ranges is selected according to the observation purpose, such as white light …”) and a second light source image (Oosake, Para 0064; “in a case where the video 38 and the static image 39 are images obtained by light (special light) in the above-described specific wavelength range, the video 38 and the static image 39 are special light images.”) obtained by photographing the evaluation target illuminated by the second light source (Oosake, Para 0057; “… light in various wavelength ranges is selected according to the observation purpose, such as … or light in one or a plurality of specific wavelength ranges, or a combination thereof. The specific wavelength range is a range narrower than the white-light wavelength range”);
a step for causing the first processor to input the first light source image to the first learning model (Oosake, Para 0082; “The first recognizer 41 is a portion that performs recognition of the image (video 38 and static image 39) captured during the observation of the body cavity …”) and calculate a determination certainty factor (Oosake, Para 0094; “the difference between the highest score and the second highest score”) from a determination result (Oosake, Para 0093; “… the score (confidence level) of “neoplastic” is 80%, the score of “non-neoplastic” is 15%, and the score of “others” is 5% …”. Here the different scores correspond to the “determination result” of Application) of the first learning model; and
a step for causing the first processor to decide whether to use the determination result of the first learning model or to use a determination result of the second learning model (Oosake, Para 0096; “in a case where the recognition confidence level determination device 43 determines that the confidence level for the recognition result by the first recognizer 41 is “low”, the control unit 44 causes the second recognizer 42 to perform the recognition process of the medical image of which the confidence level is determined to be “low”.”) based on the determination certainty factor and the certainty-factor determination value stored in the first memory (Oosake, Para 0094; “The result of determining whether the confidence level is “high” or “low” can be decided by whether the difference between the highest score and the second highest score is equal to or greater than a first reference value …”. Here “the difference …” corresponds to the “determination certainty factor”, and “a first reference value” corresponds to the “certainty-factor determination value” of Application, respectively.).
Oosake does not clearly and explicitly disclose using two sets of learning data obtained by photographing a target illuminated by two different light sources to train the two learning models respectively.
Xu in the same field of endeavor discloses using two sets of learning data obtained by photographing a target illuminated by two different light sources (Xu, Page 1001, Column 1, Para 1; 2,898 WLI images and 1,881 NBI images) to train the two learning models respectively (Xu, Page 1002, Column 2, Para 2; “… we trained and tested two traditional DCNNs as shown in Figure 2: DCNN-1 - only using WLI images for training and testing, and DCNN-2 - only using NBI images for training and testing.”. Here DCNN-1 and DCNN-2 correspond to the two learning models). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Oosake, as suggested by Xu, in order to train the two learning models using white-light images and narrow-band light images respectively. One of ordinary skill in the art would have been motivated to make the modification for the benefit of improved diagnostic capability by utilizing white-light endoscopy’s high repeatability (Xu, Page 1001, Column 1, Para 2; “the intraobserver and interobserver agreement of malignancies is strong in WLI endoscopy”) and narrow-band light endoscopy’s high differential ability between benign and malignant lesions (Xu, Page 1000, Column 2, Para 1; “NBI endoscopy can significantly enhance the ability of differential diagnosis between NPC and nasopharyngeal benign hyperplasia with specificity up to 91.3%”).
With regard to Claim 12, Oosake and Xu disclose the actuation method of the image processing apparatus according to Claim 11. Oosake further discloses comprising:
a step for causing the first processor to output the determination result of the first learning model (Oosake, Para 0092; “the recognition confidence level determination device 43 inputs the recognition result (in this example, three scores) of the first recognizer 41, and in a case where the difference between the highest score among the three scores and the other scores is large, the recognition confidence level determination device 43 classifies the image into a category having the highest score, and determines that the confidence level for the category classification is “high”.”) (Oosake, Para 0102; “… a control unit that causes the display unit 16 to display required information other than images, such as at least one of the first recognition result by the first recognizer 41 or the second recognition result by the second recognizer 42.”) when the determination certainty factor is greater than or equal to the certainty-factor determination value (Oosake, Para 0094; “The result of determining whether the confidence level is “high” or “low” can be decided by whether the difference between the highest score and the second highest score is equal to or greater than a first reference value”), and output the determination result of the second learning model when the determination certainty factor is smaller than the certainty-factor determination value (Oosake, Para 0096; “… in a case where the recognition confidence level determination device 43 determines that the confidence level for the recognition result by the first recognizer 41 is “low”, the control unit 44 causes the second recognizer 42 to perform the recognition process …”).
With regard to Claim 13, Oosake and Xu disclose the actuation method of the image processing apparatus according to Claim 11.
Oosake further discloses wherein the certainty-factor determination value stored in the first memory comprises a first certainty-factor determination value (the reference value used at the time of determining the confidence level for the recognition result obtained by the first recognizer 41) and a second certainty-factor determination value (the reference value used at the time of determining the confidence level for the recognition result obtained by the second recognizer 42) smaller than the first certainty-factor determination value (Oosake, Para 0095; “… the reference value used at the time of determining the confidence level for the recognition result obtained by the second recognizer 42 may be smaller than the reference value used at the time of determining the confidence level for the recognition result obtained by the first recognizer 41.”), and
wherein the actuation method of the image processing apparatus further comprises:
a step for causing the first processor to output the determination result of the first learning model when the determination certainty factor is greater than or equal to the first certainty-factor determination value (Oosake, Para 0092; “the recognition confidence level determination device 43 inputs the recognition result (in this example, three scores) of the first recognizer 41, and in a case where the difference between the highest score among the three scores and the other scores is large, the recognition confidence level determination device 43 classifies the image into a category having the highest score, and determines that the confidence level for the category classification is “high”.”), and output the determination result of the second learning model when the determination certainty factor is smaller than the second certainty-factor determination value (Oosake, Para 0096; “… in a case where the recognition confidence level determination device 43 determines that the confidence level for the recognition result by the first recognizer 41 is “low”, the control unit 44 causes the second recognizer 42 to perform the recognition process …”. Note that in Oosake, in a case where the confidence level of the result by the first recognizer 41 is lower than the second reference value, then the level is lower than the first reference value, and as a result, the second learning model would be executed and its determination result be outputted, which is the same as the Application.).
With regard to Claim 16, Oosake and Xu disclose all the limitations of Claim 1. Oosake further discloses wherein the first processor controls an endoscope to photograph the first light source image and the second light source image respectively by using the first light source and the second light source (Oosake, Para 0057; “The light source device 11 supplies illumination light to the light guide 35 of the endoscope 10 via the connector 37A. As the illumination light, light in various wavelength ranges is selected according to the observation purpose, such as white light (light in white-light wavelength range or light in a plurality of wavelength ranges) or light in one or a plurality of specific wavelength ranges, or a combination thereof.”), and
wherein the first determination result and the second determination result are related to a lesion included in the target (Oosake, Para 0110; Each of the first recognizer 41 and the second recognizer 42 has a function of detecting the position of the lesion candidate from the medical image”).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Oosake and Xu, in view of Ito et al (US 20210113075 A1; hereafter Ito).
With regard to Claim 5, Oosake and Xu disclose all the limitations of Claim 1 as discussed above, but do not clearly and explicitly disclose wherein the first processor is configured to
alternately and successively acquire the first light source image and the second light source image, and
select whether to output the first light source image to the first learning model or to output the second light source image to the second learning model based on the determination certainty factor and the certainty-factor determination value stored in the first memory.
Oosake further discloses wherein the first processor is configured to select whether to output the first light source image to the first learning model (Oosake, Para 0082; “The first recognizer 41 is a portion that performs recognition of the image (video 38 and static image 39) captured during the observation of the body cavity …”) or to the second learning model based on the determination certainty factor and the certainty-factor determination value stored in the first memory (Oosake, Para 0096; “in a case where the recognition confidence level determination device 43 determines that the confidence level for the recognition result by the first recognizer 41 is “low”, the control unit 44 causes the second recognizer 42 to perform the recognition process of the medical image of which the confidence level is determined to be “low”.”).
Oosake and Xu as discussed above do not clearly and explicitly disclose wherein the first processor is configured to
alternately and successively acquire the first light source image and the second light source image, and
output the second light source image to the second learning model.
Xu further discloses wherein the first processor is configured to output the second light source image to the second learning model (Xu, Page 1002, Column 2, Para 2; “… we trained and tested … DCNN-2 - only using NBI images for training and testing.”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Oosake and Xu, as further suggested by Xu, in order to cause the second learning model to analyze the second light source image. One of ordinary skill in the art would have been motivated to make the modification for the benefit of improved diagnostic capability by analyzing images containing more disease-related characteristics (Xu, Page 1004, Column 1, Para 2; “… the NBI mode can better observe such slight microvessel changes”) and training the learning model by images of the same type.
Oosake and Xu as discussed above do not clearly and explicitly disclose wherein the first processor is configured to alternately and successively acquire the first light source image and the second light source image.
Ito in the same field of endeavor discloses wherein the first processor is configured to alternately and successively acquire the first light source image and the second light source image (Ito, Para 0089; “The endoscope apparatus acquires both the white light image and the NBI image in one cycle from T1 to T1′. At this time, the image sensor performs image capturing operation twice. That is, one cycle includes two imaging frames.” Para 0092; “The illumination light control circuit 150 continuously repeats operation in one cycle described above.”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Oosake and Xu, as suggested by Ito, in order to alternatively and successively acquire the first and the second light source image. One of ordinary skill in the art would have been motivated to make the modification for the benefit of increasing detection rate of abnormal tissues by continuously imaging tissue, e.g. using white-light illustration, with images that a human operator may timely view, check and even interpret (Ito, Para 0081; “In the normal light observation mode, an observation image is acquired with white light generally used in the endoscope apparatus …”; Para 0132; “As for generation and display of the observation image for the operator, most suitable image information is selected … to generate an image with quality as desired by the operator in color reproductivity, image quality, resolution, contrast of the lesion part, or the like.”) and simultaneously acquiring images that contain more disease-related characteristics (Ito, Para 0081; “In the NBI observation mode, narrowband light corresponding to absorption characteristics of hemoglobin is used as illumination light to display a blood vessel especially in a surface layer and an intermediate layer of mucosa with high contrast.”).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Oosake and Xu, in view of Oosake (US 20210186315 A1; hereafter Oosake 2).
With regard to Claim 9, Oosake and Xu disclose all the limitations of Claim 1 as discussed above.
Oosake further discloses wherein the first processor acquires a plurality of determination results with respect to a plurality of successive first light source images (Oosake, Para 0105; “The first recognizer 41 includes a feature extraction unit and a recognition process unit, performs image recognition for each of frame images 38a … constituting the input video 38 …”. Examples of determination results for individual frames are shown in Fig. 4 and also in Para 0109; “The information display control unit 45B displays various information (in this example, imaging time, category classification, and recognition result) in a region on the right side of each of the screens 17A, 17B, and 17C.”) from the first learning model that successively receives the first light source images (Oosake, Para 0104; “In FIG. 4, the video 38 captured by the endoscope system 9 is input to the first recognizer 41 …”).
Oosake and Xu do not clearly and explicitly disclose calculating the determination certainty factor based on the plurality of determination results.
Oosake 2 in the same field of endeavor discloses calculating the determination certainty factor based on the plurality of determination results (Oosake 2, Para 0107; “The continuous detection determination unit 70 can determine that the detector 15 continuously detects the detection target when the detector 15 detects the detection target consecutively within a certain time range longer than the detection interval of the detector 15 (the period of one frame of the moving image or the period of a plurality of frames of the moving image).” In this disclosure, whether the detector 15 continuously detects the detect target (as the result of yes or no, or 1 or 0), as determined by the unit 70, corresponds to the “determination certainty factor” of the Application, and is determined based on detection results from a plurality of image frames, which again is the same as the Application). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Oosake and Xu, as suggested by Oosake 2, in order to determine a determination certainty factor based on determination results from a plurality of images. One of ordinary skill in the art would have been motivated to make the modification for the benefit of increased diagnostic efficiency by switching to a second imaging and/or analysis method only when a result is consistently obtained in multiple successive images by a first method.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Oosake and Xu, in view of Brocker et al (Weather Forecast. 2007; 22: 651 - 661; hereafter Brocker).
With regard to Claim 14, Oosake and Xu disclose the actuation method of the image processing apparatus according to Claim 11. Oosake further discloses wherein the first memory stores the calculated certainty-factor determination value (Oosake, Para 0094; “the first reference value or the second reference value … may be a preset fixed value or a value set by a user.” The reference values are inherently stored in the memory.).
Oosake and Xu as discussed above do not clearly and explicitly disclose wherein the image processing apparatus further comprises:
a third processor; and
a second memory storing a plurality of first evaluation data sets obtained by photographing a plurality of locations of the target illuminated by the first light source and a plurality of second evaluation data sets obtained by photographing same locations among the plurality of locations of the target illuminated by the second light source,
wherein the third processor inputs the first evaluation data sets and the second evaluation data sets respectively to the first learning model and the second learning model and acquire a plurality of first determination results and a plurality of second determination results from the first learning model and the second learning model;
wherein the third processor calculates a plurality of first determination certainty factors from the plurality of first determination results;
wherein the third processor calculates a plurality of first determination accuracies and a plurality of second determination accuracies from the plurality of first determination results and the plurality of second determination results, respectively; and
wherein the third processor calculates the certainty-factor determination value based on a relationship between a plurality of determination accuracy difference and the plurality of first determination certainty factors, the determination accuracy difference indicating differences between the plurality of first determination accuracies and the plurality of the second determination accuracies at the same locations of the target.
Xu further discloses wherein the image processing apparatus further comprises:
a third processor (Xu, Page 1004, Column 2, Para 2; “our personal computer (3.6 GHz Intel(R) Core(TM) i7-9700K CPU and an NVIDIA GeForce RTX 2080TI GPU).”); and
a second memory storing a plurality of first evaluation data sets obtained by photographing a plurality of locations of the target illuminated by the first light source (only using WLI images for … testing) and a plurality of second evaluation data sets (only using NBI images for … testing) obtained by photographing same locations among the plurality of locations of the target illuminated by the first light source and the second light source (Xu, Page 1002, Column 2, Para 2; “during testing, for each NBI image, we also randomly select a corresponding patient’s WLI image for pairing … we trained and tested two traditional DCNNs as shown in Figure 2: DCNN-1 - only using WLI images for training and testing, and DCNN-2 - only using NBI images for training and testing.”. Xu discloses that for each of the patients, the WLI and NBI images are acquired for a same lesion, as in Page 1001, Column 2, Para 2; “A total of 4,783 nasopharyngoscopy images of the lesion were collected, including 2,898 WLI images and 1,885 NBI images. Each patient contains at least one WLI image and one NBI image.”),
wherein the third processor inputs the first evaluation data sets and the second evaluation data sets respectively to the first learning model and the second learning model and acquire a plurality of first determination results and a plurality of second determination results from the first learning model and the second learning model (Xu, Page 1002, Table II summarizes the determination results by the two models, DCNN-1 (trained and tested by WLI images) and DCNN-2 (trained and tested by NBI images).);
wherein the third processor calculates a plurality of first determination certainty factors from the plurality of first determination results (Xu, Page 1002, Column 2, Para 5; “In image-level prediction, DCNN outputs the predicted probability value y = (p1, p2) for each input. If p1 ≤ p2, the prediction result is NPC, otherwise noncancer.” This disclosure indicates that for each predication, a DCNN model also produces a predicted probability value.); and
wherein the third processor calculates a plurality of first determination accuracies and a plurality of second determination accuracies from the plurality of first determination results and the plurality of second determination results, respectively (Xu, Page 1002, Column 2, Para 6; “We use common evaluation metrics of binary classification to evaluate the performance of our method, which include accuracy (Acc) …”. The determination accuracy of the results by the two models are summarized in Table II of Page 1002. For example, the accuracy of DCNN-1 is averaged to be 87.0, and that of DCNN-2 is 92.8.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Oosake and Xu, as further suggested by Xu, in order to determine estimation certainty and accuracy for the two learning models based on separate evaluation datasets. One of ordinary skill in the art would have been motivated to make the modification for the benefit of further advancing clinical adoption of deep learning in analyzing endoscopic images by comprehensively assessing the performance of the relevant deep learning models (Xu, Page 999, Column 2, Para 1; “Doctors who lack endoscopic experiences are prone to misdiagnosis and misjudgment.”; Page 1001, Column 1, Para 3; “Recently, deep learning has shown great potential in visual tasks, such as image classification, detection, and segmentation. Several studies have demonstrated the effectiveness of deep learning for lesion detection (15–17) and pathological classification (7,18–25) during endoscopy.”).
Oosake and Xu as discussed above do not clearly and explicitly disclose calculating the certainty-factor determination value based on a relationship between a plurality of determination accuracy differences and the plurality of first determination certainty factor, the determination accuracy difference indicating differences between the plurality of first determination accuracies and the plurality of the second determination accuracy for a same group of targets.
Brocker in an analogous field of predictive modeling discloses calculating the certainty-factor determination value (“the distance in probability of the observed relative frequencies from that expected for a reliable forecast system”) based on a relationship (“a reliability diagram on probability paper” as shown in Fig. 4, cited below) between a plurality of determination accuracy difference (“distance in probability” along y axis for the plurality of points in Fig. 4) and the plurality of first determination certainty factor (“the forecast values”, or “forecast probabilities” along x axis for the plurality of points in Fig. 4) (Brocker, Page 654, Column 2, Para 2; “In this graph, the x axis still represents the forecast values. The y axis … represents the probability that the observed relative frequency would have been closer to the diagonal than the actual observed relative frequency if the forecast was reliable (see Fig. 4). … This graph, showing the distance in probability of the observed relative frequencies from that expected for a reliable forecast system, will be referred to as a reliability diagram on probability paper.”), the determination accuracy difference indicating differences between the plurality of first determination accuracies and the plurality of the second determination accuracy for a same group of targets (Brocker, Page 654, Column 2, Para 2; “… the probability that the observed relative frequency would have been closer to the diagonal than the actual observed relative frequency if the forecast was reliable …”. This disclosure discloses 2 prediction/classification models, one model that produces the actual observed result (e.g. the “plus signs” points in Fig. 4), and the other model that would produce “reliable forecast” (corresponding points on the zero line in Fig. 4). The y-axis corresponds to difference of estimation accuracy between the two models.).
PNG
media_image1.png
487
704
media_image1.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Oosake and Xu, as suggested by Brocker, in order to determine a certainty-factor determination value based on the relationship between accuracy difference of two learning models and determination certainty factor. One of ordinary skill in the art would have been motivated to make the modification for the benefit of improving diagnostic accuracy by comprehensively assessing and comparing performance of different methods so that a proper method can be conveniently selected for analyzing images with different features (Brocker, Page 660, Column 1, Para 1; “Both forecasts and observed relative frequencies fluctuate, and to interpret a reliability diagram correctly, these deviations have to be taken into account. We have introduced a consistency resampling method for assigning consistency bars to the diagonal of the reliability diagram, indicating the region where a reliable forecast would fall into …”) (Xu, Page 1006, Column 1, Para 2; “… when the lesion is superficial, only the capillaries on the surface of the lesion may have slight changes. Hence, it is not easy for WLI mode to find such subtle changes (32). … although the NBI mode reveals more on microvessels, in some cases when there are no microvessel changes or irregular neoplasms, the overall morphological changes of the lesion may be overlooked.”).
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Oosake, Xu and Brocker, in view of Ascari et al (Plant Reproduction. 2020; 33:205 - 219; hereafter Ascari).
With regard to Claim 15, Oosake, Xu and Brocker disclose the actuation method of the image processing apparatus according to Claim 14, including causing the third processor to calculate the certainty-factor determination value based on the relationship between the determination accuracy difference and each first determination certainty factor, with the calculated certainty-factor determination value being the first determination certainty factor when a sign of the determination accuracy difference on a linearly approximated line is reversed.
Oosake, Xu and Brocker does not clearly and explicitly disclose linearly approximating the relationship of two variables, and calculates the x-axis variable when a sign of the y-axis variable on a linearly approximated line is reversed.
PNG
media_image2.png
316
572
media_image2.png
Greyscale
Ascari in analogous field of image analysis and classification discloses disclose linearly approximating the relationship of two variables (Ascari, Page 210, Column 1, Para 2; “Bland-Altman plots were also used for measuring classification accuracy …”) (Ascari, Page 210, Column 1, Para 2; “The presence of proportional bias (when the values of the differences change in proportion to averages) in Bland-Altman plots was estimated through linear regression …”. The linear approximation is also shown as blue line in Fig. 6A as cited above.), and calculates the x-axis variable when a sign of the y-axis variable on a linearly approximated line is reversed (Ascari, Page 211, Column 2, Para 1; “The analysis of proportional bias detected a slight but significant tendency to underestimate viable and sterile pollen grains at higher average counts both for UC (respectively, y = 0.02x − 0.07, P ≤ 0.001 and y = 0.07x + 0.50, P ≤ 0.01) … ” In this disclosure (corresponding to the cited Fig. 6A), the linear regression (blue line in the figure, or y = 0.02x – 0.07) quantifies the proportional bias in the classification accuracy for the UC method (i.e. unsupervised clustering). Once the linear relationship (y = 0.02x – 0.07) is computed, accuracy bias at all x-axis values are known, including the x-axis value where the bias is zero or reverses sign.). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Oosake, Xu and Brocker, as suggested by Ascari, in order to linerly approximate the relationship between the two variable thus determining the x-axis value where y-axis value changes sign. One of ordinary skill in the art would have been motivated to make the modification for the benefit of improving the determintion accuracy by selecting a classification model with higher classification accuracy (Ascari, Page 216, Column 1, Para 3; “In the comparison between manual and automated methods, the presence of proportional bias can be interpreted as an increasing tendency to underestimate total counts at higher pollen loads.”).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LEI ZHANG whose telephone number is (571)272-7172. The examiner can normally be reached Monday-Friday 8am-5pm E.T..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pascal Bui-Pho can be reached at (571) 272-2714. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/L.Z./Examiner, Art Unit 3798
/PASCAL M BUI PHO/Supervisory Patent Examiner, Art Unit 3798
Prosecution Insights