DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 08/18/2026 has been entered.
Response to Amendment
Claims 2-4, 8-10, 12-13 and 16 have been cancelled. Claims 1, 5-7, 11 and 14-15 have been amended. Claims 1, 5-7, 11 and 14-15 remain pending.
Response to Arguments
On Pages 13-14 of Remarks, Applicant argues that reference Oosake merely causes its second recognizer to process the same medical image, and reference Xu does not disclose using a low certainty factor to control acquisition of a new image. Examiner respectfully disagrees. As pointed out in section of 35 USC 112a, the newly added feature of switching to a different light source in response to a low certainty factor is not described in Specification. To Examiner’s understanding, special light image and white light image are simultaneously input into either processing procedure or processors; in other words, the image for the second learning model is available from the very beginning. Therefore, reference Oosake discloses switching to a second learning model in response to a low certainty factor for the result of the first learning model, and Xu discloses a second learning model for analyzing a different type of image. Moreover, the feature of determining the certainty-factor determination value based on the relationship relies on a widely used statistical approach for comparing different classification models, and a reference from an analogous field, Brocker, is cited for rejection.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1, 5-7, 11 and 14-15 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 1, Lines 36-39, and Claim 11, Lines 32-35, recite “in response to the determination certainty factor being less than the certainty-factor determination value, control the light source device to switch from the narrow-band special light to the white light, control the endoscope to photograph the evaluation target illuminated by the white light to acquire a white-light image”. Specification, Para 0015, recites “when the determination certainty factor is smaller than the certainty-factor determination value, the first processor acquires the second light source image in place of the first light source image”, in which the processor acquiring the second image is not equivalent to the processor directly controlling the light source device and the endoscope as claimed. In Specification, the white light and the special light are alternately emitted for every frame in the multi-frame observation mode (Para 0035), or as an alternative, one of the two lights is selected to acquire one type of images that are analyzed by one determination unit (Para 0154). In these disclosures, switching of light sources is not in response to determination certainty factor.
Claims 5-7 and 14-15 are also rejected under 35 U.S.C. 112(a) because they inherit the deficiencies of the claim(s) they respectively depend upon.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 6-7, 11 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Oosake (US 20200193236 A1; hereafter Oosake), in view of Xu et al (Laryngoscope. 2022 May;132(5):999-1007; hereafter Xu), Brocker et al (Weather Forecast. 2007; 22: 651 – 661; hereafter Brocker) and Kamon (US 20210076917 A1; hereafter Kamon).
With regard to Claim 1, Oosake discloses an image processing apparatus (Oosake, Para 0048; “a medical image processing device 14”) comprising:
a first learning model (Oosake, Para 0082; “the first recognizer 41 includes a convolutional neural network (CNN)”) learned by using a first learning data set (Oosake, Para 0087; “Parameters of the filters used in each convolutional layer are automatically learned in advance by an amount of learning data.”);
a second learning model (Oosake, Para 0089; “ the second recognizer 42 has the same configuration as the first recognizer 41, and performs a recognition process of the captured image to obtain a second recognition result of a category classification or the like”);
a first processor (Oosake, Para 0075; “A control unit 44 corresponding to the processor (not shown) … generally controls a medical image acquisition unit 40, a first recognizer 41, a second recognizer 42, a recognition confidence level determination device 43 …”); and
a first memory (Oosake, Para 0075; “a recognition confidence level determination device 43”) storing a certainty-factor determination value (Oosake, Para 0094; “The result of determining whether the confidence level is “high” or “low” can be decided by whether the difference between the highest score and the second highest score is equal to or greater than a first reference value …”),
wherein the first processor is configured to
control a light source device to emit the narrow-band special light (Oosake, Para 0057; “The light source device 11 supplies illumination light to the light guide 35 of the endoscope 10 via the connector 37A … The specific wavelength range is a range narrower than the white-light wavelength range.”) and control an endoscope to photograph an evaluation target illuminated by the narrow-band special light to acquire a special-light image (Oosake, Para 0064; “… in a case where the video 38 and the static image 39 are images obtained by light (special light) in the above-described specific wavelength range, the video 38 and the static image 39 are special light images.”),
input the special-light image to the first learning model to acquire a first determination result related to a lesion or a disease included in the evaluation target (Oosake, Para 0082; “The first recognizer 41 is a portion that performs recognition of the image (video 38 and static image 39) captured during the observation of the body cavity …”), the first determination result including a first determination probability and a second determination probability respectively according to different states of the lesion or the disease (Oosake, Para 0093; “… the score of “neoplastic” is 33%, the score of “non-neoplastic” is 30%, and the score of “others” is 37%, among these scores, since the difference between the highest score (37%) of “others” and the scores (30% and 33%) of the other categories (particularly the second highest score 33%) is small, the confidence level for the category classification that the image to be recognized is classified to “others” is determined to be “low”.”),
calculate a determination certainty factor as an absolute value of a difference between the first determination probability and the second determination probability (Oosake, Para 0093; “… the score of “neoplastic” is 33%, the score of “non-neoplastic” is 30%, and the score of “others” is 37%, among these scores, since the difference between the highest score (37%) of “others” and the scores (30% and 33%) of the other categories (particularly the second highest score 33%) is small, the confidence level for the category classification that the image to be recognized is classified to “others” is determined to be “low”.”. Here the disclosure uses the absolute difference between the highest probability and other probabilities to determine the confidence level of classification, so the disclosed difference corresponds to the claimed determination certainty factor),
compare the determination certainty factor and the certainty-factor determination value stored in the first memory (Oosake, Para 0094; “The result of determining whether the confidence level is “high” or “low” can be decided by whether the difference between the highest score and the second highest score is equal to or greater than a first reference value …”. Here “the difference …” corresponds to the “determination certainty factor”, and “a first reference value” corresponds to the “certainty-factor determination value” of Application, respectively.),
in response to the determination certainty factor being greater than or equal to the certainty-factor determination value, output the first determination result as an image-based diagnosis result of the evaluation target (Oosake, Para 0147; “… in step S14, in a case where it is determined that the confidence level for the first recognition result of the image by the first recognizer 41 is “high”, the process proceeds to step S18.”. As illustrated in Fig. 7, step S18 is “display recognition result”), and
in response to the determination certainty factor being less than the certainty-factor determination value, input the white-light image to the second learning model to acquire a second determination result related to a lesion or a disease included in the evaluation target, and output the second determination result as the image-based diagnosis result of the evaluation target (Oosake, Para 0096; “in a case where the recognition confidence level determination device 43 determines that the confidence level for the recognition result by the first recognizer 41 is “low”, the control unit 44 causes the second recognizer 42 to perform the recognition process of the medical image of which the confidence level is determined to be “low”.”).
Oosake does not clearly and explicitly disclose
including special-light images obtained by photographing a plurality of locations of a target illuminated by narrow-band special light and ground truth data corresponding to the special-light images;
a model learned by using a second learning data set including white-light images obtained by photographing the plurality of locations of the target illuminated by white light and ground truth data corresponding to the white-light images;
wherein the certainty-factor determination value is determined based on a relationship between determination certainty factors obtained from the first learning model and determination accuracy differences between determination accuracies of the first learning model and determination accuracies of the second learning model for respective pairs of a special-light image and a white-light image captured at a same location; and
controlling the light source device to switch from the narrow-band special light to the white light and controlling the endoscope to photograph the evaluation target illuminated by the white light to acquire a white-light image.
Xu in the same field of endeavor discloses
including special-light images obtained by photographing a plurality of locations of a target illuminated by narrow-band special light and ground truth data corresponding to the special-light images (light of narrow band as used in NBI) (Xu, Page 1002, Column 2, Para 2; “… we trained and tested two traditional DCNNs as shown in Figure 2: … DCNN-2 - only using NBI images for training and testing.” The diagnosis of NPC or noncancer based on nasopharyngeal biopsy (Page 1001, Column 2, Para 1) corresponds to the claimed ground truth data); and
a model learned by using a second learning data set (2,898 WLI images (Xu, Page 1001, Column 1, Para 1)) including white-light images obtained by photographing the plurality of locations of the target illuminated by white light and ground truth data corresponding to the white-light images (Xu, Page 1002, Column 2, Para 2; “… we trained and tested two traditional DCNNs as shown in Figure 2: DCNN-1 - only using WLI images for training and testing …”; The diagnosis of NPC or noncancer based on nasopharyngeal biopsy (Page 1001, Column 2, Para 1) corresponds to the claimed ground truth data).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Oosake, as suggested by Xu, in order to train the two learning models using white-light images and narrow-band light images respectively. One of ordinary skill in the art would have been motivated to make the modification for the benefit of improved diagnostic capability by utilizing white-light endoscopy’s high repeatability (Xu, Page 1001, Column 1, Para 2; “the intraobserver and interobserver agreement of malignancies is strong in WLI endoscopy”) and narrow-band light endoscopy’s high differential ability between benign and malignant lesions (Xu, Page 1000, Column 2, Para 1; “NBI endoscopy can significantly enhance the ability of differential diagnosis between NPC and nasopharyngeal benign hyperplasia with specificity up to 91.3%”).
Oosake and Xu as discussed above do not explicitly and clearly disclose
wherein the certainty-factor determination value is determined based on a relationship between determination certainty factors obtained from the first learning model and determination accuracy differences between determination accuracies of the first learning model and determination accuracies of the second learning model for respective pairs of a special-light image and a white-light image captured at a same location; and
controlling the light source device to switch from the narrow-band special light to the white light and controlling the endoscope to photograph the evaluation target illuminated by the white light to acquire a white-light image.
Fig. 4 of Brocker
PNG
media_image1.png
487
704
media_image1.png
Greyscale
Brocker in an analogous field of endeavor discloses wherein the certainty-factor determination value is determined based on a relationship (“a reliability diagram on probability paper” as shown in Fig. 4; cited above) between determination certainty factors obtained from a first model (Brocker, Page 654, Column 2, Para 2; “In this graph, the x axis still represents the forecast values. The y axis … represents the probability that the observed relative frequency would have been closer to the diagonal than the actual observed relative frequency if the forecast was reliable (see Fig. 4). … This graph, showing the distance in probability of the observed relative frequencies from that expected for a reliable forecast system, will be referred to as a reliability diagram on probability paper.” Here the disclosed “forecast values” on x axis correspond to the claimed determination certainty factors) and determination accuracy differences between determination accuracies of the first model and determination accuracies of a second model for a same group of target measurements (Brocker, Page 654, Column 2, Para 2; “… the probability that the observed relative frequency would have been closer to the diagonal than the actual observed relative frequency if the forecast was reliable …”. This disclosure discloses 2 prediction/classification models, one model that produces the actual observed result (e.g. the “plus signs” points in Fig. 4), and the other model that would produce “reliable forecast” (the zero line in Fig. 4). The disclosed “distance in probability” on y axis corresponds to the claimed difference of estimation accuracy between the two models). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Oosake and Xu, as suggested by Brocker, in order to determine a certainty-factor determination value based on the relationship between accuracy difference of two determination units and determination certainty factor. One of ordinary skill in the art would have been motivated to make the modification for the benefit of improving diagnostic accuracy by comprehensively assessing and comparing performance of different methods so that a proper method can be conveniently selected for analyzing images with different features (Brocker, Page 660, Column 1, Para 1; “Both forecasts and observed relative frequencies fluctuate, and to interpret a reliability diagram correctly, these deviations have to be taken into account. We have introduced a consistency resampling method for assigning consistency bars to the diagonal of the reliability diagram, indicating the region where a reliable forecast would fall into …”).
Oosake, Xu and Brocker as discussed above do not clearly and explicitly disclose controlling the light source device to switch from the narrow-band special light to the white light and controlling the endoscope to photograph the evaluation target illuminated by the white light to acquire a white-light image.
Kamon in the same field of endeavor discloses controlling the light source device to switch from the narrow-band special light to the white light and controlling the endoscope to photograph the evaluation target illuminated by the white light to acquire a white-light image (Kamon, Para 0010; “… one of the first observation light and the second observation light may be white light and the other may be narrow-band light …”; Para 0011; “… the first image captured by using the first observation light and the second image captured by using the second observation light are acquired.”; Para 0100; “the image acquiring unit 204A controls the light source control unit 350 under control by the image acquisition control unit 204C to cause the blue light source 310B to emit light and irradiate the subject with blue light (second observation light) as narrow-band light instead of white light (first observation light)”. In these disclosures, light source control unit causes change of light sources (from white light to narrow-band light, or from narrow-band light to white light) to acquire 2 different images). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Oosake, Xu and Brocker, as suggested by Kamon, in order to switch light source in real-time imaging. One of ordinary skill in the art would have been motivated to make the modification for the benefit of visualizing different anatomic features that are respectively more clear in images acquired with light of different wavelength ranges (Kamon, Para 0115; “in the first still image 701 using white light as observation light, thick blood vessels 601 are clearly seen but thin blood vessels 602 are not clearly seen as illustrated in FIG. 17A, for example. In contrast, in the second still image 702 using blue narrow-band light as observation light, the thick blood vessels 601 are not clearly seen but the thin blood vessels 602 are clearly seen as illustrated in FIG. 17B …”).
With regard to Claim 6, Oosake, Xu, Brocker and Kamon disclose the image processing apparatus according to Claim 1. Oosake further discloses wherein each of the first determination result and the second determination result indicates a presence or an absence of a lesion, a classification of the lesion, a degree of severity of the lesion, or remission or non-remission of an inflammatory disease (Oosake, Para 0022; “each of the first recognizer and the second recognizer may detect a position of a lesion candidate from the medical image”; Para 0023; “… classify the medical image into any category of a plurality of categories relating to a lesion”).
With regard to Claim 7, Oosake, Xu, Brocker and Kamon disclose the image processing apparatus according to Claim 1.
Oosake further discloses comprising:
a third memory storing the first learning model and the second learning model (Oosake, Para 0073; “a hardware structure which executes various controls of the medical image processing device 14”. The diagram of the medical image processing device 14 in Fig. 2 shows including of the two recognizers); and
one or two second processors (Oosake, Para 0075; “… A control unit 44 corresponding to the processor (not shown) of the medical image processing device 14 …”) configured to:
read the first learning model from the third memory, input the special-light image to the first learning model, and cause the first learning model to output the first determination result; or
read the second learning model from the third memory, input the white-light image to the second learning model, and cause the second learning model to output the second determination result (Oosake, Para 0097; “… the control unit 44 causes the first recognizer 41 to perform the image recognition process … the control unit 44 causes the second recognizer 42 to perform the image recognition process.”).
With regard to Claim 11, Oosake discloses an actuation method (Oosake, Para 0141; “… a medical image processing method according to an embodiment of the invention …”) of an image processing apparatus comprising a first learning model (Oosake, Para 0082; “the first recognizer 41 includes a convolutional neural network (CNN)”) learned by using a first learning data set (Oosake, Para 0087; “Parameters of the filters used in each convolutional layer are automatically learned in advance by an amount of learning data.”), a second learning model (Oosake, Para 0089; “ the second recognizer 42 has the same configuration as the first recognizer 41, and performs a recognition process of the captured image to obtain a second recognition result of a category classification or the like”), a first processor (Oosake, Para 0075; “A control unit 44 corresponding to the processor (not shown) … generally controls a medical image acquisition unit 40, a first recognizer 41, a second recognizer 42, a recognition confidence level determination device 43 …”), and a first memory (Oosake, Para 0075; “a recognition confidence level determination device 43”) storing a certainty-factor determination value (Oosake, Para 0094; “The result of determining whether the confidence level is “high” or “low” can be decided by whether the difference between the highest score and the second highest score is equal to or greater than a first reference value …”), the actuation method of the image processing apparatus comprising:
causing the first processor to control a light source device to emit the narrow-band special light (Oosake, Para 0057; “The light source device 11 supplies illumination light to the light guide 35 of the endoscope 10 via the connector 37A … The specific wavelength range is a range narrower than the white-light wavelength range.”) and control an endoscope to photograph an evaluation target illuminated by the narrow-band special light to acquire a special-light image (Oosake, Para 0064; “… in a case where the video 38 and the static image 39 are images obtained by light (special light) in the above-described specific wavelength range, the video 38 and the static image 39 are special light images.”);
causing the first processor to input the special-light image to the first learning model to acquire a first determination result related to a lesion or a disease included in the evaluation target (Oosake, Para 0082; “The first recognizer 41 is a portion that performs recognition of the image (video 38 and static image 39) captured during the observation of the body cavity …”), the first determination result including a first determination probability and a second determination probability respectively according to different states of the lesion or the disease (Oosake, Para 0093; “… the score of “neoplastic” is 33%, the score of “non-neoplastic” is 30%, and the score of “others” is 37%, among these scores, since the difference between the highest score (37%) of “others” and the scores (30% and 33%) of the other categories (particularly the second highest score 33%) is small, the confidence level for the category classification that the image to be recognized is classified to “others” is determined to be “low”.”);
causing the first processor to calculate a determination certainty factor as an absolute value of a difference between the first determination probability and the second determination probability (Oosake, Para 0093; “… the score of “neoplastic” is 33%, the score of “non-neoplastic” is 30%, and the score of “others” is 37%, among these scores, since the difference between the highest score (37%) of “others” and the scores (30% and 33%) of the other categories (particularly the second highest score 33%) is small, the confidence level for the category classification that the image to be recognized is classified to “others” is determined to be “low”.”. Here the disclosure uses the absolute difference between the highest probability and other probabilities to determine the confidence level of classification, so the disclosed difference corresponds to the claimed determination certainty factor);
causing the first processor to compare the determination certainty factor and the certainty-factor determination value stored in the first memory (Oosake, Para 0094; “The result of determining whether the confidence level is “high” or “low” can be decided by whether the difference between the highest score and the second highest score is equal to or greater than a first reference value …”. Here “the difference …” corresponds to the “determination certainty factor”, and “a first reference value” corresponds to the “certainty-factor determination value” of Application, respectively.);
causing the first processor, in response to the determination certainty factor being greater than or equal to the certainty-factor determination value, to output the first determination result as an image-based diagnosis result of the evaluation target (Oosake, Para 0147; “… in step S14, in a case where it is determined that the confidence level for the first recognition result of the image by the first recognizer 41 is “high”, the process proceeds to step S18.”. As illustrated in Fig. 7, step S18 is “display recognition result”); and
causing the first processor, in response to the determination certainty factor being less than the certainty-factor determination value, to input the white-light image to the second learning model to acquire a second determination result related to a lesion or a disease included in the evaluation target, and output the second determination result as the image-based diagnosis result of the evaluation target (Oosake, Para 0096; “in a case where the recognition confidence level determination device 43 determines that the confidence level for the recognition result by the first recognizer 41 is “low”, the control unit 44 causes the second recognizer 42 to perform the recognition process of the medical image of which the confidence level is determined to be “low”.”).
Oosake does not clearly and explicitly disclose
including special-light images obtained by photographing a plurality of locations of a target illuminated by narrow-band special light and ground truth data corresponding to the special-light images;
a model learned by using a second learning data set including white-light images obtained by photographing the plurality of locations of the target illuminated by white light and ground truth data corresponding to the white-light images;
wherein the certainty-factor determination value is determined based on a relationship between determination certainty factors obtained from the first learning model and determination accuracy differences between determination accuracies of the first learning model and determination accuracies of the second learning model for respective pairs of a special-light image and a white-light image captured at a same location; and
controlling the light source device to switch from the narrow-band special light to the white light and controlling the endoscope to photograph the evaluation target illuminated by the white light to acquire a white-light image.
Xu in the same field of endeavor discloses
including special-light images obtained by photographing a plurality of locations of a target illuminated by narrow-band special light and ground truth data corresponding to the special-light images (light of narrow band as used in NBI) (Xu, Page 1002, Column 2, Para 2; “… we trained and tested two traditional DCNNs as shown in Figure 2: … DCNN-2 - only using NBI images for training and testing.” The diagnosis of NPC or noncancer based on nasopharyngeal biopsy (Page 1001, Column 2, Para 1) corresponds to the claimed ground truth data); and
a model learned by using a second learning data set (2,898 WLI images (Xu, Page 1001, Column 1, Para 1)) including white-light images obtained by photographing the plurality of locations of the target illuminated by white light and ground truth data corresponding to the white-light images (Xu, Page 1002, Column 2, Para 2; “… we trained and tested two traditional DCNNs as shown in Figure 2: DCNN-1 - only using WLI images for training and testing …”; The diagnosis of NPC or noncancer based on nasopharyngeal biopsy (Page 1001, Column 2, Para 1) corresponds to the claimed ground truth data).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Oosake, as suggested by Xu, in order to train the two learning models using white-light images and narrow-band light images respectively. One of ordinary skill in the art would have been motivated to make the modification for the benefit of improved diagnostic capability by utilizing white-light endoscopy’s high repeatability (Xu, Page 1001, Column 1, Para 2; “the intraobserver and interobserver agreement of malignancies is strong in WLI endoscopy”) and narrow-band light endoscopy’s high differential ability between benign and malignant lesions (Xu, Page 1000, Column 2, Para 1; “NBI endoscopy can significantly enhance the ability of differential diagnosis between NPC and nasopharyngeal benign hyperplasia with specificity up to 91.3%”).
Oosake and Xu as discussed above do not explicitly and clearly disclose
wherein the certainty-factor determination value is determined based on a relationship between determination certainty factors obtained from the first learning model and determination accuracy differences between determination accuracies of the first learning model and determination accuracies of the second learning model for respective pairs of a special-light image and a white-light image captured at a same location; and
controlling the light source device to switch from the narrow-band special light to the white light and controlling the endoscope to photograph the evaluation target illuminated by the white light to acquire a white-light image.
Brocker in an analogous field of endeavor discloses wherein the certainty-factor determination value is determined based on a relationship (“a reliability diagram on probability paper” as shown in Fig. 4) between determination certainty factors obtained from a first model (Brocker, Page 654, Column 2, Para 2; “In this graph, the x axis still represents the forecast values. The y axis … represents the probability that the observed relative frequency would have been closer to the diagonal than the actual observed relative frequency if the forecast was reliable (see Fig. 4). … This graph, showing the distance in probability of the observed relative frequencies from that expected for a reliable forecast system, will be referred to as a reliability diagram on probability paper.” Here the disclosed “forecast values” on x axis correspond to the claimed determination certainty factors) and determination accuracy differences between determination accuracies of the first model and determination accuracies of a second model for a same group of target measurements (Brocker, Page 654, Column 2, Para 2; “… the probability that the observed relative frequency would have been closer to the diagonal than the actual observed relative frequency if the forecast was reliable …”. This disclosure discloses 2 prediction/classification models, one model that produces the actual observed result (e.g. the “plus signs” points in Fig. 4), and the other model that would produce “reliable forecast” (the zero line in Fig. 4). The disclosed “distance in probability” on y axis corresponds to the claimed difference of estimation accuracy between the two models). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Oosake and Xu, as suggested by Brocker, in order to determine a certainty-factor determination value based on the relationship between accuracy difference of two determination units and determination certainty factor. One of ordinary skill in the art would have been motivated to make the modification for the benefit of improving diagnostic accuracy by comprehensively assessing and comparing performance of different methods so that a proper method can be conveniently selected for analyzing images with different features (Brocker, Page 660, Column 1, Para 1; “Both forecasts and observed relative frequencies fluctuate, and to interpret a reliability diagram correctly, these deviations have to be taken into account. We have introduced a consistency resampling method for assigning consistency bars to the diagonal of the reliability diagram, indicating the region where a reliable forecast would fall into …”).
Oosake, Xu and Brocker as discussed above do not clearly and explicitly disclose controlling the light source device to switch from the narrow-band special light to the white light and controlling the endoscope to photograph the evaluation target illuminated by the white light to acquire a white-light image.
Kamon in the same field of endeavor discloses controlling the light source device to switch from the narrow-band special light to the white light and controlling the endoscope to photograph the evaluation target illuminated by the white light to acquire a white-light image (Kamon, Para 0010; “… one of the first observation light and the second observation light may be white light and the other may be narrow-band light …”; Para 0011; “… the first image captured by using the first observation light and the second image captured by using the second observation light are acquired.”; Para 0100; “the image acquiring unit 204A controls the light source control unit 350 under control by the image acquisition control unit 204C to cause the blue light source 310B to emit light and irradiate the subject with blue light (second observation light) as narrow-band light instead of white light (first observation light)”. In these disclosures, light source control unit causes change of light sources (from white light to narrow-band light, or from narrow-band light to white light) to acquire 2 different images). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Oosake, Xu and Brocker, as suggested by Kamon, in order to switch light source in real-time imaging. One of ordinary skill in the art would have been motivated to make the modification for the benefit of visualizing different anatomic features that are respectively more clear in images acquired with light of different wavelength ranges (Kamon, Para 0115; “in the first still image 701 using white light as observation light, thick blood vessels 601 are clearly seen but thin blood vessels 602 are not clearly seen as illustrated in FIG. 17A, for example. In contrast, in the second still image 702 using blue narrow-band light as observation light, the thick blood vessels 601 are not clearly seen but the thin blood vessels 602 are clearly seen as illustrated in FIG. 17B …”).
With regard to Claim 14, Oosake, Xu, Brocker and Kamon disclose the actuation method of the image processing apparatus according to Claim 11, including determining or calculating the certainty-factor determination value based on the relationship. Oosake further discloses wherein the first memory stores the calculated certainty-factor determination value (Oosake, Para 0094; “the first reference value or the second reference value … may be a preset fixed value or a value set by a user.” The reference values are inherently stored in the memory.).
Oosake, Xu, Brocker and Kamon as discussed above do not clearly and explicitly disclose
a third processor; and
a second memory storing a plurality of first evaluation data sets including special-light images obtained by photographing a plurality of locations of the target illuminated by the narrow-band special light and a plurality of second evaluation data sets obtained by photographing same locations among the plurality of locations of the target illuminated by the white light,
wherein the third processor inputs the first evaluation data sets and the second evaluation data sets respectively to the first learning model and the second learning model and acquires a plurality of first determination results and a plurality of second determination results from the first learning model and the second learning model, wherein the third processor calculates a plurality of first determination certainty factors from the plurality of first determination results; and
wherein the third processor calculates a plurality of first determination accuracies and a plurality of second determination accuracies from the plurality of first determination results and the plurality of second determination results, respectively.
Xu further discloses
a third processor (Xu, Page 1004, Column 2, Para 2; “our personal computer (3.6 GHz Intel(R) Core(TM) i7-9700K CPU and an NVIDIA GeForce RTX 2080TI GPU).”); and
a second memory storing a plurality of first evaluation data sets including special-light images obtained by photographing a plurality of locations of the target illuminated by the narrow-band special light (only using NBI images for … testing) and a plurality of second evaluation data sets (only using WLI images for … testing) obtained by photographing same locations among the plurality of locations of the target illuminated by the first light source and the white light (Xu, Page 1002, Column 2, Para 2; “during testing, for each NBI image, we also randomly select a corresponding patient’s WLI image for pairing … we trained and tested two traditional DCNNs as shown in Figure 2: DCNN-1 - only using WLI images for training and testing, and DCNN-2 - only using NBI images for training and testing.”. Xu discloses that for each of the patients, the WLI and NBI images are acquired for a same lesion, as in Page 1001, Column 2, Para 2; “A total of 4,783 nasopharyngoscopy images of the lesion were collected, including 2,898 WLI images and 1,885 NBI images. Each patient contains at least one WLI image and one NBI image.”),
wherein the third processor inputs the first evaluation data sets and the second evaluation data sets respectively to the first learning model and the second learning model and acquire a plurality of first determination results and a plurality of second determination results from the first learning model and the second learning model (Xu, Page 1002, Table II summarizes the determination results by the two models, DCNN-1 (trained and tested by WLI images) and DCNN-2 (trained and tested by NBI images).);
wherein the third processor calculates a plurality of first determination certainty factors from the plurality of first determination results (Xu, Page 1002, Column 2, Para 5; “In image-level prediction, DCNN outputs the predicted probability value y = (p1, p2) for each input. If p1 ≤ p2, the prediction result is NPC, otherwise noncancer.” This disclosure indicates that for each predication, a DCNN model also produces a predicted probability value.); and
wherein the third processor calculates a plurality of first determination accuracies and a plurality of second determination accuracies from the plurality of first determination results and the plurality of second determination results, respectively (Xu, Page 1002, Column 2, Para 6; “We use common evaluation metrics of binary classification to evaluate the performance of our method, which include accuracy (Acc) …”. The determination accuracy of the results by the two models are summarized in Table II of Page 1002. For example, the accuracy of DCNN-1 is averaged to be 87.0, and that of DCNN-2 is 92.8.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Oosake, Xu, Brocker and Kamon, as further suggested by Xu, in order to determine estimation certainty and accuracy for the two learning models based on separate evaluation datasets. One of ordinary skill in the art would have been motivated to make the modification for the benefit of further advancing clinical adoption of deep learning in analyzing endoscopic images by comprehensively assessing the performance of the relevant deep learning models (Xu, Page 999, Column 2, Para 1; “Doctors who lack endoscopic experiences are prone to misdiagnosis and misjudgment.”; Page 1001, Column 1, Para 3; “Recently, deep learning has shown great potential in visual tasks, such as image classification, detection, and segmentation. Several studies have demonstrated the effectiveness of deep learning for lesion detection (15–17) and pathological classification (7,18–25) during endoscopy.”).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Oosake, Xu, Brocker and Kamon, in view of Ito et al (US 20210113075 A1; hereafter Ito).
With regard to Claim 5, Oosake, Xu, Brocker and Kamon disclose the image processing apparatus according to Claim 1, but do not clearly and explicitly disclose wherein the first processor is configured to alternately and successively acquire the special light image and the white-light image.
Ito in the same field of endeavor discloses wherein the first processor is configured to alternately and successively acquire the special-light image and the white-light image (Ito, Para 0089; “The endoscope apparatus acquires both the white light image and the NBI image in one cycle from T1 to T1′. At this time, the image sensor performs image capturing operation twice. That is, one cycle includes two imaging frames.” Para 0092; “The illumination light control circuit 150 continuously repeats operation in one cycle described above.”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Oosake, Xu, Brocker and Kamon, as suggested by Ito, in order to alternatively and successively acquire two images with different light source. One of ordinary skill in the art would have been motivated to make the modification for the benefit of increasing detection rate of abnormal tissues by continuously imaging tissue, e.g. using white-light illustration, with images that a human operator may timely view, check and even interpret (Ito, Para 0081; “In the normal light observation mode, an observation image is acquired with white light generally used in the endoscope apparatus …”; Para 0132; “As for generation and display of the observation image for the operator, most suitable image information is selected … to generate an image with quality as desired by the operator in color reproductivity, image quality, resolution, contrast of the lesion part, or the like.”) and simultaneously acquiring images that contain more disease-related characteristics (Ito, Para 0081; “In the NBI observation mode, narrowband light corresponding to absorption characteristics of hemoglobin is used as illumination light to display a blood vessel especially in a surface layer and an intermediate layer of mucosa with high contrast.”).
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Oosake, Xu, Brocker and Kamon, in view of Ascari et al (Plant Reproduction. 2020; 33:205 - 219; hereafter Ascari).
With regard to Claim 15, Oosake, Xu, Brocker and Kamon disclose the actuation method of the image processing apparatus according to Claim 14, including causing the third processor to calculate the certainty-factor determination value based on the relationship between the plurality of determination accuracy differences and the plurality of the first determination certainty factor.
Oosake, Xu, Brocker and Kamon do not clearly and explicitly disclose linearly approximating the relationship of two variables, and calculating the x-axis variable when a sign of the y-axis variable on a linearly approximated line is reversed.
Fig. 6A of Ascari
PNG
media_image2.png
316
572
media_image2.png
Greyscale
Ascari in analogous field of image analysis and classification discloses linearly approximating the relationship of two variables (Ascari, Page 210, Column 1, Para 2; “Bland-Altman plots were also used for measuring classification accuracy …”) (Ascari, Page 210, Column 1, Para 2; “The presence of proportional bias (when the values of the differences change in proportion to averages) in Bland-Altman plots was estimated through linear regression …”. The linear approximation is also shown as blue line in Fig. 6A as cited above.), and calculating the x-axis variable when a sign of the y-axis variable on a linearly approximated line is reversed (Ascari, Page 211, Column 2, Para 1; “The analysis of proportional bias detected a slight but significant tendency to underestimate viable and sterile pollen grains at higher average counts both for UC (respectively, y = 0.02x − 0.07, P ≤ 0.001 and y = 0.07x + 0.50, P ≤ 0.01) … ” In this disclosure (corresponding to the cited Fig. 6A), the linear regression (blue line in the figure, or y = 0.02x – 0.07) quantifies the proportional bias in the classification accuracy for the UC method (i.e. unsupervised clustering). Once the linear relationship (y = 0.02x – 0.07) is computed, accuracy bias at all x-axis values are known, including the x-axis value where the bias is zero or reverses sign.). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Oosake, Xu, Brocker and Kamon, as suggested by Ascari, in order to linearly approximate the relationship between the two variables thus determining the x-axis value where y-axis value changes sign. One of ordinary skill in the art would have been motivated to make the modification for the benefit of improving the determination accuracy by selecting a classification model with higher classification accuracy (Ascari, Page 216, Column 1, Para 3; “In the comparison between manual and automated methods, the presence of proportional bias can be interpreted as an increasing tendency to underestimate total counts at higher pollen loads.”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LEI ZHANG whose telephone number is (571)272-7172. The examiner can normally be reached Monday-Friday 8am-5pm E.T..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pascal Bui-Pho can be reached at (571) 272-2714. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/L.Z./ Examiner, Art Unit 3798
/PASCAL M BUI PHO/ Supervisory Patent Examiner, Art Unit 3798
