DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3-8 and 12-18 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 3 recites the limitation "the respective types" in line 5. There is insufficient antecedent basis for this limitation in the claim.
Claim 4 recites the limitation "the respective types" in line 5. There is insufficient antecedent basis for this limitation in the claim.
Claim 5 recites the limitation "the respective types" in line 4. There is insufficient antecedent basis for this limitation in the claim.
Claim 6 recites the limitation "the respective types" in lines 4-6. There is insufficient antecedent basis for this limitation in the claim.
Claim 7 recites the limitation "determine a condition of each of the endoscopic images and upon detecting a plurality of the diagnosis results having an identical type, acquire the diagnostic results for the respective types” it is unclear what is meant by condition of the images limitations. The examiner questions if it’s the shooting or acquisition condition or the type of examination being performed. Additionally, it is unclear what “an identical type” limitation refers to. The examiner questions if the identical type is the lesion type, lesion identification (presence or absence) result type, or image appearance type? The examiner interprets the “condition” limitation as the type of examination being performed for example lesion detection in the stomach region and “an identical type” limitation as images classified with lesion present in them.
Claim 7 recites the limitation "the respective types" in line 5. There is insufficient antecedent basis for this limitation in the claim.
Claim 8 recites the limitation "the respective types" in line 3. There is insufficient antecedent basis for this limitation in the claim.
Claims 12-18 recite the limitation "the respective types". There is insufficient antecedent basis for this limitation in the claim.
Claim 16 recites the limitation "determining a condition of each of the endoscopic images and upon detecting a plurality of the diagnosis results having an identical type, acquire the diagnostic results for the respective types” it is unclear what is meant by condition of the images limitations. The examiner questions if it’s the shooting or acquisition condition or the type of examination being performed. Additionally, it is unclear what “an identical type” limitation refers to. The examiner questions if the identical type is the lesion type, lesion identification (presence or absence) result type, or image appearance type? The examiner interprets the “condition” limitation as the type of examination being performed for example lesion detection in the stomach region and “an identical type” limitation as images classified with lesion present in them.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Watanabe et al. (WO 2024/013848, however, the US 2025/0372264 version is used for examination purposes) in the view of Higa et al. (US 2023/0245304).
Regarding claim 1, Watanabe teaches an endoscopic image diagnosis apparatus comprising (para. 0031; an endoscopic examination system 100. The endoscopic examination system 100 makes a qualitative diagnosis to classify a part (lesion part) suspected of a lesion in an examination target and provides the classification result to an examiner such as a doctor who conducts examination or treatment using an endoscope.):
at least one memory configured to store instructions (para. 0044; The processor 11 executes a predetermined process by executing a program or the like stored in the memory 12.), and
at least one processor configured to execute the instructions to (para. 0044; The processor 11 executes a predetermined process by executing a program or the like stored in the memory 12.):
acquire, for endoscopic images used for diagnosis of a subject, identification information on a lesion appearing in each of the endoscopic images (paras. 0049 and 0053; an outline of the process of detecting a lesion part by the image processing device 1 will be described. In summary, the image processing device 1 classifies the lesion part based on a variable number of time series endoscopic images Ia. For example, on the basis of a model (also referred to as “lesion detection model”) configured to detect a lesion image, the lesion image acquisition unit 30 may acquire the lesion image. In this case, the parameters of the lesion detection model are stored in the memory 12 or the like in advance. The lesion image acquisition unit 30 builds the lesion detection model by referring to the above-described parameters and inputs an endoscope image Ia supplied from the endoscope 3 to the lesion detection model. Then, on the basis of the information outputted by the lesion detection model in response to the inputted endoscopic image Ia, the lesion image acquisition unit 30 determines whether or not the inputted endoscopic image Ia is a lesion image. In this case, the lesion detection model is, for example, a classification model that is trained to output a classification result regarding the presence or absence of a lesion part in the endoscopic image Ia upon receiving an endoscopic image Ia. The examiner notes that the processor determines identification information of the lesion using a lesion detection model that classifies images based on lesion appearance in the images as present or absent.), and
diagnosis results representing a quality of the lesion based on each of the endoscopic images (paras. 0056, 0058, and 0077; an example of calculating the classification score will be described. FIG. 4 illustrates graphs showing the transition of the classification scores. In this example, the image processing device 1 starts the process at the time “t0” and calculates classification scores of three candidate classes (“adenoma”, “hyperplastic polyp”, and “invasive cancer”) on the basis of the lesion images (the lesion images A to the lesion images D) that the lesion image acquisition unit 30 acquires at the time “t1,” “t2,” “t3,” and “t4,” respectively. Here, the graph G1 shows the transition of the classification score of the candidate class “adenoma”, the graph G2 shows the transition of the classification score of the candidate class “hyperplastic polyp”, and the graph G3 shows the transition of the classification score of the candidate class “invasive cancer”. The examiner notes after the processor determine identification information of the lesion, the processor classifies images based on lesion category, such as adenoma, hyperplastic polyp, or invasive cancer.); and
integrate the diagnosis results for each lesion for which the identification information is identical (paras. 0062 and 0068-0070; The second calculation unit 312 calculates a likelihood ratio (also referred to as “integrated likelihood ratio”) obtained by integrating the likelihood ratios calculated in time series, and determines the classification score based on the integrated likelihood ratio. The classification score may be the integrated likelihood ratio itself or may be a function including the integrated likelihood ratio as a variable. Next, a description will be given of the calculation of the integrated likelihood ratio of each candidate class in the case where classification (multi-class classification) of three or more classes is performed. Assuming that the number of candidate classes is “M” (M is an integer of 3 or more), the score calculation unit 31 calculates the integrated likelihood ratio between the k-th (k=1, 2, . . . , M) candidate class and all remaining classes among the M candidate classes. In this case, for example, the score calculation unit 31 calculates the integrated likelihood ratio using the equation (1) while replacing the denominators of the first term and the second term on the right side of the equation (1) with the maximum likelihood among all candidate classes other than the k-th candidate class. In this case, the score calculation unit 31 may calculate the integrated likelihood ratio using the sum of the likelihoods of all candidate classes other than the k-th candidate class, instead of using the maximum likelihood. Therefore, for example, the score calculation unit 31 calculates the integrated likelihood ratio of each candidate class based on the likelihood ratio (i.e., the likelihood ratio shown on the right side of the equation (1)) of each candidate class outputted by the likelihood ratio calculation model upon inputting the target N lesion images of feature extraction by the feature extractor or the features thereof to the likelihood ratio calculation model. Based on the classification score calculated by the second calculation unit 312, the classification unit 32 performs a classification regarding the lesion part, and supplies the classification result to the display control unit 33. In this instance, the classification unit 32 compares the classification score of the lesion part for each candidate class with a predetermined threshold value (also referred to as “threshold value Th”), and determines whether or not there is a candidate class having the classification score equal to or larger than the threshold value Th. The examiner notes that the processor integrates the likelihood ratio for the multiple candidate classes of the images having lesion present and generate a classification score for each class and compares it to a threshold).
However, Watanabe fails to explicitly teach that endoscopic images used for diagnosis of a subject have a respective types of appearance.
Higa, in the same field of endeavor, teaches acquiring endoscopic images with a respective types of appearance for diagnosis of a subject and acquire identification information on a lesion appearing in each endoscopic image (paras. 0017, 0065, and 0153-0165; it is preferable that the observation image includes a normal light image captured by using normal light and a special light image captured by using special light, and the processor acquires an observation mode indicating whether the observation image is the normal light image or the special light image together with the distance information of the observation image. In FIG. 14, the processor 22 acquires an observation image in which an observation region in the body is imaged by the endoscope through the image acquisition unit 110 (step S10). Subsequently, the processor 22 acquires distance information related to a distance from the endoscope to the observation region by the distance information-acquisition unit 112 (step S12). The distance information-acquisition unit 112 of this example uses a learning model (third learning model) that estimates the distance from the observation image, and inputs the observation image to the third learning model to acquire the distance information from the third learning model. The processor 22 estimates a state of the observation region (for example, a lesion region existing in the observation region) based on the observation image acquired in step S10 and the distance information acquired in step S12 (step S14). The state estimation unit 114 of the observation region of the processor 22 selects any estimation mode from a plurality of estimation modes (for example, near view mode and distant view mode) for estimating the lesion region based on the distance information, estimates the lesion region by the selected estimation mode, or weight-averages estimation results of the lesion region respectively estimated by the plurality of the estimation modes according to the acquired distance information to obtain the final output.).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the time series endoscopic images of Watanabe to include endoscopic images acquired with different illumination modes and imaging distances of Higa. Doing so would increase the accuracy of the diagnoses because the appearance of the lesion changes according to the illumination mode and imaging distance as disclosed within Higa in paras. 0007, 0014, and 0017.
Regarding claim 2, Watanabe teaches the endoscopic image diagnosis apparatus according to claim 1, however fails to explicitly teach wherein the at least one processor is configured to further execute the instructions to identify a kind of a light source used to capture each of the endoscopic images, and determine the respective types of appearance, based on the identified kind of the light source.
Higa, in the same field of endeavor, teaches at least one processor is configured to further execute the instructions to identify a kind of a light source used to capture each of the endoscopic images, and determine the respective types of appearance, based on the identified kind of the light source (para. 0140; The observation mode acquisition unit 132 is a portion that acquires an observation mode indicating whether the observation image 100 is a normal light image or a special light image, and outputs an observation mode signal indicating the acquired current observation mode to the detection mode/discrimination mode selection unit 134.).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the system of Watanabe to include identification of illumination mode and imaging distance associated with the endoscopic images of Higa. Doing so would increase the accuracy of the diagnoses because both imaging distance and illumination mode alter the appearance and diagnostically useful characteristics of the same lesion, therefore, classifying the images by observation mode and distance and selecting or weighting the corresponding model outputs improve the accuracy and reliability of the integrated diagnoses to as disclosed within Higa in paras. 0007, 0014, 0017, and 0051-0054.
Regarding claim 3, Watanabe teaches the endoscopic image diagnosis apparatus according to claim 1, however fails to explicitly teach wherein the at least one processor is configured to further execute the instructions to identify at least one of perspective appearance of the lesion or magnification of each of the endoscopic images, and determine the respective types, based on at least one of the identified perspective appearance or magnification.
Higa, in the same field of endeavor, teaches the at least one processor is configured to further execute the instructions to identify at least one of perspective appearance of the lesion or magnification of each of the endoscopic images, and determine the respective types, based on at least one of the identified perspective appearance or magnification (para. 0139; The distance information-acquisition unit 130 is a portion that inputs the observation image 100 and acquires the distance information related to the distance from the endoscope to the observation region based on the input observation image 100, and outputs the acquired distance information to the detection mode/discrimination mode selection unit 134.).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the system of Watanabe to include identification of illumination mode and imaging distance associated with the endoscopic images of Higa. Doing so would increase the accuracy of the diagnoses because both imaging distance and illumination mode alter the appearance and diagnostically useful characteristics of the same lesion, therefore, classifying the images by observation mode and distance and selecting or weighting the corresponding model outputs improve the accuracy and reliability of the integrated diagnoses to as disclosed within Higa in paras. 0007, 0014, 0017, and 0051-0054.
Regarding claim 4, Watanabe teaches the endoscopic image diagnosis apparatus according to claim 1, wherein the diagnosis result indicates a class to which the quality belongs and a score for each class (paras. 0077 and 0079; FIG. 4 illustrates graphs showing the transition of the classification scores. In this example, the image processing device 1 starts the process at the time “t0” and calculates classification scores of three candidate classes (“adenoma”, “hyperplastic polyp”, and “invasive cancer”) on the basis of the lesion images (the lesion images A to the lesion images D) that the lesion image acquisition unit 30 acquires at the time “t1,” “t2,” “t3,” and “t4,” respectively. Here, the graph G1 shows the transition of the classification score of the candidate class “adenoma”, the graph G2 shows the transition of the classification score of the candidate class “hyperplastic polyp”, and the graph G3 shows the transition of the classification score of the candidate class “invasive cancer”. Then, as shown in the graph G1, at the time t4, since the classification score of the candidate class “adenoma” is equal to or larger than the threshold value TH, the image processing device 1 generates such a classification result that the lesion part appearing in the lesion image A to the lesion image D obtained from the time to the time t4 is “adenoma”.), and the at least one processor is configured to execute the instructions to compute, based on the diagnosis results for the respective types, a representative value of the scores for each class, and generate an integrated diagnosis based on the representative value of each class (fig. 7 and paras. 0079 and 0089; Then, as shown in the graph G1, at the time t4, since the classification score of the candidate class “adenoma” is equal to or larger than the threshold value TH, the image processing device 1 generates such a classification result that the lesion part appearing in the lesion image A to the lesion image D obtained from the time to to the time t4 is “adenoma”. In the second display example, the classification unit 32 determines that the classification score of the candidate class “adenoma” supplied from the score calculation unit 31 has reached the threshold value TH, and supplies the classification result indicating that it is classified into the candidate class “adenoma” to the display control unit 33. Then, in this case, in the classification result display area 72, based on the classification result described above, the display control unit 33 displays a text message indicating that an adenoma is likely to exist.).
Regarding claim 5, Watanabe teaches the endoscopic image diagnosis apparatus according to claim 1, wherein the diagnosis result indicates a class to which the quality belongs (paras. 0077 and 0079; FIG. 4 illustrates graphs showing the transition of the classification scores. In this example, the image processing device 1 starts the process at the time “t0” and calculates classification scores of three candidate classes (“adenoma”, “hyperplastic polyp”, and “invasive cancer”) on the basis of the lesion images (the lesion images A to the lesion images D) that the lesion image acquisition unit 30 acquires at the time “t1,” “t2,” “t3,” and “t4,” respectively. Here, the graph G1 shows the transition of the classification score of the candidate class “adenoma”, the graph G2 shows the transition of the classification score of the candidate class “hyperplastic polyp”, and the graph G3 shows the transition of the classification score of the candidate class “invasive cancer”. Then, as shown in the graph G1, at the time t4, since the classification score of the candidate class “adenoma” is equal to or larger than the threshold value TH, the image processing device 1 generates such a classification result that the lesion part appearing in the lesion image A to the lesion image D obtained from the time to the time t4 is “adenoma”.), and the at least one processor is configured to execute the instructions to generate an integrated diagnosis result from the diagnosis results for the respective types in accordance with rule information indicating a priority among the classes (fig. 7 and paras. 0079 and 0089; Then, as shown in the graph G1, at the time t4, since the classification score of the candidate class “adenoma” is equal to or larger than the threshold value TH, the image processing device 1 generates such a classification result that the lesion part appearing in the lesion image A to the lesion image D obtained from the time to to the time t4 is “adenoma”. In the second display example, the classification unit 32 determines that the classification score of the candidate class “adenoma” supplied from the score calculation unit 31 has reached the threshold value TH, and supplies the classification result indicating that it is classified into the candidate class “adenoma” to the display control unit 33. Then, in this case, in the classification result display area 72, based on the classification result described above, the display control unit 33 displays a text message indicating that an adenoma is likely to exist.).
Regarding claim 6, Watanabe teaches the endoscopic image diagnosis apparatus according to claim 1, wherein the diagnosis result indicates classes to which the quality belongs (paras. 0077 and 0079; FIG. 4 illustrates graphs showing the transition of the classification scores. In this example, the image processing device 1 starts the process at the time “t0” and calculates classification scores of three candidate classes (“adenoma”, “hyperplastic polyp”, and “invasive cancer”) on the basis of the lesion images (the lesion images A to the lesion images D) that the lesion image acquisition unit 30 acquires at the time “t1,” “t2,” “t3,” and “t4,” respectively. Here, the graph G1 shows the transition of the classification score of the candidate class “adenoma”, the graph G2 shows the transition of the classification score of the candidate class “hyperplastic polyp”, and the graph G3 shows the transition of the classification score of the candidate class “invasive cancer”. Then, as shown in the graph G1, at the time t4, since the classification score of the candidate class “adenoma” is equal to or larger than the threshold value TH, the image processing device 1 generates such a classification result that the lesion part appearing in the lesion image A to the lesion image D obtained from the time to the time t4 is “adenoma”.), and the at least one processor is configured to execute the instructions to generate an integrated diagnosis result from the diagnosis results for the respective types in accordance with rule information indicating a prioritized class among combinations of the classes indicated by the diagnosis results for the respective types (fig. 7 and paras. 0079 and 0089; Then, as shown in the graph G1, at the time t4, since the classification score of the candidate class “adenoma” is equal to or larger than the threshold value TH, the image processing device 1 generates such a classification result that the lesion part appearing in the lesion image A to the lesion image D obtained from the time to to the time t4 is “adenoma”. In the second display example, the classification unit 32 determines that the classification score of the candidate class “adenoma” supplied from the score calculation unit 31 has reached the threshold value TH, and supplies the classification result indicating that it is classified into the candidate class “adenoma” to the display control unit 33. Then, in this case, in the classification result display area 72, based on the classification result described above, the display control unit 33 displays a text message indicating that an adenoma is likely to exist.).
Regarding claim 7, Watanabe teaches the endoscopic image diagnosis apparatus according to claim 1, wherein the at least one processor is configured to further execute the instructions to determine a condition of each of the endoscopic images (paras. 0036-0037 and 0060; the first calculation unit 311 sets the number N in accordance with the type of the examination target. For example, in such a case that the examination target is an organ (for example, the stomach) in which the endoscope can be moved to a certain extent, the first calculation unit 311 sets the number N to a value smaller than the value in the case of the other types of the examination target since the correlation between the lesion images becomes relatively small. On the other hand, in such a case that the examination target is an organ (e.g., esophagus) in which the endoscope cannot be almost moved, the first calculation unit 311 sets the number N to a value larger than the value in the case of the other types of the examination target since the correlation between the lesion images becomes relatively large. Thus, the first calculation unit 311 can calculate the likelihood ratio more accurately. Similarly, likelihood ratio calculation models may be prepared for respective types of the examination target. In this case, the likelihood ratio calculation models are trained for respective types of the examination target, and parameters obtained through the training are stored in advance in the memory 12 or the like for each type of the examination target. The image processing device 1 may recognize the type of the examination target based on an external input or the like by the input unit 14 prior to the endoscopic examination, or may automatically recognize the examination target by applying any image recognition technique to the endoscopic image Ia obtained at the beginning of the endoscopic examination.), and upon detecting a plurality of the diagnosis results having an identical type, acquire the diagnosis results for the respective types, by integrating the plurality of the diagnosis results, based on a determination result on the conditions of the endoscopic images used for the plurality of the diagnosis results (paras. 0060-0062; As the first calculation information, the first calculation unit 311 stores, in the first calculation information storage unit D1, the calculated likelihood ratio and data used for calculation of the likelihood ratio by the first calculation unit 311. The “data used for calculation of the likelihood ratio” may be lesion images used for calculation of the likelihood ratio, or may be features extracted from the lesion images. The second calculation unit 312 calculates a likelihood ratio (also referred to as “integrated likelihood ratio”) obtained by integrating the likelihood ratios calculated in time series, and determines the classification score based on the integrated likelihood ratio. The classification score may be the integrated likelihood ratio itself or may be a function including the integrated likelihood ratio as a variable.).
Regarding claim 8, Watanabe teaches the endoscopic image diagnosis apparatus according to claim 1, wherein the at least one processor is configured to further execute the instructions to display or output as audio a diagnosis result obtained by integrating the diagnosis results for the respective types (para. 0090; the display control unit 33 may instruct the audio output unit 16 to output the audio guidance or a predetermined warning sound to inform the examiner of the classification result. Thereby, the display control unit 33 also allows the examiner to grasp the classification result.).
Regarding claim 9, Watanabe teaches the endoscopic image diagnosis apparatus according to claim 1, wherein the at least one processor is configured to further execute the instructions to input the endoscopic images into a qualitative diagnosis model and acquire the diagnosis results output by the qualitative diagnosis, wherein the qualitative diagnosis model is trained, through machine learning, to learn a relationship between an endoscopic image including a lesion region and a quality of a lesion in the endoscopic image (paras. 0031, 0053, and 0058-0059; the first calculation unit 311 calculates the likelihood ratio using the likelihood ratio calculation model that has been trained to output the likelihood ratio for the inputted N lesion images when N lesion images are inputted to the likelihood ratio calculation model, for example. The likelihood ratio calculation model may be a deep learning model, or any other machine learning model or a statistical model. The endoscopic examination system 100 makes a qualitative diagnosis to classify a part (lesion part) suspected of a lesion in an examination target and provides the classification result to an examiner such as a doctor who conducts examination or treatment using an endoscope. The endoscopic examination system 100.).
Regarding claim 10, Watanabe teaches an endoscopic image diagnosis method executed by a computer, comprising(para. 0031; an endoscopic examination system 100. The endoscopic examination system 100 makes a qualitative diagnosis to classify a part (lesion part) suspected of a lesion in an examination target and provides the classification result to an examiner such as a doctor who conducts examination or treatment using an endoscope.):
acquiring, for endoscopic images used for diagnosis of a subject, identification information on a lesion appearing in each of the endoscopic images (paras. 0049 and 0053; an outline of the process of detecting a lesion part by the image processing device 1 will be described. In summary, the image processing device 1 classifies the lesion part based on a variable number of time series endoscopic images Ia. For example, on the basis of a model (also referred to as “lesion detection model”) configured to detect a lesion image, the lesion image acquisition unit 30 may acquire the lesion image. In this case, the parameters of the lesion detection model are stored in the memory 12 or the like in advance. The lesion image acquisition unit 30 builds the lesion detection model by referring to the above-described parameters and inputs an endoscope image Ia supplied from the endoscope 3 to the lesion detection model. Then, on the basis of the information outputted by the lesion detection model in response to the inputted endoscopic image Ia, the lesion image acquisition unit 30 determines whether or not the inputted endoscopic image Ia is a lesion image. In this case, the lesion detection model is, for example, a classification model that is trained to output a classification result regarding the presence or absence of a lesion part in the endoscopic image Ia upon receiving an endoscopic image Ia. The examiner notes that the processor determines identification information of the lesion using a lesion detection model that classifies images based on lesion appearance in the images as present or absent.), and
diagnosis results representing a quality of the lesion based on each of the endoscopic images (paras. 0056, 0058, and 0077; an example of calculating the classification score will be described. FIG. 4 illustrates graphs showing the transition of the classification scores. In this example, the image processing device 1 starts the process at the time “t0” and calculates classification scores of three candidate classes (“adenoma”, “hyperplastic polyp”, and “invasive cancer”) on the basis of the lesion images (the lesion images A to the lesion images D) that the lesion image acquisition unit 30 acquires at the time “t1,” “t2,” “t3,” and “t4,” respectively. Here, the graph G1 shows the transition of the classification score of the candidate class “adenoma”, the graph G2 shows the transition of the classification score of the candidate class “hyperplastic polyp”, and the graph G3 shows the transition of the classification score of the candidate class “invasive cancer”. The examiner notes after the processor determine identification information of the lesion, the processor classifies images based on lesion category, such as adenoma, hyperplastic polyp, or invasive cancer.); and
integrating the diagnosis results for each lesion for which the identification information is identical (paras. 0062 and 0068-0070; The second calculation unit 312 calculates a likelihood ratio (also referred to as “integrated likelihood ratio”) obtained by integrating the likelihood ratios calculated in time series, and determines the classification score based on the integrated likelihood ratio. The classification score may be the integrated likelihood ratio itself or may be a function including the integrated likelihood ratio as a variable. Next, a description will be given of the calculation of the integrated likelihood ratio of each candidate class in the case where classification (multi-class classification) of three or more classes is performed. Assuming that the number of candidate classes is “M” (M is an integer of 3 or more), the score calculation unit 31 calculates the integrated likelihood ratio between the k-th (k=1, 2, . . . , M) candidate class and all remaining classes among the M candidate classes. In this case, for example, the score calculation unit 31 calculates the integrated likelihood ratio using the equation (1) while replacing the denominators of the first term and the second term on the right side of the equation (1) with the maximum likelihood among all candidate classes other than the k-th candidate class. In this case, the score calculation unit 31 may calculate the integrated likelihood ratio using the sum of the likelihoods of all candidate classes other than the k-th candidate class, instead of using the maximum likelihood. Therefore, for example, the score calculation unit 31 calculates the integrated likelihood ratio of each candidate class based on the likelihood ratio (i.e., the likelihood ratio shown on the right side of the equation (1)) of each candidate class outputted by the likelihood ratio calculation model upon inputting the target N lesion images of feature extraction by the feature extractor or the features thereof to the likelihood ratio calculation model. Based on the classification score calculated by the second calculation unit 312, the classification unit 32 performs a classification regarding the lesion part, and supplies the classification result to the display control unit 33. In this instance, the classification unit 32 compares the classification score of the lesion part for each candidate class with a predetermined threshold value (also referred to as “threshold value Th”), and determines whether or not there is a candidate class having the classification score equal to or larger than the threshold value Th. The examiner notes that the processor integrates the likelihood ratio for the multiple candidate classes of the images having lesion present and generate a classification score for each class and compares it to a threshold).
However, Watanabe fails to explicitly teach that endoscopic images used for diagnosis of a subject have a respective types of appearance.
Higa, in the same field of endeavor, teaches acquiring endoscopic images with a respective types of appearance for diagnosis of a subject and acquire identification information on a lesion appearing in each endoscopic image (paras. 0017, 0065, and 0153-0165; it is preferable that the observation image includes a normal light image captured by using normal light and a special light image captured by using special light, and the processor acquires an observation mode indicating whether the observation image is the normal light image or the special light image together with the distance information of the observation image. In FIG. 14, the processor 22 acquires an observation image in which an observation region in the body is imaged by the endoscope through the image acquisition unit 110 (step S10). Subsequently, the processor 22 acquires distance information related to a distance from the endoscope to the observation region by the distance information-acquisition unit 112 (step S12). The distance information-acquisition unit 112 of this example uses a learning model (third learning model) that estimates the distance from the observation image, and inputs the observation image to the third learning model to acquire the distance information from the third learning model. The processor 22 estimates a state of the observation region (for example, a lesion region existing in the observation region) based on the observation image acquired in step S10 and the distance information acquired in step S12 (step S14). The state estimation unit 114 of the observation region of the processor 22 selects any estimation mode from a plurality of estimation modes (for example, near view mode and distant view mode) for estimating the lesion region based on the distance information, estimates the lesion region by the selected estimation mode, or weight-averages estimation results of the lesion region respectively estimated by the plurality of the estimation modes according to the acquired distance information to obtain the final output.).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the time series endoscopic images of Watanabe to include endoscopic images acquired with different illumination modes and imaging distances of Higa. Doing so would increase the accuracy of the diagnoses because the appearance of the lesion changes according to the illumination mode and imaging distance as disclosed within Higa in paras. 0007, 0014, and 0017.
Regarding claim 11, Watanabe teaches the endoscopic image diagnosis method according to claim 10, however fails to explicitly teach identifying a kind of a light source used to capture each of the endoscopic images, and determining the respective types of appearance, based on the identified kind of the light source.
Higa, in the same field of endeavor, teaches identifying a kind of a light source used to capture each of the endoscopic images, and determining the respective types of appearance, based on the identified kind of the light source (para. 0140; The observation mode acquisition unit 132 is a portion that acquires an observation mode indicating whether the observation image 100 is a normal light image or a special light image, and outputs an observation mode signal indicating the acquired current observation mode to the detection mode/discrimination mode selection unit 134.).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the system of Watanabe to include identification of illumination mode and imaging distance associated with the endoscopic images of Higa. Doing so would increase the accuracy of the diagnoses because both imaging distance and illumination mode alter the appearance and diagnostically useful characteristics of the same lesion, therefore, classifying the images by observation mode and distance and selecting or weighting the corresponding model outputs improve the accuracy and reliability of the integrated diagnoses to as disclosed within Higa in paras. 0007, 0014, 0017, and 0051-0054.
Regarding claim 12, Watanabe teaches the endoscopic image diagnosis method according to claim 10, however fails to explicitly teach identifying at least one of perspective appearance of the lesion or magnification of each of the endoscopic images, and determining the respective types, based on at least one of the identified perspective appearance or magnification.
Higa, in the same field of endeavor, teaches identifying at least one of perspective appearance of the lesion or magnification of each of the endoscopic images, and determining the respective types, based on at least one of the identified perspective appearance or magnification (para. 0139; The distance information-acquisition unit 130 is a portion that inputs the observation image 100 and acquires the distance information related to the distance from the endoscope to the observation region based on the input observation image 100, and outputs the acquired distance information to the detection mode/discrimination mode selection unit 134.).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the system of Watanabe to include identification of illumination mode and imaging distance associated with the endoscopic images of Higa. Doing so would increase the accuracy of the diagnoses because both imaging distance and illumination mode alter the appearance and diagnostically useful characteristics of the same lesion, therefore, classifying the images by observation mode and distance and selecting or weighting the corresponding model outputs improve the accuracy and reliability of the integrated diagnoses to as disclosed within Higa in paras. 0007, 0014, 0017, and 0051-0054.
Regarding claim 13, Watanabe teaches the endoscopic image diagnosis method according to claim 10, wherein the diagnosis result indicates a class to which the quality belongs and a score for each class (paras. 0077 and 0079; FIG. 4 illustrates graphs showing the transition of the classification scores. In this example, the image processing device 1 starts the process at the time “t0” and calculates classification scores of three candidate classes (“adenoma”, “hyperplastic polyp”, and “invasive cancer”) on the basis of the lesion images (the lesion images A to the lesion images D) that the lesion image acquisition unit 30 acquires at the time “t1,” “t2,” “t3,” and “t4,” respectively. Here, the graph G1 shows the transition of the classification score of the candidate class “adenoma”, the graph G2 shows the transition of the classification score of the candidate class “hyperplastic polyp”, and the graph G3 shows the transition of the classification score of the candidate class “invasive cancer”. Then, as shown in the graph G1, at the time t4, since the classification score of the candidate class “adenoma” is equal to or larger than the threshold value TH, the image processing device 1 generates such a classification result that the lesion part appearing in the lesion image A to the lesion image D obtained from the time to the time t4 is “adenoma”.), and the integrating the diagnosis results comprises computing, based on the diagnosis results for the respective types, a representative value of the scores for each class, and generating an integrated diagnosis based on the representative value of each class (fig. 7 and paras. 0079 and 0089; Then, as shown in the graph G1, at the time t4, since the classification score of the candidate class “adenoma” is equal to or larger than the threshold value TH, the image processing device 1 generates such a classification result that the lesion part appearing in the lesion image A to the lesion image D obtained from the time to to the time t4 is “adenoma”. In the second display example, the classification unit 32 determines that the classification score of the candidate class “adenoma” supplied from the score calculation unit 31 has reached the threshold value TH, and supplies the classification result indicating that it is classified into the candidate class “adenoma” to the display control unit 33. Then, in this case, in the classification result display area 72, based on the classification result described above, the display control unit 33 displays a text message indicating that an adenoma is likely to exist.).
Regarding claim 14, Watanabe teaches the endoscopic image diagnosis method according to claim 10, wherein the diagnosis result indicates a class to which the quality belongs (paras. 0077 and 0079; FIG. 4 illustrates graphs showing the transition of the classification scores. In this example, the image processing device 1 starts the process at the time “t0” and calculates classification scores of three candidate classes (“adenoma”, “hyperplastic polyp”, and “invasive cancer”) on the basis of the lesion images (the lesion images A to the lesion images D) that the lesion image acquisition unit 30 acquires at the time “t1,” “t2,” “t3,” and “t4,” respectively. Here, the graph G1 shows the transition of the classification score of the candidate class “adenoma”, the graph G2 shows the transition of the classification score of the candidate class “hyperplastic polyp”, and the graph G3 shows the transition of the classification score of the candidate class “invasive cancer”. Then, as shown in the graph G1, at the time t4, since the classification score of the candidate class “adenoma” is equal to or larger than the threshold value TH, the image processing device 1 generates such a classification result that the lesion part appearing in the lesion image A to the lesion image D obtained from the time to the time t4 is “adenoma”.), and the integrating the diagnosis results comprises generating an integrated diagnosis result from the diagnosis results for the respective types in accordance with rule information indicating a priority among the classes (fig. 7 and paras. 0079 and 0089; Then, as shown in the graph G1, at the time t4, since the classification score of the candidate class “adenoma” is equal to or larger than the threshold value TH, the image processing device 1 generates such a classification result that the lesion part appearing in the lesion image A to the lesion image D obtained from the time to to the time t4 is “adenoma”. In the second display example, the classification unit 32 determines that the classification score of the candidate class “adenoma” supplied from the score calculation unit 31 has reached the threshold value TH, and supplies the classification result indicating that it is classified into the candidate class “adenoma” to the display control unit 33. Then, in this case, in the classification result display area 72, based on the classification result described above, the display control unit 33 displays a text message indicating that an adenoma is likely to exist.).
Regarding claim 15, Watanabe teaches the endoscopic image diagnosis method according to claim 10, wherein the diagnosis result indicates classes to which the quality belongs (paras. 0077 and 0079; FIG. 4 illustrates graphs showing the transition of the classification scores. In this example, the image processing device 1 starts the process at the time “t0” and calculates classification scores of three candidate classes (“adenoma”, “hyperplastic polyp”, and “invasive cancer”) on the basis of the lesion images (the lesion images A to the lesion images D) that the lesion image acquisition unit 30 acquires at the time “t1,” “t2,” “t3,” and “t4,” respectively. Here, the graph G1 shows the transition of the classification score of the candidate class “adenoma”, the graph G2 shows the transition of the classification score of the candidate class “hyperplastic polyp”, and the graph G3 shows the transition of the classification score of the candidate class “invasive cancer”. Then, as shown in the graph G1, at the time t4, since the classification score of the candidate class “adenoma” is equal to or larger than the threshold value TH, the image processing device 1 generates such a classification result that the lesion part appearing in the lesion image A to the lesion image D obtained from the time to the time t4 is “adenoma”.), and the integrating the diagnosis results comprises generating an integrated diagnosis result from the diagnosis results for the respective types in accordance with rule information indicating a prioritized class among combinations of the classes indicated by the diagnosis results for the respective types (fig. 7 and paras. 0079 and 0089; Then, as shown in the graph G1, at the time t4, since the classification score of the candidate class “adenoma” is equal to or larger than the threshold value TH, the image processing device 1 generates such a classification result that the lesion part appearing in the lesion image A to the lesion image D obtained from the time to to the time t4 is “adenoma”. In the second display example, the classification unit 32 determines that the classification score of the candidate class “adenoma” supplied from the score calculation unit 31 has reached the threshold value TH, and supplies the classification result indicating that it is classified into the candidate class “adenoma” to the display control unit 33. Then, in this case, in the classification result display area 72, based on the classification result described above, the display control unit 33 displays a text message indicating that an adenoma is likely to exist.).
Regarding claim 16, Watanabe teaches the endoscopic image diagnosis method according to claim 10, determining a condition of each of the endoscopic images (paras. 0036-0037 and 0060; the first calculation unit 311 sets the number N in accordance with the type of the examination target. For example, in such a case that the examination target is an organ (for example, the stomach) in which the endoscope can be moved to a certain extent, the first calculation unit 311 sets the number N to a value smaller than the value in the case of the other types of the examination target since the correlation between the lesion images becomes relatively small. On the other hand, in such a case that the examination target is an organ (e.g., esophagus) in which the endoscope cannot be almost moved, the first calculation unit 311 sets the number N to a value larger than the value in the case of the other types of the examination target since the correlation between the lesion images becomes relatively large. Thus, the first calculation unit 311 can calculate the likelihood ratio more accurately. Similarly, likelihood ratio calculation models may be prepared for respective types of the examination target. In this case, the likelihood ratio calculation models are trained for respective types of the examination target, and parameters obtained through the training are stored in advance in the memory 12 or the like for each type of the examination target. The image processing device 1 may recognize the type of the examination target based on an external input or the like by the input unit 14 prior to the endoscopic examination, or may automatically recognize the examination target by applying any image recognition technique to the endoscopic image Ia obtained at the beginning of the endoscopic examination.), and upon detecting a plurality of the diagnosis results having an identical type, acquire the diagnosis results for the respective types, by integrating the plurality of the diagnosis results, based on a determination result on the conditions of the endoscopic images used for the plurality of the diagnosis results (paras. 0060-0062; As the first calculation information, the first calculation unit 311 stores, in the first calculation information storage unit D1, the calculated likelihood ratio and data used for calculation of the likelihood ratio by the first calculation unit 311. The “data used for calculation of the likelihood ratio” may be lesion images used for calculation of the likelihood ratio, or may be features extracted from the lesion images. The second calculation unit 312 calculates a likelihood ratio (also referred to as “integrated likelihood ratio”) obtained by integrating the likelihood ratios calculated in time series, and determines the classification score based on the integrated likelihood ratio. The classification score may be the integrated likelihood ratio itself or may be a function including the integrated likelihood ratio as a variable.).
Regarding claim 17, Watanabe teaches the endoscopic image diagnosis method according to claim 10, further comprising displaying a diagnosis result obtained by integrating the diagnosis results for the respective types (para. 0090; once the classification is determined, the display control unit 33 outputs information indicating the classification result (text message in the classification result display area 72 in this case). Thus, the display control unit 33 can suitably notify the examiner of the classification result of the lesion part.).
Regarding claim 18, Watanabe teaches the endoscopic image diagnosis method according to claim 10, wherein the at least one processor is configured to further execute the instructions to display or output as audio a diagnosis result obtained by integrating the diagnosis results for the respective types (para. 0090; the display control unit 33 may instruct the audio output unit 16 to output the audio guidance or a predetermined warning sound to inform the examiner of the classification result. Thereby, the display control unit 33 also allows the examiner to grasp the classification result.).
Regarding claim 19, Watanabe teaches the endoscopic image diagnosis method according to claim 10, inputting the endoscopic images into a qualitative diagnosis model and acquire the diagnosis results output by the qualitative diagnosis, wherein the qualitative diagnosis model is trained, through machine learning, to learn a relationship between an endoscopic image including a lesion region and a quality of a lesion in the endoscopic image (paras. 0031, 0053, and 0058-0059; the first calculation unit 311 calculates the likelihood ratio using the likelihood ratio calculation model that has been trained to output the likelihood ratio for the inputted N lesion images when N lesion images are inputted to the likelihood ratio calculation model, for example. The likelihood ratio calculation model may be a deep learning model, or any other machine learning model or a statistical model. The endoscopic examination system 100 makes a qualitative diagnosis to classify a part (lesion part) suspected of a lesion in an examination target and provides the classification result to an examiner such as a doctor who conducts examination or treatment using an endoscope. The endoscopic examination system 100.).
Regarding claim 20, Watanabe teaches a non-transitory computer readable storage medium storing a program executed by a computer, the program causing the computer to: (para. 0112; the program is stored by any type of a non-transitory computer-readable medium (non-transitory computer readable medium) and can be supplied to a control unit or the like that is a computer.):
acquiring, for endoscopic images used for diagnosis of a subject, identification information on a lesion appearing in each of the endoscopic images (paras. 0049 and 0053; an outline of the process of detecting a lesion part by the image processing device 1 will be described. In summary, the image processing device 1 classifies the lesion part based on a variable number of time series endoscopic images Ia. For example, on the basis of a model (also referred to as “lesion detection model”) configured to detect a lesion image, the lesion image acquisition unit 30 may acquire the lesion image. In this case, the parameters of the lesion detection model are stored in the memory 12 or the like in advance. The lesion image acquisition unit 30 builds the lesion detection model by referring to the above-described parameters and inputs an endoscope image Ia supplied from the endoscope 3 to the lesion detection model. Then, on the basis of the information outputted by the lesion detection model in response to the inputted endoscopic image Ia, the lesion image acquisition unit 30 determines whether or not the inputted endoscopic image Ia is a lesion image. In this case, the lesion detection model is, for example, a classification model that is trained to output a classification result regarding the presence or absence of a lesion part in the endoscopic image Ia upon receiving an endoscopic image Ia. The examiner notes that the processor determines identification information of the lesion using a lesion detection model that classifies images based on lesion appearance in the images as present or absent.), and
diagnosis results representing a quality of the lesion based on each of the endoscopic images (paras. 0056, 0058, and 0077; an example of calculating the classification score will be described. FIG. 4 illustrates graphs showing the transition of the classification scores. In this example, the image processing device 1 starts the process at the time “t0” and calculates classification scores of three candidate classes (“adenoma”, “hyperplastic polyp”, and “invasive cancer”) on the basis of the lesion images (the lesion images A to the lesion images D) that the lesion image acquisition unit 30 acquires at the time “t1,” “t2,” “t3,” and “t4,” respectively. Here, the graph G1 shows the transition of the classification score of the candidate class “adenoma”, the graph G2 shows the transition of the classification score of the candidate class “hyperplastic polyp”, and the graph G3 shows the transition of the classification score of the candidate class “invasive cancer”. The examiner notes after the processor determine identification information of the lesion, the processor classifies images based on lesion category, such as adenoma, hyperplastic polyp, or invasive cancer.); and
integrating the diagnosis results for each lesion for which the identification information is identical (paras. 0062 and 0068-0070; The second calculation unit 312 calculates a likelihood ratio (also referred to as “integrated likelihood ratio”) obtained by integrating the likelihood ratios calculated in time series, and determines the classification score based on the integrated likelihood ratio. The classification score may be the integrated likelihood ratio itself or may be a function including the integrated likelihood ratio as a variable. Next, a description will be given of the calculation of the integrated likelihood ratio of each candidate class in the case where classification (multi-class classification) of three or more classes is performed. Assuming that the number of candidate classes is “M” (M is an integer of 3 or more), the score calculation unit 31 calculates the integrated likelihood ratio between the k-th (k=1, 2, . . . , M) candidate class and all remaining classes among the M candidate classes. In this case, for example, the score calculation unit 31 calculates the integrated likelihood ratio using the equation (1) while replacing the denominators of the first term and the second term on the right side of the equation (1) with the maximum likelihood among all candidate classes other than the k-th candidate class. In this case, the score calculation unit 31 may calculate the integrated likelihood ratio using the sum of the likelihoods of all candidate classes other than the k-th candidate class, instead of using the maximum likelihood. Therefore, for example, the score calculation unit 31 calculates the integrated likelihood ratio of each candidate class based on the likelihood ratio (i.e., the likelihood ratio shown on the right side of the equation (1)) of each candidate class outputted by the likelihood ratio calculation model upon inputting the target N lesion images of feature extraction by the feature extractor or the features thereof to the likelihood ratio calculation model. Based on the classification score calculated by the second calculation unit 312, the classification unit 32 performs a classification regarding the lesion part, and supplies the classification result to the display control unit 33. In this instance, the classification unit 32 compares the classification score of the lesion part for each candidate class with a predetermined threshold value (also referred to as “threshold value Th”), and determines whether or not there is a candidate class having the classification score equal to or larger than the threshold value Th. The examiner notes that the processor integrates the likelihood ratio for the multiple candidate classes of the images having lesion present and generate a classification score for each class and compares it to a threshold).
However, Watanabe fails to explicitly teach that endoscopic images used for diagnosis of a subject have a respective types of appearance.
Higa, in the same field of endeavor, teaches acquiring endoscopic images with a respective types of appearance for diagnosis of a subject and acquire identification information on a lesion appearing in each endoscopic image (paras. 0017, 0065, and 0153-0165; it is preferable that the observation image includes a normal light image captured by using normal light and a special light image captured by using special light, and the processor acquires an observation mode indicating whether the observation image is the normal light image or the special light image together with the distance information of the observation image. In FIG. 14, the processor 22 acquires an observation image in which an observation region in the body is imaged by the endoscope through the image acquisition unit 110 (step S10). Subsequently, the processor 22 acquires distance information related to a distance from the endoscope to the observation region by the distance information-acquisition unit 112 (step S12). The distance information-acquisition unit 112 of this example uses a learning model (third learning model) that estimates the distance from the observation image, and inputs the observation image to the third learning model to acquire the distance information from the third learning model. The processor 22 estimates a state of the observation region (for example, a lesion region existing in the observation region) based on the observation image acquired in step S10 and the distance information acquired in step S12 (step S14). The state estimation unit 114 of the observation region of the processor 22 selects any estimation mode from a plurality of estimation modes (for example, near view mode and distant view mode) for estimating the lesion region based on the distance information, estimates the lesion region by the selected estimation mode, or weight-averages estimation results of the lesion region respectively estimated by the plurality of the estimation modes according to the acquired distance information to obtain the final output.).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the time series endoscopic images of Watanabe to include endoscopic images acquired with different illumination modes and imaging distances of Higa. Doing so would increase the accuracy of the diagnoses because the appearance of the lesion changes according to the illumination mode and imaging distance as disclosed within Higa in paras. 0007, 0014, and 0017.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZAINAB M ALDARRAJI whose telephone number is (571)272-8726. The examiner can normally be reached Monday-Thursday7AM-5PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Carey Michael can be reached at (571) 270-7235. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ZAINAB MOHAMMED ALDARRAJI/Patent Examiner, Art Unit 3797
Prosecution Insights