DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 02/26/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Amendment
Claim Interpretation: The amended claims filed on 05/28/2026 overcomes the Claim Interpretation in the previous office action.
With respect to 35 U.S.C. 112(b) Rejection: The amended claims filed on 05/28/2026 overcomes the Claim 35 U.S.C. 112(b) rejection in the previous office action.
Response to Arguments
This Action is in response to Applicant’s response filed on 05/28/2026. Claims 1, 3-5, 9, 11-13, 15, 17-19, 21-23 and newly adding claims 24-26 are still pending in the present application. Claims 2, 6-8, 10, 14, 16 and 20 are canceled and/or withdrawn.
With respect to 35 U.S.C. 101 Rejection: Applicant argues that the amended claims 1, 9 and 15 to include patent-eligible subject matter which include additional elements that integrate the judicial exception into a practical application that avoids problems when an object is not appropriately detected from a medical image. After reviewing the amendments and argument filed on 05/28/2026 , the Examiner has withdrawn the previous 101 rejection for the following reason: The claims recites steps and features that an additional element (or combination of elements) may have integrated the exception into a practical application include: Implementing a judicial exception with, or using a judicial exception in conjunction with, a particular machine or manufacture that is integral to the claims. However, upon further consideration of amendments, a new ground(s) of 35 U.S.C. 112(b) and 35 USC § 101 rejection is made. Since this new ground of rejection was not necessitated by the amendment, this action is NON-FINAL rejection.
With respect to 35 U.S.C. 103 Rejection: Applicant's arguments filed on 05/28/2026 have been fully considered but are moot in view of the new ground(s) rejection in view of Kanda et al (U.S. 20120051640; Kanda).
Claim Status
Claims 3-5 and 21-24 is/are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph.
Claim(s) 3-5 and 21-24 is/are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Claim(s) 1, 3, 9, 11, 15, 17 and 21-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Takenouchi (WO-2020054543 A1), in view of Kanda et al (U.S. 20120051640; Kanda).
Claim(s) 4-5, 12-13, 18-19 and 23-26 is/are rejected under 35 U.S.C. 103 as being unpatentable over Takenouchi (WO-2020054543 A1), in view of Kanda et al (U.S. 20120051640; Kanda), and in further view of Hsieh et al (U.S. 20190340470 A1; Hsieh).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3-5 and 21-24 is/are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 3 recites the limitation “ The computer-readable medium” in line 1. There is insufficient antecedent basis for this limitation in the claim. A previous recitation for “The computer-readable medium” cannot be found in either claim 1 or claim 3.
Appropriate correction is required.
Claim 4 recites the limitation “ The computer-readable medium” in line 1. There is insufficient antecedent basis for this limitation in the claim. A previous recitation for “The computer-readable medium” cannot be found in either claim 1 or claim 4.
Appropriate correction is required.
Claim 5 recites the limitation “ The computer-readable medium” in line 1. There is insufficient antecedent basis for this limitation in the claim. A previous recitation for “The computer-readable medium” cannot be found in either claim 1 or claim 4 or claim 5.
Appropriate correction is required.
Claim 21 recites the limitation “ The computer-readable medium” in line 1. There is insufficient antecedent basis for this limitation in the claim. A previous recitation for “The computer-readable medium” cannot be found in either claim 1 or claim 21.
Appropriate correction is required.
Claim 22 recites the limitation “ The computer-readable medium” in line 1. There is insufficient antecedent basis for this limitation in the claim. A previous recitation for “The computer-readable medium” cannot be found in either claim 1 or claim 22.
Appropriate correction is required.
Claim 23 recites the limitation “ The computer-readable medium” in line 1. There is insufficient antecedent basis for this limitation in the claim. A previous recitation for “The computer-readable medium” cannot be found in either claim 1 or claim 23.
Appropriate correction is required.
Claim 24 recites the limitation “ The computer-readable medium” in line 1. There is insufficient antecedent basis for this limitation in the claim. A previous recitation for “The computer-readable medium” cannot be found in either claim 1 or claim 24.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 3-5 and 21-24 is/are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
The claim(s) 3-5 and 21-24 does/do not fall within at least one of the four categories of patent eligible subject matter because the claim is directed to a [ computer-readable medium] that does not exclude transitory forms of signal transmission (often referred to as "signals perse"), such as a propagating electrical or electromagnetic signal or carrier wave, and therefore does not fall within at least one of the four categories (a process, machine, manufacture, or composition of matter). It is suggested that amending the claim language to define the a processor-readable storage medium as “a non-transitory computer-readable medium” to satisfy the requirements and limit the claimed invention to eligible subject matter.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 3, 9, 11, 15, 17 and 23-25 is/are rejected under 35 U.S.C. 103 as being unpatentable over Takenouchi (WO-2020054543 A1), in view of Kanda et al (U.S. 20120051640; Kanda).
Regarding claim 1, Takenouchi discloses a non-transitory computer-readable medium storing a computer program executed by a computer processor (Paragraph 36: “The CPU 70 controls each unit in the processor device 16 and totally controls the entire endoscope system 10. The ROM 72 stores various programs and control data for controlling the operation of the processor device 16. The program and data executed by the CPU 70 are temporarily stored in the RAM 74.”) to execute a process comprising:
acquiring a medical image generated based on a signal detected by a catheter inserted into a luminal organ; (Paragraphs 51-53: “ while inserting the insertion section 20 of the electronic endoscope 12 into the body cavity and illuminating the inside of the body cavity with the illumination light from the light source device 14, an image of the inside of the body cavity captured by the imaging element 62 is displayed on the screen of the display device 18. … visualization in which a blood vessel in a specific depth region of the observation target is emphasized. In this mode, an image is generated and an image suitable for observing a blood vessel is displayed on the display device 18.”; Paragraph 99: “In step S11, the medical image processing apparatus 160 receives the current image via the image acquisition unit 162. The image acquired by the image acquisition unit 162 is a medical image including a subject image captured using the electronic endoscope 12, and is one image of a time-series image sequentially captured in a time-series manner.)
determining whether to execute object region detection processing on the medical image based on output values indicating a class classification for each pixel in the medical image (Paragraphs 100-101; Paragraphs 164-167: “the medical image analysis processing unit is a region that should be noted based on the feature amount of the pixel of the medical image. …The medical image analysis processing unit detects the presence or absence of a target to be noted based on the feature amount of the pixel of the medical image, and the medical image analysis result obtaining unit obtains an analysis result of the medical image analysis processing unit. “; Paragraphs 68: “the availability determination unit 164 determines whether the image acquired from the image acquisition unit 162 is an image inappropriate for recognition. The availability determination unit 164 includes a recognition unit 164A. The recognizing unit 164A recognizes whether the input image is appropriate or inappropriate for image recognition. Here, “appropriate for image recognition” means that the image is suitable for recognition processing for classifying lesions, which is the main purpose of recognition. “Inappropriate for image recognition” means that the image is inappropriate for recognition such as classification of lesions, … the recognizing unit 164A is configured using a first learned model learned by machine learning so as to perform a task of two classifications of an image suitable for recognition and an image unsuitable for recognition. … and determine whether or not the image is inappropriate for recognition using the calculated feature amount.”, Examiner interpreted “ the medical image” and “the acquired medical image” as one medical image. Also, “image recognition as classification of lesions region” interpreted as “object region”);
inputting the acquired medical image into a first model trained (The classification unit 170) for detecting the object region included in the medical image (Paragraphs 90-91: “ For the classification processing of the classification unit 170 shown in FIG. 4, for example, a convolutional neural network (CNN) is used. The classification unit 170 is configured using a second learned model learned by machine learning so as to perform an image classification task of classifying the image into a specific class … When the determination result obtained from the availability determination unit 164 is “an image suitable for recognition”, the classification unit 170 executes a classification process. The classification unit 170 extracts a feature amount from the image and classifies the image. The classification unit 170 may detect a region of interest (eg, a lesion region), detect a lesion region, and / or perform segmentation based on the calculated feature amount. Further, the classification unit 170 may perform the classification process using the feature amount calculated by the recognition unit 164A.” ; Paragraphs 101-104 : “If the availability determination unit 164 determines in the determination process of step S14 that the classification is possible, the process proceeds to steps S16 and S20. … in step S20, the classification unit 170 performs processing for recognizing a lesion area from within the image and classifying the lesion area into a predetermined class.”); and
outputting a warning when it is determined that the object region is not detected in the acquired medical image (Paragraphs 67-71: “Inappropriate for image recognition” means that the image is inappropriate for recognition such as classification of lesions, which is the main purpose. …. when the availability determination unit 164 determines that the classification is difficult, the process proceeds to the notification control unit 172 without performing the processes by the motion estimation unit 166, the behavior determination unit 168, and the classification unit 170.”)
However, Takenouchi does not disclose determining whether a ratio of a number of pixels in which the output values are within a predetermined range to a total number of pixels of the medical image is equal to or greater than a threshold;
when the ratio of the number of pixels in which the output values are within the predetermined range to the total number of pixels of the medical image is less than the threshold;
outputting when it is determined that the object region is not detected in the acquired medical image when the ratio of pixels in which the output values to the total number of pixels of the medical image is equal to or greater than the threshold.
Kanda discloses acquiring a medical image generated based on a signal detected by a catheter inserted into a luminal organ; (Paragraph 35: “the image processing apparatus 10 is integrated into the endoscope. The image processing apparatus 10 receives the intraluminal image captured by the endoscope and outputs an abnormal portion detection result obtained by processing the intraluminal image.”)
determining whether to execute object region detection processing on the medical image based on output values indicating a class classification for each pixel in the medical image (Paragraphs 38-41: “The validity evaluating unit 22 evaluates whether or not the approximate value is valid on the pixel value of the examination area. The validity evaluating unit 22 includes an evaluation value calculating unit 221 that calculates an evaluation value representing an approximation degree of the approximate value of the pixel value of the pixel of the examination area. … The abnormal portion detecting unit 25 detects the abnormal portion based on the pixel value of each pixel of the intraluminal image and the approximate value of each pixel that has been evaluated as valid by the validity evaluating unit 22.”) by determining whether a ratio of a number of pixels in which the output values are within a predetermined range to a total number of pixels of the medical image is equal to or greater than a threshold; (Fig.1: validity Evaluating unit 22, the abnormal portion detecting unit 25; Fig. 7-10; Paragraphs 62-75: “In step a7, the validity evaluating unit 22 executes a validity evaluating process and evaluates whether or not the approximate value in the examination area is valid. Next, in step c3, the variance value calculating unit 222 calculates the variance value .sigma., inside the examination area, of the difference between the approximate value z' calculated in step b13 of FIG. 7 and the actual pixel value z on each pixel of the examination area to be processed. … Next, the evaluation value calculating unit 221 sets the variance value calculated in step c3 as the evaluation value representing the approximation degree of the approximate value of the pixel value of the pixel of the examination area to be processed (step c5). Then, the validity evaluating unit 22 performs threshold processing on the evaluation value of the examination area to be processed, evaluates that the approximate value of the examination area to be processed is not valid when the evaluation value is larger than a predetermined threshold value set in advance, and evaluates that the approximate value of the examination area to be processed is valid when the evaluation value is equal to or less than the threshold value (step c7). … Meanwhile, when it is determined that the approximate values of all of the examination areas have been evaluated as valid (Yes in step a9), the abnormal portion detecting unit 25 executes an abnormal portion detecting process and detects the abnormal portion from the intraluminal image to be processed (step a15).”)
inputting the acquired medical image into a first model trained for detecting the object region included in the medical image when the ratio of the number of pixels in which the output values are within the predetermined range to the total number of pixels of the medical image is less than the threshold; (Paragraphs 37-41: “ The approximate value calculating unit 21 calculates an approximate value that becomes consecutive inside an examination area on a pixel value of each pixel of the examination area based on the pixel value of the pixel of the examination area. The examination area refers to the entire area of the intraluminal image or a partial area of the intraluminal image … The validity evaluating unit 22 evaluates whether or not the approximate value is valid on the pixel value of the examination area. The validity evaluating unit 22 includes an evaluation value calculating unit 221 that calculates an evaluation value representing an approximation degree of the approximate value of the pixel value of the pixel of the examination area. …. The abnormal portion detecting unit 25 detects the abnormal portion based on the pixel value of each pixel of the intraluminal image and the approximate value of each pixel that has been evaluated as valid by the validity evaluating unit 22.”; Paragraph 64: “the evaluation value calculating unit 221 sets the variance value calculated in step c3 as the evaluation value representing the approximation degree of the approximate value of the pixel value of the pixel of the examination area to be processed (step c5) … evaluates that the approximate value of the examination area to be processed is valid when the evaluation value is equal to or less than the threshold value (step c7).”; Paragraph 75: “when it is determined that the approximate values of all of the examination areas have been evaluated as valid (Yes in step a9), the abnormal portion detecting unit 25 executes an abnormal portion detecting process and detects the abnormal portion from the intraluminal image to be processed (step a15)”) and
outputting a warning when it is determined that the object region is not detected in the acquired medical image when the ratio of pixels in which the output values to the total number of pixels of the medical image is equal to or greater than the threshold. (Paragraphs 64-68: “the validity evaluating unit 22 performs threshold processing on the evaluation value of the examination area to be processed, evaluates that the approximate value of the examination area to be processed is not valid when the evaluation value is larger than a predetermined threshold value set in advance … the area dividing unit 23 executes a dividing process and divides the examination area in which the approximate value has been evaluated as invalid. … the examination areas in which the approximate value has been evaluated as invalid in the validity evaluating process of FIG. 8 executed immediately before are sequentially set as being processed”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Takenouchi by including the calculation unit is implemented by hardware such as a central processing unit (CPU) and performs a variety of calculation processes for detecting the abnormal portion from the intraluminal image that is taught by Kanda, to make the invention that an image processing apparatus for detecting an abnormal portion from an image; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving the determine the abnormal portion with a higher degree of accuracy such as the abnormal portion having the pixel value different from the neighboring pixels can be detected with a high degree of accuracy as well as reducing a doctor's burden on observation of an image of the inside of a lumen of the body.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claim 3, Takenouchi, as modified by Kanda, discloses all the claims invention. Takenouchi further discloses the determination information includes an output of an activation function included in the first model; and acquiring an output of the activation function included in the first model using the first model. (Paragraph 68: “The recognition unit 164A can be configured using, for example, a convolutional neural network (CNN). For example, the recognizing unit 164A is configured using a first learned model learned by machine learning so as to perform a task of two classifications of an image suitable for recognition and an image unsuitable for recognition. … and determine whether or not the image is inappropriate for recognition using the calculated feature amount.”, the person or one having ordinary skill in the art can understand that the output layer of CNN included activation function. )
Regarding claim 9, Takenouchi discloses an information processing device (Paragraph 5: “image processing that provides information that supports diagnosis by processing time-series medical images.”)comprising: a processor configured to:
acquire a medical image generated based on a signal detected by a catheter inserted into a luminal organ; (Paragraphs 51-53: “ while inserting the insertion section 20 of the electronic endoscope 12 into the body cavity and illuminating the inside of the body cavity with the illumination light from the light source device 14, an image of the inside of the body cavity captured by the imaging element 62 is displayed on the screen of the display device 18. … visualization in which a blood vessel in a specific depth region of the observation target is emphasized. In this mode, an image is generated and an image suitable for observing a blood vessel is displayed on the display device 18.”; Paragraph 99: “In step S11, the medical image processing apparatus 160 receives the current image via the image acquisition unit 162. The image acquired by the image acquisition unit 162 is a medical image including a subject image captured using the electronic endoscope 12, and is one image of a time-series image sequentially captured in a time-series manner.)
determine whether to execute object region detection processing on the medical image based on output values indicating a class classification for each pixel in the medical image (Paragraphs 100-101; Paragraphs 164-167: “the medical image analysis processing unit is a region that should be noted based on the feature amount of the pixel of the medical image. …The medical image analysis processing unit detects the presence or absence of a target to be noted based on the feature amount of the pixel of the medical image, and the medical image analysis result obtaining unit obtains an analysis result of the medical image analysis processing unit. “; Paragraphs 68: “the availability determination unit 164 determines whether the image acquired from the image acquisition unit 162 is an image inappropriate for recognition. The availability determination unit 164 includes a recognition unit 164A. The recognizing unit 164A recognizes whether the input image is appropriate or inappropriate for image recognition. Here, “appropriate for image recognition” means that the image is suitable for recognition processing for classifying lesions, which is the main purpose of recognition. “Inappropriate for image recognition” means that the image is inappropriate for recognition such as classification of lesions, … the recognizing unit 164A is configured using a first learned model learned by machine learning so as to perform a task of two classifications of an image suitable for recognition and an image unsuitable for recognition. … and determine whether or not the image is inappropriate for recognition using the calculated feature amount.”, Examiner interpreted “ the medical image” and “the acquired medical image” as one medical image. Also, “image recognition as classification of lesions region” interpreted as “object region”)
input the acquired medical image into a first model trained (The classification unit 170) for detecting the object region included in the medical image (Paragraphs 90-91: “ For the classification processing of the classification unit 170 shown in FIG. 4, for example, a convolutional neural network (CNN) is used. The classification unit 170 is configured using a second learned model learned by machine learning so as to perform an image classification task of classifying the image into a specific class … When the determination result obtained from the availability determination unit 164 is “an image suitable for recognition”, the classification unit 170 executes a classification process. The classification unit 170 extracts a feature amount from the image and classifies the image. The classification unit 170 may detect a region of interest (eg, a lesion region), detect a lesion region, and / or perform segmentation based on the calculated feature amount. Further, the classification unit 170 may perform the classification process using the feature amount calculated by the recognition unit 164A.” ; Paragraphs 101-104 : “If the availability determination unit 164 determines in the determination process of step S14 that the classification is possible, the process proceeds to steps S16 and S20. … in step S20, the classification unit 170 performs processing for recognizing a lesion area from within the image and classifying the lesion area into a predetermined class.”)
output a warning when it is determined that the object region is not detected in the acquired medical image (Paragraphs 67-71: “Inappropriate for image recognition” means that the image is inappropriate for recognition such as classification of lesions, which is the main purpose. …. when the availability determination unit 164 determines that the classification is difficult, the process proceeds to the notification control unit 172 without performing the processes by the motion estimation unit 166, the behavior determination unit 168, and the classification unit 170.”)
However, Takenouchi does not disclose determining whether a ratio of a number of pixels in which the output values are within a predetermined range to a total number of pixels of the medical image is equal to or greater than a threshold;
when the ratio of the number of pixels in which the output values are within the predetermined range to the total number of pixels of the medical image is less than the threshold;
outputting when it is determined that the object region is not detected in the acquired medical image when the ratio of pixels in which the output values to the total number of pixels of the medical image is equal to or greater than the threshold.
Kanda discloses acquire a medical image generated based on a signal detected by a catheter inserted into a luminal organ; (Paragraph 35: “the image processing apparatus 10 is integrated into the endoscope. The image processing apparatus 10 receives the intraluminal image captured by the endoscope and outputs an abnormal portion detection result obtained by processing the intraluminal image.”)
determine whether to execute object region detection processing on the medical image based on output values indicating a class classification for each pixel in the medical image (Paragraphs 38-41: “The validity evaluating unit 22 evaluates whether or not the approximate value is valid on the pixel value of the examination area. The validity evaluating unit 22 includes an evaluation value calculating unit 221 that calculates an evaluation value representing an approximation degree of the approximate value of the pixel value of the pixel of the examination area. … The abnormal portion detecting unit 25 detects the abnormal portion based on the pixel value of each pixel of the intraluminal image and the approximate value of each pixel that has been evaluated as valid by the validity evaluating unit 22.”) by determining whether a ratio of a number of pixels in which the output values are within a predetermined range to a total number of pixels of the medical image is equal to or greater than a threshold; (Fig.1: validity Evaluating unit 22, the abnormal portion detecting unit 25; Fig. 7-10; Paragraphs 62-75: “In step a7, the validity evaluating unit 22 executes a validity evaluating process and evaluates whether or not the approximate value in the examination area is valid. Next, in step c3, the variance value calculating unit 222 calculates the variance value .sigma., inside the examination area, of the difference between the approximate value z' calculated in step b13 of FIG. 7 and the actual pixel value z on each pixel of the examination area to be processed. … Next, the evaluation value calculating unit 221 sets the variance value calculated in step c3 as the evaluation value representing the approximation degree of the approximate value of the pixel value of the pixel of the examination area to be processed (step c5). Then, the validity evaluating unit 22 performs threshold processing on the evaluation value of the examination area to be processed, evaluates that the approximate value of the examination area to be processed is not valid when the evaluation value is larger than a predetermined threshold value set in advance, and evaluates that the approximate value of the examination area to be processed is valid when the evaluation value is equal to or less than the threshold value (step c7). … Meanwhile, when it is determined that the approximate values of all of the examination areas have been evaluated as valid (Yes in step a9), the abnormal portion detecting unit 25 executes an abnormal portion detecting process and detects the abnormal portion from the intraluminal image to be processed (step a15).”)
input the acquired medical image into a first model trained for detecting the object region included in the medical image when the ratio of the number of pixels in which the output values are within the predetermined range to the total number of pixels of the medical image is less than the threshold; (Paragraphs 37-41: “ The approximate value calculating unit 21 calculates an approximate value that becomes consecutive inside an examination area on a pixel value of each pixel of the examination area based on the pixel value of the pixel of the examination area. The examination area refers to the entire area of the intraluminal image or a partial area of the intraluminal image … The validity evaluating unit 22 evaluates whether or not the approximate value is valid on the pixel value of the examination area. The validity evaluating unit 22 includes an evaluation value calculating unit 221 that calculates an evaluation value representing an approximation degree of the approximate value of the pixel value of the pixel of the examination area. …. The abnormal portion detecting unit 25 detects the abnormal portion based on the pixel value of each pixel of the intraluminal image and the approximate value of each pixel that has been evaluated as valid by the validity evaluating unit 22.”; Paragraph 64: “the evaluation value calculating unit 221 sets the variance value calculated in step c3 as the evaluation value representing the approximation degree of the approximate value of the pixel value of the pixel of the examination area to be processed (step c5) … evaluates that the approximate value of the examination area to be processed is valid when the evaluation value is equal to or less than the threshold value (step c7).”; Paragraph 75: “when it is determined that the approximate values of all of the examination areas have been evaluated as valid (Yes in step a9), the abnormal portion detecting unit 25 executes an abnormal portion detecting process and detects the abnormal portion from the intraluminal image to be processed (step a15)”) and
outputting when it is determined that the object region is not detected in the acquired medical image when the ratio of pixels in which the output values to the total number of pixels of the medical image is equal to or greater than the threshold. (Paragraphs 64-68: “the validity evaluating unit 22 performs threshold processing on the evaluation value of the examination area to be processed, evaluates that the approximate value of the examination area to be processed is not valid when the evaluation value is larger than a predetermined threshold value set in advance … the area dividing unit 23 executes a dividing process and divides the examination area in which the approximate value has been evaluated as invalid. … the examination areas in which the approximate value has been evaluated as invalid in the validity evaluating process of FIG. 8 executed immediately before are sequentially set as being processed”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Takenouchi by including the calculation unit is implemented by hardware such as a central processing unit (CPU) and performs a variety of calculation processes for detecting the abnormal portion from the intraluminal image that is taught by Kanda, to make the invention that an image processing apparatus for detecting an abnormal portion from an image; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving the determine the abnormal portion with a higher degree of accuracy such as the abnormal portion having the pixel value different from the neighboring pixels can be detected with a high degree of accuracy as well as reducing a doctor's burden on observation of an image of the inside of a lumen of the body.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claim 11, Takenouchi, as modified by Kanda, discloses all the claims invention. Takenouchi further discloses the determination information includes an output of an activation function included in the first model; and the processor is configured to acquire an output of the activation function included in the first model using the first model. (Paragraph 68: “The recognition unit 164A can be configured using, for example, a convolutional neural network (CNN). For example, the recognizing unit 164A is configured using a first learned model learned by machine learning so as to perform a task of two classifications of an image suitable for recognition and an image unsuitable for recognition. … and determine whether or not the image is inappropriate for recognition using the calculated feature amount.”, the person or one having ordinary skill in the art can understand that the output layer of CNN included activation function. )
Regarding claim 15, Takenouchi discloses an information processing method (Paragraph 5: “image processing that provides information that supports diagnosis by processing time-series medical images.”) comprising:
acquiring a medical image generated based on a signal detected by a catheter inserted into a luminal organ; (Paragraphs 51-53: “ while inserting the insertion section 20 of the electronic endoscope 12 into the body cavity and illuminating the inside of the body cavity with the illumination light from the light source device 14, an image of the inside of the body cavity captured by the imaging element 62 is displayed on the screen of the display device 18. … visualization in which a blood vessel in a specific depth region of the observation target is emphasized. In this mode, an image is generated and an image suitable for observing a blood vessel is displayed on the display device 18.”; Paragraph 99: “In step S11, the medical image processing apparatus 160 receives the current image via the image acquisition unit 162. The image acquired by the image acquisition unit 162 is a medical image including a subject image captured using the electronic endoscope 12, and is one image of a time-series image sequentially captured in a time-series manner.)
determining whether to execute object region detection processing on the medical image based on output values indicating a class classification for each pixel in the medical image (Paragraphs 100-101; Paragraphs 164-167: “the medical image analysis processing unit is a region that should be noted based on the feature amount of the pixel of the medical image. …The medical image analysis processing unit detects the presence or absence of a target to be noted based on the feature amount of the pixel of the medical image, and the medical image analysis result obtaining unit obtains an analysis result of the medical image analysis processing unit. “; Paragraphs 68: “the availability determination unit 164 determines whether the image acquired from the image acquisition unit 162 is an image inappropriate for recognition. The availability determination unit 164 includes a recognition unit 164A. The recognizing unit 164A recognizes whether the input image is appropriate or inappropriate for image recognition. Here, “appropriate for image recognition” means that the image is suitable for recognition processing for classifying lesions, which is the main purpose of recognition. “Inappropriate for image recognition” means that the image is inappropriate for recognition such as classification of lesions, … the recognizing unit 164A is configured using a first learned model learned by machine learning so as to perform a task of two classifications of an image suitable for recognition and an image unsuitable for recognition. … and determine whether or not the image is inappropriate for recognition using the calculated feature amount.”, Examiner interpreted “ the medical image” and “the acquired medical image” as one medical image. Also, “image recognition as classification of lesions region” interpreted as “object region”)
inputting the acquired medical image into a first model trained (The classification unit 170) for detecting the object region included in the medical image (Paragraphs 90-91: “ For the classification processing of the classification unit 170 shown in FIG. 4, for example, a convolutional neural network (CNN) is used. The classification unit 170 is configured using a second learned model learned by machine learning so as to perform an image classification task of classifying the image into a specific class … When the determination result obtained from the availability determination unit 164 is “an image suitable for recognition”, the classification unit 170 executes a classification process. The classification unit 170 extracts a feature amount from the image and classifies the image. The classification unit 170 may detect a region of interest (eg, a lesion region), detect a lesion region, and / or perform segmentation based on the calculated feature amount. Further, the classification unit 170 may perform the classification process using the feature amount calculated by the recognition unit 164A.” ; Paragraphs 101-104 : “If the availability determination unit 164 determines in the determination process of step S14 that the classification is possible, the process proceeds to steps S16 and S20. … in step S20, the classification unit 170 performs processing for recognizing a lesion area from within the image and classifying the lesion area into a predetermined class.”) and
outputting a warning when it is determined that the object region is not detected in the acquired medical image (Paragraphs 67-71: “Inappropriate for image recognition” means that the image is inappropriate for recognition such as classification of lesions, which is the main purpose. …. when the availability determination unit 164 determines that the classification is difficult, the process proceeds to the notification control unit 172 without performing the processes by the motion estimation unit 166, the behavior determination unit 168, and the classification unit 170.”)
However, Takenouchi does not disclose determining whether a ratio of a number of pixels in which the output values are within a predetermined range to a total number of pixels of the medical image is equal to or greater than a threshold;
when the ratio of the number of pixels in which the output values are within the predetermined range to the total number of pixels of the medical image is less than the threshold;
outputting when it is determined that the object region is not detected in the acquired medical image when the ratio of pixels in which the output values to the total number of pixels of the medical image is equal to or greater than the threshold.
Kanda discloses acquiring a medical image generated based on a signal detected by a catheter inserted into a luminal organ; (Paragraph 35: “the image processing apparatus 10 is integrated into the endoscope. The image processing apparatus 10 receives the intraluminal image captured by the endoscope and outputs an abnormal portion detection result obtained by processing the intraluminal image.”)
determining whether to execute object region detection processing on the medical image based on output values indicating a class classification for each pixel in the medical image (Paragraphs 38-41: “The validity evaluating unit 22 evaluates whether or not the approximate value is valid on the pixel value of the examination area. The validity evaluating unit 22 includes an evaluation value calculating unit 221 that calculates an evaluation value representing an approximation degree of the approximate value of the pixel value of the pixel of the examination area. … The abnormal portion detecting unit 25 detects the abnormal portion based on the pixel value of each pixel of the intraluminal image and the approximate value of each pixel that has been evaluated as valid by the validity evaluating unit 22.”) by determining whether a ratio of a number of pixels in which the output values are within a predetermined range to a total number of pixels of the medical image is equal to or greater than a threshold; (Fig.1: validity Evaluating unit 22, the abnormal portion detecting unit 25; Fig. 7-10; Paragraphs 62-75: “In step a7, the validity evaluating unit 22 executes a validity evaluating process and evaluates whether or not the approximate value in the examination area is valid. Next, in step c3, the variance value calculating unit 222 calculates the variance value .sigma., inside the examination area, of the difference between the approximate value z' calculated in step b13 of FIG. 7 and the actual pixel value z on each pixel of the examination area to be processed. … Next, the evaluation value calculating unit 221 sets the variance value calculated in step c3 as the evaluation value representing the approximation degree of the approximate value of the pixel value of the pixel of the examination area to be processed (step c5). Then, the validity evaluating unit 22 performs threshold processing on the evaluation value of the examination area to be processed, evaluates that the approximate value of the examination area to be processed is not valid when the evaluation value is larger than a predetermined threshold value set in advance, and evaluates that the approximate value of the examination area to be processed is valid when the evaluation value is equal to or less than the threshold value (step c7). … Meanwhile, when it is determined that the approximate values of all of the examination areas have been evaluated as valid (Yes in step a9), the abnormal portion detecting unit 25 executes an abnormal portion detecting process and detects the abnormal portion from the intraluminal image to be processed (step a15).”)
inputting the acquired medical image into a first model trained for detecting the object region included in the medical image when the ratio of the number of pixels in which the output values are within the predetermined range to the total number of pixels of the medical image is less than the threshold; (Paragraphs 37-41: “ The approximate value calculating unit 21 calculates an approximate value that becomes consecutive inside an examination area on a pixel value of each pixel of the examination area based on the pixel value of the pixel of the examination area. The examination area refers to the entire area of the intraluminal image or a partial area of the intraluminal image … The validity evaluating unit 22 evaluates whether or not the approximate value is valid on the pixel value of the examination area. The validity evaluating unit 22 includes an evaluation value calculating unit 221 that calculates an evaluation value representing an approximation degree of the approximate value of the pixel value of the pixel of the examination area. …. The abnormal portion detecting unit 25 detects the abnormal portion based on the pixel value of each pixel of the intraluminal image and the approximate value of each pixel that has been evaluated as valid by the validity evaluating unit 22.”; Paragraph 64: “the evaluation value calculating unit 221 sets the variance value calculated in step c3 as the evaluation value representing the approximation degree of the approximate value of the pixel value of the pixel of the examination area to be processed (step c5) … evaluates that the approximate value of the examination area to be processed is valid when the evaluation value is equal to or less than the threshold value (step c7).”; Paragraph 75: “when it is determined that the approximate values of all of the examination areas have been evaluated as valid (Yes in step a9), the abnormal portion detecting unit 25 executes an abnormal portion detecting process and detects the abnormal portion from the intraluminal image to be processed (step a15)”) and
outputting when it is determined that the object region is not detected in the acquired medical image when the ratio of pixels in which the output values to the total number of pixels of the medical image is equal to or greater than the threshold. (Paragraphs 64-68: “the validity evaluating unit 22 performs threshold processing on the evaluation value of the examination area to be processed, evaluates that the approximate value of the examination area to be processed is not valid when the evaluation value is larger than a predetermined threshold value set in advance … the area dividing unit 23 executes a dividing process and divides the examination area in which the approximate value has been evaluated as invalid. … the examination areas in which the approximate value has been evaluated as invalid in the validity evaluating process of FIG. 8 executed immediately before are sequentially set as being processed”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Takenouchi by including the calculation unit is implemented by hardware such as a central processing unit (CPU) and performs a variety of calculation processes for detecting the abnormal portion from the intraluminal image that is taught by Kanda, to make the invention that an image processing apparatus for detecting an abnormal portion from an image; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving the determine the abnormal portion with a higher degree of accuracy such as the abnormal portion having the pixel value different from the neighboring pixels can be detected with a high degree of accuracy as well as reducing a doctor's burden on observation of an image of the inside of a lumen of the body.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claim 17, Takenouchi, as modified by Kanda, discloses all the claims invention. Takenouchi further discloses the determination information includes an output of an activation function included in the first model, and the method further comprises: acquiring an output of an activation function included in the first model using the first model. (Paragraph 68: “The recognition unit 164A can be configured using, for example, a convolutional neural network (CNN). For example, the recognizing unit 164A is configured using a first learned model learned by machine learning so as to perform a task of two classifications of an image suitable for recognition and an image unsuitable for recognition. … and determine whether or not the image is inappropriate for recognition using the calculated feature amount.”, the person or one having ordinary skill in the art can understand that the output layer of CNN included activation function. )
Regarding claim 21, Takenouchi, as modified by Kanda, discloses all the claims invention. Takenouchi further discloses further comprising: displaying the acquired medical image generated based on the signal detected by the catheter inserted into the luminal organ on a screen of a display device. (Paragraph 21: “The image data converted by the processor device 16 is displayed on the display device 18 as an endoscopic photographed image (observed image).”; (Paragraphs 51-53: “ while inserting the insertion section 20 of the electronic endoscope 12 into the body cavity and illuminating the inside of the body cavity with the illumination light from the light source device 14, an image of the inside of the body cavity captured by the imaging element 62 is displayed on the screen of the display device 18. … visualization in which a blood vessel in a specific depth region of the observation target is emphasized. In this mode, an image is generated and an image suitable for observing a blood vessel is displayed on the display device 18.”;)
Regarding claim 22, Takenouchi, as modified by Kanda, discloses all the claims invention. Takenouchi further discloses further comprising: outputting the warning indicating that the object region is not detected from the medical image on a screen of a display device or via an audio warning. (Paragraphs 67-71: “Inappropriate for image recognition” means that the image is inappropriate for recognition such as classification of lesions, which is the main purpose. …. when the availability determination unit 164 determines that the classification is difficult, the process proceeds to the notification control unit 172 without performing the processes by the motion estimation unit 166, the behavior determination unit 168, and the classification unit 170.”; Paragraph 61: “The notification control unit 172 may include the display control circuit 78 described with reference to FIG.4”)
Claim(s) 4-5, 12-13, 18-19 and 23-26 is/are rejected under 35 U.S.C. 103 as being unpatentable over Takenouchi (WO-2020054543 A1), in view of Kanda et al (U.S. 20120051640; Kanda), and in further view of Hsieh et al (U.S. 20190340470 A1; Hsieh).
Regarding claim 4, Takenouchi, as modified by Kanda, discloses all the claims invention except wherein the determination information includes a presence or an absence of an artifact in the medical image; and acquiring the presence or the absence of the artifact in the medical image by inputting the acquired medical image into a third model trained for detecting the presence or the absence of the artifact in the medical image.
Hsieh discloses the determination information includes presence or absence of an artifact in the medical image; and acquiring the presence or absence of the artifact in the medical image by inputting the acquired medical image into a third model trained for detecting the presence or absence of the artifact in the medical image. (Paragraph 145: “each DDLD 1522, 1532, 1542 determines a signature. For example, the DDLD 1522, 1532, 1542 determines signature(s) for machine (e.g., imaging device 1410, information subsystem 1420, etc.) service issues, clinical issues related to patient health, noise texture issues, artifact issues, etc.”; Paragraph 230: “a regression and/or classification method can be used to generate image quality metrics by labeling the training data with an absolute value and/or level of the corresponding image IQ metric. That is, metrics can include quantitative measures of image quality (e.g., noise level, detectability, etc.), descriptive measures of image quality (e.g., Likert score, etc.), a classification of image quality (e.g., whether the image is diagnostic or not, has artifacts or not, etc.)”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Takenouchi and Kanda by including learning and testing/evaluation phases for an image quality deep learning network that is taught by Hsieh, to make the invention that image quality assessment and feedback using a deployed network model; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving the determine image quality and reconstruction feedback based on acquired image data as well as improve operation of imaging and/or other healthcare systems using a plurality of deep learning and/or other machine learning techniques.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claim 5, Takenouchi, as modified by Kanda and Hsieh, discloses all the claims invention. Hsieh further discloses the third model includes a model trained by unsupervised learning using a medical image with no artifact. (Paragraph 214: “hyper parameters are tuned by inputting unlabeled images 2221 to the unsupervised learning layer 2213 and then to the supervised learning layers of the convolutional network 2215. After classification 2217, one or more image quality indices 2219 are generated”; Paragraph 230: “a regression and/or classification method can be used to generate image quality metrics by labeling the training data with an absolute value and/or level of the corresponding image IQ metric. That is, metrics can include quantitative measures of image quality (e.g., noise level, detectability, etc.), descriptive measures of image quality (e.g., Likert score, etc.), a classification of image quality (e.g., whether the image is diagnostic or not, has artifacts or not, etc.)”)
Regarding claim 12, Takenouchi, as modified by Kanda, discloses all the claims invention except the determination information includes a presence or an absence of an artifact in the medical image; and the processor is configured to acquire the presence or the absence of the artifact in the medical image by inputting the acquired medical image into a third model trained for detecting the presence or the absence of the artifact in the medical image.
Hsieh discloses the determination information includes presence or absence of an artifact in the medical image; and acquiring the presence or absence of the artifact in the medical image by inputting the acquired medical image into a third model trained for detecting the presence or absence of the artifact in the medical image. (Paragraph 145: “each DDLD 1522, 1532, 1542 determines a signature. For example, the DDLD 1522, 1532, 1542 determines signature(s) for machine (e.g., imaging device 1410, information subsystem 1420, etc.) service issues, clinical issues related to patient health, noise texture issues, artifact issues, etc.”; Paragraph 230: “a regression and/or classification method can be used to generate image quality metrics by labeling the training data with an absolute value and/or level of the corresponding image IQ metric. That is, metrics can include quantitative measures of image quality (e.g., noise level, detectability, etc.), descriptive measures of image quality (e.g., Likert score, etc.), a classification of image quality (e.g., whether the image is diagnostic or not, has artifacts or not, etc.)”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Takenouchi and Kanda by including learning and testing/evaluation phases for an image quality deep learning network that is taught by Hsieh, to make the invention that image quality assessment and feedback using a deployed network model; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving the determine image quality and reconstruction feedback based on acquired image data as well as improve operation of imaging and/or other healthcare systems using a plurality of deep learning and/or other machine learning techniques.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claim 13, Takenouchi, as modified by Kanda and Hsieh, discloses all the claims invention. Hsieh further discloses the third model includes a model trained by unsupervised learning using a medical image with no artifact. (Paragraph 214: “hyper parameters are tuned by inputting unlabeled images 2221 to the unsupervised learning layer 2213 and then to the supervised learning layers of the convolutional network 2215. After classification 2217, one or more image quality indices 2219 are generated”; Paragraph 230: “a regression and/or classification method can be used to generate image quality metrics by labeling the training data with an absolute value and/or level of the corresponding image IQ metric. That is, metrics can include quantitative measures of image quality (e.g., noise level, detectability, etc.), descriptive measures of image quality (e.g., Likert score, etc.), a classification of image quality (e.g., whether the image is diagnostic or not, has artifacts or not, etc.)”)
Regarding claim 18, Takenouchi, as modified by Kanda, discloses all the claims invention except the determination information includes a presence or an absence of an artifact in the medical image, and the method further comprises: acquiring the presence or the absence of the artifact in the medical image by inputting the acquired medical image into a third model trained for detecting the presence or the absence of the artifact in the medical image.
Hsieh discloses the determination information includes presence or absence of an artifact in the medical image; and acquiring the presence or absence of the artifact in the medical image by inputting the acquired medical image into a third model trained for detecting the presence or absence of the artifact in the medical image. (Paragraph 145: “each DDLD 1522, 1532, 1542 determines a signature. For example, the DDLD 1522, 1532, 1542 determines signature(s) for machine (e.g., imaging device 1410, information subsystem 1420, etc.) service issues, clinical issues related to patient health, noise texture issues, artifact issues, etc.”; Paragraph 230: “a regression and/or classification method can be used to generate image quality metrics by labeling the training data with an absolute value and/or level of the corresponding image IQ metric. That is, metrics can include quantitative measures of image quality (e.g., noise level, detectability, etc.), descriptive measures of image quality (e.g., Likert score, etc.), a classification of image quality (e.g., whether the image is diagnostic or not, has artifacts or not, etc.)”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Takenouchi and Kanda by including learning and testing/evaluation phases for an image quality deep learning network that is taught by Hsieh, to make the invention that image quality assessment and feedback using a deployed network model; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving the determine image quality and reconstruction feedback based on acquired image data as well as improve operation of imaging and/or other healthcare systems using a plurality of deep learning and/or other machine learning techniques.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claim 19, Takenouchi, as modified by Kanda and Hsieh, discloses all the claims invention. Hsieh further discloses the third model includes a model trained by unsupervised learning using a medical image with no artifact. (Paragraph 214: “hyper parameters are tuned by inputting unlabeled images 2221 to the unsupervised learning layer 2213 and then to the supervised learning layers of the convolutional network 2215. After classification 2217, one or more image quality indices 2219 are generated”; Paragraph 230: “a regression and/or classification method can be used to generate image quality metrics by labeling the training data with an absolute value and/or level of the corresponding image IQ metric. That is, metrics can include quantitative measures of image quality (e.g., noise level, detectability, etc.), descriptive measures of image quality (e.g., Likert score, etc.), a classification of image quality (e.g., whether the image is diagnostic or not, has artifacts or not, etc.)”)
Regarding claim 24, Takenouchi, as modified by Kanda, discloses all the claims invention except wherein comprising: inputting the acquired medical image into a second model trained for outputting an evaluation index regarding detection accuracy of the object region included in the medical image.
Hsieh discloses acquiring detection accuracy of the object region included in the medical image by inputting the acquired medical image into a second model (the convolutional network 2245) trained for outputting the detection accuracy of the object region included in the medical image. (Fig. 22B; Paragraph 217: “hyper parameters are tuned by inputting unlabeled images 2251 to the learning layers of the convolutional network 2245. After classification 2247, one or more image quality indices 2259 are generated.”; Paragraph 203: “image quality is generated for computer analysis as a change in probabilistic values of image classification. On a scale of 1-5, for example, a 3 indicates the image is diagnosable (e.g., is of diagnostic quality), a 5 indicates a perfect image (e.g., probably at too high of a dose), and a 1 indicates the image data is not usable for diagnosis. As a result, a preferred score is 3-4. The DDLD 1532 can generate an IQI based on acquired image data by mimicking radiologist behavior and the 1-5 scale. Using image data attributes, the DDLD 1532 can analyze an image and determine features (e.g., a small lesion) and evaluate diagnostic quality of each feature in the image data”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Takenouchi and Kanda by including learning and testing/evaluation phases for an image quality deep learning network that is taught by Hsieh, to make the invention that image quality assessment and feedback using a deployed network model; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving the determine image quality and reconstruction feedback based on acquired image data as well as improve operation of imaging and/or other healthcare systems using a plurality of deep learning and/or other machine learning techniques.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claim 23, Takenouchi, as modified by Kanda and Hsieh, discloses all the claims invention. Hsieh further discloses the second model includes an input layer to which the medical image is input, an intermediate layer that extracts a feature amount of the medical image, and an output layer that outputs output data indicating the evaluation index for the medical image; (Figs. 1-2; Paragraph 73: “ neural network 100 includes layers 120, 140, 160, and 180. The layers 120 and 140 are connected with neural connections 130. …Data flows forward via inputs 112, 114, 116 from the input layer 120 to the output layer 180 and to an output 190.”; Paragraph 208: “the convolution 2120 is applied to the input images 2110 to generate a feature map, and pooling 2130 reduces image size to isolate portions 2125 of the images 2110 including features of interest to form a fully connected layer 2140. A classifier 2150 (e.g., a softmax classifier, etc.) associates weights with nodes representing features of interest. The classifier 2150 provides weighted features that can be used to generate a known image quality index 2160.”) and the input layer includes a plurality of nodes that receive an input of a pixel value of each pixel included in the medical image, and deliver the input pixel value to the intermediate layer, the intermediate layer includes a plurality of nodes that extract a feature amount of input data and deliver the feature amount extracted using various parameters to the output layer, and the output layer outputs continuous values indicating the evaluation index. (Figs. 1-2; Paragraph 74: “ The layer 120 is an input layer that, in the example of FIG. 1, includes a plurality of nodes 122, 124, 126. The layers 140 and 160 are hidden layers and include, the example of FIG. 1, nodes 142, 144, 146, 148, 162, 164, 166, 168. … the layer 180 is an output layer and includes, in the example of FIG. 1A, a node 182 with an output 190.”; Figs. 21A-21B; Paragraph 208: “FIGS. 21A-21B illustrate example learning and testing/evaluation phases for an image quality deep learning network. As shown in the example of FIG. 21A, known, labeled images 2110 are applied to a convolution network 2120. The images 2110 are obtained using multiple users, and their image quality indices are known. As discussed above with respect to FIGS. 1-3, the convolution 2120 is applied to the input images 2110 to generate a feature map, and pooling 2130 reduces image size to isolate portions 2125 of the images 2110 including features of interest to form a fully connected layer 2140. A classifier 2150 (e.g., a softmax classifier, etc.) associates weights with nodes representing features of interest. The classifier 2150 provides weighted features that can be used to generate a known image quality index 2160.”; Paragraph 211)
Regarding claim 25, Takenouchi, as modified by Kanda, discloses all the claims invention except wherein the processor is configured to: input the acquired medical image into a second model trained for outputting an evaluation index regarding detection accuracy of the object region included in the medical image, and wherein the second model includes an input layer to which the medical image is input, an intermediate layer that extracts a feature amount of the medical image, and an output layer that outputs output data indicating the evaluation index for the medical image; and the input layer includes a plurality of nodes that receive an input of a pixel value of each pixel included in the medical image, and deliver the input pixel value to the intermediate layer, the intermediate layer includes a plurality of nodes that extract a feature amount of input data and deliver the feature amount extracted using various parameters to the output layer, and the output layer outputs continuous values indicating the evaluation index.
Hsieh discloses the processor is configured to: input the acquired medical image into a second model trained(the convolutional network 2245) for outputting an evaluation index regarding detection accuracy of the object region included in the medical image, (Fig. 22B; Paragraph 217: “hyper parameters are tuned by inputting unlabeled images 2251 to the learning layers of the convolutional network 2245. After classification 2247, one or more image quality indices 2259 are generated.”; Paragraph 203: “image quality is generated for computer analysis as a change in probabilistic values of image classification. On a scale of 1-5, for example, a 3 indicates the image is diagnosable (e.g., is of diagnostic quality), a 5 indicates a perfect image (e.g., probably at too high of a dose), and a 1 indicates the image data is not usable for diagnosis. As a result, a preferred score is 3-4. The DDLD 1532 can generate an IQI based on acquired image data by mimicking radiologist behavior and the 1-5 scale. Using image data attributes, the DDLD 1532 can analyze an image and determine features (e.g., a small lesion) and evaluate diagnostic quality of each feature in the image data”) and wherein the second model includes an input layer to which the medical image is input, an intermediate layer that extracts a feature amount of the medical image, and an output layer that outputs output data indicating the evaluation index for the medical image; (Figs. 1-2; Paragraph 74: “ The layer 120 is an input layer that, in the example of FIG. 1, includes a plurality of nodes 122, 124, 126. The layers 140 and 160 are hidden layers and include, the example of FIG. 1, nodes 142, 144, 146, 148, 162, 164, 166, 168. … the layer 180 is an output layer and includes, in the example of FIG. 1A, a node 182 with an output 190.”; Paragraph 208: “the convolution 2120 is applied to the input images 2110 to generate a feature map, and pooling 2130 reduces image size to isolate portions 2125 of the images 2110 including features of interest to form a fully connected layer 2140. A classifier 2150 (e.g., a softmax classifier, etc.) associates weights with nodes representing features of interest. The classifier 2150 provides weighted features that can be used to generate a known image quality index 2160.”) and the input layer includes a plurality of nodes that receive an input of a pixel value of each pixel included in the medical image, and deliver the input pixel value to the intermediate layer, the intermediate layer includes a plurality of nodes that extract a feature amount of input data and deliver the feature amount extracted using various parameters to the output layer, and the output layer outputs continuous values indicating the evaluation index. (Figs. 1-2; Paragraph 73: “ neural network 100 includes layers 120, 140, 160, and 180. The layers 120 and 140 are connected with neural connections 130. …Data flows forward via inputs 112, 114, 116 from the input layer 120 to the output layer 180 and to an output 190.”Figs. 21A-21B; Paragraph 208: “FIGS. 21A-21B illustrate example learning and testing/evaluation phases for an image quality deep learning network. As shown in the example of FIG. 21A, known, labeled images 2110 are applied to a convolution network 2120. The images 2110 are obtained using multiple users, and their image quality indices are known. As discussed above with respect to FIGS. 1-3, the convolution 2120 is applied to the input images 2110 to generate a feature map, and pooling 2130 reduces image size to isolate portions 2125 of the images 2110 including features of interest to form a fully connected layer 2140. A classifier 2150 (e.g., a softmax classifier, etc.) associates weights with nodes representing features of interest. The classifier 2150 provides weighted features that can be used to generate a known image quality index 2160.”; Paragraph 211)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Takenouchi and Kanda by including learning and testing/evaluation phases for an image quality deep learning network that is taught by Hsieh, to make the invention that image quality assessment and feedback using a deployed network model; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving the determine image quality and reconstruction feedback based on acquired image data as well as improve operation of imaging and/or other healthcare systems using a plurality of deep learning and/or other machine learning techniques.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claim 26, Takenouchi, as modified by Kanda, discloses all the claims invention except further comprising: inputting the acquired medical image into a second model trained for outputting an evaluation index regarding detection accuracy of the object region included in the medical image, and wherein the second model includes an input layer to which the medical image is input, an intermediate layer that extracts a feature amount of the medical image, and an output layer that outputs output data indicating the evaluation index for the medical image; and the input layer includes a plurality of nodes that receive an input of a pixel value of each pixel included in the medical image, and deliver the input pixel value to the intermediate layer, the intermediate layer includes a plurality of nodes that extract a feature amount of input data and deliver the feature amount extracted using various parameters to the output layer, and the output layer outputs continuous values indicating the evaluation index.
Hsieh discloses further comprising: inputting the acquired medical image into a second model trained (the convolutional network 2245) for outputting an evaluation index regarding detection accuracy of the object region included in the medical image, (Fig. 22B; Paragraph 217: “hyper parameters are tuned by inputting unlabeled images 2251 to the learning layers of the convolutional network 2245. After classification 2247, one or more image quality indices 2259 are generated.”; Paragraph 203: “image quality is generated for computer analysis as a change in probabilistic values of image classification. On a scale of 1-5, for example, a 3 indicates the image is diagnosable (e.g., is of diagnostic quality), a 5 indicates a perfect image (e.g., probably at too high of a dose), and a 1 indicates the image data is not usable for diagnosis. As a result, a preferred score is 3-4. The DDLD 1532 can generate an IQI based on acquired image data by mimicking radiologist behavior and the 1-5 scale. Using image data attributes, the DDLD 1532 can analyze an image and determine features (e.g., a small lesion) and evaluate diagnostic quality of each feature in the image data”) and wherein the second model includes an input layer to which the medical image is input, an intermediate layer that extracts a feature amount of the medical image, and an output layer that outputs output data indicating the evaluation index for the medical image; (Figs. 1-2; Paragraph 74: “ The layer 120 is an input layer that, in the example of FIG. 1, includes a plurality of nodes 122, 124, 126. The layers 140 and 160 are hidden layers and include, the example of FIG. 1, nodes 142, 144, 146, 148, 162, 164, 166, 168. … the layer 180 is an output layer and includes, in the example of FIG. 1A, a node 182 with an output 190.”; Paragraph 208: “the convolution 2120 is applied to the input images 2110 to generate a feature map, and pooling 2130 reduces image size to isolate portions 2125 of the images 2110 including features of interest to form a fully connected layer 2140. A classifier 2150 (e.g., a softmax classifier, etc.) associates weights with nodes representing features of interest. The classifier 2150 provides weighted features that can be used to generate a known image quality index 2160.”) and the input layer includes a plurality of nodes that receive an input of a pixel value of each pixel included in the medical image, and deliver the input pixel value to the intermediate layer, the intermediate layer includes a plurality of nodes that extract a feature amount of input data and deliver the feature amount extracted using various parameters to the output layer, and the output layer outputs continuous values indicating the evaluation index. (Figs. 1-2; Paragraph 73: “ neural network 100 includes layers 120, 140, 160, and 180. The layers 120 and 140 are connected with neural connections 130. …Data flows forward via inputs 112, 114, 116 from the input layer 120 to the output layer 180 and to an output 190.”Figs. 21A-21B; Paragraph 208: “FIGS. 21A-21B illustrate example learning and testing/evaluation phases for an image quality deep learning network. As shown in the example of FIG. 21A, known, labeled images 2110 are applied to a convolution network 2120. The images 2110 are obtained using multiple users, and their image quality indices are known. As discussed above with respect to FIGS. 1-3, the convolution 2120 is applied to the input images 2110 to generate a feature map, and pooling 2130 reduces image size to isolate portions 2125 of the images 2110 including features of interest to form a fully connected layer 2140. A classifier 2150 (e.g., a softmax classifier, etc.) associates weights with nodes representing features of interest. The classifier 2150 provides weighted features that can be used to generate a known image quality index 2160.”; Paragraph 211)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Takenouchi and Kanda by including learning and testing/evaluation phases for an image quality deep learning network that is taught by Hsieh, to make the invention that image quality assessment and feedback using a deployed network model; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving the determine image quality and reconstruction feedback based on acquired image data as well as improve operation of imaging and/or other healthcare systems using a plurality of deep learning and/or other machine learning techniques.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Podilchuk et al (U.S. 20180053300 A1), “Method and System of Computer-Aided Detection Using Multiple Images From Different Views of a Region of Interest to Improve Detection Accuracy”, teaches about A system and method of computer-aided detection (CAD or CADe) of medical images that utilizes persistence between images of a sequence to identify regions of interest detected with low interference from artifacts to reduce false positives and improve probability of detection of true lesions, thereby providing improved performance over static CADe methods for automatic ROI lesion detection.
Kamiyama et al (U.S. 20180070798 A1), “Image Processing Apparatus, Image Processing Method , and Computer-Readable Recording Medium”, teaches about an image processing method includes: detecting, from an image acquired by imaging inside a lumen of a living body, a candidate region for a specific region that is a region where a specific part in the lumen has been captured; acquiring information related to the detected candidate region; determining an identification means for identification of, based on the information related to the candidate region, whether or not the candidate region is the specific region; and identifying whether or not the candidate region is the specific region by using the determined identification means.
Liang et al (U.S. 20180225820 A1), “Method, Systems, and Media for Simultaneously Monitoring Colonoscopic Video Quality And Detecting Polyps in Colonoscopy”, teaches about the mechanisms can include a quality monitoring system that uses a first trained classifier to monitor image frames from a colonoscopic video to determine which image frames are informative frames and which image frames are non-informative frames. The informative image frames can be passed to an automatic polyp detection system that uses a second trained classifier to localize and identify whether a polyp or any other suitable object is present in one or more of the informative image frames.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Duy A Tran whose telephone number is (571)272-4887. The examiner can normally be reached Monday-Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ONEAL R MISTRY can be reached at (313)-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUY TRAN/ Examiner, Art Unit 2674
/ONEAL R MISTRY/ Supervisory Patent Examiner, Art Unit 2674
Prosecution Insights