DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 10/25/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 6-11 and 16-18 are rejected under 35 U.S.C. 103 as being unpatentable over Cai et al. US PG-Pub(US 20220399101 A1) in view of Takahashi et al. US PG-Pub(US 20210358129 A1).
Regarding Claim 1, Cai teaches an image processing apparatus(Fig.1) comprising: a processor(¶[0111] discloses a processor) configured to: output a first output image based on a first feature map extracted by inputting a learning input image to a first sub-model in a learning model including the first sub-model and a second sub-model(¶[0007], “receiving, by processing circuitry, a first medical image having a first resolution, applying, by the processing circuitry, a neural network to the first medical image, the neural network including a first subset of layers and, subsequently, a second subset of layers”, ¶[0007] discloses inputting the medical image into a neural network which has a first layer for processing and a second layer as well.); output a second output image having a higher resolution than the first output image(¶[0007] discloses “wherein the first resolution is lower than the second resolution” the second images resolution is higher than the first resolution of the first image.), based on a second feature map extracted by inputting the first feature map to the second sub-model(¶[0007], “the first subset of layers of the neural network generating, from the first medical image, a second medical image having a second resolution and the second subset of layers of the neural network generating, from the second medical image”, ¶[0007] discloses the second layer is processing the output of the first layer to generate a second medical image.)); calculate an evaluation result by using the second output image; update the learning model by using the evaluation result to set the learning model as a learned model including a first sub-learned model that is the first sub-model that has performed learning and a second sub-learned model that is the second sub-model that has performed learning ([0062] “In an embodiment, the “fine” high resolution medical image estimation 532 from the output layer can then be compared with the concurrently obtained and/or generated target high resolution medical image 533 and a loss function can be minimized therebetween. The loss function may evaluate a difference between the estimated “fine” high resolution medical image 532 and the target high resolution medical image 533. If, upon evaluation of the loss function and comparison of the resultant value against a criterion at 534, it is determined that the criterion is met and the loss function has been minimized (i.e., there is an acceptable difference between the estimated “fine” high resolution medical image and the target high resolution medical image), the second subset of layers 515 of the neural network 530 is determined to be sufficiently trained and ready for implementation with unknown degraded data”, discloses calculating an evaluation result between the images generated and retraining the learned model based on the loss between the high resolution image and the target image.)
Cai does not explicitly teach output the first output image as an inference result image based on the first feature map extracted by inputting an inference input image to the first sub-learned model in the learned model.
Takahashi teaches output the first output image as an inference result image based on the first feature map extracted by inputting an inference input image to the first sub-learned model in the learned model.(¶[0038], “The image reduction unit 11 reads n pies of training images included in one mini-batch in accordance with the allocation determined by the learning control unit 17 and reduces the size of each image to a predetermined size. Next, the learning execution unit 12 executes learning of the FCN based on the plurality of reduced training images and ground truths respectively corresponding to the original training images (Step S13).”, ¶[0038] discloses training images are input into the learning model and ¶[0036] discloses “at the time of learning, a large number of set images are used. Each set is composed of a training image that is a target of segmentation and a ground truth that is a label image as a result of the appropriate segmentation of the training image.”, a ground truth/inference image is output during the time of learning.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Cai with Takahashi in order to outputting a ground truth/inference image based on inputting training images into the first learned model. One skilled in the art would have been motivated to modify Cai in this manner in order for performing semantic segmentation on an image by supervised machine learning. (Takahashi, ¶[0001])
Regarding Claim 6, the combination of Cai and Takahashi teach the image processing apparatus according to claim 1, where Cai further teaches wherein the resolution of the second output image is lower than a resolution of the learning input image. (¶[0121], “a third medical image having a third resolution, and outputting, by the processing circuitry, the third medical image, wherein the first resolution is lower than the second resolution and the second resolution is lower than the third resolution”, ¶[0121] discloses the second resolution of the output image is lower than the resolution of a third input image.)
Regarding Claim 7, the combination of Cai and Takahashi teach the image processing apparatus according to claim 1, where Cai further teaches wherein the first sub-model and the second sub-model are constituted by using a convolutional neural network. (¶[0052] discloses using a CNN, “the super-resolution method may be based on a convolutional neural network super-resolution method selected from the group including but not limited to Super-Resolution Convolutional Neural Network (SRCNN),”)
Regarding Claim 8, the combination of Cai and Takahashi teach the image processing apparatus according to claim 1, where Cai further teaches wherein a resolution of the first output image is lower than a resolution of the learning input image. (¶[0007] discloses “wherein the first resolution is lower than the second resolution” the second images resolution is higher than the first resolution of the first image.),
Regarding Claim 9, the combination of Cai and Takahashi teach the image processing apparatus according to claim 1, where Cai further teaches wherein the processor is configured to: further output an intermediate feature map having a higher resolution than the first feature map by using the first sub-model; and further input the intermediate feature map to the second sub-model. (¶[0039], “The low resolution CT image, or degraded image, can be provided to a first subset of layers of the neural network 130 and converted to a second medical image having a second resolution, referred to interchangeably herein as an intermediate image or as a “coarse” high resolution image”, ¶[0039] discloses processing the low resolution image through a plurality of layers to generate a higher resolution image.)
Regarding Claim 10, the combination of Cai and Takahashi teach the image processing apparatus according to claim 1, where Cai further teaches wherein the learning input image and the inference input image are medical images. ([0038] “FIG. 1 provides a high-level implementation of the methods of the present disclosure. At step 105 of method 100, a first medical image having a first resolution, referred to interchangeably herein as a low resolution medical image, can be obtained from an imaging modality. The low resolution medical image may be a 2D slice(s) or a 3D image volume from any region of the body of a patient and from any perspective of the body of the patient. In an example, the low resolution medical image is a low resolution CT image obtained from, as the imaging modality, a CT scanner.”, ¶[0038] disclose using medical images to input into the learning model.)
Regarding Claim 11, the combination of Cai and Takahashi teach the image processing apparatus according to claim 1, where Cai further teaches wherein the inference input image is an image acquired in time-series order. (¶[0038] discloses the medical images are 2d or 3d slices of a region of a patient which would inherently mean they are time series images.)
Regarding Claim 16, claim 16 is considered an apparatus claim substantially corresponding to claim 1. Please see the discussion of claim 1 above for a discussion of similar limitations.
Regarding Claim 17, claim 17 is considered an apparatus claim substantially corresponding to claim 1. Please see the discussion of claim 1 above for a discussion of similar limitations
Regarding Claim 18, claim 17 is considered an apparatus claim substantially corresponding to claim 1. Please see the discussion of claim 1 above for a discussion of similar limitations.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Cai et al. US PG-Pub(US 20220399101 A1) in view of Takahashi et al. US PG-Pub(US 20210358129 A1) in view of Choi et al. US PG-Pub(US 20220084167 A1).
Regarding Claim 5, while the combination of Cai and Takahashi teach the image processing apparatus according to claim 1, they do not explicitly teach wherein the resolution of the second output image is same as a resolution of the learning input image.
Choi teaches wherein the resolution of the second output image is same as a resolution of the learning input image. (¶[0015], “The memory is configured to further store a second artificial intelligence model, wherein the processor is configured to: obtain an output image having the same resolution as the input image by inputting the input image to the second artificial intelligence model, and control the communication interface to transmit the output image to another electronic apparatus,”, discloses the output image has the same resolution as the input image.))
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Cai and Takahashi with Choi in order to have the same resolutions between the output image and learning image. One skilled in the art would have been motivated to modify Cai and Takahashi in this manner in order to improve image quality. (Choi, ¶[0027])
Claims 2, 12 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Cai et al. US PG-Pub(US 20220399101 A1) in view of Takahashi et al. US PG-Pub(US 20210358129 A1) in view of Machida et al. US PG-Pub(US 20210158218 A1).
Regarding Claim 2, while the combination of Cai and Takahashi teach the image processing apparatus according to claim 1, they do not explicitly teach wherein the processor is configured to calculate the evaluation result by comparing the second output image with a learning correct answer image corresponding to the learning input image, and the learning correct answer image is a correct answer label image in which a correct answer label is attached for each of regions constituting the learning correct answer image.
Machida teaches wherein the processor is configured to calculate the evaluation result by comparing the second output image with a learning correct answer image corresponding to the learning input image, and the learning correct answer image is a correct answer label image in which a correct answer label is attached for each of regions constituting the learning correct answer image. (¶[0052], “In the evaluating unit 104, evaluation data is input to an evaluation data input unit 401 from the evaluation data holding unit 105. Learned parameters, which represent a learning result, are input to a learning result input unit 402 from the learning unit 103. A comparison unit 403 compares correct-answer data in the evaluation data with a result obtained by processing the medical image in the evaluation data using the learned parameters. A calculating unit 404 calculates the match rate between the correct-answer data in the evaluation data and the learned data of the learning result. An output unit 405 outputs the result of the calculation performed by the calculating unit 404.”, ¶[0052], discloses comparing the evaluation data with a correct answer data to determine if the output is correct or not.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Cai and Takahashi with Machida in order to compare the output with a correct answer. One skilled in the art would have been motivated to modify Cai and Takahashi in this manner in order to improve the image analysis accuracy
Regarding Claim 12, while the combination of Cai and Takahashi teach the image processing apparatus according to claim 1, they do no explicitly teach wherein the processor is configured to: generate report information based on information of the inference result image; generate a report image based on the report information; and perform control to display the report image.
Machida teaches wherein the processor is configured to: generate report information based on information of the inference result image; generate a report image based on the report information; and perform control to display the report image. (¶[0048], evaluation data is constituted by a radiation image prepared in advance and irradiation field information thereof (correct-answer data). Note that the evaluation method that is performed by the evaluating unit 104 will be described in detail later. In step S205, the display unit 106 displays the evaluation result obtained by the evaluating unit 104, on a display device. FIG. 3 shows a display example of an evaluation result displayed by the display unit 106. FIG. 3 indicates that the accuracy of irradiation field recognition that is performed using the irradiation field recognition function has improved from 80% to 90%. In other words, the display unit 106 can display whether a learning result of machine learning (the accuracy of irradiation field recognition) has improved or deteriorated.”, ¶[0045] discloses generating a report and displaying the results of the learning.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Cai and Takahashi with Machida in order to display the evaluation results to the user. One skilled in the art would have been motivated to modify Cai and Takahashi in this manner in order to improve the image analysis accuracy. (Machida, ¶[0003])
Regarding Claim 13, while the combination of Cai, Takahashi and Machida teach the image processing apparatus according to claim 11, where Machida further teaches wherein the report image is generated to display the report information so as to be superimposed on the inference input image or an image acquired later than the inference input image in time series.(¶[0039],“The display unit 106 displays the evaluation result for the learning result. The display unit 106 can also display a medical image. The display unit 106 can display whether the learning result of machine learning (e.g., the accuracy of image recognition, the accuracy of image processing, or the accuracy of diagnosis support) has improved or deteriorated.”, ¶[0039] discloses displaying the evaluation result of the machine learning to the user.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Cai and Takahashi with Machida in order to display the evaluation results to the user. One skilled in the art would have been motivated to modify Cai and Takahashi in this manner in order to improve the image analysis accuracy. (Machida, ¶[0003])
Claims 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Cai et al. US PG-Pub(US 20220399101 A1) in view of Takahashi et al. US PG-Pub(US 20210358129 A1) in view of Machida et al. US PG-Pub(US 20210158218 A1) in view of Sakashita et al. US PG-Pub(US 20220358640 A1).
Regarding Claim 14, while the combination of Cai, Takahashi and Machida teach the image processing apparatus according to claim 12, they do not explicitly teach wherein the report image is generated so as to display the inference input image or an image acquired later than the inference input image in time series and the report information at positions different from each other.
Sakashita teaches wherein the report image is generated so as to display the inference input image or an image acquired later than the inference input image in time series and the report information at positions different from each other. ([0084] “Note that the predicted disease image acquisition screen 9 illustrated in FIG. 3 is merely an example. That is, the configuration of the predicted disease image acquisition screen 9 can be appropriately changed. First of all, the configuration of the pre-modification image 50 can be changed. In a pre-modification image, only the position of the lesion 31 (for example, the position of the center of the lesion 31) may be displayed. In this case, the range of the lesion 31 may be displayed with a parameter (for example, a numerical value indicating the breadth of the range) other than the image. In addition, the position and range of the lesion 31 may be displayed in different pre-modification images. When, for example, the position of the lesion 31 is determined in advance, only the range (shape and size) of the lesion 31 may be displayed in the pre-modification image.”, ¶[0084] discloses displaying the lesion at different positions for the user to view.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Cai, Takahashi and Machida with Shakashita in order to display the evaluation results with the position of the lesion to the user. One skilled in the art would have been motivated to modify Cai, Takahashi and Machida in this manner in order to display the predicted disease image and the pre-modification image simultaneously or in a switching manner. (Sakashita, Abstract)
Regarding Claim 15, while the combination of Cai, Takahashi and Machida the image processing apparatus according to claim 13, they do not explicitly teach wherein the report information is position information of a specific shape surrounding a region indicating a feature included in the inference input image
Sakashita teaches wherein the report information is position information of a specific shape surrounding a region indicating a feature included in the inference input image. (¶[0084], “the range of the lesion 31 may be displayed with a parameter (for example, a numerical value indicating the breadth of the range) other than the image. In addition, the position and range of the lesion 31 may be displayed in different pre-modification images. When, for example, the position of the lesion 31 is determined in advance, only the range (shape and size) of the lesion 31 may be displayed in the pre-modification image.”, ¶[0084] discloses determining the shape of the lesion in the image.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Cai, Takahashi and Machida with Shakashita in order to display the evaluation results with the position of the lesion to the user. One skilled in the art would have been motivated to modify Cai, Takahashi and Machida in this manner in order to display the predicted disease image and the pre-modification image simultaneously or in a switching manner. (Sakashita, Abstract)
Allowable Subject Matter
Claims 3 and 4 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HAN D HOANG whose telephone number is (571)272-4344. The examiner can normally be reached Monday-Friday 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JOHN M VILLECCO can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HAN HOANG/Primary Examiner, Art Unit 2661
Prosecution Insights