DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 6 and 16 are objected to. Claim 6 recites “neigher2neighbor” in line 1. This appears to be a typographical error that should read “neighbor2neighbor” instead. The same applies to claim 16. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2-4, 8, 11, 12-14 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Zhang et al. (WO2021/041125) and Tan et al. (US 2021/0383223).
Regarding claim 1, Zhang et al. discloses a computer-implemented method for improving image quality comprising:
(a) a deep learning network model (“The model network may be a trained model for enhancing the quality of PET images” at paragraph 35, line 1) based at least in part on an original medical image of a subject, wherein the original medical image has a low quality (“For instance, image data from clinical database may be used to generate low quality image data mimicking the image data acquired with shortened scan time.” at paragraph 70, line 5; though not explicit, the image data from the clinical database could feasibly have relatively low image quality due to scan constraints; “For example, the fast-scanned PET images 101 with low image quality” at paragraph 33, line 6);
(b) generating a pair of images of low-quality from the original medical image (as in paragraph 70, the generated low quality images data includes at least a pair of images; “In the illustrated example, a variety of filters may be applied to the low quality images obtained from accelerated PET images prior to improving the image quality through the deep learning network. Non-limiting examples of these filters can be convolutional filters (for example Roberts edge enhancing filter, Gaussian smoothing filter, Gaussian sharpening filter, Sobel edge detection filter, etc.) or morphological filters (for example erosion, dilation, segmentation filters, etc.) or various other filters” at paragraph 46, line 6; the filtering does pre-processing to perhaps remove a few bits of noise but would not constitute generation of high quality images, as the filtered images are still relatively low quality);
(c) training the deep learning network model based on the pair of images of low-quality, wherein the deep learning network model has the architecture identified in (a) (“In some embodiments, the model for enhancing image quality may be trained using supervised learning” at paragraph 36, line 1); and
(d) making inference with deep learning network model to output an enhanced medical image (“The image enhancement module 404 may be configured to enhance image quality using a trained model obtained from the training module. The image enhancement module may implement the trained model for making inferences, i.e., generating PET images with improved quality. For instance, the image enhancement module may take one or more fast- scanned PET image data collected from a PET scanner as input and output PET image data with improved quality. In some cases, the image enhancement module and/or the adaptive mixing and filtering module 406 may implement the method as described in FIG. 2 to further improve the performance of the system” at paragraph 55).
Zhang et al. does not explicitly disclose (a) identifying architecture for a deep learning network model based at least in part on an original medical image of a subject and (d) making inference with deep learning network model with dropout enabled.
Tan et al. teaches a method in the same field of endeavor of machine learning, comprising:
(a) identifying architecture for a deep learning network model based at least in part on an original image (“Generally, the present disclosure is directed to systems and methods to perform joint architecture and hyper-parameter search for machine learning models” at paragraph 0015, line 1; looking at paragraph 0077, the disclosure states that image data may be processed to generate modified image data, which is interpreted to be applicable to image enhancement; therefore the architecture search will be done with the objective of optimizing the image enhancement);
(d) a deep learning network model (“Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks” at paragraph 0067, last sentence) to output an enhanced image with dropout enabled (“The model trainer 160 can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained” at paragraph 0071, second sentence).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the joint model searching as taught by Tan et al. to choose the deep learning model of Zhang et al. for “combining joint hyper-parameter and architecture optimization, resulting in improved model performance and, compared to existing architecture and hyper-parameter search techniques, faster searching with reduced computational requirements” (Tan et al. at paragraph 0015, line 8).
Regarding claim 11, Zhang et al. discloses a non-transitory computer-readable storage medium including instructions that, when executed by one or more processors (“a non-transitory computer-readable storage medium including instructions that, when executed by one or more processors, cause the one or more processors to perform operations” at paragraph 7, line 1), cause the one or more processors to perform operations comprising:
(a) a deep learning network model (“The model network may be a trained model for enhancing the quality of PET images” at paragraph 35, line 1) based at least in part on an original medical image of a subject, wherein the original medical image has a low quality (“For instance, image data from clinical database may be used to generate low quality image data mimicking the image data acquired with shortened scan time.” at paragraph 70, line 5; though not explicit, the image data from the clinical database could feasibly have relatively low image quality due to scan constraints; “For example, the fast-scanned PET images 101 with low image quality” at paragraph 33, line 6);
(b) generating a pair of images of low-quality from the original medical image (as in paragraph 70, the generated low quality images data includes at least a pair of images; “In the illustrated example, a variety of filters may be applied to the low quality images obtained from accelerated PET images prior to improving the image quality through the deep learning network. Non-limiting examples of these filters can be convolutional filters (for example Roberts edge enhancing filter, Gaussian smoothing filter, Gaussian sharpening filter, Sobel edge detection filter, etc.) or morphological filters (for example erosion, dilation, segmentation filters, etc.) or various other filters” at paragraph 46, line 6; the filtering does pre-processing to perhaps remove a few bits of noise but would not constitute generation of high quality images, as the filtered images are still relatively low quality);
(c) training the deep learning network model based on the pair of images of low-quality, wherein the deep learning network model has the architecture identified in (a) (“In some embodiments, the model for enhancing image quality may be trained using supervised learning” at paragraph 36, line 1); and
(d) making inference with deep learning network model to output an enhanced medical image (“The image enhancement module 404 may be configured to enhance image quality using a trained model obtained from the training module. The image enhancement module may implement the trained model for making inferences, i.e., generating PET images with improved quality. For instance, the image enhancement module may take one or more fast- scanned PET image data collected from a PET scanner as input and output PET image data with improved quality. In some cases, the image enhancement module and/or the adaptive mixing and filtering module 406 may implement the method as described in FIG. 2 to further improve the performance of the system” at paragraph 55).
Zhang et al. does not explicitly disclose (a) identifying architecture for a deep learning network model based at least in part on an original medical image of a subject and (d) making inference with deep learning network model with dropout enabled.
Tan et al. teaches a non-transitory computer-readable storage medium including instructions in the same field of endeavor of machine learning that, when executed by one or more processors (“The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 114 can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 which are executed by the processor 112 to cause the user computing device 102 to perform operations” at paragraph 0060), comprising:
(a) identifying architecture for a deep learning network model based at least in part on an original image (“Generally, the present disclosure is directed to systems and methods to perform joint architecture and hyper-parameter search for machine learning models” at paragraph 0015, line 1; looking at paragraph 0077, the disclosure states that image data may be processed to generate modified image data, which is interpreted to be applicable to image enhancement; therefore the architecture search will be done with the objective of optimizing the image enhancement);
(d) a deep learning network model (“Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks” at paragraph 0067, last sentence) to output an enhanced image with dropout enabled (“The model trainer 160 can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained” at paragraph 0071, second sentence).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the joint model searching as taught by Tan et al. to choose the deep learning model of Zhang et al. for “combining joint hyper-parameter and architecture optimization, resulting in improved model performance and, compared to existing architecture and hyper-parameter search techniques, faster searching with reduced computational requirements” (Tan et al. at paragraph 0015, line 8).
Regarding claims 2 and 12, Zhang et al. discloses a method and medium wherein the medical image is acquired using a medical imaging apparatus with shortened scanning time (“For example, the fast-scanned PET images 101 with low image quality” at paragraph 33, line 6) or reduced amount of tracer dose.
Regarding claims 3 and 13, Tan et al. discloses a method and medium wherein the architecture for the deep learning network model is identified by employing a natural architecture search algorithm (“In some implementations, the input to the machine-learned model(s) of the present disclosure can be text or natural language data. The machine-learned model(s) can process the text or natural language data to generate an output” at paragraph 0078, line 1).
Regarding claims 4 and 14, Tan et al. discloses a method and medium wherein the natural architecture search algorithm comprises reinforcement learning (“Some example implementations of the present disclosure can apply reinforcement learning together with weight sharing to search over the discretized space” at paragraph 0050, line 1) with a recurrent neural network controller (“In some implementations, the user computing device 102 can store or include one or more machine-learned models 120. For example, the machine-learned models 120 can be or can otherwise include various machine-learned models such as neural networks (e.g., deep neural networks) or other types of machine-learned models, including non-linear models and/or linear models. Neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or other forms of neural networks” at paragraph 0061).
Regarding claims 8 and 18, Zhang et al. discloses a method and medium wherein the training dataset for training the deep learning network model includes medical images of low quality only (“In some embodiments, the model may be trained using unsupervised learning or semi-supervised learning that may not require abundant labeled data. High quality medical image datasets or paired dataset can be hard to collect. In some cases, the provided method may utilize unsupervised training approach allowing the deep learning method to train and apply on existing datasets (e.g., unpaired dataset) that are already available in clinical database” at paragraph 37, line 1).
Claim(s) 5, 6, 9, 10, 15, 16, 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Zhang et al. and Tan et al. as applied to claims 1 and 11 above, and further in view of Quan et al. (“Self2Self With Dropout: Learning Self-Supervised Denoising From Single Image”).
Regarding claims 5 and 15, the Zhang et al. and Tan et al. combination discloses the elements of claims 1 and 11 above.
The Zhang et al. and Tan et al. combination does not explicitly disclose that the pair of images of low-quality is generated using a sampler method.
Quan et al. teaches a method and medium in the same field of endeavor of image denoising, wherein the pair of images of low-quality is generated using a sampler method (“As the NN is trained only on a single noisy image y, we need to generate multiple image pairs from y, which are different from y yet cover most of its information. With this goal, we generate a set of Bernoulli sampled instances of y, denoted by {bym}Mm =1” at section 3.2, line 1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the sampling as taught by Quan et al. to generate the low-quality images of the Zhang et al. and Tan et al. combination as a way to generate the augmented training data with having only limited image data.
Regarding claims 6 and 16, Quan et al. discloses a method and medium wherein the sampler method is selected from a group consisting of self2self sampler method (“Self2Self With Dropout: Learning Self-Supervised Denoising From Single Image” at title) and neigher2neighbor sampler method.
Regarding claims 9 and 19, the Zhang et al. and Tan et al. combination discloses the elements of claims 1 and 11 above.
The Zhang et al. and Tan et al. combination does not explicitly disclose that the deep learning network model is trained using self-supervised learning.
Quan et al. teaches a method and medium in the same field of endeavor of image denoising, wherein the deep learning network model is trained using self-supervised learning (“Based on the discussion above, we propose a dropoutbased scheme for the single-image self-supervised learning of denoising NNs” at page 1891, left column, last paragraph, line 1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the sampling as taught by Quan et al. to generate the low-quality images of the Zhang et al. and Tan et al. combination as a way to generate the augmented training data with having only limited unlabeled image data.
Regarding claims 10 and 20, the Zhang et al. and Tan et al. combination discloses the elements of claims 1 and 11 above.
The Zhang et al. and Tan et al. combination does not explicitly disclose that the enhanced medical image is an average of multiple inferences made by the deep learning network model by dropping nodes in the deep learning network model randomly.
Quan et al. teaches a method and medium in the same field of endeavor of image denoising, wherein the enhanced medical image is an average of multiple inferences made by the deep learning network model by dropping nodes in the deep learning network model randomly (“Our recipe for reducing the variance of prediction when training a denoising NN on a single noisy image is a dropout-based scheme. Dropouts are used during training as well as test, in terms of both dropping nodes in the NN and dropping pixels (Bernoulli sampling) in the input noisy image” at section 5, line 4; “Dropout [24] is a widely-used regularization technique for deep NNs. It refers to randomly dropping out nodes when training an NN, which can be viewed as using a single NN to approximate a large number of different NNs” at page 1891, left column, third full paragraph, line 2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the particular dropout as taught by Quan et al. in the training of Zhang et al. and Tan et al. combination as it “provides a computationally-efficient way to train and maintain multiple NN models for prediction” (Quan et al. at page 1891, left column, third full paragraph, line 7).
Allowable Subject Matter
Claims 7 and 17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: the prior art does not teach or disclose that the sampler method is selected based at least in part on a noise distribution in the medical image as required by claims 7 and 17.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATRINA R FUJITA whose telephone number is (571)270-1574. The examiner can normally be reached Monday - Friday 9:30-5:30 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 5712723638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATRINA R FUJITA/ Primary Examiner, Art Unit 2672