DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3, 5-6, and 12-14 are rejected under 35 U.S.C. 102 (a)(1) as being anticipated by Wu et. al (“Iterative Low-Dose CT Reconstruction with Priors Trained by Artificial Neural Network”).
Regarding Claim 1, Wu teaches a computer-implemented method of tomographic reconstruction, the computer-implemented method comprising:
Introduction, pg. 1: “In this work, we proposed a novel iterative CT reconstruction method based on priors learned by a k-sparse autoencoder [13], which will be further explain in section II-B.”
receiving an input dataset comprising tomographic projection data of an object; and
B. Problem Modeling, pg. 2: “A is the system matrix and b is the logarithm sinogram.”
A. Datasets, pg. 3: “The experiments were carried on 2016 Low-dose CT Grand Challenge datasets, which contained projection and image data of the chest and abdomen from 10 different patients. The data were acquired with Siemens Somatom Definition CT scanners, under 120kVp and 200 effective mAs.”
Explanation: The claimed input dataset corresponds to CT projection measurements (sinogram data) represented by b.
reconstructing an image of the object using an iterative reconstruction technique, wherein the iterative reconstruction technique seeks to minimize a cost function, the cost function including a first regularizer and a second regularizer, wherein the first regularizer is a first trained machine learning model, trained using image data associated with the object and extracted from a first direction and the second regularizer is a second trained machine learning model, trained using image data associated with the object and extracted from a second direction.
Introduction, pg. 1: “Compared to conventional filtered back projection (FBP) algorithms, iterative image reconstruction methods have great potential in noise reduction and information preservation, by incorporating photon statistics and prior information of the image to be reconstructed [2]–[4].”
C. Optimization Algorithm: “A monotonically non-increasing alternating optimization algorithm was used to solve (3) [7].”
PNG
media_image1.png
226
362
media_image1.png
Greyscale
PNG
media_image2.png
148
364
media_image2.png
Greyscale
D. Three-Dimensional Reconstruction, pg. 3: “Due to the large number of variables in the fully connected neural network, it was impractical to train the autoencoders on 3D patches. Instead, three independent autoencoders were trained from axial, sagittal and coronal slices, and the reconstruction problem was reformed as Equation 10, where s stands for one of the three directions. Psm is the patch extraction matrix along the sth direction.”
PNG
media_image3.png
90
350
media_image3.png
Greyscale
B. Training of KSAE, pg. 4: “3 independent KSAEs were trained for axial, sagittal and coronal directions, where 1 million 16 × 16 patches were randomly extracted from 5 of the 10 patients’ data for each direction.”
D. Three-Dimensional Reconstruction, pg. 3: “Ds (·) and Es (·) are the decoder and encoder trained for the sth direction respectively.”
PNG
media_image4.png
572
367
media_image4.png
Greyscale
Explanation: The reconstruction process is explicitly iterative. Equation 3 is an explicit cost function minimized during reconstruction. As shown in Equation 10, the reconstruction objective includes multiple learned prior terms: one learned prior from a first direction, another learned prior from a second direction, and a third learned prior from a third direction. Thus, the cost function includes at least a first and second regularizer. A first KSAE is trained using image patches extracted from a first direction (e.g., axial). A second independent KSAE is trained using image data extracted from a different direction (e.g., sagittal). Figure 1 shows the trained machine-learning regularizer architecture which illustrates the encoder, decoder, and neural-network training architecture. Figure 2 supports that the learned regularizer is incorporated into the iterative optimization.
Regarding Claim 2, Wu teaches the computer-implemented method of claim 1, wherein the cost function includes a third trained machine learning model as a third regularizer, trained using image data associated with the object and extracted a third direction, and wherein the first direction, second direction, and third direction are mutually orthogonal.
D. Three-Dimensional Reconstruction, pg. 3: “Due to the large number of variables in the fully connected neural network, it was impractical to train the autoencoders on 3D patches. Instead, three independent autoencoders were trained from axial, sagittal and coronal slices, and the reconstruction problem was reformed as Equation 10, where s stands for one of the three directions. Psm is the patch extraction matrix along the sth direction.”
PNG
media_image3.png
90
350
media_image3.png
Greyscale
B. Training of KSAE, pg. 4: “3 independent KSAEs were trained for axial, sagittal and coronal directions, where 1 million 16 × 16 patches were randomly extracted from 5 of the 10 patients’ data for each direction.”
Explanation: Axial, sagittal, and coronal directions are mutually orthogonal anatomical planes. The reference teaches the first ML regularizer (axial), the second ML regularizer (sagittal), and the third ML regularizer (coronal), which are all incorporated into the reconstruction cost function.
Regarding Claim 3, Wu teaches the computer-implemented method of claim 1, wherein the tomographic reconstruction is medical tomographic reconstruction, and wherein the object is a patient.
Abstract: “Dose reduction in computed tomography (CT) is essential for decreasing radiation risk in clinical applications.”
A. Datasets, pg. 3: “The experiments were carried on 2016 Low-dose CT Grand Challenge datasets, which contained projection and image data of the chest and abdomen from 10 different patients.”
Regarding Claim 5, Wu teaches the computer-implemented method of claim 1, wherein the iterative reconstruction technique uses a gradient descent algorithm.
C. Optimization Algorithm: “(3) Due to the random patch extraction strategy, the cost to keep track of ym was too high. Instead, the initial value of ym was first estimated with Equation 9 then several steps of the gradient descend (7) was carried out.”
PNG
media_image5.png
95
337
media_image5.png
Greyscale
Explanation: Steepest descent is a gradient-descent optimization algorithm.
Regarding Claim 6, Wu teaches the computer-implemented method of claim 1, wherein at least one of the first regularizer or the second regularizer are trained using a stochastic gradient descent algorithm.
B. Training of KSAE, pg. 4: “The KSAEs had the structure in figure 1, where 3 fully connect layers with 1024 units was used for both encoder and decoder. 50 epoches of ADAM algorithm with a step size of 0.0001 with other parameters same as in [30]. A minibatch size of 100 was used for the training.”
Explanation: ADAM is a stochastic gradient-based optimization algorithm that updates neural network parameters using gradients computed from mini-batches of training data. Wu teaches training the KSAEs using ADAM and further teaches a mini-batch size of 100, thereby teaching training the regularizer models using a stochastic gradient descent algorithm.
Regarding Claim 12, Wu teaches A computer-implemented method of generating a training dataset for a machine learning model, the computer-implemented method comprising:
receiving one or more computed tomography (CT) images of a patient as one or more ground truth images;
Abstract: “A manifold was learned from normal-dose images and the distance between the reconstructed image and the manifold was minimized along with data fidelity during reconstruction.”
A. Datasets, pg. 3: “The experiments were carried on 2016 Low-dose CT Grand Challenge datasets, which contained projection and image data of the chest and abdomen from 10 different patients.”
Explanation: Ground truth CT images of patients are disclosed.
generating one or more training images by adding quantum noise to the one or more CT images; and
A. Datasets, pg. 4: “Quarter-dose data were provided by the challenge committee, which were simulated from the normal-dose data by noise injection.”
Explanation: Noise injection into normal-dose CT images to create low-dose training data.
splitting the one or more ground truth images and the one or more training images into a training set and a testing set.
B. Training of KSAE, pg. 4: “3 independent KSAEs were trained for axial, sagittal and coronal directions, where 1 million 16 × 16 patches were randomly extracted from 5 of the 10 patients’ data for each direction.”
C. Reconstruction Setups, pg. 4: “The data from the patients other than the training set were used for reconstruction study.”
Explanation: The reference explicitly separates data into training patients and non-training/testing patients which corresponds to training/test splitting.
Regarding Claim 13, Wu teaches all of the limitations of claim 1 above. Wu further teaches the recited data processing apparatus with a memory storing computer-executable instructions and a processor configured to execute the computer-executable instructions as Wu discloses processor-executed reconstruction algorithms utilizing trained neural-network priors.
Regarding Claim 14, Wu teaches all of the limitations of claim 1 above. Wu further teaches a non-transitory computer-readable medium comprising instructions which, when performed by a processor of a computer, cause the processor to perform substantially the same steps as claim 1 as Wu discloses processor-executed reconstruction algorithms utilizing trained neural-network priors.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Wu et. al in view of Karimi and Ward (“A hybrid stochastic-deterministic gradient descent algorithm for image reconstruction in cone-beam computed tomography”).
Regarding Claim 4, Wu teaches the computer-implemented method of claim 1, but fails to teach that the tomographic projection data is cone-beam computed tomography (CBCT) data.
However, Karimi and Ward state that “we apply the proposed algorithm on simulated and real cone-beam CT projections and compare it with several other algorithms,” (Abstract) and “in this paper, we propose a new algorithm for image reconstruction for cone-beam computed tomography (CBCT)… commercial CBCT scanners use the well-known Feldkamp–Davis–Kress (FDK) filtered-back projection algorithm for image reconstruction” (Introduction, pg. 1).
Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to apply the machine-learning based iterative reconstruction technique of Wu to cone-beam CT projection data as taught by Karimi and Ward because both references are directed to iterative tomographic reconstruction of CT image data and seek to improve reconstruction quality from projection measurements. Karimi teaches that iterative reconstruction methods for CBCT can produce improved reconstruction quality from projection data, stating that “most of these algorithms are based on a regularized least-squares minimization with a regularization that promotes sparsity of the image gradient…recently proposed iterative reconstruction methods are able to reconstruct high-quality images from a much smaller number of projections” (Introduction, pg. 2). A person of ordinary skill in the art would have recognized CBCT reconstruction as a comparable tomographic reconstruction environment and would have reasonably expected that applying Wu’s trained machine-learning regularizers to CBCT projection data would predictably improve reconstruction quality in the same manner that Wu improves CT reconstruction generally. Therefore, it would have been obvious to modify Wu to utilize CBCT projection data as taught by Karimi and Ward with a reasonable expectation of success. This represents the use of a known technique to improve similar devices (methods or products) in the same way.
Claims 7 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Wu et. al in view of Mukherjee et. al (“Learned reconstruction methods with convergence guarantees”).
Regarding Claim 7, Wu teaches the computer-implemented method of claim 1, but fails to teach that at least one of the first regularizer or the second regularizer are trained in a weakly supervised manner.
However, Mukherjee teaches weakly supervised learning for learned image reconstruction regularizers, stating that “weakly supervised learning: This refers to the case where samples of measurement data and ground-truth are available, but they are not paired… notable methods that fall within the weakly supervised training paradigm are adversarial regularizer (AR) [20], [22] and its convex variant given by adversarial convex regularizer (ACR) [23]…both of these seek to learn a regularizer in a variational model as a critic that can tell apart noisy reconstructions (obtained through some simple baseline approach, e.g., by applying the pseudo-inverse of A on the measurements) from the ground-truth image samples xi” (B. Data driven reconstruction, pg. 7).
Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to train Wu’s learned regularizers using the weakly supervised training paradigm taught by Mukherjee because both references are directed to machine-learning based image reconstruction employing learned regularizers. Applying the known weakly supervised training technique of Mukherjee to Wu’s learned regularizer would have predictably provided an alternative means of training the regularizer while reducing the need for paired training datasets, with a reasonable expectation of success. This represents the use of a known technique to improve similar devices (methods or products) in the same way.
Regarding Claim 8, Wu teaches the computer-implemented method of claim 1, but fails to teach that at least one of the first regularizer or the second regularizer are adversarial convex regularizers.
However, Mukherjee teaches adversarial convex regularizers, stating that “notable methods that fall within the weakly supervised training paradigm are adversarial regularizer (AR) [20], [22] and its convex variant given by adversarial convex regularizer (ACR) [23]…both of these seek to learn a regularizer in a variational model as a critic that can tell apart noisy reconstructions (obtained through some simple baseline approach, e.g., by applying the pseudo-inverse of A on the measurements) from the ground-truth image samples xi” (B. Data driven reconstruction, pg. 7). Additionally, Mukherjee states that “its convex counterpart (abbreviated as ACR) models the regularizer using an input-convex neural network (ICNN) [30]” (A. Learned regularization methods, pg. 12).
Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to substitute Wu’s learned regularizer with the known adversarial convex regularizer taught by Mukherjee because both regularizers perform the same function of providing learned prior information within an iterative reconstruction objective. Such substitution would have predictably yielded a functioning learned reconstruction system while obtaining the known benefits associated with convex regularization, including the convergence properties discussed by Mukherjee, with a reasonable expectation of success. This represents the simple substitution of one known element for another to obtain predictable results.
Claims 9-11 are rejected under 35 U.S.C. 103 as being unpatentable over Wu et. al in view of Zhang et. al (“Improving CBCT quality to CT level using deep learning with generative adversarial network”).
Regarding Claim 9, Wu teaches the computer-implemented method of claim 1, but fails to teach that the image data associated with the object and extracted from the first direction includes a first training subset comprising higher quality data providing ground truth data and a second training subset comprising lower quality data providing training input data.
However, Zhang teaches exactly such a training arrangement. Specifically, Zhang states that “150 paired pelvic CT and CBCT scans were used for model training and validation… a total of 12000 slice pairs of CT and CBCT were used for model training, while 10-cross validation was applied to verify model robustness” (Abstract) and explains that “the Generator was used to generate synthetic CT (sCT) from the original CBCT, and the Discriminator was used to distinguish the synthetic CT (sCT) from the reference CT (rCT)… the sCT and rCT slices were used as input” (2.2 Pix2pix GAN Architecture with Feature Matching, pg. 3). Additionally, Zhang defines the reference CT as ground truth, stating that “all the deep-learning generated synthetic CTs (sCT) were compared to this reference” (2.1 Data Acquisition and Preprocessing, pg. 3). Thus, Zhang teaches a first training subset comprising higher-quality CT/reference CT data serving as ground-truth data and a second training subset comprising lower-quality CBCT data serving as training input data.
Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to apply Zhang’s known higher quality/ground-truth versus lower-quality/input training methodology to the machine-learning reconstruction framework of Wu because both references are directed to improving tomographic image reconstruction quality using machine learning. Applying Zhang’s training technique to Wu’s learned reconstruction prior would have predictable improved the quality and robustness of the learned regularizer by training the model using higher-quality ground-truth image data while utilizing lower-quality image data as training inputs, thereby yielding improved reconstruction performance with a reasonable expectation of success. This represents the use of a known technique to improve similar devices (methods or products) in the same way.
Regarding Claim 10, Wu in view of Zhang teaches the computer-implemented method of claim 9, and Zhang further teaches that the first training subset comprises CT scan data and the second training subset comprises CBCT scan data.
Abstract: “150 paired pelvic CT and CBCT scans were used for model training and validation… a total of 12000 slice pairs of CT and CBCT were used for model training, while 10-cross validation was applied to verify model robustness.”
2.1 Data Acquisition and Preprocessing: “For each patient, the CT images were mapped to each set of CBCT images using Velocity (Varian Medical Systems, Palo Alto, CA) with multi-pass B-spline based free form deformation to create a reference CT (rCT).”
2.2. Pix2pix GAN Architecture with Feature Matching: “The Generator was used to generate synthetic CT (sCT) from the original CBCT, and the Discriminator was used to distinguish the synthetic CT (sCT) from the reference CT (rCT)… then the synthesize CT (sCT) slices were used as the input of the Discriminator with the reference CT (rCT) slices as ground truth.”
Regarding Claim 11, Wu in view of Zhang teaches the computer-implemented method of claim 9, and Wu further teaches that the second training subset is simulated data generated from the first training subset.
A. Datasets, pg. 4: “Quarter-dose data were provided by the challenge committee, which were simulated from the normal-dose data by noise injection.”
Explanation: Normal-dose data = first dataset and quarter-dose data = simulated second dataset.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Park et. al (“Low-Dose CT Image Reconstruction with a Deep Learning Prior”) teaches a deep PWLS method by combining it with a deep learning prior. The proposed model is then reformulated as a constrained optimization problem and solved using the alternating optimization (AO) algorithm. While training for the noise reduction function is performed in an iterative way, the corrected image is obtained directly from the trained noise reduction function.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WILLIAM ADU-JAMFI whose telephone number is (571)272-9298. The examiner can normally be reached M-T 8:00-6:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WILLIAM ADU-JAMFI/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights