DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-4, 7-8, 11-12, 14-15, 18, and 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Krebs (PGPUB: US 20190205766 A1 ) in view of Tang’940 (PGPUB: 20200065940 A1), and further in view of TSUYUKI (20200219252 A1).
Regarding claims 1 and 12. Krebs teaches an image generation system comprising:
at least one processor; and at least one non-transitory, computer-readable memory accessible by the processor and having instructions that, when executed by the processor (see Fig. 7), cause the processor to:
receive image data acquired from a patient that includes at least a first patient image associated with a patient (see Fig. 2, paragraph 41, Two input images I from different times t are input. For training, many examples of these pairs of images are input) and a second patient image associated with the patient (see Fig. 2, paragraph 41, Two input images I from different times t are input. For training, many examples of these pairs of images are input) (see paragraph 6 and 8, first and second sets of scan data representing a patient are acquired; pairs of images representing the patient at different times, from scans with different settings, and/or from scans with different modalities are obtained. In the examples below, images acquired at different times from a same scan of the patient are used as a fixed image and a moving image);
train an untrained model based on the first patient image and the second patient image (see Fig. 3, paragraph 48, a machine (e.g., image processor) trains the defined neural network arrangement. The training data (i.e., pairs of images) are used to train the neural network to determine a diffeomorphic deformation field. Given pairs of images, the network is trained by the training data to estimate the displacements between later input unseen images) to generate a trained neural network (see Fig. 4, paragraph 60, the machine outputs a trained neural network. The machine-learned network incorporates the deep learned features for the various units and/or layers of the network. The collection of individual features forms a feature or feature set for estimating deformation between two images);
provide the first patient image to the model (see Fig. 1-4, paragraph 63, using the learned features, the machine-learned network may detect the velocities, displacements, and/or warped image in response to input of a pair of images, such as medical images for a patient. Once the network is trained, the network may be applied. The network with defined or learned features is used to extract from input images. The machine-learned network uses the extracted features from the images to estimate the diffeomorphic displacements);
wherein the system trains the untrained model with training data only comprising the image data acquired from the patient (see Fig. 2 and 3, paragraph 48, a machine (e.g., image processor) trains the defined neural network arrangement. The training data (i.e., pairs of images) are used to train the neural network to determine a diffeomorphic deformation field. Given pairs of images, the network is trained by the training data to estimate the displacements between later input unseen images. Other outputs may be trained, such as training to estimate a velocity field and/or warped image).
However, Krebs does not expressly teach:
receive a third patient image from the model; and
output the third patient image to at least one of a storage system or a display.
Tang’940 teaches that an image reconstructor 34 receives sampled and digitized x-ray data from DAS 22 and performs high speed image reconstruction. The reconstructed image is output to a computer 36 which stores the image in a computer storage device 38 (see Fig. 2, paragraph 60); Display 42 allows the operator to observe the reconstructed image and other data from computer 36. The operator supplied commands and parameters are used by computer 36 to provide control signals and information to DAS 22, x-ray controller 28, and gantry motor controller 32 (see Fig. 2, paragraph 61).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Krebs by Tang’940 for providing an image reconstructor 34 receives sampled and digitized x-ray data from DAS 22 and performs high speed image reconstruction, as receive a third patient image from the model; providing display 42 allows the operator to observe the reconstructed image and other data from computer 36, as output the third patient image to at least one of a storage system or a display. Therefore, combining the elements from prior arts according to known methods and technique would yield predictable results.
However, the combination does not expressly teach to generate a trained model.
TSUYUKI teaches that the learning function 352 is able to generate the trained model M1 functioned to receive an input of a TDC and timing and to output information indicating to which one of the classes among “early”, “appropriate”, and “late”, the input timing corresponds. Further, the learning function 352 stores the generated trained model M1 into the model memory 34 (see Fig. 1 and 4, paragraph 94); the processing circuitry 35 acquires information about appropriateness of the timing determined at step S105, on the basis of the CT image data and an examination purpose (step S108). After that, the processing circuitry 35 generates a trained model M1 by using the TDC acquired at step S102 and the timing determined at step S105 as input-side learning data and using the information about appropriateness acquired at step S108 as output-side learning data (step S109) (see Fig. 1 and 7, paragraph 123).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination by TSUYUKI to obtain the processing circuitry 35 acquires information about appropriateness of the timing determined at step S105, on the basis of the CT image data and an examination purpose (step S108). After that, the processing circuitry 35 generates a trained model M1 by using the TDC acquired at step S102 and the timing determined at step S105 as input-side learning data, in order to provide to generate a trained model. Therefore, combining the elements from prior arts according to known methods and technique would yield predictable results.
Regarding claims 3 and 14. The combination teaches the system of claim 1, wherein the untrained model comprises a fully convolutional neural network (see Krebs, Fig. 2 and 3).
However, the combination does not expressly teach wherein the fully convolutional neural network does not include a pooling layer.
The examiner is taking "Official Notice" that the limitation about the fully convolutional neural network does not include a pooling layer is well known in the art.
Therefore, it would have been obvious to a person having ordinary skill in the art at the time the invention was made to have modified the combination so that the convolutional neural network does not include a pooling layer would be available in the convolutional neural network.
Regarding claims 4 and 15. The combination teaches the system of claim 1, wherein the untrained model comprises a neural network comprising an encoder path and a decoder path, the neural network further comprising a number of identity mapping layers comprising skip connections, the identity mapping layers coupled between the encoder path and the decoder path (see Fig. 3, paragraph 42).
Regarding claims 7. The combination teaches the system of claim 1, wherein the system is coupled to an imaging system, and wherein the system is further configured receive the first patient image from the imaging system and receive the second patient image from the imaging system (see Krebs, paragraph 48).
Regarding claims 8 and 18. The combination teaches the system of claim 1, wherein the first patient image is a magnetic resonance image (see Tang’940, paragraph 52, high quality medical image data can be acquired using one or more imaging modalities, such as x-ray, computed tomography (CT), molecular imaging and computed tomography (MICT), magnetic resonance imaging (MRI)), the second patient image is a noisy positron emission tomography image (see Krebs, paragraph 53, the network may be trained for computed tomography, magnetic resonance, ultrasound, x-ray, positron emission tomography), and the third patient image is a denoised positron emission tomography image (see Krebs, paragraph 37).
Regarding claim 11. The system of claim 1, wherein the system is further configured to generate a report based on the third patient image and output the report to the display (see Tang’940, Fig. 13 and 16, paragraph 117 and 118).
Regarding claim 21. The method of claim 12, wherein at least two of the first patient image, the second patient image (see Krebs, Fig. 1, paragraph 24, pairs of images representing the patient at different times, from scans with different settings, and/or from scans with different modalities are obtained), and the third patient image are images produced by different medical imaging modalities (see Tang’940, Fig. 2, paragraph 60, an image reconstructor 34 receives sampled and digitized x-ray data from DAS 22 and performs high speed image reconstruction. The reconstructed image is output to a computer 36 which stores the image in a computer storage device 38; see Krebs, paragraph 87, the images are acquired as part of any application including deformable image registration. For example, a registration application is provided as part of radiation therapy imaging, motion compensation during reconstruction (e.g., multi-modality CT and PET or SPECT), a pharmaco-kinetic study (e.g., MR-based study), or for image guided therapy).
Claim(s) 2 and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Krebs (PGPUB: US 20190205766 A1 ) in view of Tang’940 (PGPUB: 20200065940 A1), in view of TSUYUKI (20200219252 A1), and further in view of Tang’747 (PGPUB: 20210248747).
Regarding claims 2 and 13. The combination teaches the system of claim 1, wherein the untrained model comprises a neural network comprising a number of feature maps, wherein at least a portion of the feature maps (see Krebs, paragraph 27 and 78, three-dimensional datasets are obtained. In alternative embodiments, two-dimensional datasets representing planes are obtained; the encoder of the neural network has four convolutional layers with strides (2, 2, 2, 1)).
However, the combination does not expressly teach downsampled by a 3 x 3 x 3 three dimensional convolutional layer having a stride of 2 x 2 x 2.
Tang’747 teaches that the 3D convolution module 432a performs a spatial convolution over the input signal composed of volumes. The “stride” parameter specifies the strides of the convolution along each spatial dimension. The “kernel” parameter specifies the depth, width and height of the 3D convolution window. As shown as an example, the kernel is set to 3×3×3 to specify the volume of the convolution window. The stride parameter is set to 2 (two), in this example, to specify two strides of convolution. This layer creates a convolution kernel that is convolved with the layer input to produce a tensor of outputs (see Fig. 2, paragraph 47); the stride parameter is set equal to 2 (two), which specifies two strides of convolution along each spatial dimension. The kernel parameter is set equal to 2×2×2, which specifies the depth, width, and height of the transposed convolution window, as an example. The transposed convolution module 636 increases the spatial resolution of the volume, resulting in an output tensor of the same size as the downsample input of the previous output (see Fig. 6, paragraph 53).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination by Tang’747 for providing The transposed convolution module 636 increases the spatial resolution of the volume, resulting in an output tensor of the same size as the downsample input of the previous output; The “kernel” parameter specifies the depth, width and height of the 3D convolution window. As shown as an example, the kernel is set to 3×3×3 to specify the volume of the convolution window; the stride parameter is set equal to 2 (two), which specifies two strides of convolution along each spatial dimension. The kernel parameter is set equal to 2×2×2, which specifies the depth, width, and height of the transposed convolution window, as downsampled by a 3 x 3 x 3 three dimensional convolutional layer having a stride of 2 x 2 x 2. Therefore, combining the elements from prior arts according to known methods and technique, such as downsample technique and stride of 2 x 2 x 2, would yield predictable results.
Claim(s) 5 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Krebs (PGPUB: US 20190205766 A1 ) in view of Tang’940 (PGPUB: 20200065940 A1), in view of TSUYUKI (20200219252 A1), and further in view of Jin (PGPUB: 20200043163 A1).
Regarding claims 5 and 16. The combination teaches the system of claim 1, wherein the first patient image is a magnetic resonance image (see Krebs, paragraph 25, the images are captured using x-ray, computed tomography (CT), fluoroscopy, angiography, ultrasound, positron emission tomography (PET), or single photon emission computed tomography (SPECT)), and the third patient image is a reconstructed positron emission tomography image (see Tang’940, Fig. 2, paragraph 60, an image reconstructor 34 receives sampled and digitized x-ray data from DAS 22 and performs high speed image reconstruction. The reconstructed image is output to a computer 36 which stores the image in a computer storage device 38).
However, the combination does not expressly teach the second patient image is a raw sinogram.
Jin teaches that Layers 109, 113, 121 can include a plurality of filters of various different sizes. The inclusion of 1×1 filters allows for a reduction in the number of parameters and reduces the computational cost of training, for example. The example inception network 100 can include varying numbers of layers 108, 109, 112, 113, 120, 121 depending on a desired quality of output, for example. The two inputs shown originate from the same PET image, one input 102 being the raw sinogram while the other input 104 is an attenuation correction of the same PET scan/image (see Fig. 1, paragraph 34).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination by Jin for providing The two inputs shown originate from the same PET image, one input 102 being the raw sinogram while the other input 104 is an attenuation correction of the same PET scan/image, as the second patient image is a raw sinogram. Therefore, combining the elements from prior arts according to known methods and technique, such as one input 102 being the raw sinogram while the other input 104 is an attenuation correction of the same PET scan/image, would yield predictable results.
Claim(s) 9-10 and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Krebs (PGPUB: US 20190205766 A1 ) in view of Tang’940 (PGPUB: 20200065940 A1), in view of TSUYUKI (20200219252 A1), and further in view of Zhang (PGPUB: 20190130572).
Regarding claims 9 and 19. The combination teaches the system of claim 1, wherein the system is configured to train the untrained model by:
solving an objection function for a predetermined number of epochs based on the first patient image and the second patient image (see Krebs, paragraph 36) (see Tang, paragraph 94, the training can stop after pre-set epochs are reached. Thus, the CNN can be used in iterative training to pass through the training data set followed by testing with a verification set to form a single epoch. Multiple epochs can be executed to train the network model for denoising deployment),
wherein for each epoch the system is configured to: solve a first subproblem of the objection function for a first predetermined number of iterations (see Krebs, paragraph 78, the numbers of iterations in the exponentiation layer is set to N=4); and
solve a second subproblem of the objection function for iterations to a second predetermined number of update the untrained model (see Tang, paragraph 94 and 99, the training can stop after pre-set epochs are reached. Thus, the CNN can be used in iterative training to pass through the training data set followed by testing with a verification set to form a single epoch. Multiple epochs can be executed to train the network model for denoising deployment; the training neural network model 1220 processes the noisy image data through convolution, filtering, etc., to identify the noise in the noisy image data. The training neural network model 1220 extracts and outputs the noise, which is fed to the comparator 1230. The comparator 1230 compares the noise extracted by the training neural network 1220 with an expected, known, or “ground truth” noise value to determine an accuracy of the network model 1220. Feedback from the comparator 1230 indicating how close or how far the neural network model 1220 is from correctly identifying the known noise in the noisy image can be provided to the network weight updater 1240, which can adjust network weights of the training model 1220 to adjust operation of the model 1220 with respect to the noisy image data).
However, the combination does not expressly teach using the limited memory Broyden-Fletcher-Goldfarb-Shanno algorithm.
Zhang teaches that the iteration solution is performed by the limited memory Broyden-Fletcher-Goldfarb-Shanno method (L-BFGS method) or the gradient descent method, the transformation is performed by using the B-spline Free-form Deformation model, and the interpolation process is performed by the bilinear interpolation method or the B-spline interpolation method (see paragraph 20).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination by Zhang for providing that the iteration solution is performed by the limited memory Broyden-Fletcher-Goldfarb-Shanno method (L-BFGS method) or the gradient descent metho, as using the limited memory Broyden-Fletcher-Goldfarb-Shanno algorithm. Therefore, combining the elements from prior arts according to known methods and technique, such as the iteration solution is performed by the limited memory Broyden-Fletcher-Goldfarb-Shanno method, would yield predictable results.
Regarding claims 10 and 20. The combination teaches the system of claim 9, wherein the first subproblem is solved based on a Poisson distribution (see Tang’940, paragraph 2, In x-ray and/or computed tomography imaging, quantum noise due to Poisson statistics of x-ray photons can be a source of noise in an image, for example. Noise or interference in an image can corrupt actual image data and/or otherwise obscure features in a resulting image).
Response to Arguments
Regarding limitations of Claims of the instant case in view of the amended Claims and upon further consideration, a new ground(s) of rejection, necessitated by the amendments is made in view of different interpretation of the previously applied references and new prior art as presented in this Office action. Therefore, Applicant’s arguments are moot. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIN JIA whose telephone number is (571)270-5536. The examiner can normally be reached 9:00 am-7:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at (571)272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XIN JIA/Primary Examiner, Art Unit 2663