DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-26 are pending.
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claim(s) 1-6, 10-13, 15 and 20-25 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mailhe et al (US20200408864A1) in view of Cachovan et al (US20210081778A1).
Regarding claim 1, Mailhe teaches a method for reconstructing an image from sensor data, comprising:
receiving, by a computer system, sensor data from an imaging system;
(Mailhe, "A high-resolution image is reconstructed using a generator of a progressive generative adversarial network", [0006]; Cachovan, "Some embodiments provide generation of a high-resolution reconstructed volume based on an input low-resolution reconstructed volume and an input higher-resolution reconstructed volume.", [0011]; reconstructing images/volumes from input sensor data; Mailhe, "The MR system scans a patient with an MR sequence. The scanning results in k-space measurements.", [0007], acquiring sensor data; Cachovan, "emission data1-k is acquired via a first imaging modality. For example, emission data1-k may be acquired by a PET or SPECT scanner", [0014])
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the specific multimodal sensor data of Cachovan into the system or method of Mailhe in order to enhance image reconstruction. The combination of Mailhe and Cachovan also teaches other enhanced capabilities.
The combination of Mailhe and Cachovan further teaches:
accessing a machine learning model with the computer system, wherein the machine learning model comprises
a first subnetwork that receives sensor data as an input and generates an intermediate image as a first output, and
(Mailhe, "The system uses a machine-learned model in reconstruction.", [0031], accessing a machine-learned model configured for reconstruction; "A first generative adversarial network is machine trained at a first resolution for image denoising in the reconstruction.", [0012]; "Image or object domain data is input", [0031]; a first subnetwork (first GAN at a first resolution) receives the sensor data to output an intermediate denoised reconstruction image)
a second subnetwork that receives the first output from the first subnetwork and generates an enhanced image as a second output;
(Mailhe, "A second generative adversarial network is progressively machine trained at a second resolution greater than the first resolution.", [0012]; "The GANs in the progression incorporate the previous GAN of the progression.", [0059]; a second subnetwork (second GAN at a greater resolution) incorporates the first network's output to generate an enhanced, higher-resolution image)
inputting the sensor data to the machine learning model using the computer system, generating an enhanced image as an output; and
(Mailhe, "Image or object domain data is input, and image or object domain data with less artifacts are output.", [0031]; providing the sensor data into the model to generate the enhanced, artifact-reduced output image)
presenting the enhanced image to a user with the computer system.
(Mailhe, "The MR image is displayed.", [0007]; "A display is configured to display an image of the region from the reconstructed representation.", [0017]; presenting the enhanced reconstructed image to a user via a display)
Regarding claims 2 and 20, the combination of Mailhe and Cachovan teaches its/their respective base claim(s).
The combination further teaches the method of claim 1, wherein the first subnetwork comprises
a first component and a second component;
wherein the first component converts the sensor data from a sensor domain to an image domain, generating image-domain data as an output; and
wherein the second component extracts images features from the image-domain data.
(Mailhe, "The k-space measurements resulting from the scan sequence are transformed from the frequency domain to the spatial domain in reconstruction.", [0040]; "The encoder is formed from hidden layers and downsampling layers 304, 308, 312.", [0063]; converting the sensor data (k-space) to an image domain spatial distribution (first component/Fourier transform), and using an encoder (second component) to extract image features.)
Regarding claims 3 and 21, the combination of Mailhe and Cachovan teaches its/their respective base claim(s).
The combination further teaches the method of claim 2, wherein the first component comprises an artificial neural network comprising at least two fully connected layers.
(Mailhe, "A fully connected network structure may be used.", [0063]; using a fully connected network structure, which inherently comprises at least two fully connected layers)
Regarding claims 4 and 22, the combination of Mailhe and Cachovan teaches its/their respective base claim(s).
The combination further teaches the method of claim 2, wherein the second component comprises an artificial neural network comprising a convolutional autoencoder.
(Mailhe, "The image-to-image network includes an encoder formed from layers (e.g., 302-312) with downsampling to the bottleneck layer or layers 314 and a decoder formed from layers (e.g., 316-326) with upsampling from the bottleneck layer or layers 314.", [0062]; the second component comprises an encoder-decoder architecture, which constitutes a convolutional autoencoder)
Regarding claims 5 and 23, the combination of Mailhe and Cachovan teaches its/their respective base claim(s).
The combination further teaches the method of claim 1, wherein the second subnetwork comprises a convolutional neural network.
(Mailhe, "For example, a convolutional neural network (CNN) is used.", [0044]; the subnetwork/generator comprises a convolutional neural network)
Regarding claim 6, the combination of Mailhe and Cachovan teaches its/their respective base claim(s).
The combination further teaches the method of claim 1, wherein the sensor data comprise functional imaging data.
(Cachovan, "emission data1-k may be acquired by a PET or SPECT scanner", [0014]; using functional imaging data (PET/SPECT) as the sensor data)
Regarding claim 10, the combination of Mailhe and Cachovan teaches its/their respective base claim(s).
The combination further teaches the method of claim 1, further comprising accessing structural imaging data with the computer system and inputting the structural imaging data as an additional input to the machine learning model.
(Cachovan, "Also input to network 110 is a higher-resolution volume which is generated using a different imaging modality... the higher resolution may comprise a CT volume", [0016]; accessing and inputting structural imaging data (CT volume) as an additional input to the model)
Regarding claim 11, the combination of Mailhe and Cachovan teaches its/their respective base claim(s).
The combination further teaches the method of claim 10, wherein the structural imaging data are input to the second subnetwork of the machine learning model.
(Mailhe, "The input at a higher resolution than for layers 310 and 318 is then used to train.", [0064]; Cachovan, "the higher resolution may comprise a CT volume acquired using a CT scanner.", [0016]; inputting the structural (CT) imaging data of Cachovan into the higher-resolution stages (second subnetwork) of Mailhe's progressive network would lead to guide the generation of fine details)
Regarding claim 12, the combination of Mailhe and Cachovan teaches its/their respective base claim(s).
The combination further teaches the method of claim 10, wherein the first subnetwork comprises
a first component and a second component;
wherein the first component converts the sensor data from a sensor domain to an image domain, generating image-domain data as an output;
wherein the second component extracts images features from the image-domain data; and
wherein the structural imaging data are input to the second component.
(Mailhe, "The k-space measurements resulting from the scan sequence are transformed from the frequency domain to the spatial domain in reconstruction.", [0040]; "The encoder is formed from hidden layers and downsampling layers 304, 308, 312.", [0063]; Cachovan, "Also input to network 110 is a higher-resolution volume which is generated using a different imaging modality... the higher resolution may comprise a CT volume", [0016]; Combining Mailhe and Cachovan teaches converting sensor data to an image domain, extracting features, and inputting the structural imaging data into the feature-extracting component of the network)
Regarding claim 13, the combination of Mailhe and Cachovan teaches its/their respective base claim(s).
The combination further teaches the method of claim 10, wherein the structural imaging data comprise x-ray imaging data.
(Cachovan, "the higher resolution may comprise a CT volume acquired using a CT scanner.", [0016]; structural imaging data comprises x-ray (CT) imaging data)
Regarding claim 15, Mailhe teaches a method for training a machine learning model for multimodal image reconstruction, the method comprising:
(Mailhe, "A method of machine training for reconstruction in medical imaging", [0012]; Cachovan, "The artificial neural network is trained to generate a high-resolution data set based on the plurality of sets of low-resolution emission data and respective ones of the higher-resolution data set.", [0028])
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the multimodal training approach of Cachovan into the system or method of Mailhe in order to improve cross-modality resolution enhancement. The combination of Mailhe and Cachovan also teaches other enhanced capabilities.
The combination of Mailhe and Cachovan further teaches:
(a) accessing first imaging data with a computer system, wherein the first imaging data were acquired from a group of subjects;
(Cachovan, "Training system 210 uses Q sets of emission data1-k", [0020]; "The Q sets of emission data1-k and high-resolution volumes1-Q may represent many different patients, phantoms, or other subjects.", [0025]; accessing first imaging data acquired from a group of subjects)
(b) accessing second imaging data with the computer system, wherein the second imaging data were acquired from the group of subjects and have a higher spatial resolution than the first imaging data;
(Cachovan, "Also input to network 110 is a higher-resolution volume which is generated using a different imaging modality than the modality used to generate emission data1-k.", [0016]; "The Q sets of emission data1-k and high-resolution volumes1-Q may represent many different patients, phantoms, or other subjects.", [0025]; accessing higher spatial resolution second imaging data acquired from the subjects)
(c) assembling the first imaging data into a first training dataset and the second imaging data into a second training data set;
(Mailhe, "To machine train, training data is gathered or accessed. The training data includes many sets of data", [0054]; Cachovan, "Training system 210 uses Q sets of emission data1-k and high-resolution volumes1-Q to train artificial neural network 110.", [0020]; Mailhe teaches assembling the gathered image data into training datasets. Cachovan teaches assembling the first (e.g., emission) and second (higher-resolution) imaging data into training datasets)
(d) accessing a machine learning model with the computer system, wherein the machine learning model comprises
a first subnetwork that receives first imaging data as an input and generates an intermediate image as a first output, and
a second subnetwork that receives the first output from the first subnetwork and generates an enhanced image as a second output;
(Mailhe, "A first generative adversarial network is machine trained at a first resolution for image denoising in the reconstruction. A second generative adversarial network is progressively machine trained at a second resolution greater than the first resolution.", [0012]; "The GANs in the progression incorporate the previous GAN of the progression.", [0059]; accessing a machine learning model comprising first and second progressive subnetworks, where the second subnetwork receives the output of the first subnetwork to generate an enhanced, higher-resolution image)
(e) training the first subnetwork on the first training dataset;
(Mailhe, "A first generative adversarial network is machine trained at a first resolution for image denoising in the reconstruction.", [0012]; Cachovan, "Training system 210 uses Q sets of emission data1-k... to train artificial neural network 110.", [0020]; Mailhe teaches training the first subnetwork at a first resolution. Training this first subnetwork may use the first training dataset (lower-resolution/first modality data) of Cachovan)
(f) training the second subnetwork on the second training data set; and
(Mailhe, "A second generative adversarial network is progressively machine trained at a second resolution greater than the first resolution.", [0012]; Cachovan, "Training system 210 uses... high-resolution volumes1-Q to train artificial neural network 110.", [0020]; Mailhe teaches training the second subnetwork at a greater resolution. Training this second subnetwork may use the second training dataset (higher-resolution structural data) of Cachovan)
(g) storing the trained first subnetwork and the trained second subnetwork as a trained machine learning model.
(Mailhe, "A generator of the second generative adversarial network is stored after the progressive machine training of the second generative adversarial network.", [0012]; storing the progressively trained networks as a trained machine learning model)
Regarding claim 24, the combination of Mailhe and Cachovan teaches its/their respective base claim(s).
The combination further teaches the method of claim 15, wherein the first imaging data are acquired with a first imaging modality and the second imaging data are acquired with a second imaging modality that is different from the first imaging modality.
(Cachovan, "higher-resolution volume which is generated using a different imaging modality than the modality used to generate emission data1-k.", [0016]; using first and second imaging data acquired with different imaging modalities)
Regarding claim 25, the combination of Mailhe and Cachovan teaches its/their respective base claim(s).
The combination further teaches the method of claim 24, wherein the first imaging modality is a functional imaging modality and the second imaging modality is a structural imaging modality.
(Cachovan, "emission data1-k may be acquired by a PET or SPECT scanner... the higher resolution may comprise a CT volume", [0016]; the first modality => a functional modality (PET/SPECT) and the second modality => a structural modality (CT))
Claim(s) 7-9, 14 and 26 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mailhe et al (US20200408864A1) in view of Cachovan et al (US20210081778A1) and further in view of Fang (US20200015744A1),.
Regarding claim 7, the combination of Mailhe and Cachovan teaches its/their respective base claim(s).
The combination does not expressly disclose but Fang teaches the method of claim 6, wherein the functional imaging data comprise optical imaging data.
(Fang, "The functional imaging technique may be an ill-posed imaging technique utilizing model-based reconstructions. The ill-posed imaging technique can be diffuse optical tomography.", [0009])
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the functional optical imaging data of Fang into the multi-modal framework of Cachovan in order to benefit from portable, low-cost functional measurements without the need for radioactive tracers. The combination of Mailhe, Cachovan and Fang also teaches other enhanced capabilities.
Regarding claim 8, the combination of Mailhe, Cachovan and Fang teaches its/their respective base claim(s).
The combination further teaches the method of claim 7, wherein the optical imaging data comprise diffuse optical tomography (DOT) data.
(Cachovan, "emission data1-k may be acquired by a PET or SPECT scanner", [0014]; Fang, "The ill-posed imaging technique can be diffuse optical tomography.", [0009]; incorporating Fang's diffuse optical tomography (DOT) data into Cachovan's multi-modal framework would provide high-contrast functional maps of tissue properties like hemoglobin concentration)
Regarding claim 9, the combination of Mailhe, Cachovan and Fang teaches its/their respective base claim(s).
The combination further teaches the method of claim 8, wherein the DOT data comprise three-dimensional (3D) DOT data.
(Fang, "For every speculative tumor locations, the 3D physiological images of total hemoglobin concentration (HbT), oxygen saturation (SO2) and the reduced scattering coefficient (μs′) can be obtained by a reconstruction method", [0043]; "The output of the metric function, M, becomes a 3D volumetric image, Δμ(r0)", [0049]; the reconstructed DOT data comprise three-dimensional (3D) spatial data/images)
Regarding claim 14, the combination of Mailhe and Cachovan teaches its/their respective base claim(s).
The combination of Mailhe, Cachovan and Fang further teaches the method of claim 13, wherein the sensor data comprise diffuse optical tomography (DOT) data.
(Cachovan, "the higher resolution may comprise a CT volume", [0016]; Fang, "Multi-modal breast imaging combining mammography with diffuse optical tomography (DOT) has been shown as a viable approach... fusing high-resolution x-ray tissue anatomy with functional optical measurements", [0027]; Cachovan teaches combining functional sensor data with high-resolution X-ray (CT) data. implementing Fang's DOT data as the functional sensor data of Cachovan would lead to successfully correlating optical functional characteristics with high-resolution X-ray anatomy)
Regarding claim 26, the combination of Mailhe and Cachovan teaches its/their respective base claim(s).
The combination of Mailhe, Cachovan and Fang further teaches the method of claim 25, wherein the first imaging modality comprises diffuse optical tomography and the second imaging modality comprises x-ray imaging.
(Cachovan, "emission data1-k may be acquired by a PET or SPECT scanner... the higher resolution may comprise a CT volume", [0016]; Fang, "Multi-modal breast imaging combining mammography with diffuse optical tomography (DOT) has been shown as a viable approach... fusing high-resolution x-ray tissue anatomy with functional optical measurements", [0027]; combining Cachovan's multi-modal framework with Fang's specific pairing of diffuse optical tomography (first modality) and x-ray imaging (second modality) would improve positive predictive value and tissue characterization)
Claim(s) 16-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mailhe et al (US20200408864A1) in view of Cachovan et al (US20210081778A1) and further in view of Kaufhold et al 6 (US20190080205A1).
Regarding claim 16, the combination of Mailhe and Cachovan teaches its/their respective base claim(s).
The combination does not expressly disclose but Kaufhold teaches the method of claim 15, wherein assembling the first training dataset includes generating noise-added imaging data from the first imaging data with the computer system and storing the noise-added imaging data as the first training dataset.
(Kaufhold, "Deep Model Image Generation... generates images that are similar to a given image with slight variations.", [0094]; "training a DMTG generator using noise as an initialization... Gaussian noise 705", [0101]; Cachovan, "Training system 210 uses Q sets of emission data1-k... to train artificial neural network 110.", [0020]; Cachovan teaches using sensor emission data to train the network. Kaufhold teaches generating variations of training images using noise to create supplementary images. It would be obvious to combine Cachovan's training data assembly with Kaufhold's supplementary noise-added generation to provide a robust training dataset that improves network generalizability to noise.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the supplementary noise-added generation of Kaufhold into the training data assembly of Cachovan in order to provide a robust training dataset that improves network generalizability to noise. The combination of Mailhe, Cachovan and Kaufhold also teaches other enhanced capabilities.
Regarding claim 17, the combination of Mailhe, Cachovan and Kaufhold teaches its/their respective base claim(s).
The combination further teaches the method of claim 16, wherein the noise-added imaging data are generated by inputting the first imaging data to a generative adversarial network (GAN) to add realistic noise to the first imaging data.
(Kaufhold, "training a DMTG generator using noise as an initialization. The DMTG 707 includes a generator 709, which is composed of a GAN 711", [0101]; "The generative neural network 715 is not given data to begin with; instead it is initialized with Gaussian noise 705... The two networks play an adversarial game”, [0101]; “eventually the generative network 715 is able to generate images 717 that statistically are similar to the training images 701.", [0123]; generating supplementary training data by using a Generative Adversarial Network (GAN) initialized with noise to create realistic variations of the original images; incorporating this GAN-based noise addition to Cachovan's dataset would allow generating realistic noisy training data)
Claim(s) 18-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mailhe et al (US20200408864A1) in view of Cachovan et al (US20210081778A1) and further in view of Chan et al 4 (US20190365341A1).
Regarding claim 18, the combination of Mailhe and Cachovan teaches its/their respective base claim(s).
The combination does not expressly disclose but Chan teaches the method of claim 15, wherein the first subnetwork is trained on the first training dataset using a prior-weighted loss function that penalizes more heavily on inaccuracies within a region-of-interest (ROI) in the first training dataset.
(Chan, "a background region. This balances the competing objectives of learning to preserve desired small features while suppressing noise in the background.", [0072]; Cachovan, "The total loss is back-propagated from loss layer component 430 to network 410.", [0033]; Cachovan teaches training the network utilizing a loss function)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the prior-weighted loss function of Chan into the method of training the network utilizing a loss function of Cachovan in order to penalize errors more heavily in a specific ROI (lesions) to ensure the network preserves clinically significant small features rather than over-smoothing them as background noise. The combination of Mailhe, Cachovan and Chan also teaches other enhanced capabilities.
Regarding claim 19, the combination of Mailhe, Cachovan and Chan teaches its/their respective base claim(s).
The combination further teaches the method of claim 18, wherein the prior-weighted loss function receives as an input prior knowledge of a location and size of abnormalities in the first training dataset.
(Chan, "To generate the weight maps, the lesions/regions of interest are first segmented in the target images... in order to create lesion masks ... The weight maps are generated from the lesion masks by assigning Nb/N1 times higher weights in the lesions, where Nb and N1 are the total number of voxels in the background and the total number of voxels in the lesions, respectively.", [0080]; utilizing segmented lesion masks providing prior knowledge of the location (segmented region) and size (number of voxels) of abnormalities to construct the weight map inputted into the loss function)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANXUN YANG whose telephone number is (571)272-9874. The examiner can normally be reached on MON-FRI: 8AM-5PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272- 1000.
/JIANXUN YANG/
Primary Examiner, Art Unit 2662 7/25/2026
Prosecution Insights