DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-17 are pending.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 17 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Claim 15 recites limitation “A computer program product…”. A computer program per se, is not directed to one of the statutory categories, Gottschalk v. Benson, 409 U.S. at 72, 175 USPQ at 676-77. See MPEP § 2106(I).
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claim(s) 1-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Marinescu et al (Bayesian Image Reconstruction, 2021) in view of Hong et al (3D-StyleGAN, 2021).
Regarding claims 1 and 16-17, Marinescu teaches a computer-implemented method for generating a non-affected 3D image of a human body part of a patient from an affected 3D image of said human body part of said patient, said method comprising:
(Marinescu, Fig. 1; “reconstruct an image that has undergone a corruption process”, p2; “Given a corrupted input image I, we reconstruct it as G(w∗)”, p3; generating a clean (non-affected) image from a corrupted (affected) image; while Marinescu demonstrates this on 2D slices ("2D slices from a collection of 5 brain datasets", p7), Hong teaches the 3D implementation; Hong, “we extend the ... StyleGAN2 model, which natively works with two-dimensional images, to enable 3D image synthesis”, p1; “synthesizing realistic 3D brain MRIs”, p2; generating 3D images of human body parts, specifically brains; it would be obvious to apply the reconstruction method of Marinescu to the 3D images of Hong to handle 3D medical data, as Hong explicitly aims to overcome the limitations of 2D slice generation)
The combination of Marinescu and Hong further teaches:
receiving a trained generative model configured to receive a latent variable as input and to generate a non-affected 3D image as output, wherein said generative model is obtained by training a generative network using a library of human body part 3D scans of reference;
(Marinescu, “uses a single pre-trained generator model to solve different image restoration tasks”, p1; “Given an input corrupted image I, we aim to reconstruct the clean image I_cln”, “The prior model p(I_cln) has been trained a-priori ... I_cln = G(w), where w = ... is the latent vector of StyleGAN2 ... G ... is the deterministic function”, p4; “demonstrate BRGM on three large and diverse datasets ... (iii) a combined collection of 5 brain MRI datasets”, p1; teaching a trained generative model (StyleGAN2) receiving a latent variable w to generate a clean image, trained on reference scans; Hong, “extend the state-of-the-art StyleGAN2 model ... to enable 3D image synthesis”, p1; “trained different configurations of 3D-StyleGAN on a collection of ~12,000 T1 MRI brain scans”, “synthesizing realistic 3D brain MRIs”, p2; the generative model is a 3D generative network trained on a library of 3D brain scans)
receive said affected 3D image of a patient comprising at least one portion of said human body part, wherein said affected 3D image comprises at least one affected area;
(Marinescu, “Given a pre-trained generator G (here StyleGAN2), a known corruption model f, and a corrupted image I to be restored”, “Given a corrupted input image I, we reconstruct it as G(w∗)”, p3; “reconstruct an image that has undergone a corruption process”, p2; receiving an image with an affected ("corrupted") area, such as one requiring in-painting; Hong, “3D volumetric images”, p1; “synthesizing realistic 3D brain MRIs”, p2; the images are 3D images of a human body part (brain))
define (defining) a subset of said affected 3D image by excluding the content of at least one affected area from said affected 3D image;
(Marinescu, “In-painting with arbitrary mask: f_in is implemented as an operator that performs pixelwise multiplication with a mask M”, “produces a cropped-out (corrupted) image I_cor = I_cln⊙M”, p5; using a mask M (corruption model f) to exclude content (the affected/missing area) from the image processing. The likelihood term is computed based on the masked image, effectively defining a subset for comparison)
generating an optimal latent variable from minimizing a distance between said defined subset of the affected 3D image and a candidate subset;
(Marinescu, “reconstruct the image by estimating the Bayesian maximum a-posteriori (MAP) estimate over the input latent vector that generated the reconstructed image”, p1; “an optimisation over w: ... w* = ...”, eq. (6); “... as a weighted sum of four loss terms ...: w* = ... Given the Bayesian MAP solution w*, the clean image is returned as I*_cln = f ◦ G(w*)”, p5; optimizing the latent variable w (candidate latent variable) to minimize the distance (e.g., L_pixel, L2 norm) between the input image I (defined subset) and the corrupted generation f*G(w) (candidate subset); Hong, “projected to the latent space by finding w ... that minimize a distance between an input image I and a generated image I and a generated image g(w,n)”, p4; optimizing a latent variable to minimize distance for 3D images)
wherein said candidate subset is defined by excluding the content of said at least one affected area from a candidate image generated by said generative model fed with a candidate latent variable;
(Marinescu, Fig. 1, “use generator G to generate clean images G(w), followed by a corruption model f to generate a corrupted image f ◦ G(w)”; “f_in is implemented as an operator that performs pixelwise multiplication with a mask M”, p5; the candidate subset (f ◦ G(w)) is created by applying the mask (f, excluding content) to the candidate image (G(w)) generated from the latent variable)
generate a non-affected 3D image of the human body part of said patient with the generative model using said optimal latent variable as input.
(Marinescu, “Given the Bayesian MAP solution w*, the clean image is returned as I*_cln = G(w∗)”, p5; generating the final non-affected ("clean") image using the optimized latent variable; Hong, “enable 3D image synthesis”, “demonstrate the 3D-StyleGAN’s performance ... with 12,000 three-dimensional full brain MR T1 images”, p1; the generated image is a 3D image)
Regarding claim 2, the combination of Marinescu and Hong teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein the generative network is the generator of a generative adversarial network.
(Hong, Fig. 1, 3D-StyleGAN)
Regarding claim 3, the combination of Marinescu and Hong teaches its/their respective base claim(s).
The combination further teaches the method according to claim 2, wherein the generative adversarial network uses 3D convolutions.
(Hong, Fig. 1, 3D-StyleGAN; 3x3x3 Conv layers, “The style-based generator architecture of 3D-StyleGAN. The mapping network m, made of 8 fully-convolutional layers, maps a normalized latent vector z to an intermediate latent vector w)
Regarding claim 4, the combination of Marinescu and Hong teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein the distance is a Euclidean distance calculated over all the voxels of the subset of said affected 3D and the candidate subset.
(Marinescu, loss L_pixel term in eq. (6), p5, is a L2 norm distance (Euclidean distance); minimizing the pixelwise L2 norm (Euclidean distance) between the input image and the generated image processed by a corruption function f (which includes masking for in-painting); Hong, Fig. 1, “3D-StyleGAN”; “We modified StyleGAN2 for 3D image synthesis by replacing the 2D convolutions, upsampling and downsampling with 3D operations”, “used two mean squared error (MSE) losses”, p4; applying MSE (equivalent to Euclidean distance minimization) in a 3D voxel context)
Regarding claim 5, the combination of Marinescu and Hong teaches its/their respective base claim(s).
The combination further teaches the method according to claim 4, wherein the latent variable is a latent vector of dimension N and the latent variable is optimized on a mapped space of dimension N of distributions of the vectors used during training.
(Marinescu, “the latent space of StyleGAN2 consists of many vectors w = [w1, ..., wL]”, “ensures the w vectors lie in the same region as the vectors used during training”, “distribution for p(w): p(w) =... (eq. (2)”, p4; “optimisation over w: w∗ = arg max_w p(w)p(I | w)”, p5; optimizing the latent vector w to result in w* in a latent space which corresponds to the mapped space distributions used during training; eq. (2), constraining the latent variable using the distribution statistics (mu, sigma) derived from the training mapping network)
Regarding claim 6, the combination of Marinescu and Hong teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein Gaussian noise is added and optimized during generating the optimal latent variable as a second set of latent variables.
(Hong, “Image Projection: An image ... can be projected to the latent space by finding w and stochastic noise n at each layer that minimize a distance”, p4; optimizing the stochastic noise (n) alongside the latent variable during the projection/reconstruction process; stochastic noise can often be interpreted or modeled as Gaussian noise)
Regarding claim 7, the combination of Marinescu and Hong teaches its/their respective base claim(s).
The combination further teaches the method according to claim 4, wherein regularization terms are added to the Euclidean distance.
(Marinescu, eq. (6): “w* = (argmin_w L_w + lambda_colin L_colin) + lambda_pixel L_pixel +...”, p5; adding regularization terms (argmin_w L_w + lambda_colin L_colin) to the pixelwise Euclidean loss lambda_pixel L_pixel)
Regarding claim 13, the combination of Marinescu and Hong teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1 in which the generative network is trained by using a first library of T1 MRI scans or a second library of T2 MRI scans.
(Hong, “we used 11,820 brain MR T1 images ... Among those images, 7,392 images were used for training”, p5; training the network using a library of T1 MRI scans)
Regarding claim 14, the combination of Marinescu and Hong teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein the human body part is a brain.
(Marinescu, “combined collection of 5 brain MRI datasets”, p1; Hong, “12,000 three-dimensional full brain MR T1 images”, p1)
Regarding claim 15, the combination of Marinescu and Hong teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein said at least one affected area comprises at least one of: a lesion, an artifact, a resolution lower than a predetermined threshold.
(Marinescu, Fig. 2, “Top row shows super resolution, while bottom row shows in-painting”; in-painting => affected areas like artifacts)
Allowable Subject Matter
Claim(s) 8-12 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening Claim(s).
The following is a statement of reasons for the indication of allowable subject matter:
Claim(s) 8 recite(s) limitation(s) related to structured variational regularization of latent spaces. There are no explicit teachings to the above limitation(s) found in the prior art cited in this office action and from the prior art search.
Claim(s) 9-12 depend on claim 5.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANXUN YANG whose telephone number is (571)272-9874. The examiner can normally be reached on MON-FRI: 8AM-5PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, Applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JIANXUN YANG/
Primary Examiner, Art Unit 2662 1/27/2026