DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments at page 8 of Remarks, filed 8 April 2026, with respect to the objection to the drawings have been fully considered and are persuasive. The objection has been withdrawn.
Applicant’s arguments at pages 8-9 of Remarks, filed 8 April 2026, with respect to the objections to claims 10 and 15 for minor informalities have been fully considered and are persuasive. The objections to claims 10 and 15 are withdrawn.
Applicant’s arguments at page 9 of Remarks, filed 8 April 2026, with respect to the rejections of claim 16 under 35 U.S.C. 112(b) and 35 U.S.C. 112(d) have been fully considered and are persuasive. The rejections are withdrawn.
Applicant’s arguments at pages 9-14 of Remarks, filed 8 April 2026, with respect to the rejections under 35 U.S.C. 103 have been fully considered but are moot. Applicant’s arguments are directed to the subject matter concerning bounding boxes that was added to the claims. Han is relied upon to teach these features. Accordingly, as Applicant’s amendment necessitated the inclusion of Han and none of Applicant’s arguments are specifically directed to Han, the claims remain rejected under 35 U.S.C. 103 for the reasons set forth below.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
Claims 3, 4 and 21 are rejected under 35 U.S.C. 112(a) as failing to comply with the written description requirement. The claim contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor at the time the application was filed, had possession of the claimed invention.
[Paragraph 20] For example, the descriptor may be formatted as one or more spheres (or other objects) indicating one or more size, volume, and/or structural characteristics for the abnormality and/or affected tissue of the anatomy surrounding the tissue. In some implementations, the spheres may be concentric. The center of the concentric spheres may indicate a position for a center-of-mass of the abnormality. The volume of each of the one or more spheres may correspond to the volume of a particular segment of the abnormality. For example for a tumor type abnormality, a first sphere may indicate a core density for a tumor, and a second sphere may indicate a density of another peripheral region of the tumor, a third sphere may indicate another volume characteristic of the tumor and/or a volume of non-tumor tissue affected by the insertion of the tumor. Various spheres may be included. In some cases, the spheres may further indicate the total volume of the abnormality and/or the affected tissue region ... (emphasis added).
[Paragraph 50] Mass effect and tissue deformation in the surrounding region of the tumor may be synthesized using abnormality mask segments. For example, outer spheres of the concentric sphere descriptor may designate the volume of regions of non-tumor tissue that are affected by the presence of the tumor. Alternatively, a bounding box or mask applied at the second stage may be used to designate the extent of the effect of the tumor. For example, a bounding box or mask larger than the tumor mask may be defined. The region in the bounding box or mask but outside the tumor mask is the affected region proximate to the tumor, where mass effect and tissue deformation may be present. The region outside the bounding box or mask may be the same as the original pre-abnormality image ... (emphasis added).
[Claim 1] A system for synthesizing a medical image of a synthesized medical abnormality, the system including: synthesis circuitry configured to: obtain a descriptor input for the synthesized medical abnormality, the descriptor input detailing a selected characteristic for the synthesized medical abnormality; denoise, using a first diffusion model machine-learned network, a first noise input to generate an abnormality spatial mask and a bounding box surrounding the abnormality spatial mask within a defined multidimensional space, the bounding box defining a spatial extent outside the abnormality spatial mask of effects due to the synthesized medical abnormality; and obtain a pre-abnormality image mapped into the defined multidimensional space; denoise, using a second diffusion model machine-learned network, a second noise input to generate the medical image with the synthesized medical abnormality positioned in accord with the abnormality spatial mask, the pre-abnormality image being preserved in regions outside of the bounding box; and machine learning control circuitry configured to provide the abnormality spatial mask and the medical image for a medical machine learning system.
[Claim 2] The system of claim 1, where the descriptor input includes a descriptor in a predefined format associated with the first diffusion model machine-learned network.
[Claim 3] The system of claim 2, where the predefined format includes: one or more concentric spheres positioned within the defined multidimensional space; and/or a vector indicating one or medical classifications of the synthesized medical abnormality.
[Claim 4] The system of claim 1, where the descriptor input includes one or more spheres positioned within the defined multidimensional space to indicate one or more selected volume characteristics and/or a center-of-mass of the synthesized medical abnormality.
[Claim 20] A denoising method for synthesizing a medical image of a synthesized medical abnormality with a selected characteristic, the method including: obtaining a descriptor of at least the selected characteristic of the synthesized medical abnormality in a predefined format associated with a first diffusion model machine-learned network; providing the descriptor and a first noise input to the first diffusion model machine-learned network; denoising, via the first diffusion model machine-learned network, the first noise input to obtain: an abnormality spatial mask that spatially defines the synthesized medical abnormality with the selected characteristic; and a bounding box surrounding the abnormality spatial mask, the bounding box defining a spatial extent outside the abnormality spatial mask of effects due to the synthesized medical abnormality; after obtaining the abnormality spatial mask, denoising, using a pre-abnormality image and a second diffusion model machine-learned network, second noise input to obtain the medical image of the synthesized medical abnormality, the medical image consistent with pre-abnormality image modified to include the synthesized medical abnormality inserted in accord with the abnormality spatial mask, the pre-abnormality image being preserved in regions outside of the bounding box; and providing the medical image to a training interface for training interaction.
[Claim 21] The denoising method of claim 20, where the predefined format includes: one or more concentric spheres positioned within the defined multidimensional space; and/or a vector indicating one or medical classifications of the synthesized medical abnormality.
The instant specification describes using bounding boxes as alternatives to concentric spheres for confining the spatial boundaries of the abnormality in the synthesized image, and not used together in a single embodiment. The specification does not explicitly or implicitly describe an embodiment with a descriptor of a selected characteristic used with a first diffusion network to generate or obtain a mask and bounding box and generating or obtaining one or more spheres or concentric spheres. At best, the specification supports one alternative or another, but not both simultaneously. For example, Figure 4 provides concentric sphere descriptor 420 in the first denoising stage 410, but no bounding box descriptor. Paragraphs 36 and 50 describe using a bounding box in relation to the second stage, not the first stage. While combinations of bounding boxes and spheres are conceivable, there is no indication that Applicant intended such a combination as claimed to be within the scope of the described embodiments, which raises doubt as to whether Applicant had possession of the claimed invention at the time of filing. Therefore, claims 3, 4 and 21 are rejected for failing to comply with the written description requirement.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed inventions absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-4 and 6-13 are rejected under 35 U.S.C. 103 as being unpatentable over Synthesis of Brain Tumor MR Images for Learning Data Augmentation (published 22 March 2021) to Kim et al. (hereinafter “Kim”) in view of Mask-conditioned latent diffusion for generating gastrointestinal polyp images (published 19 July 2023) to Machacek et al. (hereinafter “Machacek”) and in further view of Synthesizing Diverse Lung Nodules Wherever Massively: 3D Multi-Conditional GAN-based CT Image Augmentation for Object Detection to Han et al. (hereinafter “Han”).
Regarding claim 1, Kim teaches a system for synthesizing a medical image of a synthesized medical abnormality (Kim, Fig. 1, “Synthesized MR images of Brain Tumor”), the system including:
synthesis circuitry configured to (see Kim at section 3: the conducted experiments require a computer or general-purpose processor to produce to results presented.):
obtain a descriptor input for the synthesized medical abnormality, the descriptor input detailing a selected characteristic (tumor representation) for the synthesized medical abnormality (Kim, Abstract, “Because tumors have complex characteristics, the proposed method simplifies them into concentric circles that are easily controllable”; section I, “Real tumor masks usually have complex features, such as grade, appearance, size, and location. Thus, these features of tumor masks are condensed and simplified to concentric circles.”);
generate an abnormality spatial mask (A tumor mask for inpainting the synthetic tumor is generated from a normal brain image, its brain mask, and the concentric circles feature descriptors. see Kim at Fig. 1. During synthesis (inference), the process is reversed starting with the circles. ) within a defined multidimensional space (two-dimensional space));
obtain a pre-abnormality image mapped into the defined multidimensional space (Kim, Fig. 1, “MR Images of Normal Brain”);
generate the medical image with the synthesized medical abnormality positioned in accord with the abnormality spatial mask (Kim, Fig. 1, “Synthesized MR images of Brain Tumor”); and
machine learning control circuitry (see Kim at section 3: the conducted experiments require a computer or general-purpose processor to produce to results presented.) configured to provide the abnormality spatial mask and the medical image for a medical machine learning system (Kim, Abstract, “In terms of data augmentation, the proposed method can successfully synthesize brain tumor images that can be used to train tumor segmentation networks or other deep neural networks.”), but does not teach that which is explicitly taught by Machacek.
Machacek teaches denoise, using a first diffusion model machine-learned network (Machacek, Figure 1, “Improved Diffusion”), a first noise input to generate an abnormality spatial mask (Machacek, section 3.1, “The improved diffusion model general synthetic mask images by first adding noise to a randomly selected mask image from the training set. This noise would then be gradually reversed through multiple steps until a synthetic mask image is generated.”) within a defined multidimensional space (see Machacek at Figure 1: the synthetic masks are two-dimensional);
obtain an image mapped into the defined multidimensional space (see Machacek at Figure 1: the real polyp images are two-dimensional); and
denoise, using a second diffusion model machine-learned network (Machacek, Figure 1, “Pre-Trained Latent Diffusion”, “The green box represents the conditional latent diffusion model which is used to generate synthetic polyp conditioned on input masks.”), a second noise input to generate the medical image with the synthesized medical abnormality positioned in accord with the abnormality spatial mask (see Machacek at Figure 1: the latent diffusion model has a forward and reverse noise process as the improved diffusion model.).
Kim discloses a medical image synthesis method that synthesizes brain tumor images from real healthy brain images using a generative model for generating a larger and more diverse training image set for machine learning. Thus, Kim shows that it was known in the art before the effective filing date of the claimed invention to use generative models to create more training examples, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, expanding a limited set of training examples. Machacek discloses a medical image synthesis method that synthesizes brain tumor images from real healthy brain images using a generative model, specifically two diffusion models, for generating a larger and more diverse training image set for machine learning. Thus, Machacek shows that it was known in the art before the effective filing date of the claimed invention to use diffusion models to create more training examples, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, expanding a limited set of training examples.
A person of ordinary skill in the art would have been motivated to replace the GAN-based image synthesis disclosed by Kim with diffusion-based image synthesis as disclosed by Machacek, to thereby generate synthetic brain tumor masks within an enclosed area surrounded by real healthy brain images using a feature descriptor of concentric circles for representing tumors. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to “overcome the issue of limited annotated data and train machine learning models more effectively” (Machacek, section 3.1).
Kim in view of Machacek does not teach that which is explicitly taught by Han.
Han teaches a bounding box (Han, section 3.1, “Volumes of Interest (VOIs)”) surrounding the abnormality spatial mask within a defined multidimensional space (Han, section 3.1, “We crop/resize various nodules to 32 × 32 × 32 voxels and replace them with noise boxes from a uniform distribution between [−0.5,0.5], while maintaining their 64 × 64 × 64 surroundings as Volumes of Interest (VOIs)—using those noise boxes, instead of boxes filled with the same voxel values, improves the training robustness”), the bounding box defining a spatial extent outside the abnormality spatial mask of effects due to the synthesized medical abnormality (The surrounding anatomy is preserved, thereby having effects on the synthesized image beyond the spatial extent of the nodule itself. See Han at section 3.1 and Figure 2.).
Kim in view of Machacek discloses generating synthetic abnormalities from a simplified abnormality representation (concentric circles). Kim and Machacek are analogous to the claimed invention for the same reasons provided above. Han discloses a conditional medical image synthesis and data augmentation method that generates new medical images using abnormality spatial masks and bounding boxes as inputs. Thus, Han shows that it was known in the art before the effective filing date of the claimed invention that synthetic medical abnormality image generation benefits from a larger surrounding region that includes contextual anatomical information, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, expanding a limited set of training examples.
A person of ordinary skill in the art would have been motivated to modify the image synthesis process of Kim in view of Machacek to utilize a bounding box surrounding the generated abnormality mask as disclosed by Han, to thereby promote surrounding anatomical structures to influence synthesis of the abnormality and adjacent tissue into another image. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to improve realism of the synthesized output images.
Regarding claim 2, Kim in view of Machacek and in further view of Han teaches the system of claim 1, where the descriptor input includes a descriptor in a predefined format associated with the first diffusion model machine-learned network (Kim, Abstract, “Because tumors have complex characteristics, the proposed method simplifies them into concentric circles that are easily controllable”; see Kim at Fig. 2: The concentric circles are in a two-dimensional format, same as the real input images and same as the multi-dimensional space where the inpainting occurs.).
Regarding claim 3, Kim in view of Machacek and in further view of Han teaches the system of claim 2, where the predefined format includes:
one or more concentric circles positioned within the defined multidimensional space (Kim, Fig. 2, “concentric circles”), but does not teach that which is explicitly further taught by Kim.
Kim further teaches one or more concentric spheres positioned within the defined multidimensional space (Kim, pg. 2197, “Although the proposed method was developed for two-dimensional images in this study, the method has the potential for expansion to three-dimensional cases. If the concentric circles are expanded to concentric spheres, the tumor characteristics can be controlled in three dimensions, and brain tumors synthesized by the proposed method can have continuous and coherent shapes along slice direction. Then, the synthesized dataset would be applicable to three-dimensional tumor segmentation algorithms”).
Kim in view of Machacek and in further view of Han is analogous to the claimed invention for the reasons provided above.
A person of ordinary skill in the art would have been motivated to modify the two-dimensional space of the image data and diffusion models disclosed by Kim in view of Machacek and in further view of Han to process three-dimensional medical images and generate synthetic three-dimensional images of a real three-dimensional medical image with an inpainted three-dimensional synthetic tumor as suggested by Kim, to thereby generate synthetic brain tumor masks within real healthy brain images using a feature descriptor of concentric spheres for representing tumors. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of more accurately representing real tumors by generating synthetic tumors having shapes in three dimensions instead of just two dimensions.
Regarding claim 4, Kim in view of Machacek and in further view of Han teaches the system of claim 1, where the descriptor input includes one or more spheres positioned within the defined multidimensional space to indicate one or more selected volume characteristics and/or a center-of-mass of the synthesized medical abnormality (Kim, pg. 2197, “concentric spheres”).
The rationale for obviousness is the same as provided for claim 3.
Regarding claim 6, Kim in view of Machacek and in further view of Han teaches the system of claim 1, where the first diffusion model machine-learned network is further configured to denoise the first noise input based on an anatomical mask positioned within the defined multidimensional space (see Kim at Fig. 1: the normal brain images produce a brain mask that is used to condition/constrain the inpainted area), the anatomical mask generated based on the pre-abnormality image (see Kim at Fig. 1).
Regarding claim 7, Kim in view of Machacek and in further view of Han teaches the system of claim 6, where:
the anatomical mask includes a brain mask (see Kim at Fig. 1: the normal brain images produce a brain mask that is used to condition/constraint the inpainted area); and
the synthesized medical abnormality includes a brain tumor (Kim, Fig. 1, “Synthesized MR images of Brain Tumor”).
Regarding claim 8, Kim in view of Machacek and in further view of Han teaches the system of claim 6, where:
the anatomical mask includes one or more anatomical boundaries (see Kim at Fig. 1: the normal brain images produce a brain mask that is used to condition/constraint the inpainted area); and
the first diffusion model machine-learned network is further configured to denoise the first noise input based on an anatomical mask by positioning and/or shaping the abnormality spatial mask to disallow boundary straddling (see Kim at Fig. 1: the normal brain images produce a brain mask that is used to condition/constrain the inpainted area).
Regarding claim 9, Kim in view of Machacek and in further view of Han teaches the system of claim 1, where:
the first diffusion model machine-learned network is further configured to denoise the first noise input iteratively using multiple denoising iterations (Machacek, section 3.1, “The improved diffusion model general synthetic mask images by first adding noise to a randomly selected mask image from the training set. This noise would then be gradually reversed through multiple steps until a synthetic mask image is generated.”); and
the second diffusion model machine-learned network is further configured to denoise the second noise input iteratively using multiple denoising iterations (see Machacek at Figure 1: both diffusion models implement diffusion processes, meaning each model performs iterative denoising across multiple timesteps.).
The rationale for obviousness is the same as provided for claim 1.
Regarding claim 10, Kim in view of Machacek and in further view of Han teaches the system of claim 1, where the first diffusion model machine-learned network and/or second diffusion model machine-learned network include diffusion model machine-learned networks trained using an image set generated using a ground truth image (Machacek, section 3.1, “the first step is to obtain a training set for real mask images”) with increasing levels of noise added (Machacek, section 3.1, “This noise would then be gradually reversed through multiple steps until a synthetic mask image is generated.”).
The rationale for obviousness is the same as provided for claim 1.
Regarding claim 11, Kim in view of Machacek and in further view of Han teaches the system of claim 1, where the synthesized medical abnormality includes a tumor (Kim, Fig. 1, “Synthesized MR images of Brain Tumor”).
Regarding claim 12, Kim in view of Machacek and in further view of Han teaches the system of claim 1, where the pre-abnormality image includes a magnetic resonance imaging (MRI) image (Kim, Abstract, “Our method can synthesize a huge number of brain tumor multicontrast MR images from numerous healthy brain multicontrast MR images and various concentric circles.”).
Regarding claim 13, Kim in view of Machacek and in further view of Han teaches the system of claim 1, where the defined multidimensional space includes a two-dimensional space (see Kim at Fig. 1) or a three-dimensional space (see Kim at pg. 2197, “three dimensions”).
The rationale for obviousness is the same as provided for claim 3.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Machacek, in view of Han, in further view of U.S. Pat. Appl. Pub. No. 20230377226 (filed 19 May 2023) to Saharia et al. (hereinafter “Saharia”), and in further view of LLM Itself Can Read and Generate CXY Images (published 24 May 2023) to Lee et al. (hereinafter “Lee”).
Regarding claim 5, Kim in view of Machacek and in further view of Han teaches the system of claim 1, but does not teach that which is explicitly taught by Saharia.
Saharia teaches where:
the descriptor input includes a vector descriptor of the synthesized medical abnormality (Saharia, par. 52, “an image generation system that combines the power of text encoder neural networks (e.g., large language models (LLMs)) with a sequence of generative neural networks (e.g., diffusion-based models) to deliver text-to-image generation with a high degree of photorealism, fidelity, and deep language understanding.”); and
obtaining the descriptor input includes applying a large language model (Saharia, par. 52, “large language models (LLMs)”) to generate the vector descriptor (Saharia, par. 64, “the system 100 can generate medical images including, but limited to, magnetic resonance imaging (MM) images, computed tomography (CT) images, ultrasound images, x-ray images, and so on”; Contextual embeddings generated from a text prompt for generating a medical image visually representing the text prompt).
Kim in view of Machacek and in further view of Han is analogous to the claimed invention for the reasons provided above. Saharia discloses diffusion-based text-to-image synthesis for generating synthetic medical images with an LLM as a pre-trained natural language text encoder (see Saharia at par. 64). Thus, Saharia shows that it was known in the art before the effective filing date of the claimed invention to use LLMs to receive text prompts provided to an LLM to produce vector descriptors of the prompt for specifying what the user wants the synthesized image to include, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, expanding a limited set of training examples.
A person of ordinary skill in the art would have been motivated to add an LLM as disclosed by Saharia to the image synthesis method disclosed by Kim in view of Machacek and in further view of Han, to thereby generate text encodings from a prompt provided by a user to specify what is included in the synthesized medical image. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of increasing user-friendliness of the training interface for training interaction.
Kim in view of Machacek, in view of Han and in further view of Saharia does not teach that which is explicitly taught by Lee.
Lee teaches descriptor input includes a vector descriptor indicating one or medical classifications of the synthesized medical abnormality (Lee, Figure 1, “Generate a CXR image for the following diagnosis: left lung pneumonia”); and
obtaining the descriptor input includes applying a large language model to clinical description of a model medical abnormality to generate the vector descriptor (see Lee at Figure 1: LLM can generate text or image output based on text and/or image input tokens).
Kim in view of Machacek, in view of Han, and in further view of Saharia is analogous to the claimed invention for the reasons provided above. Lee discloses medical image synthesis that uses an LLM to generate new medical images that (ideally) correspond to model (ground truth) clinical descriptions of particular medical classifications. Thus, Lee shows that it was known in the art before the effective filing date of the claimed invention to use LLMs to receive text prompts of clinical descriptions that medically classify the expected appearance of the synthesized output image, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, expanding a limited set of training examples.
A person of ordinary skill in the art would have been motivated to finetune the LLM disclosed by Kim in view of Machacek, in view of Han, and in further view of Saharia according to the framework disclosed by Lee, to thereby generate synthetic brain tumor images responsive to user-provided prompts that include the same type of textual description used by clinicians when describing certain types and appearances of specific brain tumors. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of giving a user greater control over the specificity of the synthesized output.
Claims 14, 15 and 17-21 are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Machacek, in view of Han and in further view of U.S. Pat. Appl. Pub. No. 20240169500 (filed 22 November 2022) to Zheng et al. (hereinafter “Zheng”).
Regarding claim 14, Kim teaches a multiple-stage method for synthesizing a medical image of a synthesized medical abnormality (Kim, Fig. 1, “Synthesized MR images of Brain Tumor”), the method including:
obtain an abnormality spatial mask (see Kim at Fig. 1: tumor mask generated from normal brain image, its brain mask, and the concentric circles feature descriptors) within a defined multidimensional space (two-dimensional space);
after obtaining the abnormality spatial mask, using a pre-abnormality image and a model machine-learned network obtain the medical image of the synthesized medical abnormality (Kim, Fig. 1, “Synthesized MR images of Brain Tumor”), the medical image consistent with pre-abnormality image modified to include the synthesized medical abnormality inserted in accord with the abnormality spatial mask (see Kim at Fig. 1: compare “MR Images of Normal Brain” to the “Synthesized MR images of Brain Tumor”), but does not teach that which is explicitly taught by Machacek.
Machacek teaches denoising, using a first diffusion model machine-learned network at a first denoising stage (Machacek, Figure, 1, “Improved Diffusion”), a first noise input to obtain an abnormality spatial mask (Machacek, section 3.1, “The improved diffusion model general synthetic mask images by first adding noise to a randomly selected mask image from the training set. This noise would then be gradually reversed through multiple steps until a synthetic mask image is generated.”) within a defined multidimensional space (see Machacek at Figure 1: the synthetic masks are two-dimensional);
after obtaining the abnormality spatial mask, denoising, using an image (see Machacek at Figure 1: real polyp images) and a second diffusion model machine-learned network at a second denoising stage network (Machacek, Figure 1, “Pre-Trained Latent Diffusion”, “The green box represents the conditional latent diffusion model which is used to generate synthetic polyp conditioned on input masks.”), a second noise input to obtain the medical image of the synthesized medical abnormality (see Machacek at Figure 1: the latent diffusion model has a forward and reverse noise process as the improved diffusion model).
The rationale for obviousness is the same as provided for claim 1.
Kim in view of Machacek does not teach that which is explicitly taught by Han.
Han teaches a bounding box (Han, section 3.1, “Volumes of Interest (VOIs)”) surrounding the abnormality spatial mask within a defined multidimensional space (Han, section 3.1, “We crop/resize various nodules to 32 × 32 × 32 voxels and replace them with noise boxes from a uniform distribution between [−0.5,0.5], while maintaining their 64 × 64 × 64 surroundings as Volumes of Interest (VOIs)—using those noise boxes, instead of boxes filled with the same voxel values, improves the training robustness”), the bounding box defining a spatial extent outside the abnormality spatial mask of effects due to the synthesized medical abnormality (The surrounding anatomy is preserved, thereby having effects on the synthesized image beyond the spatial extent of the nodule itself. See Han at section 3.1 and Figure 2.).
The rationale for obviousness is the same as provided for claim 1.
Kim in view of Machacek and in further view of Han does not teach that which is explicitly taught by Zheng.
Zheng teaches providing the medical image to a training interface for training interaction (Zheng, par. 51, “user interface 220 receives an image including a first region that includes content and a second region to be inpainted. In some examples, user interface 220 provides the image as an input to diffusion model 225, where the intermediate output image is conditioned based on the first region of the image. In some examples, user interface 220 receives a user input indicating the second region to be inpainted.”; see also Zheng at pars. 121-22).
Kim in view of Machacek and in further view of Han is analogous to the claimed invention for the reasons provided above. Zheng discloses a user interface for training diffusion models for inpainting. Thus, Zheng shows that it was known in the art before the effective filing date of the claimed invention to use user interfaces for training diffusion models to generate synthetic images, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, expanding a limited set of training examples.
A person of ordinary skill in the art would have been motivated to add a user interface as disclosed by Zheng to the device performing the method of Kim in view of Machacek and in further view of Han, to thereby enable a user to visually inspect pre-abnormality images to select a subset of images for training a machine learning model to synthesize new types of synthetic tumor images or fine-tune the existing model. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of providing a simple way to use and re-train the model.
Regarding claim 15, Kim in view of Machacek, in view of Han, and in further view of Zheng teaches the multiple-stage denoising method of claim 14, where denoising the second noise input to obtain the medical image includes obtaining an image to supplement a training set of medical images with deficient occupancy for medical images with a medical abnormality with at least a selected characteristic (having the synthetic image included therein) present in the synthesized medical abnormality (see Machacek at section 2: “real-world datasets … have some limitations” including “diversity”, meaning a “narrow range of images” will not be representative of the many ways medical abnormalities are presented in real life and may incorrectly influence the model to misclassify an image object, for example mistaking a healthy fold in a GI tract as being a polyp (i.e., a deviation from a medically established relative probability for occurrences of the synthetic object produced by the model), “Annotation quality”, meaning datasets include mislabeled images and lack the synthetic images that can be assumed to always include the correctly-labeled object (i.e., an absence of medical images with the correct label of the synthetic image), and image Size”, meaning a deficiency is there are not enough images (i.e., below a threshold) in the dataset.).
The rationale for obviousness is the same as provided for claim 14.
Regarding claim 17, Kim in view of Machacek, in view of Han and in further view of Zheng teaches the multiple-stage denoising method of claim 14, where denoising the first noise input to obtain the abnormality spatial mask includes denoising the first noise input to obtain the abnormality spatial mask within an anatomical mask positioned within the defined multidimensional space (see Kim at Fig. 1: the brain tumor is synthesized within the two-dimensional space of the tumor mask within the overall brain mask).
Regarding claim 18, Kim in view of Machacek, in view of Han and in further view of Zheng teaches the multiple-stage denoising method of claim 17, where:
the anatomical mask includes one or more anatomical boundaries (see Kim at Fig. 1: the tumor mask has an outer boundary.); and
at a time that the abnormality has a center-of-mass near the one or more anatomical boundaries (see Kim at Fig. 1: the tumor mask restricts where the synthesized data is added. The center of mass of the tumor in the masked area is always somewhere near the boundary. Furthermore, the tumor is represented as concentric circles. The center of the concentric circles is an approximation of a center of mass of a tumor.):
denoising the first noise input includes shaping the abnormality spatial mask to disallow boundary straddling (see Kim at Fig. 1: the normal brain images produce a brain mask that is used to condition/constraint the inpainted area. The determined shape of a tumor mask disallows a shared boundary with the surrounding healthy brain image).
Regarding claim 19, Kim in view of Machacek, in view of Han, and in further view of Zheng teaches the multiple-stage denoising method of claim 17, where:
the anatomical mask includes a brain mask (see Kim at Fig. 1: brain mask output from the normal brain images); and
the abnormality includes a brain tumor (see Kim at Fig. 1: tumor mask).
Claim 20 substantially corresponds to claim 14, mainly differing by including claim limitations substantially similar to limitations in claims 1 and 2 (taught by Kim), and further specifying that the abnormality spatial mask spatially defines the synthesized medical abnormality with the selected characteristic (see Kim at Fig. 1: the concentric circles represent tumor characteristics within the spatially defined tumor mask area.). Therefore, claim 20 is rejected for the same reasons for obviousness as provided for claim 14.
Regarding claim 21, Kim in view of Machacek, in view of Han, and in further view of Zheng teaches denoising method of claim 20, where the predefined format includes:
one or more concentric circles positioned within the defined multidimensional space (Kim, Fig. 2, “concentric circles”). but does not teach that which is explicitly further taught by Kim.
Kim further teaches one or more concentric spheres positioned within the defined multidimensional space (Kim, pg. 2197, “Although the proposed method was developed for two-dimensional images in this study, the method has the potential for expansion to three-dimensional cases. If the concentric circles are expanded to concentric spheres, the tumor characteristics can be controlled in three dimensions, and brain tumors synthesized by the proposed method can have continuous and coherent shapes along slice direction. Then, the synthesized dataset would be applicable to three-dimensional tumor segmentation algorithms”).
The rationale for obviousness is the same as provided for claim 3.
Conclusion
Applicant's amendment necessitated the new grounds of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN P POTTS whose telephone number is (571)272-6351. The examiner can normally be reached M-F, 9am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 571-272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN P POTTS/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights