DETAILED ACTION
Notice of Pre-AIA or AIA Status
Claims 1-20 are pending in this application. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 8 and 17 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Swersky et al. (WIPO Publication WO 2025/068600 A1, hereby referred to as “Swersky” filed September 30, 2024 with priority dating back to September 28, 2023).
Consider Claims 1, 8 and 17.
Swersky teaches:
- 1. A computer-implemented method to train machine learning models, the method comprising: / 8. A computer system comprising one or more computing devices, the computing system configured to perform operations to train image generation models, the operations comprising:/ 17. One or more non-transitory computer-readable media that store a generator model that has been trained by performance of training operations, the training operations comprising: (Swersky: pages 1-3, Summary, More specifically, this specification describes how a system can train the diffusion neural network through reinforcement learning using a differentiable reward function, e.g., after the diffusion neural network has already been trained on an objective that does not make use of the reward function. To overcome this mismatch, this specification proposes an efficient approach for gradient-based reward fine-tuning based on differentiating through the diffusion sampling process. That is, a training system can perform the described training approach after "pretraining" the diffusion neural network, e.g., on a large data set of diverse data items, in order to align the diffusion neural network to enable the desired behavior after training, i.e., to cause the diffusion neural network to generate data items that have the qualities measured by the reward function after training.)
- 1. obtaining, by a computing system comprising one more computing devices, a pre-trained denoising diffusion model comprising a set of pre-trained model parameters; / 8. obtaining a training example comprising a training image; performing one or more forward diffusion steps on the training example to generate a partially noised training example; / 17. obtaining, by a computing system comprising one more computing devices, a pre-trained denoising diffusion model comprising a set of pre-trained model parameters; (Swersky: page 6, When the output data item is generated in a latent space, the system can generate a final output data item in output space by processing the output data item in the latent space using a decoder neural network, e.g., one that has been pre-trained in an auto-encoder framework. During training, the system can use an encoder neural network, e.g., one that has been pre-trained jointly with the decoder in the auto-encoder framework, to encode target data items in the output space to generate target outputs for the diffusion neural network in the latent space. FIG. 1 is a diagram of an example training system 100. The training system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented. The system 100 is a system that trains a diffusion neural network 110 that is used to generate an output (final) data item 112 given a conditioning input 102. In particular, the system 100 trains the diffusion neural network 110 so that the output (final) data item 112 that has the one or more desired properties characterized by the conditioning input 102. More specifically, the system 100 performs "fine-tuning," i.e., further training, of the diffusion neural network 110 using a reward function 120. In other words, prior being trained using the reward function 120, the system 100 or another training system has trained the diffusion neural network on a different objective, e.g., one that does not make use of the reward function 120. In general the diffusion neural network can have been trained conventionally, using any diffusion model objective. As one example, the diffusion neural network can have been trained on a set of training data items on a diffusion score matching objective or a variant thereof. The reward function 120 can be any appropriate differentiable reward function that maps an input that includes (i) a data item or (ii) a latent representation of a data item to a reward score 122. Optionally, the reward function input can also include the conditioning input 102 or a representation of the conditioning input 102.)
- 1. instantiating, by the computing system, a first instance of the pre-trained denoising diffusion model as a generator model having the set of pre-trained model parameters; / 8. performing an additional forward diffusion step on the partially noised training example to generate an additionally noised training example; / 17. instantiating, by the computing system, a first instance of the pre-trained denoising diffusion model as the generator model having the set of pre-trained model parameters; (Swersky: page 12 As another example, the diffusion neural network 110 can include one or more other types of neural network layers that are conditioned on the one or more embeddings. Examples of such layers include Feature-wise Linear Modulation (FiLM) layers, layers with conditional gated activation functions, and so on. The diffusion input at any given updating iteration can also include data defining a noise level for the iteration. Generally, each updating iteration has a corresponding time step t and the noise level for the iteration depends on the time step. For example, the noise level can be a decreasing function of the time step t. Examples of such functions include a linear function, a cosine function, and a sigmoid function. In these cases, data identifying the noise level, the time step, or both can be embedded using an appropriate neural network, e.g., a multi-layer perceptron (MLP) and used to condition the diffusion neural network 110 as described above for the conditioning input. FIG. 2 is a flow diagram of an example process 200 for training a diffusion neural network using reward scores. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 200. The system initializes a representation of a data item (step 202). For example, the system can initialize the representation by sampling the values in the representation from a distribution, e.g., a Gaussian distribution.)
- 1. instantiating, by the computing system, a second instance of the pre-trained denoising diffusion model as a discriminator model having the set of pre-trained model parameters; / 8. processing the additionally noised training example with a generator model to generate fully de-noised prediction, wherein the generator model comprises a denoising diffusion model; / 17. instantiating, by the computing system, a second instance of the pre-trained denoising diffusion model as a discriminator model having the set of pre-trained model parameters; (Swersky: page 17 By repeatedly performing this training for different sets of one or more conditioning inputs, the system effectively "fine-tunes" the diffusion neural network to generate outputs
that result in increased reward scores. FIG. 3 is a flow diagram of an example process 300 for determining a variance reduction term of the loss function. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300. The variance reduction term is also referred to as a "second" term in the loss function in this specification. The system can perform the process 300 at each of one or more noise iterations. The system samples noise for the noise iteration (step 302). For example, the system can sample the values in the noise from the same distribution used to initialize the representation of the data item, e.g., a Gaussian distribution. The system applies the noise to the final representation to generate a noisy representation (step 304). The system processes an input that includes the noisy representation using the diffusion neural network to generate a denoising output for the noise iteration (step 306). The system then updates the noisy representation using the denoising output for the noise iteration to generate an updated noisy representation that is an estimate of the final representation (step 308). The system generates a new reward input from the updated noisy representation (step 310). When the representation of the data item is a representation in the output space, the system can directly include the updated noisy representation in the new reward input. Optionally, the reward input can also include other information, e.g., the conditioning input. When the representation of the data item is a latent representation in a latent space, the system can process the updated noisy representation using a decoder neural network to generate a noisy data item and include the noisy data item in the new reward input. The system applies the reward function to the new reward input to generate a new reward score (step 312).)
- 1. and finetuning, by the computing system, at least the generator model on a finetuning dataset, / 8. performing one or more forward diffusion steps on the fully de-noised prediction to generate a partially re-noised prediction; processing the partially re-noised prediction with a discriminator model to generate a discriminator prediction; / 17. and finetuning, by the computing system, at least the generator model on a finetuning dataset, (Swersky: page 18 When the representation of the data item is a latent representation in a latent space, the system can process the updated noisy representation using a decoder neural network to generate a noisy data item and include the noisy data item in the new reward input. The system applies the reward function to the new reward input to generate a new reward score (step 312). Generally, the loss function described above includes a second term that measures the new reward scores for the one or more noise iterations. When there are a plurality of noise iterations, the second term can measure a combination of, e.g., an average of, the new reward scores. When the loss function includes the second term, training the neural network on the loss function can include backpropagating gradients of the second term through the reward function and into the noise iterations but not through any of the sampling iterations. That is, the system can insert a stop gradient to prevent from computing gradients ;of the second term with respect to the final representation or any representations generated at any preceding sampling iterations. An example technique for training the diffusion neural network (e having parameters 8 when the diffusion sampler is DDIM and there are Tis; Table 1.)
- 1. wherein finetuning, by the computing system, the generator model comprises modifying, by the computing system, / 8. and updating one or more parameter values of at least the generator model based on a loss function, wherein the loss function comprises: / 17. wherein finetuning, by the computing system, the generator model comprises modifying, by the computing system, (Swersky: page 21 As another example, the system can train a single instance of the diffusion neural network a different reward function using the LoRA training technique. In this example, the system can generate a new instance of the diffusion neural network by computing, for each network parameter in the first subset that is updated through the LoRA training, a weighted sum of the value of the network parameter after the fine-tuning and the pre-trained value of the network parameter, with the weight for each being determined by how strongly the property ( or properties) measured by the reward function should be reflected in new data items generated after training. FIG. 5 shows an example 500 of the performance of the described techniques when fine-tuned on a reward function that measures an aesthetic score. In particular, as can be seen from the example 500, variants of the described techniques outperform two existing techniques (ReFL and DDPO) as well as the pre-trained model (Stable Diffusion) and using prompt engineering across a range ofreward queries (training conditioning inputs).)
- 1. the set of pre-trained model parameters of the generator model based on a generative adversarial network loss term that provides a loss value based on an output of the discriminator model. / 8. a reconstruction loss term that generates a reconstruction loss value based on the training example and the fully de-noised prediction, and a GAN loss term that generates a GAN loss value based on the discriminator prediction. / 17. the set of pre-trained model parameters of the generator model based on a generative adversarial network loss term that provides a loss value based on an output of the discriminator model. (Swersky: page 8, As another example, the reward function 120 can include a reward that causes the diffusion model to generate adversarial examples. That is, the system can fine-tune the diffusion model such that output data items generated based on a prompt for a classy are classified as a different class y' by a pre-trained classifier for data items of the particular type. For example, as the reward, the system can use the negative cross-entropy to the target class by the pre-trained classifier. As another example, the reward function 120 can include one or more hard-coded differentiable reward functions.)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhao (US PGPub US20250045912A1, hereby referred to as “Zhao”) , in view of Swersky et al. (WIPO Publication WO 2025/068600 A1, hereby referred to as “Swersky” filed September 30, 2024 with priority dating back to September 28, 2023).
Consider Claims 1, 8 and 17.
Zhao teaches:
- 1. A computer-implemented method to train machine learning models, the method comprising: / 8. A computer system comprising one or more computing devices, the computing system configured to perform operations to train image generation models, the operations comprising:/ 17. One or more non-transitory computer-readable media that store a generator model that has been trained by performance of training operations, the training operations comprising: (Zhao: abstract, A system for synthesizing medical images including synthesizing medical abnormalities has multiple diffusion model based denoising stages. At a first denoising stage, a machine-learned network denoises a first noise input to obtain an abnormality spatial mask detailing positional and structural characteristics of the synthesized medical abnormality. At a second denoising stage, a machine-learned network denoises a second noise input based on the abnormality spatial mask and a pre-abnormality image to obtain a synthesized medical image that corresponds to the pre-abnormality image with the synthesized medical abnormality inserted consistent with the abnormality spatial mask. [0015] FIG. 1 shows an example image synthesis system (ISS) 100. The example ISS 100 includes multiple denoising stages 110, 150. Referring also to FIG. 2 , example image synthesis logic (ISL) 200 is shown. The ISL 200, which may be implemented on circuitry, may govern the operation of the ISS 100. [0016] In various implementations, the denoising stages may include diffusion model neural networks.)
- 1. obtaining, by a computing system comprising one more computing devices, a pre-trained denoising diffusion model comprising a set of pre-trained model parameters; / 8. obtaining a training example comprising a training image; performing one or more forward diffusion steps on the training example to generate a partially noised training example; / 17. obtaining, by a computing system comprising one more computing devices, a pre-trained denoising diffusion model comprising a set of pre-trained model parameters; (Zhao: [0016] In various implementations, the denoising stages may include diffusion model neural networks. The diffusion model neural networks may be trained to remove noise from a noise image. When trained to remove noise to generate a specific type of images, e.g., medical images such as MRI images, CT scans, or other medical images, a diffusion model network may generate a synthesized image of that type from a pure noise input. In various implementations, an iterative denoising process may be used. For example, for a given denoising stage, the ISL 200 may cause the denoising stages to iterative repeat the denoising process to successively remove a selected noise level from the image to progress towards a fully denoised image. In some implementations, a specific neural network for each iteration may be used. In other words, the neural network for each iteration may be specifically trained to remove noise from an image with a specific noise level to produce an image at a specific output noise level (e.g., that may correspond to the input noise level for the next neural network in the iterative chain). [0017]-[0018])
- 1. instantiating, by the computing system, a first instance of the pre-trained denoising diffusion model as a generator model having the set of pre-trained model parameters; / 8. performing an additional forward diffusion step on the partially noised training example to generate an additionally noised training example; / 17. instantiating, by the computing system, a first instance of the pre-trained denoising diffusion model as the generator model having the set of pre-trained model parameters; (Zhao: [0016], [0017] In various implementations, convolutional neural networks may be trained to perform the diffusion model denoising. [0018] The first denoising stage 110 may perform denoising on a first noise input 112 (210). The first noise input may occupy a defined multidimensional space, such as a three-dimensional (3D) volume or a two-dimensional (2D) plane. The defined multidimensional space may include the space occupied by the resultant synthesized medical image. Accordingly, the dimensionality and dimensions of the noise input (e.g., at each of the multiple denoising stages) may match the dimensionality and dimensions of the output synthesized medical image, which may, for example, be a 3D or 2D image. Thus, the output corresponds to a noise-removed input. [0019] In various implementations, the first denoising stage 110 may obtain a descriptor 120 as an input (212). As discussed above, the descriptor may detail selected characteristics of the synthesized abnormality to guide the medical image synthesis to outputs with those selected characteristics. The format of the descriptor may be specific to the denoising system. [0020] For example, the descriptor may be formatted as one or more spheres (or other objects) indicating one or more size, volume, and/or structural characteristics for the abnormality and/or affected tissue of the anatomy surrounding the tissue. In some implementations, the spheres may be concentric. The center of the concentric spheres may indicate a position for a center-of-mass of the abnormality)
- 1. instantiating, by the computing system, a second instance of the pre-trained denoising diffusion model as a discriminator model having the set of pre-trained model parameters; / 8. processing the additionally noised training example with a generator model to generate fully de-noised prediction, wherein the generator model comprises a denoising diffusion model; / 17. instantiating, by the computing system, a second instance of the pre-trained denoising diffusion model as a discriminator model having the set of pre-trained model parameters; (Zhao: [0020] For example, the descriptor may be formatted as one or more spheres (or other objects) indicating one or more size, volume, and/or structural characteristics for the abnormality and/or affected tissue of the anatomy surrounding the tissue. In some implementations, the spheres may be concentric. The center of the concentric spheres may indicate a position for a center-of-mass of the abnormality. The volume of each of the one or more spheres may correspond to the volume of a particular segment of the abnormality. For example for a tumor type abnormality, a first sphere may indicate a core density for a tumor, and a second sphere may indicate a density of another peripheral region of the tumor, a third sphere may indicate another volume characteristic of the tumor and/or a volume of non-tumor tissue affected by the insertion of the tumor. Various spheres may be included. In some cases, the spheres may further indicate the total volume of the abnormality and/or the affected tissue region. For example, in a concentric configuration, rather than the volume of each sphere indicating a volume for a segment, the shell formed by a given sphere to the next innermost sphere defines the volume for the segment. Thus, the volume of the innermost sphere defines a first segment volume, then each successive sphere going outward defines a segment volume based on the differential between the volume of that particular sphere and the volume of the sphere just a size smaller. Therefore, the volume of the outermost sphere defines the total volume of the abnormality and/or the affected tissue region. [0021] For example, the descriptor 120 may include a vector indicating one or more characteristics of the abnormality based on definitions for the various entries within the vector. In some cases, image data and/or natural language descriptions may be transformed into descriptors using one or more ML systems. Based on the descriptor, the first denoising stage may generate an abnormality spatial mask 140 with the selected characteristics of the descriptor. [0022] In various implementations, the descriptor 120 may be generated using a large language model (LLM) analysis, a keyword analysis, a natural language processing (NLP) operation, or other language analysis of a clinical description of selected characteristics to generate a descriptor 120 for input to the first denoising stage 110. For example, a LLM analysis may be used to select sphere sizes and position based on clinical notes. Similarly, a vector-type descriptor may be generated using such a language analysis. In an example, language processing may be used to create an interface in which an operator may request medical images with abnormalities with specific characteristics.)
- 1. and finetuning, by the computing system, at least the generator model on a finetuning dataset, / 8. performing one or more forward diffusion steps on the fully de-noised prediction to generate a partially re-noised prediction; processing the partially re-noised prediction with a discriminator model to generate a discriminator prediction; / 17. and finetuning, by the computing system, at least the generator model on a finetuning dataset, (Zhao: [0028] In some cases, to train the machine-learned networks to perform the denoising, a ground truth image may have set step levels of noise added. Each of the step levels may be used as a training tuple, with the images with one step less noise added being the ground truth for the image with one step more noise added. [0029] In various implementations, the first denoising stage 110 may obtain an anatomical mask 130 as an input (214). The anatomical mask 130 may detail the spatial layout of the anatomical background in the multidimensional space defined in the first noise sample. The anatomy may be used by the first denoising stage 110 to determine positioning and shape of the abnormality. For example, the anatomical mask 130 may include anatomical boundaries. The first denoising stage may shape and position the abnormality to handle the anatomical boundaries. For example, some rigid boundaries, such as a skull in the case of the brain tumor type abnormality: may disallow straddling (e.g., the tumor may not be on be on both sides), may disallow crossing (e.g., the tumor may not be outside the skull), and/or may disallow deformation (e.g., the tumor may not deform the skull). For flexible boundaries, straddling and/or crossing may be disallowed, and deformation of the boundary may be permitted. Other boundaries may allow some selection of crossing, straddling, and/or deformation while disallowing the complementary selection. [0030] In various implementations, the particular way the anatomical mask 130 is used by the first denoising stage may be controlled via the training of the one or more neural networks on which the first denoising stage is implemented. [0031] The second denoising stage 150 may denoise a second noise 152 input using a pre-abnormality image 160 and the abnormality spatial mask 140 as inputs to produce a synthesized medical image 170 (220). Thus for the second denoising stage, the ISL 200 may obtain the pre-abnormality image 160 (222) and provide the abnormality spatial mask to the second denoising stage (224). [0045]-[0046])
- 1. wherein finetuning, by the computing system, the generator model comprises modifying, by the computing system, / 8. and updating one or more parameter values of at least the generator model based on a loss function, wherein the loss function comprises: / 17. wherein finetuning, by the computing system, the generator model comprises modifying, by the computing system, (Zhao: [0037] In some implementations used for generation of ML training data, the synthesized medical image 170 and abnormality spatial mask 140 may be provided to a machine learning training interface 190 (e.g., for training other machine learning systems) as a training tuple 180 (230). The training tuple 180 may define the synthesized medical image 170 as a training input and the abnormality spatial mask 140 as a ground truth output. Thus, the training tuple provides a medical image input paired with a segmented/identified abnormality output. [0038] FIG. 3 shows an example synthesis computation environment (SCE) 300, which, for example, may operate as synthesis circuitry. The SCE 300 may include system logic 314 to support implementation of the example ISL 200. The system logic 314 may include processors 316, memory 320, and/or other circuitry, which may be used to implement the multiple denoising stages, provide inputs to the stages, pass information between the stages, iterate denoising operations, execute machine-learned networks, and/or perform other image synthesis operations. [0039] In some implementations, the SCE 300 may receive pre-abnormality images obtained from an imaging device 301. In some cases, an anatomical mask generation system 302 may generate masks from the pre-abnormality images. Various systems for anatomical segmentation of non-abnormality medical images may be used to produce the anatomical mask based on the pre-abnormality images. In some implementations, processing to generate anatomical masks may occur on processing system logically or physically separate from the SCE 300. [0045] In an illustrative example image synthesis system 400 shown in FIG. 4 , brain-tumor-type abnormalities are synthesized within medical images of brains. In the illustrative example image synthesis system 400, the two pipeline stages 410, 450 of diffusion model based image synthesis are used. The pipeline uses a set of concentric spheres 420 as the starting point. The illustrative example image synthesis system 400 first synthesizes the multi-label tumor mask 440 from the concentric spheres 512 and then synthesizes the output medical image 470 from the multi-label tumor mask. This example illustrative design enables the model to control the synthetic tumor's location, size and volume ratios of different tumor parts with the configurable parameters of the concentric spheres' location, size and volume ratios, which can synthesize tumors with specified locations, sizes and structures by configuring these parameters. [0046] In the first stage 410, the tumor mask 440 is synthesized from the Gaussian white noise 412 image by iterative denoising operations of the diffusion model 411, where T is the number of iterations. The illustrative example denoising model uses a UNet-like deep convolutional neural network, and shallow N-block unmasked transformers with alternating layers of (i) self-attention, (ii) a position-wise MLP and (iii) a cross-attention layer were added to the lower resolution levels of the UNet. The tumor mask 440 (e.g., which corresponds to the abnormality spatial mask) contains segmentation masks of different parts of tumors with different labels, such as the necrotic and non-enhancing tumor core, the peritumoral edema, the GD-enhancing tumor. The synthesis is conditioned on the anatomical mask, e.g. the brain mask, and a set of concentric spheres 420 by concatenating them with the Gaussian white noise 412 image along the channel dimension.)
- 1. the set of pre-trained model parameters of the generator model based on a generative adversarial network loss term that provides a loss value based on an output of the discriminator model. / 8. a reconstruction loss term that generates a reconstruction loss value based on the training example and the fully de-noised prediction, and a GAN loss term that generates a GAN loss value based on the discriminator prediction. / 17. the set of pre-trained model parameters of the generator model based on a generative adversarial network loss term that provides a loss value based on an output of the discriminator model.(Zhao: [0039] In some implementations, the SCE 300 may receive pre-abnormality images obtained from an imaging device 301. In some cases, an anatomical mask generation system 302 may generate masks from the pre-abnormality images. Various systems for anatomical segmentation of non-abnormality medical images may be used to produce the anatomical mask based on the pre-abnormality images. In some implementations, processing to generate anatomical masks may occur on processing system logically or physically separate from the SCE 300. [0040] The memory 320 may be used to store: image data 322, descriptors 324, and/or mask data 326 used in synthesis of medical images and abnormality masks. The memory 320 may further store parameters 321, such as machine-learned network states, parameters for abnormality characteristic generation, and/or other synthesis parameters. The memory may further store executable code 329, which may support input data handling, machine-learned network operation and/or other image synthesis functions.)
Zhao does not specifically teach the term “finetuning” from the following limitations:
- 1. and finetuning, by the computing system, at least the generator model on a finetuning dataset, / 8. performing one or more forward diffusion steps on the fully de-noised prediction to generate a partially re-noised prediction; processing the partially re-noised prediction with a discriminator model to generate a discriminator prediction; / 17. and finetuning, by the computing system, at least the generator model on a finetuning dataset,
- 1. wherein finetuning, by the computing system, the generator model comprises modifying, by the computing system, / 8. and updating one or more parameter values of at least the generator model based on a loss function, wherein the loss function comprises: / 17. wherein finetuning, by the computing system, the generator model comprises modifying, by the computing system,
Swersky teaches:
- 1. A computer-implemented method to train machine learning models, the method comprising: / 8. A computer system comprising one or more computing devices, the computing system configured to perform operations to train image generation models, the operations comprising:/ 17. One or more non-transitory computer-readable media that store a generator model that has been trained by performance of training operations, the training operations comprising: (Swersky: pages 1-3, Summary, More specifically, this specification describes how a system can train the diffusion neural network through reinforcement learning using a differentiable reward function, e.g., after the diffusion neural network has already been trained on an objective that does not make use of the reward function. To overcome this mismatch, this specification proposes an efficient approach for gradient-based reward fine-tuning based on differentiating through the diffusion sampling process. That is, a training system can perform the described training approach after "pretraining" the diffusion neural network, e.g., on a large data set of diverse data items, in order to align the diffusion neural network to enable the desired behavior after training, i.e., to cause the diffusion neural network to generate data items that have the qualities measured by the reward function after training.)
- 1. obtaining, by a computing system comprising one more computing devices, a pre-trained denoising diffusion model comprising a set of pre-trained model parameters; / 8. obtaining a training example comprising a training image; performing one or more forward diffusion steps on the training example to generate a partially noised training example; / 17. obtaining, by a computing system comprising one more computing devices, a pre-trained denoising diffusion model comprising a set of pre-trained model parameters; (Swersky: page 6, When the output data item is generated in a latent space, the system can generate a final output data item in output space by processing the output data item in the latent space using a decoder neural network, e.g., one that has been pre-trained in an auto-encoder framework. During training, the system can use an encoder neural network, e.g., one that has been pre-trained jointly with the decoder in the auto-encoder framework, to encode target data items in the output space to generate target outputs for the diffusion neural network in the latent space. FIG. 1 is a diagram of an example training system 100. The training system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented. The system 100 is a system that trains a diffusion neural network 110 that is used to generate an output (final) data item 112 given a conditioning input 102. In particular, the system 100 trains the diffusion neural network 110 so that the output (final) data item 112 that has the one or more desired properties characterized by the conditioning input 102. More specifically, the system 100 performs "fine-tuning," i.e., further training, of the diffusion neural network 110 using a reward function 120. In other words, prior being trained using the reward function 120, the system 100 or another training system has trained the diffusion neural network on a different objective, e.g., one that does not make use of the reward function 120. In general the diffusion neural network can have been trained conventionally, using any diffusion model objective. As one example, the diffusion neural network can have been trained on a set of training data items on a diffusion score matching objective or a variant thereof. The reward function 120 can be any appropriate differentiable reward function that maps an input that includes (i) a data item or (ii) a latent representation of a data item to a reward score 122. Optionally, the reward function input can also include the conditioning input 102 or a representation of the conditioning input 102.)
- 1. instantiating, by the computing system, a first instance of the pre-trained denoising diffusion model as a generator model having the set of pre-trained model parameters; / 8. performing an additional forward diffusion step on the partially noised training example to generate an additionally noised training example; / 17. instantiating, by the computing system, a first instance of the pre-trained denoising diffusion model as the generator model having the set of pre-trained model parameters; (Swersky: page 12 As another example, the diffusion neural network 110 can include one or more other types of neural network layers that are conditioned on the one or more embeddings. Examples of such layers include Feature-wise Linear Modulation (FiLM) layers, layers with conditional gated activation functions, and so on. The diffusion input at any given updating iteration can also include data defining a noise level for the iteration. Generally, each updating iteration has a corresponding time step t and the noise level for the iteration depends on the time step. For example, the noise level can be a decreasing function of the time step t. Examples of such functions include a linear function, a cosine function, and a sigmoid function. In these cases, data identifying the noise level, the time step, or both can be embedded using an appropriate neural network, e.g., a multi-layer perceptron (MLP) and used to condition the diffusion neural network 110 as described above for the conditioning input. FIG. 2 is a flow diagram of an example process 200 for training a diffusion neural network using reward scores. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 200. The system initializes a representation of a data item (step 202). For example, the system can initialize the representation by sampling the values in the representation from a distribution, e.g., a Gaussian distribution.)
- 1. instantiating, by the computing system, a second instance of the pre-trained denoising diffusion model as a discriminator model having the set of pre-trained model parameters; / 8. processing the additionally noised training example with a generator model to generate fully de-noised prediction, wherein the generator model comprises a denoising diffusion model; / 17. instantiating, by the computing system, a second instance of the pre-trained denoising diffusion model as a discriminator model having the set of pre-trained model parameters; (Swersky: page 17 By repeatedly performing this training for different sets of one or more conditioning inputs, the system effectively "fine-tunes" the diffusion neural network to generate outputs
that result in increased reward scores. FIG. 3 is a flow diagram of an example process 300 for determining a variance reduction term of the loss function. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300. The variance reduction term is also referred to as a "second" term in the loss function in this specification. The system can perform the process 300 at each of one or more noise iterations. The system samples noise for the noise iteration (step 302). For example, the system can sample the values in the noise from the same distribution used to initialize the representation of the data item, e.g., a Gaussian distribution. The system applies the noise to the final representation to generate a noisy representation (step 304). The system processes an input that includes the noisy representation using the diffusion neural network to generate a denoising output for the noise iteration (step 306). The system then updates the noisy representation using the denoising output for the noise iteration to generate an updated noisy representation that is an estimate of the final representation (step 308). The system generates a new reward input from the updated noisy representation (step 310). When the representation of the data item is a representation in the output space, the system can directly include the updated noisy representation in the new reward input. Optionally, the reward input can also include other information, e.g., the conditioning input. When the representation of the data item is a latent representation in a latent space, the system can process the updated noisy representation using a decoder neural network to generate a noisy data item and include the noisy data item in the new reward input. The system applies the reward function to the new reward input to generate a new reward score (step 312).)
- 1. and finetuning, by the computing system, at least the generator model on a finetuning dataset, / 8. performing one or more forward diffusion steps on the fully de-noised prediction to generate a partially re-noised prediction; processing the partially re-noised prediction with a discriminator model to generate a discriminator prediction; / 17. and finetuning, by the computing system, at least the generator model on a finetuning dataset, (Swersky: page 18 When the representation of the data item is a latent representation in a latent space, the system can process the updated noisy representation using a decoder neural network to generate a noisy data item and include the noisy data item in the new reward input. The system applies the reward function to the new reward input to generate a new reward score (step 312). Generally, the loss function described above includes a second term that measures the new reward scores for the one or more noise iterations. When there are a plurality of noise iterations, the second term can measure a combination of, e.g., an average of, the new reward scores. When the loss function includes the second term, training the neural network on the loss function can include backpropagating gradients of the second term through the reward function and into the noise iterations but not through any of the sampling iterations. That is, the system can insert a stop gradient to prevent from computing gradients ;of the second term with respect to the final representation or any representations generated at any preceding sampling iterations. An example technique for training the diffusion neural network (e having parameters 8 when the diffusion sampler is DDIM and there are Tis; Table 1.)
- 1. wherein finetuning, by the computing system, the generator model comprises modifying, by the computing system, / 8. and updating one or more parameter values of at least the generator model based on a loss function, wherein the loss function comprises: / 17. wherein finetuning, by the computing system, the generator model comprises modifying, by the computing system, (Swersky: page 21 As another example, the system can train a single instance of the diffusion neural network a different reward function using the LoRA training technique. In this example, the system can generate a new instance of the diffusion neural network by computing, for each network parameter in the first subset that is updated through the LoRA training, a weighted sum of the value of the network parameter after the fine-tuning and the pre-trained value of the network parameter, with the weight for each being determined by how strongly the property ( or properties) measured by the reward function should be reflected in new data items generated after training. FIG. 5 shows an example 500 of the performance of the described techniques when fine-tuned on a reward function that measures an aesthetic score. In particular, as can be seen from the example 500, variants of the described techniques outperform two existing techniques (ReFL and DDPO) as well as the pre-trained model (Stable Diffusion) and using prompt engineering across a range ofreward queries (training conditioning inputs).)
- 1. the set of pre-trained model parameters of the generator model based on a generative adversarial network loss term that provides a loss value based on an output of the discriminator model. / 8. a reconstruction loss term that generates a reconstruction loss value based on the training example and the fully de-noised prediction, and a GAN loss term that generates a GAN loss value based on the discriminator prediction. / 17. the set of pre-trained model parameters of the generator model based on a generative adversarial network loss term that provides a loss value based on an output of the discriminator model. (Swersky: page 8, As another example, the reward function 120 can include a reward that causes the diffusion model to generate adversarial examples. That is, the system can fine-tune the diffusion model such that output data items generated based on a prompt for a classy are classified as a different class y' by a pre-trained classifier for data items of the particular type. For example, as the reward, the system can use the negative cross-entropy to the target class by the pre-trained classifier. As another example, the reward function 120 can include one or more hard-coded differentiable reward functions.)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the machine learning algorithm of Zhao’s method and system for medical image synthesis to leverage the training diffusion neural network by backpropagating differential rewards using gradient descent disclosed by Swersky. The determination of obviousness is predicated upon the following findings: both references rely on machine learning for overall image processing and analysis and lend themselves for combination. One skilled in the art would have been motivated to modify Zhao in order to leverage a more accurate and computationally efficient gradient-based reward fine-tuning using a modified pre-training process as disclosed by Swersky. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Zhao, while the teaching of Swersky continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of a more accurate and computationally efficient machine learning process that relies on the gradient-descent algorithm that rewards a modified fine-tuning pre-training process as disclosed by Swersky. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claims 2 and 18.
The combination of Zhao and Swersky teaches:
2. The computer-implemented method of claim 1, wherein finetuning, by the computing system, the generator model further comprises modifying, by the computing system, the set of pre-trained model parameters of the generator model based on a reconstruction loss term that provides a loss value based on an output of the generator model./ 18. The one or more non-transitory computer-readable media of claim 17, wherein finetuning, by the computing system, the generator model further comprises modifying, by the computing system, the set of pre-trained model parameters of the generator model based on a reconstruction loss term that provides a loss value based on an output of the generator model. (Zhao: [0048]
In the second stage 450, the denoising model uses a similar diffusion model UNet-like deep convolutional neural network 451 to synthesize the synthetic medical image 470 with a synthetic tumor. The synthesis is conditioned on the tumor mask 440 and the original pre-abnormality image 460, which are concatenated to the Gaussian white noise 452 image along the channel dimension. The original input background medical image 460 guides the synthesis of surrounding tissues around the synthetic tumor and the background image. The guiding from the input background allows the model to generate mass effect around large synthetic tumors. The tumor mask guides the structure of the synthetic tumor and may be used as the ground truth segmentation of the tumor in downstream ML training for tumor segmentation or detection tasks. In some cases, this synthesis step may be conditioned on more than one pre-abnormality image along the channel dimension. They can be of the same subject using different contrasts, the same subject at different longitudinal time points, or other image information from the subject. In the example system, one of the medical images is selected to be the synthetic image's target background. The selected background image 460 may be used in the loss function of the diffusion model. Swersky: page 15 The diffusion network that are being updated during the training ( e.g. including the LoRA parameters). Later, in the Algorithm, this is given as r(x0, c), where x0 denotes the final data item. The system trains the diffusion neural network using a loss function that includes a first term that measures the reward score for the final data item (step 216). For example, the first term can be the reward score, e.g., r(sample(0, c, Xr ), c) or the negative of the reward score. As part of the training, the system can backpropagate gradients of the first term through the reward function and through a subset of the sampling iterations in order to determine a gradient of the first term with respect to the network parameters of the diffusion neural network. For example, the gradient of the first term with respect to the network parameters can be backpropagated through the subset of the sampling iterations in order to determine the gradient, g, of the first term with respect to the network parameters of the diffusion neural network, V 0r(sample(0, c, Xr ), c) (i.e. back through multiple calls of the diffusion model neural network in the sampling chain, analogously to backpropagation through time). For example, the system can backpropagate the gradient of the first term through the reward function to determine a gradient of the first term with respect to the reward input, and then backpropagate the gradient of the first term with respect to the reward input through the subset of the sampling iteration to determine the gradient with respect to the network parameters as described above. Page 17 FIG.3 is a flow diagram of an example process 300 for determining a variance reduction term of the loss function. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 100 depicted in FIG.1, appropriately programmed in accordance with this specification, can perform the process 300. The variance reduction term is also referred to as a “second” term in the loss function in this specification. The system can perform the process 300 at each of one or more noise iterations. The system samples noise for the noise iteration (step 302).
Consider Claims 3 and 19.
The combination of Zhao and Swersky teaches:
3. The computer-implemented method of claim 1, wherein finetuning, by the computing system, the generator model further comprises finetuning, by the computing system, the generator model on a text-to-image generation task. / 19. The one or more non-transitory computer-readable media of claim 17, wherein finetuning, by the computing system, the generator model further comprises finetuning, by the computing system, the generator model on a text-to-image generation task. (Zhao: [0050] Additionally or alternatively, denoising stage may use language-based descriptors as input, e.g. clinical reports, electronic health records (EHRs), Digital Imaging and Communication in Medic (DICOM) tags, sequence or protocol parameters or tumor description, and/or other language-based descriptors. The encoded text in the text information may be transformed into a text embedding sequence (e.g., a descriptor vector) using a pretrained language model. The text embedding sequence may applied to the UNet using a transformer (e.g., because the text embedding sequence does not contain image information that can be natively input to the UNet). In some implementations, a multi-head cross-attention application of the text embedding sequence may be implemented by the transformers to apply the text embedding sequence at lower resolution levels of the UNet. Accordingly, the description of the tumor within in the text information may be used to guide the synthetic tumor's appearance. Alternatively, a tumor characteristic vector can be encoded via vector embedding (e.g. rather than using transformation via a language model), and then used to guide the synthetic tumor's appearance. The vector may be applied to the UNet in the same way as the text embedding sequence. The entries of the tumor characteristic vector may represent a characteristic of the tumor, e.g. the level of heterogeneity of the tumor appearance, the contrast of the tumor, the type and subtype of the tumor, the TNM stage of the tumor, whether the tumor is post-operational, an International Classification of Diseases 10th Revision (ICD-10) code, a DICOM tag, sequence or protocol parameters, or other characteristics description. Furthermore, a vector may include a blend of language model extracted characteristics and direct vector entry input.)
Consider Claims 4 and 20.
The combination of Zhao and Swersky teaches:
4. The computer-implemented method of claim 1, wherein finetuning, by the computing system, the generator model further comprises finetuning, by the computing system, the generator model on a domain-specific downstream task. / 20. The one or more non-transitory computer-readable media of claim 17, wherein finetuning, by the computing system, the generator model further comprises finetuning, by the computing system, the generator model on a domain-specific downstream task. (Zhao: [0037] In some implementations used for generation of ML training data, the synthesized medical image 170 and abnormality spatial mask 140 may be provided to a machine learning training interface 190 (e.g., for training other machine learning systems) as a training tuple 180 (230). The training tuple 180 may define the synthesized medical image 170 as a training input and the abnormality spatial mask 140 as a ground truth output. Thus, the training tuple provides a medical image input paired with a segmented/identified abnormality output. [0038] FIG. 3 shows an example synthesis computation environment (SCE) 300, which, for example, may operate as synthesis circuitry. The SCE 300 may include system logic 314 to support implementation of the example ISL 200. The system logic 314 may include processors 316, memory 320, and/or other circuitry, which may be used to implement the multiple denoising stages, provide inputs to the stages, pass information between the stages, iterate denoising operations, execute machine-learned networks, and/or perform other image synthesis operations. [0039] In some implementations, the SCE 300 may receive pre-abnormality images obtained from an imaging device 301. In some cases, an anatomical mask generation system 302 may generate masks from the pre-abnormality images. Various systems for anatomical segmentation of non-abnormality medical images may be used to produce the anatomical mask based on the pre-abnormality images. In some implementations, processing to generate anatomical masks may occur on processing system logically or physically separate from the SCE 300. [0045] In an illustrative example image synthesis system 400 shown in FIG. 4 , brain-tumor-type abnormalities are synthesized within medical images of brains. In the illustrative example image synthesis system 400, the two pipeline stages 410, 450 of diffusion model based image synthesis are used. The pipeline uses a set of concentric spheres 420 as the starting point. The illustrative example image synthesis system 400 first synthesizes the multi-label tumor mask 440 from the concentric spheres 512 and then synthesizes the output medical image 470 from the multi-label tumor mask. This example illustrative design enables the model to control the synthetic tumor's location, size and volume ratios of different tumor parts with the configurable parameters of the concentric spheres' location, size and volume ratios, which can synthesize tumors with specified locations, sizes and structures by configuring these parameters. [0046] In the first stage 410, the tumor mask 440 is synthesized from the Gaussian white noise 412 image by iterative denoising operations of the diffusion model 411, where T is the number of iterations. The illustrative example denoising model uses a UNet-like deep convolutional neural network, and shallow N-block unmasked transformers with alternating layers of (i) self-attention, (ii) a position-wise MLP and (iii) a cross-attention layer were added to the lower resolution levels of the UNet. The tumor mask 440 (e.g., which corresponds to the abnormality spatial mask) contains segmentation masks of different parts of tumors with different labels, such as the necrotic and non-enhancing tumor core, the peritumoral edema, the GD-enhancing tumor. The synthesis is conditioned on the anatomical mask, e.g. the brain mask, and a set of concentric spheres 420 by concatenating them with the Gaussian white noise 412 image along the channel dimension. Swersky: page 21 As another example, the system can train a single instance of the diffusion neural network a different reward function using the LoRA training technique. In this example, the system can generate a new instance of the diffusion neural network by computing, for each network parameter in the first subset that is updated through the LoRA training, a weighted sum of the value of the network parameter after the fine-tuning and the pre-trained value of the network parameter, with the weight for each being determined by how strongly the property ( or properties) measured by the reward function should be reflected in new data items generated after training. FIG. 5 shows an example 500 of the performance of the described techniques when fine-tuned on a reward function that measures an aesthetic score. In particular, as can be seen from the example 500, variants of the described techniques outperform two existing techniques (ReFL and DDPO) as well as the pre-trained model (Stable Diffusion) and using prompt engineering across a range ofreward queries (training conditioning inputs).)
Consider Claim 5.
The combination of Zhao and Swersky teaches:
5. The computer-implemented method of claim 1, wherein the generator model is configured to receive and process a noise sample to generate a denoised synthetic image in a single denoising step. (Zhao: [0018]-[0019] [0028] In some cases, to train the machine-learned networks to perform the denoising, a ground truth image may have set step levels of noise added. Each of the step levels may be used as a training tuple, with the images with one step less noise added being the ground truth for the image with one step more noise added. [0029] In various implementations, the first denoising stage 110 may obtain an anatomical mask 130 as an input (214). The anatomical mask 130 may detail the spatial layout of the anatomical background in the multidimensional space defined in the first noise sample. The anatomy may be used by the first denoising stage 110 to determine positioning and shape of the abnormality. For example, the anatomical mask 130 may include anatomical boundaries. The first denoising stage may shape and position the abnormality to handle the anatomical boundaries. For example, some rigid boundaries, such as a skull in the case of the brain tumor type abnormality: may disallow straddling (e.g., the tumor may not be on be on both sides), may disallow crossing (e.g., the tumor may not be outside the skull), and/or may disallow deformation (e.g., the tumor may not deform the skull). For flexible boundaries, straddling and/or crossing may be disallowed, and deformation of the boundary may be permitted. Other boundaries may allow some selection of crossing, straddling, and/or deformation while disallowing the complementary selection. [0030] In various implementations, the particular way the anatomical mask 130 is used by the first denoising stage may be controlled via the training of the one or more neural networks on which the first denoising stage is implemented. [0031] The second denoising stage 150 may denoise a second noise 152 input using a pre-abnormality image 160 and the abnormality spatial mask 140 as inputs to produce a synthesized medical image 170 (220). Thus for the second denoising stage, the ISL 200 may obtain the pre-abnormality image 160 (222) and provide the abnormality spatial mask to the second denoising stage (224). [0045]-[0046] Swersky: page 13 At each of the plurality of sampling iterations, the system processes a diffusion input for the sampling iteration that includes the representation of the data item and a representation of the conditioning input using the diffusion neural network to generate a denoising output for the sampling iteration (step 208). Generally, the denoising output defines an estimate of the final representation given the current representation. In some implementations, the denoising output is an estimate of the noise component of the current representation, i.e., the noise that needs to be combined with, e.g., added to or subtracted to, the final representation to generate the current representation. In some other implementations, the denoising output is an estimate of the final representation given the current representation, i.e., an estimate of the data item that would result from removing the noise component of the current representation. In yet other implementations, the system parametrizes the denoising output differently, e.g., using a v-parameterization (Salimans and Ho arXiv: 2202.00512, 2022, section 4; Appendix D) or another appropriate parameterization. The system then updates the representation of the data item using the denoising output (step 210). In some implementations, the system uses the denoising output as the final denoising output for the updating iteration. In some other implementations, the system also generates one or more additional denoising outputs for the sampling iteration. For example, the system can make use of classifier-free guidance. In this example, the system processes a second diffusion input for the sampling iteration that includes the representation of the data item but not the representation of the conditioning input using the diffusion neural network to generate an unconditional denoising output for the updating iteration. For example, the second diffusion input can include the representation of the data item and a predetermined representation that indicates unconditional sampling. For example, the second diffusion input can include the representation of the data item and a predetermined representation that indicates unconditional sampling.)
Consider Claims 6 and 13.
The combination of Zhao and Swersky teaches:
6. The computer-implemented method of claim 1, wherein the pre-trained denoising diffusion model comprises a pre-trained latent diffusion model./ 13. The computer system of claim 8, wherein the generator model and the discriminator model have both been initialized from a pre-trained latent diffusion model. (Zhao: [0018]-[0019] [0028] In some cases, to train the machine-learned networks to perform the denoising, a ground truth image may have set step levels of noise added. Each of the step levels may be used as a training tuple, with the images with one step less noise added being the ground truth for the image with one step more noise added. [0029] In various implementations, the first denoising stage 110 may obtain an anatomical mask 130 as an input (214). The anatomical mask 130 may detail the spatial layout of the anatomical background in the multidimensional space defined in the first noise sample. The anatomy may be used by the first denoising stage 110 to determine positioning and shape of the abnormality. For example, the anatomical mask 130 may include anatomical boundaries. The first denoising stage may shape and position the abnormality to handle the anatomical boundaries. For example, some rigid boundaries, such as a skull in the case of the brain tumor type abnormality: may disallow straddling (e.g., the tumor may not be on be on both sides), may disallow crossing (e.g., the tumor may not be outside the skull), and/or may disallow deformation (e.g., the tumor may not deform the skull). For flexible boundaries, straddling and/or crossing may be disallowed, and deformation of the boundary may be permitted. Other boundaries may allow some selection of crossing, straddling, and/or deformation while disallowing the complementary selection. [0030] In various implementations, the particular way the anatomical mask 130 is used by the first denoising stage may be controlled via the training of the one or more neural networks on which the first denoising stage is implemented. [0031] The second denoising stage 150 may denoise a second noise 152 input using a pre-abnormality image 160 and the abnormality spatial mask 140 as inputs to produce a synthesized medical image 170 (220). Thus for the second denoising stage, the ISL 200 may obtain the pre-abnormality image 160 (222) and provide the abnormality spatial mask to the second denoising stage (224). [0045]-[0046] Swersky: page 13 At each of the plurality of sampling iterations, the system processes a diffusion input for the sampling iteration that includes the representation of the data item and a representation of the conditioning input using the diffusion neural network to generate a denoising output for the sampling iteration (step 208). Generally, the denoising output defines an estimate of the final representation given the current representation. In some implementations, the denoising output is an estimate of the noise component of the current representation, i.e., the noise that needs to be combined with, e.g., added to or subtracted to, the final representation to generate the current representation. In some other implementations, the denoising output is an estimate of the final representation given the current representation, i.e., an estimate of the data item that would result from removing the noise component of the current representation. In yet other implementations, the system parametrizes the denoising output differently, e.g., using a v-parameterization (Salimans and Ho arXiv: 2202.00512, 2022, section 4; Appendix D) or another appropriate parameterization. The system then updates the representation of the data item using the denoising output (step 210). In some implementations, the system uses the denoising output as the final denoising output for the updating iteration. In some other implementations, the system also generates one or more additional denoising outputs for the sampling iteration. For example, the system can make use of classifier-free guidance. In this example, the system processes a second diffusion input for the sampling iteration that includes the representation of the data item but not the representation of the conditioning input using the diffusion neural network to generate an unconditional denoising output for the updating iteration. For example, the second diffusion input can include the representation of the data item and a predetermined representation that indicates unconditional sampling. For example, the second diffusion input can include the representation of the data item and a predetermined representation that indicates unconditional sampling.)
Consider Claims 7 and 14.
The combination of Zhao and Swersky teaches:
7. The computer-implemented method of claim 1, wherein the pre-trained denoising diffusion model comprises a U-Net. / 14. The computer system of claim 8, wherein the generator model and the discriminator model both comprise a U-Net architecture. (Zhao: [0050] Additionally or alternatively, denoising stage may use language-based descriptors as input, e.g. clinical reports, electronic health records (EHRs), Digital Imaging and Communication in Medic (DICOM) tags, sequence or protocol parameters or tumor description, and/or other language-based descriptors. The encoded text in the text information may be transformed into a text embedding sequence (e.g., a descriptor vector) using a pretrained language model. The text embedding sequence may applied to the UNet using a transformer (e.g., because the text embedding sequence does not contain image information that can be natively input to the UNet). In some implementations, a multi-head cross-attention application of the text embedding sequence may be implemented by the transformers to apply the text embedding sequence at lower resolution levels of the UNet. Accordingly, the description of the tumor within in the text information may be used to guide the synthetic tumor's appearance. Alternatively, a tumor characteristic vector can be encoded via vector embedding (e.g. rather than using transformation via a language model), and then used to guide the synthetic tumor's appearance. The vector may be applied to the UNet in the same way as the text embedding sequence. The entries of the tumor characteristic vector may represent a characteristic of the tumor, e.g. the level of heterogeneity of the tumor appearance, the contrast of the tumor, the type and subtype of the tumor, the TNM stage of the tumor, whether the tumor is post-operational, an International Classification of Diseases 10th Revision (ICD-10) code, a DICOM tag, sequence or protocol parameters, or other characteristics description. Furthermore, a vector may include a blend of language model extracted characteristics and direct vector entry input.)
Consider Claim 9.
The combination of Zhao and Swersky teaches:
9. The computer system of claim 8, wherein processing the additionally noised training example with the generator model to generate fully de-noised prediction comprises performing only a single denoising step with the generator model to generate fully de-noised prediction from the additionally noised training example. (Zhao: [0050] Additionally or alternatively, denoising stage may use language-based descriptors as input, e.g. clinical reports, electronic health records (EHRs), Digital Imaging and Communication in Medic (DICOM) tags, sequence or protocol parameters or tumor description, and/or other language-based descriptors. The encoded text in the text information may be transformed into a text embedding sequence (e.g., a descriptor vector) using a pretrained language model. The text embedding sequence may applied to the UNet using a transformer (e.g., because the text embedding sequence does not contain image information that can be natively input to the UNet). In some implementations, a multi-head cross-attention application of the text embedding sequence may be implemented by the transformers to apply the text embedding sequence at lower resolution levels of the UNet. Accordingly, the description of the tumor within in the text information may be used to guide the synthetic tumor's appearance. Alternatively, a tumor characteristic vector can be encoded via vector embedding (e.g. rather than using transformation via a language model), and then used to guide the synthetic tumor's appearance. The vector may be applied to the UNet in the same way as the text embedding sequence. The entries of the tumor characteristic vector may represent a characteristic of the tumor, e.g. the level of heterogeneity of the tumor appearance, the contrast of the tumor, the type and subtype of the tumor, the TNM stage of the tumor, whether the tumor is post-operational, an International Classification of Diseases 10th Revision (ICD-10) code, a DICOM tag, sequence or protocol parameters, or other characteristics description. Furthermore, a vector may include a blend of language model extracted characteristics and direct vector entry input.)
Consider Claims 10.
The combination of Zhao and Swersky teaches:
10. The computer system of claim 8, wherein processing the additionally noised training example with the generator model to generate fully de-noised prediction comprises performing a text-to-image generation task on the additionally noised training example and a text prompt. (Zhao: [0050] Additionally or alternatively, denoising stage may use language-based descriptors as input, e.g. clinical reports, electronic health records (EHRs), Digital Imaging and Communication in Medic (DICOM) tags, sequence or protocol parameters or tumor description, and/or other language-based descriptors. The encoded text in the text information may be transformed into a text embedding sequence (e.g., a descriptor vector) using a pretrained language model. The text embedding sequence may applied to the UNet using a transformer (e.g., because the text embedding sequence does not contain image information that can be natively input to the UNet). In some implementations, a multi-head cross-attention application of the text embedding sequence may be implemented by the transformers to apply the text embedding sequence at lower resolution levels of the UNet. Accordingly, the description of the tumor within in the text information may be used to guide the synthetic tumor's appearance. Alternatively, a tumor characteristic vector can be encoded via vector embedding (e.g. rather than using transformation via a language model), and then used to guide the synthetic tumor's appearance. The vector may be applied to the UNet in the same way as the text embedding sequence. The entries of the tumor characteristic vector may represent a characteristic of the tumor, e.g. the level of heterogeneity of the tumor appearance, the contrast of the tumor, the type and subtype of the tumor, the TNM stage of the tumor, whether the tumor is post-operational, an International Classification of Diseases 10th Revision (ICD-10) code, a DICOM tag, sequence or protocol parameters, or other characteristics description. Furthermore, a vector may include a blend of language model extracted characteristics and direct vector entry input.)
Consider Claims 11 and 12.
The combination of Zhao and Swersky teaches:
11. The computer system of claim 8, wherein the generator model and the discriminator model have both been initialized from a pre-trained diffusion model. / 12. The computer system of claim 8, wherein the generator model and the discriminator model have both been initialized from a pre-trained diffusion model. (Zhao: [0050] Additionally or alternatively, denoising stage may use language-based descriptors as input, e.g. clinical reports, electronic health records (EHRs), Digital Imaging and Communication in Medic (DICOM) tags, sequence or protocol parameters or tumor description, and/or other language-based descriptors. The encoded text in the text information may be transformed into a text embedding sequence (e.g., a descriptor vector) using a pretrained language model. The text embedding sequence may applied to the UNet using a transformer (e.g., because the text embedding sequence does not contain image information that can be natively input to the UNet). In some implementations, a multi-head cross-attention application of the text embedding sequence may be implemented by the transformers to apply the text embedding sequence at lower resolution levels of the UNet. Accordingly, the description of the tumor within in the text information may be used to guide the synthetic tumor's appearance. Alternatively, a tumor characteristic vector can be encoded via vector embedding (e.g. rather than using transformation via a language model), and then used to guide the synthetic tumor's appearance. The vector may be applied to the UNet in the same way as the text embedding sequence. The entries of the tumor characteristic vector may represent a characteristic of the tumor, e.g. the level of heterogeneity of the tumor appearance, the contrast of the tumor, the type and subtype of the tumor, the TNM stage of the tumor, whether the tumor is post-operational, an International Classification of Diseases 10th Revision (ICD-10) code, a DICOM tag, sequence or protocol parameters, or other characteristics description. Furthermore, a vector may include a blend of language model extracted characteristics and direct vector entry input.)
Consider Claims 15.
The combination of Zhao and Swersky teaches:
15. The computer system of claim 8, wherein the reconstruction loss term evaluates a KL divergence between the training example and the fully de-noised prediction. (Zhao: [0050] Additionally or alternatively, denoising stage may use language-based descriptors as input, e.g. clinical reports, electronic health records (EHRs), Digital Imaging and Communication in Medic (DICOM) tags, sequence or protocol parameters or tumor description, and/or other language-based descriptors. The encoded text in the text information may be transformed into a text embedding sequence (e.g., a descriptor vector) using a pretrained language model. The text embedding sequence may applied to the UNet using a transformer (e.g., because the text embedding sequence does not contain image information that can be natively input to the UNet). In some implementations, a multi-head cross-attention application of the text embedding sequence may be implemented by the transformers to apply the text embedding sequence at lower resolution levels of the UNet. Accordingly, the description of the tumor within in the text information may be used to guide the synthetic tumor's appearance. Alternatively, a tumor characteristic vector can be encoded via vector embedding (e.g. rather than using transformation via a language model), and then used to guide the synthetic tumor's appearance. The vector may be applied to the UNet in the same way as the text embedding sequence. The entries of the tumor characteristic vector may represent a characteristic of the tumor, e.g. the level of heterogeneity of the tumor appearance, the contrast of the tumor, the type and subtype of the tumor, the TNM stage of the tumor, whether the tumor is post-operational, an International Classification of Diseases 10th Revision (ICD-10) code, a DICOM tag, sequence or protocol parameters, or other characteristics description. Furthermore, a vector may include a blend of language model extracted characteristics and direct vector entry input.)
Consider Claims 16.
The combination of Zhao and Swersky teaches:
16. The computer system of claim 8, wherein the operations further comprise updating one or more parameter values of the discriminator model based on a second GAN loss term that generates a second GAN loss value based on the discriminator prediction. (Zhao: [0048] In the second stage 450, the denoising model uses a similar diffusion model UNet-like deep convolutional neural network 451 to synthesize the synthetic medical image 470 with a synthetic tumor. The synthesis is conditioned on the tumor mask 440 and the original pre-abnormality image 460, which are concatenated to the Gaussian white noise 452 image along the channel dimension. The original input background medical image 460 guides the synthesis of surrounding tissues around the synthetic tumor and the background image. The guiding from the input background allows the model to generate mass effect around large synthetic tumors. The tumor mask guides the structure of the synthetic tumor and may be used as the ground truth segmentation of the tumor in downstream ML training for tumor segmentation or detection tasks. In some cases, this synthesis step may be conditioned on more than one pre-abnormality image along the channel dimension. They can be of the same subject using different contrasts, the same subject at different longitudinal time points, or other image information from the subject. In the example system, one of the medical images is selected to be the synthetic image's target background. The selected background image 460 may be used in the loss function of the diffusion model. Swersky: page 17 By repeatedly performing this training for different sets of one or more conditioning inputs, the system effectively "fine-tunes" the diffusion neural network to generate outputs that result in increased reward scores. FIG. 3 is a flow diagram of an example process 300 for determining a variance reduction term of the loss function. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300. The variance reduction term is also referred to as a "second" term in the loss function in this specification. The system can perform the process 300 at each of one or more noise iterations. The system samples noise for the noise iteration (step 302). For example, the system can sample the values in the noise from the same distribution used to initialize the representation of the data item, e.g., a Gaussian distribution.)
Conclusion
The prior art made of record in form PTO-892 and not relied upon is considered pertinent to applicant's disclosure.
PNG
media_image1.png
173
887
media_image1.png
Greyscale
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAHMINA ANSARI whose telephone number is 571-270-3379. The examiner can normally be reached on IFP Flex - Monday through Friday 9 to 5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’NEAL MISTRY can be reached on 313-446-4912. The fax phone numbers for the organization where this application or proceeding is assigned are 571-273-8300 for regular communications and 571-273-8300 for After Final communications. TC 2600’s customer service number is 571-272-2600.
Any inquiry of a general nature or relating to the status of this application or proceeding should be directed to the receptionist whose telephone number is 571-272-2600.
2674
/Tahmina Ansari/
July 3, 2026
/TAHMINA N ANSARI/Primary Examiner, Art Unit 2674
Prosecution Insights