DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on February 16th 2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Arguments
This is in response to applicant’s amendment/response filed on 02/03/2026 which have been entered and made of record.
Applicant’s arguments regarding claim rejections under 35 U.S.C. 103 have been fully considered but they are not persuasive.
Applicant argues that Jefferson, either alone or in combination with the other cited references, does not disclose or suggest all features of claim 2. Jefferson (e.g., in Para. [0031]) discloses a guidance strength input element configured to receive a guidance strength input. Jefferson, however, does not disclose or suggest that the guidance strength input is multiplied by a gradient of a guidance loss. Paras. [0094] and [0099] of Jefferson, which are cited as disclosing the guidance loss, instead refer to a training loss instead of a guidance loss computed during image generation at inferencing time. The computation of the guidance updates recited in claim 1 as currently amended differs from the computation of parameter updates during neural network training, since the guidance updates are applied to the generated image instead of to the parameters of a neural network. Jefferson only mentions loss as a quantity computed during training and does not disclose or suggest that the loss is multiplied by a guidance strength hyperparameter.
The Office action cites Paras. [0094] and [0099] of Jefferson as disclosing the estimated clean image in the rejection of claim 3. As discussed above, these paragraphs of Jefferson refer to a neural network training process rather than to inference-time guidance of image generation. Thus, the Office action incorrectly interprets the term "estimated clean image" of claim 3 as referring to a training image.
The Office action also cites Kreis in the rejection of claim 3. However, Kreis also does not disclose or suggest the steps of computing the guidance update that are recited in claim 1 as currently amended. In addition, the other references cited in the Office action do not appear to disclose or suggest the features of claims 2 and 3 that have been incorporated into claim 1, together with the other recited features. Applicant therefore
Examiner respectable disagrees. According to BRI (Broadest Reasonable Interpretation) Guidance Loss can refer to several different loss functions in machine learning. Such as the generic loss function used to guide the optimization process of training a model or a specific loss function used to guide the optimization process of modifying/processing the results of a model. Examiner recognizes from the applicants’ arguments/remarks and specification the “guidance loss” is a specific loss function used to guide the optimization process of a “guidance strength” parameter that modifies the results of the model and does not involve training the model. However, this relationship is not present in the independent claims as there is no mention of the “guidance loss” and “guidance strength” relationship or the “guidance updates, guidance loss, or guidance strength” does not involve training the model. Thus, it would be improper to import claim limitations from the specification (See MPEP 2111.01(II)).
Further Applicant recites “Jefferson does not disclose or suggest that the loss is multiplied by a guidance strength hyperparameter”. While Jefferson does not explicitly state the loss is multiplied by the guidance strength. Jefferson teaches a loss function (Para. 0093-0094 and 0099) that minimize/maximize the weights/parameters to reduce error or increase accuracy between the predicted output and the actual output. Jefferson teaches a guidance strength (Para. 0031, 0077-0078, and 0117) that directly controls the actual output. Changing the guidance strength would impact the loss as the guidance strength would modify the weights/parameters. One way the guidance strength could modify the weights/parameters is through multiplying the guidance strength to the changing weights/parameters. Thus, Jefferson teaches the loss is multiplied by a guidance strength hyperparameter.
Regarding claim 3, applicant recites “the office action incorrectly interprets the term "estimated clean image". The terminology “ estimated clean image” is broad and according to BRI can be any image that has been processed, corrected, regenerated, free of noise/artifacts/distortions, etc. and can be used in training a model. If applicant seeks a specific interpretation of “estimated clean image” examiner suggests amending claims to describe the specific “estimated clean image” sought.
Regarding the remaining arguments applicant argues with respect to the amended claim language, which is fully addressed in the prior art rejections set forth below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 4, 6, 10, 12, 15-16, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Jampani et al. U.S. Patent Application US 20240412458 A1 (hereinafter Jampani) in view of Marri et al. U.S. Patent Application US 20240338859 A1 (hereinafter Marri) in further view of Jefferson et al. U.S. Patent Application US 20250022192 A1 (hereinafter Jefferson) in further view of Kreis et al. U.S. Patent Application US 20230377099 A1(hereinafter Kreis).
Regarding claim 1, Jampani teaches a computing system (Para. 0105) comprising:
one or more processing (Processors or Computers, Para. 0105) devices configured to:
receive an image generation prompt (Text Prompt, Para. 0026, Fig. 2);
receive a reference image (Input Image, Para. 0019, 0026, and 0037, Fig. 2);
over a plurality of denoising timesteps, compute a guided image at least in part by: (Para.0008 and 0033)
iteratively applying a plurality of denoising updates to a generated image at a denoising diffusion model; (Para. 0033-0037)
and at a subset of the plurality of denoising timesteps (n Update Iterations, Para. 0043), applying respective guidance updates to the generated image based at least in part on the image generation prompt (Text Prompt, Para. 0026, Fig .2), the reference image (Input Image, Para. 0019, 0026, and 0037, Fig.2), and a generated image set that includes a current-timestep generated image and a plurality of prior-timestep generated images, wherein the one or more processing devices are configured to compute each guidance update at least in part by: (Para. 0037 and 0043)
performing a forward pass over the generated image set in a plurality of first integration timesteps, wherein performing the forward pass includes computing an estimated clean image based at least in part on the generated image set;
performing a backward pass over the generated image set in a plurality of second integration timesteps, wherein a size of the generated image set, a number of the first integration timesteps, and a number of the second integration timesteps are each equal to a predefined integration timestep count;
computing the guidance update as a product of a guidance strength hyperparameter and a gradient of a guidance loss, wherein the guidance loss is computed based at least in part on the estimated clean image and the reference image;
and output, as the guided image, a final generated image (Output Image) computed in a final denoising timestep of the plurality of denoising timesteps. (Fig.2 and Para. 0048-0050)
However, Jampani fails to explicitly teach:
and at a subset of the plurality of denoising timesteps, applying respective guidance updates to the generated image based at least in part on the image generation prompt, the reference image, and a generated image set that includes a current-timestep generated image and a plurality of prior-timestep generated images, wherein the one or more processing devices are configured to compute each guidance update at least in part by:
performing a forward pass over the generated image set in a plurality of first integration timesteps, wherein performing the forward pass includes computing an estimated clean image based at least in part on the generated image set;
and performing a backward pass over the generated image set in a plurality of second integration timesteps, wherein a size of the generated image set, a number of the first integration timesteps, and a number of the second integration timesteps are each equal to a predefined integration timestep count;
computing the guidance update as a product of a guidance strength hyperparameter and a gradient of a guidance loss, wherein the guidance loss is computed based at least in part on the estimated clean image and the reference image;
Jampani and Marri are analogous to the claimed invention because both of them are in the same field of utilizing diffusion models for text-to-image generation.
Marri teaches:
and at a subset of the plurality of denoising timesteps (t steps), applying respective guidance updates to the generated image based at least in part on the image generation prompt (Para. 0020 and 0088), the reference image (Original Image or Additional Image, Para. 0020, 0065, and 0070), and a generated image set (Para. 0089-0091) that includes a current-timestep generated image (X.Sub.1, …, X.sub.T, Para. 0089-0091) and a plurality of prior-timestep generated images (X.Sub.1, …, X.sub.T, Para. 0089-0091), wherein the one or more processing devices (Processor Unit, Para. 0040) are configured to compute each guidance update at least in part by: (Fig. 4) Diffusion models can map the current timestep image to intermediate images. X.sub.0 is the original image and X.Sub.1, …., X.sub.T are intermediate images.
performing a forward pass over the generated image set in a plurality of first integration timesteps (Para. 0090, Fig. 4), wherein performing the forward pass includes computing an estimated clean image based at least in part on the generated image set;
and performing a backward pass (Reverse Process) over the generated image set (Para. 0089-0091) in a plurality of second integration timesteps (Para. 0091, Fig. 4), wherein a size of the generated image set, a number of the first integration timesteps (Forward Process Timesteps, Para. 0089-0090), and a number of the second integration timesteps (Reverse Process, Para. 0089 and 0091) are each equal to a predefined integration timestep (Set of Diffusion Time Steps) count; (Para. 0050) The number of timesteps the diffusion model performs is based on the number of intermediate image embeddings. Both the forward and reverse processes utilize X.Sub.1, …., X.sub.T to create intermediate image embeddings, thus the size would be equal.
computing the guidance update as a product of a guidance strength hyperparameter and a gradient of a guidance loss, wherein the guidance loss is computed based at least in part on the estimated clean image and the reference image;
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jampani Iterative Diffusion Process to incorporate Marri’s Forward and Reverse Diffusion Processes. Since doing so would provide the benefit of utilizing a standard diffusion model that includes a forward and reverse process. (Marri, Para. 0065-66)
However, Jampani and Marri fail to explicitly teach:
wherein performing the forward pass includes computing an estimated clean image based at least in part on the generated image set;
computing the guidance update as a product of a guidance strength hyperparameter and a gradient of a guidance loss, wherein the guidance loss is computed based at least in part on the estimated clean image and the reference image;
Jampani, Marri, and Jefferson are analogous to the claimed invention because all of them are in the same field of utilizing diffusion models for text-to-image generation.
Jefferson teaches:
wherein performing the forward pass includes computing an estimated clean image based at least in part on the generated image set;
computing the guidance update as a product of a guidance strength (Guidance Strength Input, Para. 0031,0077-0078, and 0099) hyperparameter and a gradient of a guidance loss (Para. 0031, 0094, and 0099), wherein the guidance loss is computed based at least in part on the estimated clean image (Predicted Outputs or Current Results, Para. 0094 and 0099) and the reference image (Actual Targets or Target Results, Para. 0094 and 0099);
According to BRI (Broadest Reasonable Interpretation) Guidance Loss can refer to several different loss functions in machine learning. Such as the generic loss function used to guide the optimization process of training a model or a specific loss function used to guide the optimization process of modifying/processing the results of a model. Examiner recognizes from the applicants’ arguments/remarks and specification the “guidance loss” is a specific loss function used to guide the optimization process of a “guidance strength” parameter that modifies the results of the model and does not involve training the model. However, this relationship is not present in the independent claims as there is no mention of the “guidance loss” and “guidance strength” relationship or the “guidance updates, guidance loss, or guidance strength” does not involve training the model. Thus, it would be improper to import claim limitations from the specification (See MPEP 2111.01(II)).
Further Applicant recites “Jefferson does not disclose or suggest that the loss is multiplied by a guidance strength hyperparameter”. While Jefferson does not explicitly state the loss is multiplied by the guidance strength. Jefferson teaches a loss function (Para. 0093-0094 and 0099) that minimize/maximize the weights/parameters to reduce error or increase accuracy between the predicted output and the actual output. Jefferson teaches a guidance strength (Para. 0031, 0077-0078, and 0117) that directly controls the actual output. Changing the guidance strength would impact the loss as the guidance strength would modify the weights/parameters. One way the guidance strength could modify the weights/parameters is through multiplying the guidance strength to the changing weights/parameters. Thus, Jefferson teaches the loss is multiplied by a guidance strength hyperparameter.
Regarding claim 3, applicant recites “the office action incorrectly interprets the term "estimated clean image". The terminology “ estimated clean image” is broad and according to BRI can be any image that has been processed, corrected, regenerated, free of noise/artifacts/distortions, etc. and can be used in training a model. If applicant seeks a specific interpretation of “estimated clean image” examiner suggests amending claims to describe the specific “estimated clean image” sought.
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jampani’s Diffusion model altered by Marri to incorporate Jefferson’s guidance strength and guidance loss. Since doing so would provide the benefit of enhancing the diffusion model to allow the user to control the guidance strength for each timestep.
However Jefferson fails to teach:
wherein performing the forward pass includes computing an estimated clean image based at least in part on the generated image set;
Jampani, Marri, Jefferson, and Kreis are analogous to the claimed invention because all of them are in the same field of utilizing diffusion models for text-to-image generation.
Kreis teaches:
wherein performing the forward pass (Para. 0027) includes computing an estimated clean image (Para. 0030) based at least in part on the generated image set (Para. 0035-0037);
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jampani’s Diffusion model altered by Marri and Jefferson to incorporate Kreis’s Clean Image. Since doing so would provide the benefit of better estimating the guidance loss by utilizing a clean image.
Regarding claim 4, Jampani, Marri, and Jefferson fail to teach the computing system of claim 1, wherein:
performing the forward pass includes numerically solving a first ordinary differential equation (ODE) over the plurality of first integration timesteps;
and performing the backward pass includes numerically solving a second ODE over the plurality of second integration timesteps.
Jampani, Marri, Jefferson, and Kreis are analogous to the claimed invention because all of them are in the same field of utilizing diffusion models for text-to-image generation.
However, Kreis teaches teach the computing system of claim 1, wherein:
performing the forward pass (Para. 0035-0037) includes numerically solving a first ordinary differential equation (ODE) (Para. 0034 and 0035-0037) over the plurality of first integration timesteps;
and performing the backward pass (Reverse Diffusion, Para. 0035-0037) includes numerically solving a second ODE (Para. 0034 and 0035--0037) over the plurality of second integration timesteps.
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jampani’s Diffusion model altered by Marri to incorporate Ordinary Differential Equations. Since doing so would provide the benefit of increasing the speed of the synthesis. (Kreis, Para. 0023)
Regarding claim 6, Jampani teaches the computing system of claim 1, wherein the one or more processing devices are configured to compute the guidance update based at least in part on a noise scheduling hyperparameter (Noise Schedule Function) that varies over the plurality of denoising timesteps. (Para. 0034)
Regarding claim 10, Jampani fails to explicitly teach the computing system of claim 1, wherein the reference image is a style transfer reference image or a subject personalization reference image.
However, Marri teaches the computing system of claim 1, wherein the reference image (Original Image or Additional Image, Para. 0020, 0065, and 0070) is a style transfer reference image (Style Attributes from Conditional Image Generation) or a subject personalization reference image. (Para. 0003, 0020, and 0063) Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jampani’s Reference Image to incorporate Marri’s Reference Image. Since doing so would provide the benefit of utilizing various types of references images in the diffusion model. Diffusion models are used to generate images with specific styles, features, colors, content, and object locations. (Marri, Para. 0003)
Regarding claim 12, has similar limitations as of claim 1, therefore it is rejected under the same rationale as claim 1.
Regarding claim 15, has similar limitations as of claim 4, therefore it is rejected under the same rationale as claim 4.
Regarding claim 16, has similar limitations as of claim 6, therefore it is rejected under the same rationale as claim 6.
Regarding claim 18, has similar limitations as of claim 10, therefore it is rejected under the same rationale as claim 10.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Jampani et al. U.S. Patent Application US 20240412458 A1 (hereinafter Jampani) in view of Marri et al. U.S. Patent Application US 20240338859 A1 (hereinafter Marri), Jefferson et al. U.S. Patent Application US 20250022192 A1 (hereinafter Jefferson), and Kreis et al. U.S. Patent Application US 20230377099 A1(hereinafter Kreis) in further view of Pandey et al. U.S. Patent Application US 20250103930 A1 (hereinafter Pandey).
Regarding claim 5, Jampani, Marri, and Jefferson, Kreis fail to teach the computing system of claim 4, wherein the one or more processing devices are configured to solve the first ODE and the second ODE using a symplectic Euler method or a symplectic Runge-Kutta method.
Jampani, Marri, Jefferson, Kreis and Pandey are analogous to the claimed invention because all of them are in the same field of utilizing diffusion models to generate images.
Pandey teaches he computing system of claim 4, wherein the one or more processing devices are configured to solve the first ODE (Para. 0021-0028) and the second ODE (Para. 0021-0028) using a symplectic Euler method (Para. 0046-0047) or a symplectic Runge-Kutta method. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jampani’s Diffusion Model altered by Marri, Jefferson, and Kreis to incorporate Pandey’s Symplectic Euler Method. Since, doing so would provide the benefit of yielding better results than the standard Euler method that Kreis utilizes.
Claim(s) 7-9 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Jampani et al. U.S. Patent Application US 20240412458 A1 (hereinafter Jampani) in view of Marri et al. U.S. Patent Application US 20240338859 A1 (hereinafter Marri) in further view of Jefferson et al. U.S. Patent Application US 20250022192 A1 (hereinafter Jefferson) in further view of Kreis et al. U.S. Patent Application US 20230377099 A1(hereinafter Kreis) in further view of NPL IDS Reference “FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model” by Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang (hereinafter Yu).
Regarding claim 7, Jampani, Marri, Jefferson, and Kreis fail to teach the computing system of claim 1, wherein the one or more processing devices are configured to apply the guidance updates at a plurality of intermediate denoising timesteps that are preceded by a plurality of initial denoising timesteps and followed by a plurality of subsequent denoising timesteps.
Jampani, Marri, Jefferson, Kreis, and Yu are analogous to the claimed invention because all of them are in the same field of utilizing diffusion models to generate images from text.
Yu teaches the computing system of claim 1, wherein the one or more processing devices are configured to apply the guidance updates at a plurality of intermediate denoising timesteps (Semantic Stage 800-500, Fig.3) that are preceded by a plurality of initial denoising timesteps (Chaotic Stage 1000-800, Fig.3) and followed by a plurality of subsequent denoising timesteps (Refinement Stage 500-1, Fig.3).Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jampani’s Diffusion model altered by Marri, Jefferson, and Kreis to incorporate Yu’s Initial and Subsequent denoising timesteps. Since doing so would provide the benefit of configuring different sets of timesteps differently increasing the flexibility of the diffusion model.
Regarding claim 8, Jampani, Marri, Jefferson and Kreis fail to teach the computing system of claim 1, wherein, at one or more of the denoising timesteps, the one or more processing devices are configured to repeat denoising and noise addition for the generated image at each of a plurality of self-recurrence timesteps.
However, Yu teaches the computing system of claim 1, wherein, at one or more of the denoising timesteps, the one or more processing devices are configured to repeat denoising (Resampling t-th Timesteps Again) and noise addition for the generated image (Refined Generated Results) at each of a plurality of self-recurrence (Time Travel Strategy) timesteps. (Section: 4.2. Efficient Time-Travel Strategy) Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jampani’s Diffusion Model altered by Marri, Jefferson, and Kreis to incorporate Yu’s Time Travel Strategy. Since doing so would provide the benefit of refining generated results at specific or multiple timesteps. (Yu, Section: 4.2 Efficient Time-Travel Strategy)
Regarding claim 9, Jampani, Marri, Jefferson, and Kreis fail to teach the computing system of claim 8, wherein the one or more processing devices are configured to compute and apply the guidance update at one or more of the self-recurrence timesteps.
However, Yu teaches the computing system of claim 8, wherein the one or more processing devices are configured to compute and apply the guidance update at one or more of the self-recurrence (Time Travel Strategy) timesteps. (Section: 4.2 Efficient Time-Travel Strategy) Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jampani’s Diffusion Model altered by Marri, Jefferson, and Kreis to incorporate Yu’s Time Travel Strategy. Since doing so would provide the benefit of refining generated results at specific or multiple timesteps. (Yu, Section: 4.2 Efficient Time-Travel Strategy)
Regarding claim 17, has similar limitations as of claim 7, therefore it is rejected under the same rationale as claim 7.
Claim(s) 11 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Jampani et al. U.S. Patent Application US 20240412458 A1 (hereinafter Jampani) in view of Marri et al. U.S. Patent Application US 20240338859 A1 (hereinafter Marri) ) in further view of Jefferson et al. U.S. Patent Application US 20250022192 A1 (hereinafter Jefferson) in further view of Kreis et al. U.S. Patent Application US 20230377099 A1(hereinafter Kreis) in further view of Petersen et al. U.S. Patent Application US 20240378698 A1 (hereinafter Petersen).
Regarding claim 11, Jampani, Marri, Jefferson, and Kreis fail to teach the computing system of claim 1, wherein the one or more processing devices are further configured to:
receive an input video including a plurality of frames;
compute respective depth maps of the frames of the input video;
based at least in part on the depth maps, compute a guided video including a plurality of guided frames;
and output the guided video.
Jampani, Marri, Jefferson, Kreis, and Peterson are analogous to the claimed invention because all of them are in the same field of utilizing diffusion models to generate images.
However, Petersen teaches the computing system of claim 1, wherein the one or more processing devices are further configured to:
receive an input video including a plurality of frames; (Para. 0003 and 0006)
compute respective depth maps of the frames of the input video; (Para. 0049, 0060, and 0098)
based at least in part on the depth maps, compute a guided video (Consecutive Frames of the Video, Para. 0087 and 0089) including a plurality of guided frames (Output Unsampled frame, Para. 0066); (Para. 0062)
and output the guided video (Consecutive Frames of the Video, Para. 0087 and 0089). The consecutive frame of the video can be the enhanced frames, resulting in a new generated video of enhanced frames.
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jampani’s Diffusion Model altered by Marri, Jefferson, and Kreis to incorporate Petersen’s Video Input and Enhancement. Since doing so would provide the benefit of utilizing a diffusion model to generate videos.
Regarding claim 19, has similar limitations as of claim 11, therefore it is rejected under the same rationale as claim 11.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Jampani et al. U.S. Patent Application US 20240412458 A1 (hereinafter Jampani) in view of Marri et al. U.S. Patent Application US 20240338859 A1 (hereinafter Marri) in further view of NPL “DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models” by Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee (hereinafter Fan) ) in further view of Jefferson et al. U.S. Patent Application US 20250022192 A1 (hereinafter Jefferson) in further view of Kreis et al. U.S. Patent Application US 20230377099 A1(hereinafter Kreis).
Regarding claim 20, Jampani teaches a computing system (Para. 0105) comprising:
one or more processing devices (Processors or Computers, Para. 0105) configured to:
receive an image generation prompt (Text Prompt, Para. 0026, Fig. 2);
over a plurality of denoising timesteps, compute a guided image at least in part by: (Para.0008 and 0033)
iteratively applying a plurality of denoising updates to a generated image at a denoising diffusion model; (Para. 0033-0037)
and at a subset of the plurality of denoising timesteps (n Update Iterations, Para. 0043):
at a trained feedback model, computing a feedback model reward value based at least in part on a current-timestep generated image;
and applying respective guidance updates to the generated image based at least in part on the image generation prompt, the feedback model reward value, and a generated image set that includes a current-timestep generated image and a set of prior-timestep generated images, wherein the one or more processing devices are configured to compute each guidance update at least in part by:
performing a forward pass over the generated image set in a plurality of first integration timesteps, wherein performing the forward pass includes computing an estimated clean image based at least in part on the generated image set;
performing a backward pass over the generated image set in a plurality of second integration timesteps, wherein a size of the generated image set, a number of the first integration timesteps, and a number of the second integration timesteps are each equal to a predefined integration timestep count; and
computing the guidance update as a product of a guidance strength hyperparameter and a gradient of a guidance loss, wherein the guidance loss is computed based at least in part on the estimated clean image and the reference image;
and output, as the guided image, a final generated image (Output Image) computed in a final denoising timestep of the plurality of denoising timesteps. (Fig.2 and Para. 0048-0050)
However, Jampani fails to teach:
at a trained feedback model, computing a feedback model reward value based at least in part on a current-timestep generated image;
and applying respective guidance updates to the generated image based at least in part on the image generation prompt, the feedback model reward value, and a generated image set that includes a current-timestep generated image and a set of prior-timestep generated images, wherein the one or more processing devices are configured to compute each guidance update at least in part by:
performing a forward pass over the generated image set in a plurality of first integration timesteps, wherein performing the forward pass includes computing an estimated clean image based at least in part on the generated image set;
performing a backward pass over the generated image set in a plurality of second integration timesteps, wherein a size of the generated image set, a number of the first integration timesteps, and a number of the second integration timesteps are each equal to a predefined integration timestep count; and
computing the guidance update as a product of a guidance strength hyperparameter and a gradient of a guidance loss, wherein the guidance loss is computed based at least in part on the estimated clean image and the reference image;
Jampani and Marri are analogous to the claimed invention because both of them are in the same field of utilizing diffusion models for text-to-image generation.
Marri teaches:
at a trained feedback model, computing a feedback model reward value based at least in part on a current-timestep generated image;
and applying respective guidance updates to the generated image based at least in part on the image generation prompt (Para. 0020 and 0088), the feedback model reward value, and a generated image set (Para. 0089-0091) that includes a current-timestep generated image (X.Sub.1, …, X.sub.T, Para. 0089-0091) and a set of prior-timestep generated images (X.Sub.1, …, X.sub.T, Para. 0089-0091), wherein the one or more processing devices (Processor Unit, Para. 0040) are configured to compute each guidance update at least in part by: (Fig. 4) Diffusion models can map the current timestep image to intermediate images. X.sub.0 is the original image and X.Sub.1, …., X.sub.T are intermediate images.
performing a forward pass over the generated image set in a plurality of first integration timesteps (Para. 0090, Fig. 4), wherein performing the forward pass includes computing an estimated clean image based at least in part on the generated image set;
and performing a backward pass (Reverse Process) over the generated image set (Para. 0089-0091) in a plurality of second integration timesteps (Para. 0091, Fig. 4), wherein a size of the generated image set, a number of the first integration timesteps (Forward Process Timesteps, Para. 0089-0090), and a number of the second integration timesteps (Reverse Process, Para. 0089 and 0091) are each equal to a predefined integration timestep (Set of Diffusion Time Steps) count; (Para. 0050) The number of timesteps the diffusion model performs is based on the number of intermediate image embeddings. Both the forward and reverse processes utilize X.Sub.1, …., X.sub.T to create intermediate image embeddings, thus the size would be equal.
computing the guidance update as a product of a guidance strength hyperparameter and a gradient of a guidance loss, wherein the guidance loss is computed based at least in part on the estimated clean image and the reference image;
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jampani iterative diffusion process to incorporate Marri’s forward and reverse diffusion processes. Since doing so would provide the benefit of utilizing a standard diffusion model that includes a forward and reverse process. (Marri, Para. 0065-66)
However, Marri fails to teach:
at a trained feedback model, computing a feedback model reward value based at least in part on a current-timestep generated image;
and applying respective guidance updates to the generated image based at least in part on the feedback model reward value.
wherein performing the forward pass includes computing an estimated clean image based at least in part on the generated image set;
computing the guidance update as a product of a guidance strength hyperparameter and a gradient of a guidance loss, wherein the guidance loss is computed based at least in part on the estimated clean image and the reference image;
Jampani, Marri, and Fan are analogous to the claimed invention because all of them are in the same field of utilizing diffusion models for image generation.
Fan teaches:
at a trained feedback model (ImageReward Open-Source Reward Model Page 2 Para. 1-2 and Fig.1), computing a feedback model (Reward Scores or Reward Function, Fig. 1) reward value based at least in part on a current-timestep generated image (Fig.1 and Section: 4.1 RL Fine-tuning with KL Regularization);
and applying respective guidance updates to the generated image based at least in part on the feedback model (ImageReward Open-Source Reward Model Page 2 Para. 1-2 and Fig.1) reward value. The diffusion model is trained and modified based on the imageReward model. (Section: 4.1 RL Fine-tuning with KL Regularization)
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jampani’s Diffusion Model altered by Marri’s Forward/Reverse Process to incorporate Fan’s ImageReward Model. Since doing so would provide the benefit of ensuring better quality for the generated images. (Fan, Fig.2, Fig.3, and Section: 5.5 Fine-Tuning on Multiple Prompts)
However, Fan fails to explicitly teach:
wherein performing the forward pass includes computing an estimated clean image based at least in part on the generated image set;
computing the guidance update as a product of a guidance strength hyperparameter and a gradient of a guidance loss, wherein the guidance loss is computed based at least in part on the estimated clean image and the reference image;
Jampani, Marri, Fan, and Jefferson are analogous to the claimed invention because all of them are in the same field of utilizing diffusion models for image generation.
Jefferson teaches:
wherein performing the forward pass includes computing an estimated clean image based at least in part on the generated image set;
computing the guidance update as a product of a guidance strength (Guidance Strength Input, Para. 0031,0077-0078, and 0099) hyperparameter and a gradient of a guidance loss (Para. 0031, 0094, and 0099), wherein the guidance loss is computed based at least in part on the estimated clean image (Predicted Outputs or Current Results, Para. 0094 and 0099) and the reference image (Actual Targets or Target Results, Para. 0094 and 0099);
According to BRI (Broadest Reasonable Interpretation) Guidance Loss can refer to several different loss functions in machine learning. Such as the generic loss function used to guide the optimization process of training a model or a specific loss function used to guide the optimization process of modifying/processing the results of a model. Examiner recognizes from the applicants’ arguments/remarks and specification the “guidance loss” is a specific loss function used to guide the optimization process of a “guidance strength” parameter that modifies the results of the model and does not involve training the model. However, this relationship is not present in the independent claims as there is no mention of the “guidance loss” and “guidance strength” relationship or the “guidance updates, guidance loss, or guidance strength” does not involve training the model. Thus, it would be improper to import claim limitations from the specification (See MPEP 2111.01(II)).
Further Applicant recites “Jefferson does not disclose or suggest that the loss is multiplied by a guidance strength hyperparameter”. While Jefferson does not explicitly state the loss is multiplied by the guidance strength. Jefferson teaches a loss function (Para. 0093-0094 and 0099) that minimize/maximize the weights/parameters to reduce error or increase accuracy between the predicted output and the actual output. Jefferson teaches a guidance strength (Para. 0031, 0077-0078, and 0117) that directly controls the actual output. Changing the guidance strength would impact the loss as the guidance strength would modify the weights/parameters. One way the guidance strength could modify the weights/parameters is through multiplying the guidance strength to the changing weights/parameters. Thus, Jefferson teaches the loss is multiplied by a guidance strength hyperparameter.
Regarding claim 3, applicant recites “the office action incorrectly interprets the term "estimated clean image". The terminology “ estimated clean image” is broad and according to BRI can be any image that has been processed, corrected, regenerated, free of noise/artifacts/distortions, etc. and can be used in training a model. If applicant seeks a specific interpretation of “estimated clean image” examiner suggests amending claims to describe the specific “estimated clean image” sought.
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jampani’s Diffusion model altered by Marri and Fan to incorporate Jefferson’s guidance strength and guidance loss. Since doing so would provide the benefit of enhancing the diffusion model to allow the user to control the guidance strength for each timestep.
However Jefferson fails to teach:
wherein performing the forward pass includes computing an estimated clean image based at least in part on the generated image set;
Jampani, Marri, Fan, Jefferson, and Kreis are analogous to the claimed invention because all of them are in the same field of utilizing diffusion models for image generation.
Kreis teaches:
wherein performing the forward pass (Para. 0027) includes computing an estimated clean image (Para. 0030) based at least in part on the generated image set (Para. 0035-0037);
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jampani’s Diffusion model altered by Marri, Fan, and Jefferson to incorporate Kreis’s Clean Image. Since doing so would provide the benefit of better estimating the guidance loss by utilizing a clean image.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
NPL “Universal Guidance for Diffusion Models” by “Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, Tom Goldstein, University of Maryland, University of North Carolina at Chapel Hill, New York University” which teaches a product of a guidance strength and guidance loss (Section: 3.1. Forward Universal Guidance). As well as clean image (3. Universal Guidance) and a style reference image (Section: Style Guidance)
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRIANNA R COCHRAN whose telephone number is (571)272-4671. The examiner can normally be reached Mon-Fri. 7:30am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at (571) 272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BRIANNA RENAE COCHRAN/Examiner, Art Unit 2615
/ALICIA M HARRINGTON/Supervisory Patent Examiner, Art Unit 2615
Prosecution Insights