Detailed Action
1. Claims 1-20 are pending in this Application.
Notice of Pre-AIA or AIA Status
2. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless -
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
3. Claims 1-7,10-17 and 20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by
Kimin Lee et al., (hereafter Kimin), “Aligning Text-to-Image Models using Human Feedback”, Google Research, pub. 02/23/2023.
As to claim 1, Kimin teaches A method for training a generative image model (Abstract a method of Fine-tuning text-to-image diffusion models with human feedback that improves text-to-image alignment)comprising:
providing, to the generative image model, an image caption associated with input image data ( Fig.1 Abstract , Fig. 1 : “The steps in our fine-tuning method. (1) Multiple images sampled from the text-to-image model using the same text prompt, followed by collection of (binary) human feedback. (2) A reward function is learned from human assessments to predict image-text alignment. We also utilize an auxiliary objective called prompt classification, which identifies the original text prompt within a set of perturbed text prompts);
receiving, from the generative image model, output image data ( as discuss above (2) A reward function is learned from human assessments to predict image-text alignment. The model output the predict image-text alignment);
providing the output image data to an image scoring model that scores images according to image quality based on an image quality metric (Figs.1 and 6 , section 3.2 1st par., section 4.3 1st par., “Auxiliary objective: prompt classification. Using this human labeled dataset, we train a reward function to predict human feedback given the image and text prompt. We propose an auxiliary task—identifying the original text prompt within a set of perturbed text prompts—to more effectively exploit human feedback for reward learning.” To measure image-text alignment, we learn a reward function rφ(x, z). Given two images from the same text prompt(x1,x2,z),we check whether our reward function generates a higher score for the human-preferred image” );
receiving image quality data associated with the output image data from the image scoring model, the image quality data characterizing image quality of the output image data according to the image quality metric (page 1 section 1 right col., 2nd par ., page 2 section 1, left col., 2nd par., “We update the model using semi-supervised learning to measure model-output quality w.r.t. The learned reward function. We show that the learned reward function predicts human assessments of the quality more accurately than the CLIP score (Radford et al., 2021). In addition, we show that rejection sampling based on our learned reward function can also significantly improve the image-text alignment”);
using a loss function, computing a loss based on at least the output image data and the image quality data; and using the loss, optimizing the generative image model to generate images with high image quality according to the image quality metric (page 2 section 1, left col.,1st par., page 4 section 3.3, left col., 1st par., “We analyze several design choices, such as using an auxiliary loss for reward learning and the effect of using “diverse” datasets for fine-tuning”. The authors use the learned reward function rφ to update the text-to-image model p with parameters θ by minimizing the loss function , see equation 2).
As to claim 2, Kimin teaches The method of claim 1, wherein the loss comprises a reconstruction loss that is computed based on the output image data (page 2 section 1, left col.,1st par., page 4 section 3.3, left col., 1st par., We analyze several design choices, such as using an auxiliary loss for reward learning and the effect of using “diverse” datasets for fine-tuning”. The output image data corresponds the to the learned reward function rφ ), and the loss further comprises a refinement loss that is computed based on the image quality data (as discus in claim 1 above, the authors use the learned reward function rφ to update the text-to-image model p with parameters θ by minimizing the loss function , see equation 2).
As to claim 3, Kimin teaches the output image data comprises a plurality of output images ( Fig.1 page 1 section 1 right col., 2nd par ., page 2 section 1, left col., 2nd par., “The steps in our fine-tuning method. (1) Multiple images sampled from the text-to-image model using the same text prompt, followed by collection of (binary) human feedback.” (2) A reward function is learned from human assessments to predict image-text alignment.)and the image quality data comprises a ranking of the plurality of output images according to the image quality metric ((2) page 1 section 1 right col., 2nd par ., page 2 section 1, left col., 2nd par., A reward function is learned from human assessments to predict image-text,. The model output the predict image-text alignment. We show that the learned reward function predicts human assessments of the quality more accurately than the CLIP score (Radford et al., 2021). In addition, we show that rejection sampling based on our learned reward function can also significantly improve the image-text alignment )
As to claim 4, Kimin teaches ranking the plurality of output images generates a plurality of ranked images, the method further comprising providing the plurality of ranked images as training data to the generative image model for performing a training process on the generative image model(page 1 section 1 right col., 2nd par ., page 2 section 1, left col., 2nd par., section 3.1, 1st par., To test specific capabilities of a given text-to-image model, we consider three categories of text prompts that generate objects with a specified count, color, or background.2 For each category, we generate prompts by combining a word or phrase from that category with some object; We also consider combinations of the three categories (e.g., two green dogs in a city). From each prompt, we generate up to 60 images using a pretrained text-to-image model—in this work, we use Stable Diffusion).
As to claim 5, Kimin teaches providing the input image data to the generative image model, wherein the loss is computed further based on the input image data( page 4 left col., 1st and 2nd pars., To effectively exploit the feedback dataset, we design a simple data augmentation scheme and auxiliary loss for reward learning. For each image-text pair that has been labeled good, we generate N 1 text prompts with different semantics than the original text prompt. We use the augmented prompts in an auxiliary task, namely, classifying the original prompt for reward learning. Our prompt classifier uses the reward function rφ . Our experiments show this auxiliary loss Lpc (see equation1) improves the generalization to unseen images and text prompts. Finally, we define the combined loss Lrewared as show in equation below).
As to claim 6, Kimin teaches receiving scoring training data comprising a plurality of images and a corresponding plurality of image scores (Figure 1. The steps in our fine-tuning method. (1) Multiple images sampled from the text-to-image model using the same text prompt, followed by collection of (binary) human feedback. (2) A reward function is learned from human assessments to predict image-text alignment. We also utilize an auxiliary objective called prompt classification, which identifies the original text prompt within a set of perturbed text prompts. ); and
performing a training process using the scoring training data to train the image scoring model ( Figs.1 and 6 , section 3.2 1st par., section 4.3 1st par., “Auxiliary objective: prompt classification. Using this human labeled dataset, we train a reward function to predict human feedback given the image and text prompt. We propose an auxiliary task—identifying the original text prompt within a set of perturbed text prompts—to more effectively exploit human feedback for reward learning.” To measure image-text alignment, we learn a reward function rφ(x, z). Given two images from the same text prompt(x1,x2,z),we check whether our reward function generates a higher score for the human-preferred image).
As to claim 7, Kimin teaches receiving image training data comprising a plurality of images and a corresponding plurality of image captions (Fig.5, (a) Fine-tuned model only with human-labeled dataset. (c) Fine-tuned model with human-labeled, unlabeled and pre-training datasets.); and
performing a training process using the image training data to train the generative image model ( pages 3-4 section 4.1, 1st par., For our baseline generative model, we use stable diffusion v1.5 , which has been pre-trained on large image-text datasets. For fine-tuning, we freeze the CLIP language encoder and fine-tune only the diffusion module. For the reward model, we use ViT-L/14 CLIP model to extract image and text embeddings and train a MLP using these embeddings as input).
As to claim 10, Kimin teaches the generative image model is one of a Generative Adversarial Network (GAN), a Variational Autoencoder (VAE), an autoregressive model, a diffusion model (Fig. 2, page 1 section 1 left col., last par., pages 3-4 section 4.1, 1st par., We fine-tune the stable diffusion model (Rombach et al.,2022) using 27K image-text pairs with human feedback. Our fine-tuned model shows improvement in generating objects with specified colors, counts, and backgrounds. Moreover, it improves compositional generation)and a transformer-based architecture).
As to claim 11, Kimin teaches A non-transitory computer-readable medium storing a program for training a generative image model, which when executed by a computer, configures the computer to: provide (Fig. 2, page 19, teaches Stable Diffusion v1.5, Reward Learning model and an algorithm (Algorithm 1) that that describe Reward Learning Pseudocode. It is inherent that the Stable Diffusion v1.5 requires both memory (RAM and VRAM) and a processor (CPU and GPU) to run the algorithm 1 that describe the Reward Learning Pseudocode.);
regarding the remaining limitation of claim 11, the remaining limitations are rejected the same as claim 1 except claim 11 is directed to a computer program claim. All the remaining limitations of claim 11 are addressed in claim 1. Thus, argument analogous to that presented above for claim 1 is applicable to remaining limitation of claim 11.
Claim 12 is rejected the same as claim 2 except claim 12 is directed to a computer program claim. All the limitations of claim 12 are addressed in claim 2. Thus, argument analogous to that presented above for claim 2 is applicable to claim12.
Claim 13 is rejected the same as claim 3 except claim 13 is directed to a computer program claim. All the limitations of claim 13 are addressed in claim 3. Thus, argument analogous to that presented above for claim 3 is applicable to claim13.
Claim 14 is rejected the same as claim 4 except claim 14 is directed to a computer program claim. All the limitations of claim 14 are addressed in claim 4. Thus, argument analogous to that presented above for claim 4 is applicable to claim14.
Claim 15 is rejected the same as claim 5 except claim 15 is directed to a computer program claim. All the limitations of claim 15 are addressed in claim 5. Thus, argument analogous to that presented above for claim 5 is applicable to claim15.
Claim 16 is rejected the same as claim 6 except claim 16 is directed to a computer program claim. All the limitations of claim 16 are addressed in claim 6. Thus, argument analogous to that presented above for claim 6 is applicable to claim16.
Claim 17 is rejected the same as claim 7 except claim 17 is directed to a computer program claim. All the limitations of claim 17 are addressed in claim 7. Thus, argument analogous to that presented above for claim 7 is applicable to claim17.
Claim 20 is rejected the same as claim 11 except claim 20 is directed to a system claim. All the limitations of claim 20 are addressed in claim 11. Thus, argument analogous to that presented above for claim 11 is applicable to claim 20.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
4. Claim 8-9, 18 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Kimin, “Aligning Text-to-Image Models using Human Feedback”, in view of Andres Monroy-Hernandez ( hereafter Andres), “Designing for Remixing: Supporting an Online Community of Amateur Creators”, Ph.D Thesis, pub. 08/20/2012.
Regarding claim 8, while Kimin, teaches the image quality metric, claim, but fails to teach
“wherein the image quality metric is computed from engagement data of images from a social media platform, the engagement data comprising clicks, views, likes, saves, downloads, favorites, shares, and remixes”
On the other hand Andres teaches engagement data of images from a social media platform, the engagement data comprising clicks, views, likes, saves, downloads, favorites, shares, and remixes (Remixing Systems (see page 24-36), The Scratch Online Community ( see page 37-54). The specific online community and creative programming database that uniquely features remixes alongside views, likes, saves, downloads, favorites, and shares for interactive media projects is the Scratch Online Community. Sharing a friend's post on Facebook, automatically attributed. ( page 32), the "retweet" button, automating attribution on Twitter( page 31), Download button and license statement of a YouTube video ( page 25). See also Figs. 1-2, 1-3,2-2 ,2-3, 3-1,3-2 and 3-8 for example)
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the database teachings of remixing systems and online creative communities like Scratch taught Andres into an image diffusion model framework using human feedback taught by Kimin . Doing so provides a collaborative repository of iterative user modifications and preference data to further enhance text-to-image alignment and continuous model refinement.
Regarding claim 9, while Kimin teaches the image quality metric, but fails to teach
“usage data of images from a stock image database , the usage data comprising searches, views, favorites, saves, downloads, purchases , and resolutions”.
On the other hand Andres teaches usage data of images from a stock image database , the usage data comprising searches, views, favorites, saves, downloads, purchases , and resolutions(Remixing Systems (see page 24-36), The Scratch Online Community ( see page 37-54). The specific online community and creative programming database that uniquely features remixes alongside views, likes, saves, downloads, favorites, and shares for interactive media projects is the Scratch Online Community. Sharing a friend's post on Facebook, automatically attributed. ( page 32), the "retweet" button, automating attribution on Twitter( page 31), Download button and license statement of a YouTube video ( page 25). See also Figs. 1-2, 1-3,2-2 ,2-3, 3-1,3-2 and 3-8 for example).
Claim 18 is rejected the same as claim 8 except claim 18 is directed to a computer program claim. All the limitations of claim 18 are addressed in claim 8. Thus, argument analogous to that presented above for claim 8 is applicable to claim18.
Claim 19 is rejected the same as claim 9 except claim 19 is directed to a computer program claim. All the limitations of claim 19 are addressed in claim 9. Thus, argument analogous to that presented above for claim 9 is applicable to claim19.
Prior art not used in rejections but pertinent to the claims or disclosure.
a. “UPainting: Unified Text-to-Image Diffusion Generation with Cross-modal Guidance” to Wei et al.,
In this paper, we systematically study the problem of text-conditional image generation for both simple and complex scenes, and propose a simple yet effective method to unify them. We find that effectively combining cross-modal matching models with pre-trained transformer language models can greatly improve sample fidelity and image-text alignment for diffusion image generation, which gives the model a general ability to generate images for both simple and complex scenes. To more comprehensively compare different text-to-image generation models, we also propose a comprehensive and challenging evaluation benchmark for both simple and complex scene image generation. On this benchmark, UPainting greatly outperforms other strong models such as Stable Diffusion and Disco Diffusion, on both image-text alignment and image fidelity(see abstract).
b. “Optimizing Prompts for Text-to-Image Generation”, Advances in Neural Information Processing Systems 36 (NeurIPS 2023(, pub. 2023, to Yaru Hao et al.. disclosed
Well-designed prompts can guide text-to-image models to generate amazing images. However, the performant prompts are often model-specific and misaligned with user input. Instead of laborious human engineering, we propose prompt adaptation, a general framework that automatically adapts original user input to model-preferred prompts. Specifically, we first perform supervised fine-tuning with a pretrained language model on a small collection of manually engineered prompts. Then we use reinforcement learning to explore better prompts. We define a reward function that encourages the policy to generate more aesthetically pleasing images while preserving the original user intentions. Experimental results on Stable Diffusion show that our method outperforms manual prompt engineering in terms of both automatic metrics and human preference ratings. Moreover, reinforcement learning further boosts performance, especially on out-of-domain prompts.” See Abstract.
Contact Information
Any inquiry concerning this communication or earlier communication from the examiner should be directed to Mekonen Bekele whose telephone number is (469) 295-9077.The examiner can normally be reached on Monday -Friday from 9:00AM to 6:50 PM Eastern Time.
If attempt to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Eng, George can be reached on (571) 272-7495.The fax phone number for the organization where the application or proceeding is assigned is 571-237-8300. Information regarding the status of an application may be obtained from the patent Application Information Retrieval (PAIR) system. Status information for published application may be obtained from either Private PAIR or Public PAIR.
Status information for unpublished application is available through Privet PAIR only.
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have question on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866.217-919 (tool-free)
/MEKONEN T BEKELE/Primary Examiner, Art Unit 2699
Prosecution Insights