DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-4, 6, 7, 13-16, 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ramesh et al. (U.S. Patent 11,922,550) in view of Bachmann et al. (4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities) and further in view of Greenen et al. (U.S. Patent Application 20240221242).
In regards to claim 1, Ramesh teaches a method [Fig. 2; e.g. method, c.10 L.12-30] comprising:
obtaining an input prompt that indicates an image element [e.g. Accessing a text description may include at least one of retrieving, requesting, receiving, acquiring, or obtaining a text description. Text descriptions corresponding to the set of images may include captions, subtitles, or details that may explain features of an image or represent the image in a written form, c.8 L.29-54, also see c.5 L.35-59];
encoding, using an image generation prior model [e.g. the first sub-model comprises a prior model, c.9 L.52-c.10 L.11], the input prompt to obtain a prior embedding that represents the image element [e.g. the first sub-model is configured to encode, prior to generating the corresponding image embedding, at least one of the text description or the text embedding, via a transformer, as a sequence of tokens predicted autoregressively, c.10 L.31-60]; and
generating, using an image generation model, a synthetic image based on the prior embedding [e.g. The operations may include inputting at least one of the text description or the corresponding image embedding, generated by the first sub-model, into a second sub-model configured to generate, based on at least one of the text description or the corresponding image embedding, an output image, c.2 L.9-34].
Ramesh does not explicitly teach
a sparsity level;
a prior embedding that represents the sparsity level;
wherein the synthetic image depicts a pattern including the image element and the sparsity level.
However, Bachmann teaches
a sparsity level [e.g. crowdedness score, SAM clutter score, or COCO clutter score, see section titled “Modalities” in page 4];
a prior embedding [e.g. Semantic metadata, see sections titled “Modalities” and “Modality-specific accommodations” in pages 4 and 25] that represents the sparsity level [e.g. crowdedness score, SAM clutter score, or COCO clutter score, see section titled “Modalities” in page 4];
wherein the synthetic image depicts a scene [Fig. 4; e.g. picture of a Swiss mountain scene, see section titled “Steerable multimodal generation” in page 6] including the image element [Fig. 4; e.g. humans, mountain, see section titled “Steerable multimodal generation” in page 6] and the sparsity level [Fig. 4; e.g. crowdedness score, SAM clutter score, or COCO clutter score, see sections titled “Modalities” and “Steerable multimodal generation” in pages 4 and 6].
Therefore, it would have been obvious to one of ordinary skill in the art to have modified Ramesh’s method with the features of
a sparsity level;
a prior embedding that represents the sparsity level;
wherein the synthetic image depicts a scene including the image element and the sparsity level;
in the same conventional manner as taught by Bachmann because Bachmann provides a more direct and steerable method of controlling the multimodal data generation
process, enabling exciting further research into generative dataset design [see section titled “Steerable multimodal generation” in page 6 and Figure 4].
Ramesh as modified by Bachmann does not explicitly teach
wherein the synthetic image depicts a pattern (emphasis added).
However, Greenen teaches
wherein the synthetic image depicts a pattern [Fig. 3A; e.g. output content includes image content comprising tiles, 0033-0034, also see 0001].
Therefore, it would have been obvious to one of ordinary skill in the art to have modified the combination of Ramesh’s method and the teachings of Bachmann with the features of wherein the synthetic image depicts a pattern in the same conventional manner as taught by Greenen because generating patterns is well known and commonly used in the art of generative AI systems [0001].
In regards to claim 2, Ramesh does not explicitly teach the method of claim 1, wherein obtaining the input prompt comprises:
obtaining a content prompt and a sparsity indicator;
identifying a sparsity token based on the sparsity indicator; and
combining the content prompt with the sparsity token to obtain the input prompt.
However, Bachmann teaches the method of claim 1, wherein obtaining the input prompt [see rejection of claim 1 above] comprises:
obtaining a content prompt [Fig. 4; e.g. caption input, see section titled “Steerable multimodal generation” in page 6] and a sparsity indicator [e.g. crowdedness score, SAM clutter score, or COCO clutter score, see section titled “Modalities” in page 4];
identifying a sparsity token [e.g. special tokens, see section titled “Tokenization” in page 5] based on the sparsity indicator [e.g. based on the metadata, see section titled “Tokenization” in page 5]; and
combining the content prompt with the sparsity token to obtain the input prompt [e.g. We chose to model metadata using interleaved pairs of special tokens, where the first one specifies the type of metadata modality, and the second specifies its value. For example, a crowdedness score of 3 and a brightness of 120 would be specified as the sequence v1=5 v0=3 v1=10 v0=120. During training the number of metadata entries and their order is randomized. All of this results in a sequence prediction formulation. All modalities are first transformed into sequences of discrete tokens using modality-specific tokenizers, see sections titled “Method”, “Tokenization of sequence modalities” in pages 3, 25].
Therefore, it would have been obvious to one of ordinary skill in the art to have modified Ramesh’s method with the features of wherein obtaining the input prompt comprises:
obtaining a content prompt and a sparsity indicator;
identifying a sparsity token based on the sparsity indicator; and
combining the content prompt with the sparsity token to obtain the input prompt
in the same conventional manner as taught by Bachmann because Bachmann provides a more direct and steerable method of controlling the multimodal data generation
process, enabling exciting further research into generative dataset design [see section titled “Steerable multimodal generation” in page 6 and Figure 4].
In regards to claim 3, Ramesh teaches the method of claim 1, wherein: the input prompt comprises a sequence of text tokens [e.g. sequence of tokens, c.2 L.63-67]; and the prior embedding comprises a representation of the input prompt in an image embedding space [e.g. Joint representation may be a latent space representation of the text embedding and the image embedding, c.9 L.33-51].
In regards to claim 4, Ramesh teaches the method of claim 1, wherein: the prior embedding encodes the pattern [e.g. The image embedding is used to generate a modified instance of the output image through interpolation. For example, a car with modern features, c.17 L.4-42].
In regards to claim 6, Ramesh teaches the method of claim 1, wherein generating the synthetic image comprises: obtaining a noise input [e.g. The diffusion process may involve adding noise to an input, c.10 L.61-c.11 L.53]; and denoising the noise input based on the prior embedding to obtain the synthetic image [e.g. a diffusion model may involve a neural network which denoises an image by reversing a diffusion process, c.10 L.61-c.11 L.53].
In regards to claim 7, Ramesh does not explicitly teach the method of claim 1, wherein: the image generation model is trained using training data including a training prompt that indicates the sparsity level.
However, Bachmann teaches the method of claim 1, wherein: the image generation model is trained using training data [e.g. trained on CC12M datasets including the semantic metadata, see sections titled “Modalities” and “Training details” in pages 4, 5] including a training prompt that indicates the sparsity level [e.g. To address the sparsity and number of different types of metadata, the metadata modalities are all grouped together as a single modality during training, see section titled “Modality-specific accommodations” in page 25].
Therefore, it would have been obvious to one of ordinary skill in the art to have modified Ramesh’s method with the features of wherein: the image generation model is trained using training data including a training prompt that indicates the sparsity level in the same conventional manner as taught by Bachmann because Bachmann provides a more direct and steerable method of controlling the multimodal data generation
process, enabling exciting further research into generative dataset design [see section titled “Steerable multimodal generation” in page 6 and Figure 4].
In regards to claim 13, the claim recites similar limitations as claim 1 but in the form of an apparatus comprising:
at least one processor; at least one memory storing instructions executable by the at least one processor for performing the method of claim 1.
Furthermore, Ramesh teaches an apparatus [e.g. system, c.5 L.14-34] comprising:
at least one processor [e.g. at least one processor, c.5 L.14-34];
at least one memory [e.g. at least one memory, c.5 L.14-34] storing instructions [e.g. instructions, c.5 L.14-34] executable by the at least one processor for performing the method of claim 1.
Therefore, the same rationale as claim 1 is applied.
In regards to claim 14, Ramesh teaches the apparatus of claim 13, further comprising: a text encoder [e.g. text encoder, c.2 L.9-34] configured to encode the input prompt to generate a text embedding [e.g. text embedding, c.2 L.9-34].
In regards to claim 15, Ramesh teaches the apparatus of claim 13, further comprising: an image encoder [e.g. image encoder, c.2 L.35-45] configured to encode a training image [e.g. training an image encoder on the first data set, c.2 L.35-45] to generate an image embedding [e.g. image embedding, c.2 L.35-45].
In regards to claim 18, Ramesh teaches the apparatus of claim 13, wherein: the image generation model comprises a diffusion model [e.g. diffusion model, c.2 L.60-62].
In regards to claim 19, Ramesh teaches the apparatus of claim 13, wherein: the image generation prior model comprises a diffusion model [e.g. diffusion prior may comprise a diffusion model, c.10 L.61-64].
In regards to claim 20, Ramesh does not explicitly teach the apparatus of claim 13, further comprising: a color extractor configured to decode an output from the image generation model to obtain a color palette.
However, Bachmann teaches the apparatus of claim 13, further comprising: a color extractor [e.g. PyPalette, see section titled “Modalities” in page 3] configured to decode an output [e.g. decoding tokens, see section titled “Steerable multimodal generation” in page 6] from the image generation model to obtain a color palette [e.g. color palette, see section titled “Modalities” in page 3].
Therefore, it would have been obvious to one of ordinary skill in the art to have modified Ramesh’s method with the features of a color extractor configured to decode an output from the image generation model to obtain a color palette in the same conventional manner as taught by Bachmann because Bachmann provides a more direct and steerable method of controlling the multimodal data generation process, enabling exciting further research into generative dataset design [see section titled “Steerable multimodal generation” in page 6 and Figure 4].
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ramesh et al. (U.S. Patent 11,922,550) in view of Bachmann et al. (4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities) and further in view of Greenen et al. (U.S. Patent Application 20240221242) as applied to claim 1 above, and further in view of Khamis et al. (U.S. Patent Application 20240371096).
In regards to claim 5, Ramesh as modified by Bachmann and Greenen does not explicitly teach the method of claim 1, further comprising: obtaining a color input indicating one or more colors, wherein the synthetic image is generated based on the color input to include the one or more colors.
However, Khamis teaches the method of claim 1, further comprising: obtaining a color input indicating one or more colors [e.g. receiving, as an input, a color code, 0040], wherein the synthetic image is generated based on the color input to include the one or more colors [e.g. generating an adjusted image based on the color code, 0024, 0040].
Therefore, it would have been obvious to one of ordinary skill in the art to have modified the combination of Ramesh’s method and the teachings of Bachmann and Greenen with the features of obtaining a color input indicating one or more colors, wherein the synthetic image is generated based on the color input to include the one or more colors in the same conventional manner as taught by Khamis because generating a synthetic image based on color inputs is well known and commonly used in the art of generative AI systems.
Allowable Subject Matter
Claims 8-12 are allowed.
Claims 16, 17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
In regards to claim 8, Ramesh teaches a method of training a machine learning model [e.g. method for training a model, c.7 L.43-c.8 L.28], the method comprising:
obtaining training data [e.g. training data, c.7 L.43-c.8 L.28] including an image [e.g. image data set, c.7 L.43-c.8 L.28] and a training prompt that indicates an image element [e.g. text data set may comprise captions corresponding to images in image data set, c.7 L.43-c.8 L.28];
generating a predicted prior embedding [e.g. predicted image embedding, c.10 L.31-60] based on the training prompt [e.g. based on the text description, c.10 L.31-60]; and
training, using the training data, an image generation prior model [e.g. training the autoregressive prior, c.10 L.31-60] to generate a prior embedding [e.g. image embedding, c.10 L.31-60] that represents the image element [e.g. elements or components within image embedding, c.7 L.43-c.8 L.28].
Ramesh does not explicitly teach
classifying the image using a sparsity classifier to obtain a sparsity level;
generating a predicted prior embedding based on the sparsity level (emphasis added); and
a prior embedding that represents a pattern with the sparsity level.
However, Bachmann teaches
obtain a sparsity level [e.g. crowdedness score, SAM clutter score, or COCO clutter score, see section titled “Modalities” in page 4];
generating a predicted prior embedding based on the sparsity level; and
a prior embedding that represents a pattern with the sparsity level.
Ramesh as modified by Bachmann fails to teach or suggest
classifying the image using a sparsity classifier to obtain a sparsity level (emphasis added);
generating a predicted prior embedding based on the sparsity level; and
a prior embedding that represents a pattern with the sparsity level.
Therefore, claim 8 is allowed over the prior art of record.
In regards to claims 9-12, the claims depend on at least claim 8. Therefore, the claims 9-12 are allowed for at least the same reason as claim 8.
In regards to claim 16, the prior art of record fails to teach or suggest the apparatus of claim 13, further comprising: a sparsity classifier configured to classify a training image to obtain the sparsity level.
In regards to claim 17, the prior art of record fails to teach or suggest the apparatus of claim 13, further comprising: an aesthetic classifier configured to filter a preliminary set of training images.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW SHIN whose telephone number is (571)270-5764. The examiner can normally be reached Monday - Friday from 11:00AM to 7:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said Broome can be reached at 571-272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW SHIN/Examiner, Art Unit 2612
/Said Broome/Supervisory Patent Examiner, Art Unit 2612
Prosecution Insights