DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination (RCE) under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on May 4, 2026 has been entered.
Claims 1, 12, and 16 are amended; claim 10 is cancelled; claims 21 and 22 are added.
Claims 1-9 and 11-22 are pending and under examination.
Declaration Under 37 CFR 1.130(a)
Applicant maintains that Betker et al. ("Improving Image Generation with Better Captions", hereinafter "Betker") is not available as prior art under 35 U.S.C. 102(a)(1) by operation of the 35 U.S.C. 102(b)(1)(A) exception, relying on the declaration of James Betker and Aditya Ramesh filed November 5, 2025 and on In re Katz, 687 F.2d 450 (C.C.P.A. 1982). The argument is not persuasive, and no supplemental declaration was filed with the RCE.
In re Katz holds that co-authorship does not, by itself, raise a presumption of co-inventorship. The rejection of the declaration does not rest on a bare co-authorship presumption. It rests on specific evidence in the Betker publication that contradicts the declarants' assertion: the publication designates Gabriel Goh and Li Jing with the symbol "*" denoting "Equal contribution." Under MPEP 717.01(a)(1) and Ex parte Kroger, 219 USPQ 370 (Bd. App. 1982), "[a] mere statement from the inventor or a joint inventor, without any accompanying reasonable explanation, may not be sufficient where there is evidence to the contrary." An "equal contribution" designation on a publication describing the very subject matter relied upon (pages 5-7, 9, and 17, and Fig. 3) is evidence to the contrary that requires a reasonable explanation.
The declaration does not supply that explanation. It states only that the listed co-authors "did not conceive of or invent the elements" relied upon, without addressing why authors credited with equal contribution to the publication did not contribute to conception of the subject matter that publication describes. The declaration further misidentifies "Li Jing" as "Li Jeng" and does not explain the involvement of the equally-credited Li Jing, and is silent as to Linjie Li, a separately listed author. The declarants' assertion that James Betker and Aditya Ramesh are the inventors of the relied upon elements does not rebut the "equal contribution" evidence as to Goh and Jing. The declaration therefore does not establish that the relied-upon disclosure of Betker originated solely with the named inventors, the 35 U.S.C. 102(b)(1)(A) exception is not invoked, and Betker remains available as prior art under 35 U.S.C. 102(a)(1).
Response to Arguments
Applicant's arguments filed 11/5/2025 have been fully considered but they are not persuasive.
Claims 1-20 are pending in this application and have been considered below.
Arguments:
Applicant amended claims 1, 12, and 16 to recite "training an image generation model"
and "generating, with the image generation model, an image," and argues the claims are like
eligible Example 39 rather than ineligible Example 47 because no mathematical algorithm is
named.
Examiner’s Response:
The amendment does not remove the recited judicial exception. The claims do not merely train a model; they apply trained models to transform data: claim 1 recites "applying an image captioner model to images" and "generating, with the image generation model, an image"; claim 12 recites "applying the tuned image captioner model to images"; and claim 16 recites "upsampling the text description with a language model." Applying a trained model executes the mathematical relationships learned during training to transform input data into output data. This is the execution of mathematical operations, not the generic "training ... using the first training set" limitation found non-exceptional in Example 39 and the August 4, 2025 Memorandum. The specification confirms the underlying operations are mathematical, describing a likelihood function objective and updating of the model parameter (Spec ¶¶ 52-53). (Step 2A, Prong One: YES.)
Applicant argues the added "generating ... an image" limitation reflects the specification's described improvements and that, under Ex Parte Desjardins (Appeals Review Panel, designated precedential Nov. 4, 2025), a single claim limitation reflecting a disclosed improvement confers eligibility. This is not persuasive. Desjardins turned on a claim limitation that itself recited the improvement to model operation: "adjust the first values of the plurality of parameters to optimize performance of the machine learning model on the second machine learning task while protecting performance of the machine learning model on the first machine learning task." Amended claims 1, 12, and 16 contain no comparable limitation. They recite only generic "training an image generation model" and "generating ... an image," with no recitation of how a model is improved or any improved property of the generated image. The improvements Applicant relies on are disclosed only in the specification (¶¶ 10-12, 52, 64), not recited in the claims. The added step is the generic output of the abstract process, comparable to Example 47, Claim 2 and unlike Example 47, Claim 3 or Example 48, Claim 2. (Step 2A, Prong Two: NO.)
Claim 22 recites that the image generation model "generates images having a higher CLIP score than images generated by an image generation model trained using only ground-truth captions." This does not change the result. Unlike the limitation in Desjardins, which recited the specific parameter-adjustment operation by which the model is improved, claim 22 recites only a desired comparative result (a higher CLIP score) of the same mathematical training process. A recited outcome of the judicial exception, untethered to any recited technical means of achieving it, is not an improvement to technology and does not integrate the exception into a practical application. See MPEP 2106.04(d)(1); 2106.05(a). The CLIP score is itself a mathematical measure (a cosine similarity of embeddings), so reciting that the output scores higher characterizes the result of the mathematical process rather than improving computer functionality.
Applicant argues that training the captioner "with compressed images" is non-conventional activity amounting to significantly more. Under Berkheimer v. HP, Inc., 881 F.3d 1360 (Fed. Cir. 2018) and MPEP 2106.05(d), a bare assertion of non-conventionality is insufficient, and the record evidences conventionality: the specification describes the image embedding underlying the "compressed representation space" as a generic "numerical representation of an image, such as a vector" that "may be generated through neural networks, such as CNNs, GANs, or transformers" (Spec ¶ 51), and describes the datasets as "available as public datasets" (¶ 46). The compressed-representation training is the application of well-understood, routine, and conventional image-embedding techniques within the recited mathematical process. Likewise, "generating ... an image" with a trained image generation model is the conventional, expected output of such a model, recited at a high level of generality without any technical detail; outputting the result of the model is insignificant post-solution activity that is well-understood, routine, and conventional. See MPEP 2106.05(d), (g).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-9 and 11-22 are rejected under 35 U.S.C. § 101 because the claimed invention is directed to abstract ideas (mathematical concepts) without integration into a practical application and without providing significantly more.
Independent Claims 1, 12, and 16 Analysis
Step 1: Statutory Category
Claim 1 recites a process. Claims 12 and 16 each recite a machine. Accordingly, claims 1, 12, and
16 fall within statutory categories of 35 U.S.C. § 101. (Step 1: YES)
Step 2A, Prong One: Does the Claim Recite a Judicial Exception?
Claims 1, 12, and 16 recite limitations directed to training, tuning, and applying machine learning models to process data:
Claim 1 recites "generating a recaptioned dataset by applying an image captioner model to images in the text-to -image dataset, the image captioner model trained with compressed images corresponding to an image dataset an image dataset, a first tuning stage, and a second tuning stage; training an image generation model with the recaptioned dataset: and generating, with the image generation model, an image. "
Claim 12 recites "generating an image captioner model configured to generate captions from input images, the image captioner model trained using a text-to image dataset, wherein the text-to-image dataset comprises one or more digital compressed image-caption pairs"; "performing a first tuning stage ... training the image captioner model using a first set of captions"; "performing a second tuning stage ... training the image captioner using the set of synthetic captions;" and "generating a captioned dataset by applying the tuned image captioner model to images in a dataset; training an image generation model with the captioned dataset; and generating, with the image generation model, an image."
Claim 16 recites "upsampling the text description with a language model" and "training an image generation model with a dataset comprising image-caption pairs, wherein at least a portion of captions are generated with an image captioner model that is trained on compressed image data; and generating, with the image generation model, an image based on the upsampled text description."
These limitations recite mathematical concepts. Applying a trained model executes the mathematical operations learned during training, and training and tuning machine learning models involve optimization algorithms that iteratively adjust model parameters through mathematical calculations to minimize a loss function. The specification confirms this at ¶¶
52-53 (maximizing a likelihood function objective and updating the model parameter). See
MPEP 2106.04(a)(2)(I); July 2024 SME Example 47.
The recaptioning limitation also recites a mental process. "generating a recaptioned dataset by applying an image captioner model to images" (claim 1), "generate captions from input images" (claim 12), and the dependent recitations of generating short and descriptive captions that describe an image's main subject, background, and coloration (claims 4 and 7) encompass observation and evaluation that can be practically performed in the human mind. A person can view an image and describe its subject, surroundings, and colors. The nominal recitation of an image captioner model to perform the description does not remove the step from the mental process grouping; the model is invoked merely as a tool to carry out the evaluation. That the description is generated across a dataset rather than a single image is a matter of scale and speed and does not change the character of the act. See MPEP 2106.04(a)(2)(III), (III)(C); Electric Power Group, LLC v. Alstom S.A., 830 F.3d 1350, 1353-54 (Fed. Cir. 2016). Because the claim recites both a mathematical concept and a mental process, the limitations are treated together as a single abstract idea. See MPEP 2106.04.
(Step 2A, Prong One: YES.)
Step 2A, Prong Two: Practical Application?
The additional elements are generic computer components ( claims 12, 16: "at least one memory" and "at least one processor"), data gathering ( obtaining datasets/captions; receiving a text description), and data output. The memory and processor amount to mere instructions to apply the exception on generic components (MPEP 2106.05(f)) . The data gathering is insignificant extra-solution activity (MPEP 2106.05(g)).
Consideration of Improvement: The specification describes improvements including more efficient captioning, computational resource savings, and improved image generation (¶¶ 7, 10-12, 66, 88). However, the claims do not reflect these improvements:
Claim 1 now recites training an image generation model with the recaptioned dataset and generating an image. As explained in the Response to Arguments, this added training and generic image output do not reflect any improvement recited in the claim and do not integrate the exception; they are the generic output of the abstract process (MPEP 2106.0S(f), (g)).
Claim 12 now recites training an image generation model with the captioned dataset and generating an image. For the reasons stated for claim 1, this generic training and output does not integrate the exception.
Claim 16 now recites training an image generation model and generating an image based on the upsampled text description. For the reasons stated for claim 1, this generic training and output does not integrate the exception.
(Step 2A, Prong Two: NO)
The claims are directed to an abstract idea. (Step 2A: YES)
Step 2B: Significantly More?
The generic computer components, data gathering, and data output are well-understood, routine, and conventional (MPEP 2106.05(d)(II)). The compressed-representation training is the application of well-understood, routine, and conventional image-embedding techniques, as evidenced by Spec ¶ 51 ("numerical representation of an image, such as a vector") and ¶ 46 (conventional public datasets), and generating an image is the conventional output of a trained generation model. The claims do not include additional elements sufficient to amount to significantly more. (Step 2B: NO)
Claims 1, 12, and 16 are not eligible.
Claims 2 and 13
Claim 2 adds updating captions using the model; claim 13 adds that the image dataset is a subset of the text-to-image dataset. These further describe data manipulation or data relationships. Neither adds a practical application nor significantly more. Not eligible.
Claims 3, 6, and 17
These further describe the training/tuning process (mathematical concepts) and data organization. Obtaining captions is data gathering; updating the model is part of the mathematical training process; specifying dataset relationships is data characterization. None adds a practical application or significantly more. Not eligible.
Claims 4, 14, and 18
These describe characteristics of the model output or training data (short captions describing main subjects). Describing what type of captions the model generates further characterizes the mathematical process and the mental process of describing an image. None adds significantly more. Not eligible.
Claims 5 and 8
Claim 5 elaborates on the second tuning stage with data gathering and mathematical processes; claim 8 specifies training data generated by another machine learning model. Neither adds a practical application or significantly more. Not eligible.
Claims 7 and 19
These describe characteristics of the model output ( descriptive synthetic captions with specific content types) and the mental process of describing an image's surroundings, background, and coloration. Neither adds significantly more. Not eligible.
Claim 9
Image embeddings are mathematical representations of images. The specification at ¶ 51 describes image embeddings as "numerical representation of an image, such as a vector"
generated through neural networks. Augmenting the model with embeddings adds mathematical concepts without adding a practical application or significantly more. Not eligible.
Claims 10 and 15
Claim 10 is cancelled. Claim 15 adds training a text-to-image machine learning model with the captioned dataset. Training another model is itself a mathematical process without a recited technical outcome. Claim 15 is not eligible.
Claims 11 and 20
Claim 11 adds applying an LLM to transform text; claim 20 describes increased text length. Applying a language model to expand text is executing learned mathematical transformations; describing increased length characterizes the mathematical output. Neither adds significantly more. Not eligible.
Claims 21 and 22
Claim 21 adds generating the compressed images by providing a compressed representation space, appending classification information identifying a main subject, background, color, or theme, and generating a synthetic caption based on that information. These recite further mathematical processes (generating an image embedding) and data characterization (appending classification metadata, which is also the mental act of noting an image's attributes). Claim 21 adds no practical application and no significantly more, and is not eligible.
Claim 22 adds caption blending by selection probability, a multimodal language model in a shared embedding space, joint CLIP and likelihood pre-training, training using reinforcement learning, text-image alignment feedback, or reward-model comparisons, and a higher CLIP score. Each is a further mathematical process or a characterization of a mathematical output. For the reasons stated for the recited CLIP-score result in the Response to Arguments, the higher CLIP-score limitation recites a result and does not integrate the exception. Claim 22 is not eligible.
Conclusion
Claims 1-9 and 11-22 are rejected under 35 U.S.C. § 101 as being directed to abstract ideas without integration into a practical application and without providing significantly more.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-9 and 11 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Betker et al. (Improving Image Generation with Better Captions – hereinafter “Betker”).
Claim 1.
Betker discloses a method for enhancing a training dataset for a machine learning model (Abstract discloses “training a bespoke image captioner and use it to recaption the training dataset. We then train several text-to-image models and find that training on these synthetic captions reliably improves prompt following ability.”), the method comprising:
obtaining a text-to-image dataset comprising one or more digital image-caption pairs (p. 5, ¶1 discloses “a large quantity of pairings (t, i) where i is an image and t is text that describes that
image”); and
generating a recaptioned dataset by applying an image captioner model to images in the text-to-image dataset (p. 5, § 2 Data Recaptioning), the image captioner model trained with compressed images corresponding to an image dataset (Betker: "we need a compressed representation space. Conveniently, CLIP provides just this. Thus, given a pre-trained CLIP image embedding function F(i), we augment our language model objective" (§2.1). This teaches the captioner is trained on a CLIP-compressed image representation.), a first tuning stage (p. 6, § 2.1.1 Fine-tuning the captioner, ¶1 discloses “In our first attempt, we build a small dataset of captions that describe only the main subject of the image”; Fig. 3, short synthetic captions (SSC)), and a second tuning stage (p. 6, § 2.1.1 Fine-tuning the captioner, ¶2 discloses “We repeat this process a second time, creating a dataset of long, highly-descriptive captions describing the contents of each image in our fine-tuning dataset”; Fig. 3, descriptive synthetic captions (DSC));
training an image generation model with the recaptioned dataset (Betker: "we train DALL-E 3
... we use a mixture of 95% synthetic captions and 5% ground truth captions" (§4).); and
generating, with the image generation model, an image (Betker: DALL-E 3 generates images
from captions (§4; Fig. 6).).
Claim 2.
Betker discloses the method of claim 1, wherein generating the recaptioned dataset comprises updating one or more captions in the text-to-image dataset (Betker p. 6, Fig. 3, short synthetic captions (SSC)) using the image captioner model (Betker p. 5, §2.1, ¶1 discloses “An image captioner is very similar to a traditional language model that predicts text.”).
Claim 3.
Betker discloses the method of claim 1, wherein the first tuning stage comprises: obtaining a first set of captions corresponding to at least a first subset of the image dataset (Betker p. 6, § 2.1.1 Fine-tuning the captioner, ¶1 discloses “In our first attempt, we build a small dataset of captions that describe only the main subject of the image.”); and updating, based on the first set of captions, the image captioner model (Betker p. 6, § 2.1.1, “We then continue to train our captioner on this dataset”).
Claim 4.
Betker discloses the method of claim 3, wherein: the image captioner model is configured to generate short synthetic captions (p. 6, § 2.1.1: “We refer to captions generated by this
fine-tune as ‘short synthetic captions’.”; Fig. 3 shows SSC examples), and the first set of captions describe a main subject of an image in the image dataset (p. 6, § 2.1.1: “captions that describe only the main subject of the image”).
Claim 5.
Betker discloses the method of claim 3, wherein the second tuning stage comprises:
obtaining a second set of captions corresponding to at least a second subset of the image dataset (p. 6, § 2.1.1: “creating a dataset of long, highly-descriptive captions describing the
contents of each image in our fine-tuning dataset” i.e. second subset), wherein captions of the second set of captions have a length that is longer than captions of the first set of captions (Fig. 3: Shows Descriptive Synthetic Captions (DSC) are longer than Short Synthetic Captions (SSC)); and updating, based on the second set of captions, the image captioner model (p. 6, § 2.1.1: “We again fine-tune our base captioner on this dataset.”).
Claim 6.
Betker discloses the method of claim 5, wherein: the image dataset is a subset of the text-to-image dataset (p. 6, § 2.1.1 Fine-tuning the captioner, ¶1 discloses “In our first attempt, we build a small dataset” – small dataset i.e. subset); and the first subset and the second subset are inclusive of each other (inclusive of each other – could potentially be the same set of images (spec ¶79), or one set could contain the other; p. 6, § 2.1.1 and Fig. 3 discloses using the same images for the SSC and DSC).
Claim 7.
Betker discloses the method of claim 5, wherein: the image captioner model is configured to generate descriptive synthetic captions (p. 6, § 2.1.1: “We refer to captions generated by this
captioner as ‘descriptive synthetic captions’.”; Fig. 3 shows DSC examples), and the second set of captions describe the main subject plus at least one of surroundings, background, image text, style, or coloration of an image in the image dataset (p. 6, § 2.1.1: “These captions describe not only the main subject of the image, but also its surroundings, background, text found in the image, styles, coloration, etc.”).
Claim 8.
Betker discloses the method of claim 5, wherein at least one of the first set of captions or the second set of captions are generated with a machine learning model (p. 9, § 3.5, ¶3: “we found that GPT-4 will readily ‘upsample’ any caption into a highly descriptive one.”).
Claim 9.
Betker discloses the method of claim 1, further comprising augmenting the image captioner model with an image embedding, the image embedding corresponding to a compressed representation space (p. 5, §2.1: discloses the need for a compressed representation space and using a pre-trained CLIP image embedding function F(i) to augment the language model).
Claim 11.
Betker discloses the method of claim 1, further comprising upsampling a caption in the recaptioned dataset using a large language model (p. 9, §3.5: “utilizing a LLM to 'upsample' captions”; p. 17, Appendix C shows a prompt used with GPT-4 for upsampling).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 12-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Betker et al. (Improving Image Generation with Better Captions – hereinafter “Betker”) in view of Zhao et al. (US 20220012544 A1 – hereinafter “Zhao”).
Claim 12.
Betker discloses a system comprising:
generating an image captioner model configured to generate captions from input images, the image captioner model trained using a text-to-image dataset (p. 5, §2.1), wherein the text-to-image dataset comprises one or more digital compressed image-caption pairs (Betker: captioner conditioned on a CLIP-compressed image embedding F(i) (§2.1).);
performing a first tuning stage for the image captioner model, the first tuning stage comprising: training the image captioner model using a first set of captions corresponding to at least a first subset of an image dataset (p. 6, § 2.1.1 Fine-tuning the captioner, ¶1 discloses “In our first attempt, we build a small dataset of captions that describe only the main subject of the image” – small dataset i.e. subset; Fig. 3, short synthetic captions (SSC));
obtaining a set of synthetic captions (p. 6, §2.1.1: “We refer to captions generated by this
fine-tune as ‘short synthetic captions’.”; Fig. 3 shows SSC examples);
after the first tuning stage, performing a second tuning stage for the trained image captioner model (p. 6, § 2.1.1 Fine-tuning the captioner, ¶2 discloses “We repeat this process a second time, creating a dataset of long, highly-descriptive captions describing the contents of each image in our fine-tuning dataset”), the second tuning stage comprising:
training the image captioner using the set of synthetic captions (p. 6, § 2.1.1: “We then continue to train our captioner on this dataset.”; where the output of the first stage are referred to as synthetic captions (SSC)); and
generating a captioned dataset by applying the tuned image captioner model to images in a dataset (p. 6, § 2.1.1: “Once built, we apply our image captioner fine-tunes to every image in our text-to-image dataset, resulting in a set of synthetic captions which we use for subsequent experiments.”);
training an image generation model with the captioned dataset; and generating, with the image generation model, an image (Betker: §4; Fig. 6.).
Betker discloses all of the subject matter as described above except for specifically teaching “at least one memory storing instructions; at least one processor configured to execute the instructions to perform operations, the operations comprising.” However, Zhao in the same field of endeavor teaches at least one memory storing instructions (¶¶32-33 and Fig. 4 discloses a memory 402 storing instructions); at least one processor configured to execute the instructions to perform operations, the operations comprising (¶¶32-33, 40 and Fig. 4 discloses a processor 401 that executes instructions).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify Betker to include Zhao because such a modification is the result of combining prior art elements according to known methods to yield predictable results. More specifically, Betker as modified by Zhao can yield a predictable result of using standard computer hardware to implement Betker’s machine learning method. Thus, a person of ordinary skill would have appreciated including in Betker’s method for improving text-to-image datasets using a tuned captioner with the ability to do computer-implemented methods for augmenting caption datasets and training captioning models since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the would yield the predictable result of a computer-implemented system.
Claim 13.
The combination of Betker and Zhao discloses the system of claim 12, wherein the image dataset is a subset of the text-to-image dataset (Betker p. 6, § 2.1.1 Fine-tuning the captioner, ¶1 discloses “In our first attempt, we build a small dataset of captions that describe only the main subject of the image” – small dataset i.e. subset; Fig. 3, short synthetic captions (SSC); p. 7, §3.2: “All models were trained to 500,000 training steps at a batch size of 2048, corresponding to 1B training images total (emphasis added).”).
Claim 14.
The combination of Betker and Zhao discloses the system of claim 12, wherein the first set of captions comprises short captions, the short captions describing a main subject of an image in the image dataset (Betker p. 6, § 2.1.1 Fine-tuning the captioner, ¶1 discloses “In our first attempt, we build a small dataset of captions that describe only the main subject of the image.”).
Claim 15.
The combination of Betker and Zhao discloses the system of claim 12, further comprising training a text-to-image machine learning model with the captioned dataset (Betker p. 6, § 2.1.1, “We then continue to train our captioner on this dataset”).
Claim 16.
Betker discloses a system comprising:
receiving a text description corresponding to an image (p. 6, § 2.1.1 Fine-tuning the captioner, ¶3 discloses “ground truth … captions”; Fig. 3 discloses “alt-text accompanying selected images scraped from the internet”);
upsampling the text description with a language model (p. 9, §3.5: “utilizing a LLM to 'upsample' captions”; p. 17, Appendix C shows a prompt used with GPT-4 for upsampling); and
training an image generation model (p. 9, § 3.5), the image generation model trained with a dataset comprising image-caption pairs, wherein at least a portion of captions are generated with an image captioner model (p. 6, § 2.1.1 ) that is trained on compressed image data (Betker: captioner trained on a CLIP-compressed image embedding (§2.1); image generation model trained on recaptioned data (§4).); and
generating, with the image generation model, an image based on the upsampled text description (Betker: "GPT-4 will readily 'upsample' any caption into a highly descriptive one" (§3.5); image generation (§4; Fig. 6).).
Betker discloses all of the subject matter as described above except for specifically teaching “at least one memory storing instructions; at least one processor configured to execute the instructions to perform operations, the operations comprising.” However, Zhao in the same field of endeavor teaches at least one memory storing instructions (¶¶32-33 and Fig. 4 discloses a memory 402 storing instructions); at least one processor configured to execute the instructions to perform operations, the operations comprising (¶¶32-33, 40 and Fig. 4 discloses a processor 401 that executes instructions).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify Betker to include Zhao because such a modification is the result of combining prior art elements according to known methods to yield predictable results. More specifically, Betker as modified by Zhao can yield a predictable result of using standard computer hardware to implement Betker’s machine learning method. Thus, a person of ordinary skill would have appreciated including in Betker’s method for improving text-to-image datasets using a tuned captioner with the ability to do computer-implemented methods for augmenting caption datasets and training captioning models since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Claim 17.
The combination of Betker and Zhao discloses the system of claim 16, wherein the image captioner model is trained with a first tuning stage and a second tuning stage (Betker p. 6, § 2.1.1).
Claim 18.
The combination of Betker and Zhao discloses the system of claim 16, wherein the image captioner model is configured to generate short synthetic captions (Betker p. 6, § 2.1.1; Fig. 3, SSC).
Claim 19.
The combination of Betker and Zhao discloses the system of claim 16, wherein the image captioner model is configured to generate descriptive synthetic captions (Betker p. 6, § 2.1.1: “We refer to captions generated by this fine-tune as ‘short synthetic captions’.”; Fig. 3 shows SSC examples).
Claim 20.
The combination of Betker and Zhao discloses the system of claim 16, wherein the upsampling increases the length of the text description (Betker Fig. 3: Shows Descriptive Synthetic Captions (DSC) are longer than Short Synthetic Captions (SSC)).
Claims 21-22 are rejected under 35 U.S.C. 103 as unpatentable over Betker in view of Cho, US 2023/0153522 A1 (hereinafter "Cho").
Claim 21.
Betker discloses generating the compressed images corresponding to the image dataset by
providing a compressed representation space (Betker: "we need a compressed representation
space. Conveniently, CLIP provides just this" (§2.1).); wherein generating the recaptioned
dataset further comprises
(Betker: descriptive synthetic captions "describe not only the main subject of the image, but also its surroundings, background, text found in the image, styles, coloration" (§2.1.1).); and generating a synthetic caption for the image based at least in part on the appended classification information (Betker: §2.1.1, generating descriptive synthetic captions.).
Betker teaches a captioner that describes the main subject, background, and coloration of the image but does not specifically teach "appending classification information .. . metadata
identifying ... a main subject, a background element, a color, or a theme." However, Cho, in the same field of training image captioning models, teaches selecting an attribute-specific caption identifying a color or background of the image (Cho: "training component 410 selects an attribute-specific caption as a positive training sample. For example, a specific attribute can be a particular color or background, and an attribute-specific caption can include words relating to the particular color or background" (¶ 63).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to append Cho's attribute-specific classification information (e.g., color, background) to the images of Betker and to generate Betker's descriptive synthetic caption based on that information. This combines prior-art elements according to known methods to yield the predictable result of a caption that reliably reflects the identified attributes. The motivation would have been to bias the captioner toward including the identified color and background attributes, improving the completeness of the descriptive captions Betker uses to recaption the dataset.
Claim 22.
Betker discloses generating the recaptioned dataset by adding synthetic captions while retaining ground-truth captions (Betker: "we opted to blend synthetic captions with ground truth captions" (§3.1).); selecting between a ground-truth caption and a synthetic caption according to a predetermined selection probability, configured to reduce overfitting (Betker: "Likelihood models like our text-to-image diffusion models have a notorious tendency to overfit"; "Blending happens at data sampling time, where we randomly select either the ground truth or synthetic caption with a fixed percent chance" (§3.1).); the captioner comprising a multimodal language model in a shared embedding space, jointly pre-trained with a CLIP model and trained by optimizing a likelihood objective augmented with a CLIP embedding (Betker: "we jointly pre-train our captioner with a CLIP and a language modeling objective" using L(t,i) augmented with F(i) (§2.1, Eg. 2).); and a higher CLIP score than a model trained using only ground-truth captions (Betker: "both models trained on synthetic captions achieve slightly better CLIP score performance than the baseline model", the baseline being
trained on ground-truth captions (§3.3; Fig. 4).); and
Betker does not specifically teach "the image captioner model is further trained using at least two of reinforcement learning, text-image alignment feedback, or reward-model comparisons of captions." However, Cho teaches training a captioning network using reinforcement learning and a text-image alignment reward (Cho: "training component 410 computes a reward function based on an encoded training caption and an encoded training image"; "the parameters of image captioning network 425 are updated based on a reinforcement learning model with a self-critical baseline" (¶ 62).) Cho further teaches a reward-model comparison distinct from the image-alignment reward (Cho: "training component 410 can compute an augmented reward function as the sum of the reward function R (I, c) and the grammar score g(c)" (¶ 77).).
Cho thus teaches all three recited options, satisfying "at least two": reinforcement
learning, text-image alignment feedback. and reward-model comparisons. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further train Betker's captioner using Cho's reinforcement learning
and reward-based training, to improve the faithfulness of the generated captions to the
image, thereby improving the quality of the recaptioned dataset Betker uses to train the image
generation model.
Alternative Claim Rejections - 35 USC § 103
Claims 1-9 and 11 are rejected under 35 U.S.C. 103 as unpatentable over Segalis, "A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation," (hereinafter "Segalis") in view of Radford et al., "Learning Transferable Visual Models from Natural Language Supervision") hereinafter "Radford".
Claim 1.
Segalis discloses a method for enhancing a training dataset for a machine learning model
(Segalis: "by relabeling the corpus with a specialized automatic captioning model and
training a text-to-image model on the recaptioned dataset, the model benefits substantially
across the board" (Abstract).), comprising: obtaining a text-to-image dataset comprising one or
more digital image-caption pairs (Segalis: a subset of LAION (image, caption) pairs (§4.1).);
generating a recaptioned dataset by applying an image captioner model to images in the text-to-image dataset (Segalis: "we use this fine-tuned model to recaption the images in the training dataset of a text-to-image model" (Fig. 2).); the image captioner model trained with
(Segalis: captioner fine-tuned on a first, short caption type and a second, long caption
type, "RECAP Short" and "RECAP Long" conditioned on "a different fixed conditioning
prefix for the short vs. long captions" (§4.2).); training an image generation model with the
recaptioned dataset (Segalis: "train an image generation model with the recaptioned
dataset" (Fig. 2; §4.3).); and generating, with the image generation model, an image (Segalis:
Fig. 1; §5.).
Segalis teaches fine-tuning the captioner on a first (short) caption type and a second (long) caption type, but does not specifically teach that the captioner is "trained with compressed images." Under the broadest reasonable interpretation, consistent with the specification, the recited compressed images are CLIP image embeddings: the specification states that image embeddings may be "obtained from pre-trained models, such as Contrastive Language-Image Pretraining (CLIP)", that "CLIP may provide text and image embeddings mapped to a common representation space", and that "the image embeddings may correspond to a compressed representation space" (Spec ¶ 51 ). Radford is that CLIP reference and teaches the image embedding (Radford: "CLIP jointly trains an image encoder and a text encoder" and uses "a linear projection to map from each encoder's representation to the multi-modal embedding space" (Approach § 2.2; Fig. 1). Because the specification equates the compressed
representation with the CLIP embedding, Radford's CLIP image embedding is the recited compressed image representation.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to train Segalis's captioner on a compressed image
embedding, as taught by Radford. The motivation would have been to condition the captioner efficiently on a compact image representation rather than raw pixels (Radford), and to bias the captioner first toward concise main-subject descriptions and then toward detailed descriptions, improving sample efficiency as Segalis teaches.
Claims 2-9 and 11.
The combination discloses claim 2 (updating captions using the captioner) (Segalis: §4, Fig. 2.);
claim 3 (first tuning stage) and claim 5 (second tuning stage with longer captions) (Segalis:
RECAP Short and RECAP Long (§4.2); Aggarwal: Abstract.); claim 4 (short captions
describing the main subject) (Segalis: "a single short sentence" (§4.2).); claim 6 (image dataset
a subset; subsets inclusive) (Segalis: §4.1-4.2.); claim 7 (descriptive captions describing
surroundings, background, style, or coloration) (Segalis: RECAP Long examples (Fig. 4).);
claim 8 (captions generated with a machine learning model) (Segalis: PaLI captioner (§4.2).); claim 9 (augmenting the captioner with an image embedding corresponding to a
compressed representation space) (Radford: image embedding in a multi-modal embedding space (Approach); Segalis: CLIP encoding (§3).); and claim 11 (upsampling a caption using a large language model) (Segalis: "leverage large pre-trained language components" (§2)).
Claims 12-21 are rejected under 35 U.S.C. 103 as unpatentable over Segalis in view of Radford, and further in view of Aggarwal, US 2023/0315988 A1 (hereinafter "Aggarwal").
Claims 12-20.
Segalis in view of Radford and Aggarwal discloses the captioner-generation, first and second tuning stages, recaptioning, image-generation-model training, and image generation recited in claims 12 and 16, for the reasons given for claim 1 above, with the captioner being a multimodal model trained on a compressed image representation (image embeddings) (Segalis: §4, Fig. 2; Radford: image embedding in a multi-modal embedding space (Approach); Aggarwal: Abstract.). Segalis further teaches upsampling-type expansion of captions (Segalis: RECAP Long detailed captions (§4.2; Fig. 4).). The combination does not specifically teach at least one memory storing instructions; at least one processor configured to execute the instructions.
However, Aggarwal teaches memory storing instructions executed by a processor (Aggarwal: "a data storage device ( e.g., at least one memory) storing processor-readable instructions; and at least one processor configured to execute the instructions"(¶ 8).).
It would have been obvious to implement the method of Segalis on the standard computer system of Aggarwal, in the same caption-dataset field, to obtain a computer-implemented system, with the same reasoning as claim 12 above. Dependent claims
13-15 and 17-20 are met as set forth for the corresponding subject matter in claims 2-9 above
(caption types, tuning stages, dataset relationships, and caption length).
Claim 21.
Segalis discloses generating the compressed images by providing a compressed representation
space and appending classification information identifying a main subject, background, color, or
theme (Segalis: CLIP representation (§3); RECAP Long captions describe main subject,
background, and color, e.g. "a combination of black, grey, and green colors. it is placed on
a black background" (Fig. 4).), and generating a synthetic caption based on the appended
classification information (Segalis: §4.2). It would have been obvious, for the reasons stated for claim 1, to represent the main-subject, background, and color content of Segalis's descriptive captions as appended classification metadata, to retain that information for conditioning the captioner.
15. Claim 22 is rejected under 35 U.S.C. 103 as unpatentable over Segalis in view of Radford and Aggarwal, further in view of Cho.
Claim 22.
Segalis discloses generating the recaptioned dataset by adding synthetic captions while retaining ground-truth captions (Segalis: mixing the synthetic RECAP with the original “Alttext” (ground truth) captions (§7).);
selecting, for a digital image during training, between a ground-truth caption and a corresponding synthetic caption according to a predetermined selection probability, the selection being configured to reduce overfitting (Segalis: "we ... used a 50%-50% mixture of RECAP Short and RECAP Long captions" and discusses "different mixtures of the three caption types we have (RECAP Short, RECAP Long, and Alttext)", where Alttext is the ground-truth caption (§4.3; §7).);
and
Segalis does not specifically teach the recited captioner architecture and training. However, Cho teaches an image captioner that is a multimodal language model trained with a likelihood objective augmented with a multimodal image-text embedding and further trained with reinforcement learning and reward comparisons (Cho: “train an image-conditioned language model on an image-caption dataset” that is “trained by maximizing likelihood over ground truth captions” (¶ 4); “the image captioning network is trained by encoding training images and training captions … in a same embedding space using a multi-modal encoder, and by comparing the encoded training images and the encoded training captions” (¶ 8); the captioner is updated base on “a reinforcement learning model with a self-critical baseline” (¶ 62) and an “training component 410 can compute an augmented reward function as the sum of the reward function R (I, c) and the grammar score g(c)” (¶ 77).). Radford teaches that the multimodal model is a CLIP model whose image and text embeddings are mapped to a common representation space (Radford: "CLIP jointly trains an image encoder and a text encoder" and maps "each encoder's representation to the multi-modal embedding space" (Approach § 2.2; Fig. 1).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to train Segalis’s captioner as a CLIP-based multimodal language model using Cho’s likelihood plus reward training, as taught by Cho and Radford. The motivation would have been to improve the captioner’s faithfulness to the image via the image-text alignment reward (Cho) and to condition the captioner on the common CLIP representation space (Radford), thereby improving the quality of the recaptioned training data and the resulting image generation model.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ross Varndell whose telephone number is (571)270-1922. The examiner can normally be reached M-F, 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’Neal Mistry can be reached at (313)446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ross Varndell/Primary Examiner, Art Unit 2674
Prosecution Insights