DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This is in response to Applicant’s Arguments/Remarks filed on June 24th, 2026, which has
been entered and made of record.
Response to Arguments
Claim Rejections - 35 USC § 102/103
Applicant’s arguments regarding the current claim(s) have been fully considered. But, the
arguments/remarks are directed to the claims as amended, and so are believed to be answered
by and therefore moot in view of the new grounds of rejection presented below.
Status of Claims
Claims 1-6, 8-13, and 15-20 are pending. Claim(s) 1, 8, and 15 were amended. Claim(s) 7 and 14 were canceled. No new Claim(s) were added. Claims 1-6, 8-13, and 15-20.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3, 8, 10, 15, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Im (US 20240177507 A1) in view of Li (Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. https://arxiv.org/pdf/2301.12597) and Singh (US 20250316063 A1).
Regarding Claim 1, representative of Claim 8 and 15, Im teaches a method, comprising:
obtaining a first machine learning model, the first machine learning model([0056] S200 of controlling an image-to-text translation model so that the image-to-text translation model learns a function of extracting text related to the content of an image);
obtaining a set of training data, wherein the set of training data comprises at least one or more uncaptioned training examples that are of the high-dimensional data modality ([0060]: generating text from an image according to an example embodiment of the present disclosure may further include an operation S300 of generating image-text synthetic data for the image by combining image and text information, [0062]: generating text/image synthetic data so that a text-based image may be generated from an image-only dataset which has only images without text information);
generating a set of synthetic captions for the training data by applying the first machine learning model to the set of training data, wherein each synthetic caption describes a respective training example of the set of training data in the low-dimensional data modality ([0060]: generating text from an image according to an example embodiment of the present disclosure may further include an operation S300 of generating image-text synthetic data for the image by combining image and text information, [0062]: generating text/image synthetic data so that a text-based image may be generated from an image-only dataset which has only images without text information, see Fig. 5 caption describing the image);
for one or more iterations, training a second machine learning model, ([0066] Virtual synthetic data which describes the image may be generated for the image-only dataset without text information using the back-translation model (S300). The text-based image generation model may be trained using virtual text/image pair data generated as described above (S400)), by:
generating a set of estimations by applying a set of parameters of the second machine learning model to the synthetic captions for a batch of the training examples, wherein the set of estimations include reconstructions of the content of the high-dimensional data modality for the batch of training examples ([0089] E is an encoder, G is a decoder, Z is a codebook, and D is a discriminator. L.sub.VQ is a loss function related to codebook learning which is set so that loss is reduced when an image is reconstructed in an encoding and decoding process),
computing a loss for the batch of training examples based on the set of estimations, the loss representing the difference between the estimations and the batch of training examples ([0089] E is an encoder, G is a decoder, Z is a codebook, and D is a discriminator. L.sub.VQ is a loss function related to codebook learning which is set so that loss is reduced when an image is reconstructed in an encoding and decoding process), and
updating the set of parameters of the second model to reduce the loss ([0089]: Training may be performed to reduce the sum of the two loss functions).
Although Im teaches an image to text model and a text to image model, Im does not explicitly teach the first machine learning model comprising a visual encoder, a transformer model, and a large-language model (LLM),
wherein applying the first machine learning model to a training example comprises:
applying the visual encoder to the training example to generate an encoded image,
applying the transformer model to the encoded image to generate a prompt,
applying the LLM to the prompt to generate a synthetic caption for the training example,
wherein during a training process, parameters of the visual encoder and the LLM are frozen, and parameters of the transformer model are updated; and
wherein the second machine learning model is a stable diffusion model.
However, the first machine learning model is known as taught by Li teaching the first machine learning model comprising a visual encoder, a transformer model, and a large-language model (LLM) (see Fig. 1 including Image Encoder, Querying Transformer, and Large Language Model),
wherein applying the first machine learning model to a training example comprises:
applying the visual encoder to the training example to generate an encoded image ([Introduction, paragraph 4]: As shown in Figure 1, Q-Former is a lightweight trans former which employs a set of learnable query vectors to extract visual features from the frozen image encoder),
applying the transformer model to the encoded image to generate a prompt ([Introduction, paragraph 4]: As shown in Figure 1, Q-Former is a lightweight trans former which employs a set of learnable query vectors to extract visual features from the frozen image encoder. It acts as an information bottleneck between the frozen image encoder and the frozen LLM, [Section 3.3, paragraph 1]: As shown in Figure 3, we use a fully-connected (FC) layer to linearly project the output query embeddings Z into the same dimension as the text embedding of the LLM. The projected query embeddings are then prepended to the input text embeddings. They function as soft visual prompts that condition the LLM),
applying the LLM to the prompt to generate a synthetic caption for the training example ([Section 3.3, paragraph 2]: where the frozen LLM is tasked to generate the text conditioned on the visual representation from Q-Former),
wherein during a training process, parameters of the visual encoder and the LLM are frozen, and parameters of the transformer model are updated ([Section 3.1, paragraph 1]: Q-Former as the trainable module to bridge the gap between a frozen image encoder and a frozen LLM).
Li does not explicitly teach wherein the second machine learning model is a stable diffusion model. However, this model is taught by ---Singh teaching wherein the second machine learning model is a stable diffusion model ([0148]: The text-to-image model 35 may be any (pre-trained) text-to-image model. Possible text-to-image models that may be used include: [0149] 1. Stable Diffusion).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have modified the teachings of Im to include the teachings of Li, and further modify the combination of Im and Li to include the teachings of Singh. Im generally teaches an image to text generation model capable of synthesizing text data for training a text to image model. Referring to the image to text generation model as the first machine learning model and the text to image generation model as the second machine learning model, Im does not explicitly teach the particularities regarding these models recited in claim 1 as shown above. Li teaches an image to text generation model using a BLIP-2 model. It would have been obvious to substitute Im’s first machine learning model for Li’s BLIP-2 model to achieve the predictable result of text generated from an image. Further it would have been obvious to have substituted Im’s second machine learning model for Singh’s stable diffusion model to provide the predictable result of images generated from text.
Regarding Claim 3, representative of Claim 10 and 17, the Im, Li, and Singh combination teaches the method of claim 1. In addition, Im teaches wherein the first machine learning model is an image-to-text model ([0056]: S200 of controlling an image-to-text translation model, [0060]: S300 of generating image-text synthetic data) and the second machine learning model is a text-to-image model ([0066]: text-based image generation model may be trained using virtual text/image pair data generated as described above (S400)).
Claim(s) 2, 5-6, 9, 12-13, 16, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Im (US 20240177507 A1) in view of Li (Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. https://arxiv.org/pdf/2301.12597), Singh (US 20250316063 A1), and Karpman (US 11995803 B1).
Regarding Claim 2, representative of Claim 9 and 16, the Im, Li, and Singh combination (hereinafter referred to as the Im combination) teaches the method of claim 1. In addition, Im teaches a first and second machine learning model. However, none of the Im combination explicitly teaches the remaining limitations of Claim 2. Karpman teaches wherein the first machine learning model is trained on a different set of training data than the second machine learning model, wherein the different set of training data for the first machine learning model has a higher number of training examples than the set of training data for the second machine learning model ([0047]: the system can first execute the captioner module on the training corpus and then execute the filter module on the resulting training set in order to remove image-text pairs that include either misaligned alternate text or misaligned (e.g., inaccurate) synthetic captions generated by the captioner module, thereby increasing overall caption fidelity of the training corpus but yielding a (slightly) smaller set of training examples. Examiner notes the modification of Im with Karpman’s synthetic caption filter would reduce the resulting image/caption pairs generated for a training dataset).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have modified the teachings of the Im combination to include the teachings of Karpman by evaluating the generated synthetic captions, and filtering those determined to be misaligned/inaccurate. Doing so would improve the fidelity of the training data used in Im’s secondary processing of text to image generation.
Regarding Claim 5, representative of Claim 12 and 19, the Im combination teaches the method of claim 1. However, none of the Im combination explicitly teaches the remaining limitations of Claim 5. Karpman teaches wherein the set of training data comprises both captioned and uncaptioned training examples that are of the high-dimensional data ([0046]: system can automatically infer, generate, and/or bootstrap high-fidelity text captions for images (e.g., images without associated alternate text) retrieved by the web intelligence engine 108 and identify and correct noisy, incorrect and/or mismatched image-alternate-text pairs retrieved by the web intelligence engine 108 in order to create a large set of instructive text-image training examples. Examiner notes, captions are generated for images without alternate text (uncaptioned) and existing image-alternate text pairs may be corrected (captioned)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have modified the teachings of the Im combination to include the teachings of Karpman by substituting the dataset of uncaptioned images for a dataset that may include uncaptioned and captioned images. Doing so would provide the predictable result of a providing a training dataset.
Regarding Claim 6, representative of Claim 13 and 20, the Im combination teaches the method of claim 1. However, none of the Im combination explicitly teaches the remaining limitations of Claim 6. Karpman teaches further comprising: receiving a confidence level associated with each synthetic caption; and filtering the training data to exclude training examples where the confidence level of the corresponding synthetic caption does not exceed a threshold confidence level ([0047]: the system can first execute the captioner module on the training corpus and then execute the filter module on the resulting training set in order to remove image-text pairs that include either misaligned alternate text or misaligned (e.g. inaccurate) synthetic captions generated by the captioner module, thereby increasing overall caption fidelity).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have modified the teachings of the Im combination to include the teachings of Karpman by evaluating the generated synthetic captions, and filtering those determined to be misaligned/inaccurate. Doing so would improve the fidelity of the training data.
Claim(s) 4, 11, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Im (US 20240177507 A1), Li (Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. https://arxiv.org/pdf/2301.12597) and Singh (US 20250316063 A1), in view of Jayaswal (V. Jayaswal, S. Ji, Satyankar, V. Singh, Y. Singh and V. Tiwari, "Image Captioning Using VGG-16 Deep Learning Model," 2024 2nd International Conference on Disruptive Technologies (ICDT), Greater Noida, India, 2024, pp. 1428-1433, doi: 10.1109/ICDT61202.2024.10489470).
Regarding Claim 4, representative of Claim 11 and 18, the Im combination teaches the method of claim 3. However, none of the Im combination explicitly teaches the remaining limitations of Claim 4. Jayaswal teaches wherein the uncaptioned training examples are images and further comprising pre-processing the images to reduce resolutions of the images before applying the first machine learning model ([Section V. A.]: image captioning process involves preprocessing and data loading to convert unprocessed image and text data into a machine learning model. Preprocessing [17] involves cropping).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have modified the teachings of the Im combination to include the teachings of Jayaswal by including a preprocessing step of cropping images before captioning them. Doing so would reduce the amount of data needed to be processed, thereby improving processing speed.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JANICE VAZ whose telephone number is (703)756-4685. The examiner can normally be reached Monday-Friday 9:00-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JANICE E. VAZ/Examiner, Art Unit 2667 /MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
