DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
In response to the Final Office Action from 7/27/2026, Applicant has filed a Request for Continued Examination (RCE) on 9/15/2026. In this reply, Applicant has amended independent claims 1, 8, and 14 to add limitations regarding based on determining that an amount of training data comprising images and corresponding text in a second language is less than an amount of training data comprising images and corresponding text in the first language, generating an augmented second language batch of text by augmenting a second-language batch of text in the second language by translating a set of text in the first language to generate translated text in the second language; and fine tuning the multilingual large language model by utilizing the augmented second language batch of text to generate text embeddings corresponding to the augmented second language batch of text.
Applicant has also argued that the prior art of record fails to teach the limitations added via the instant amendment (Remarks, Pages 17-25). These arguments have been fully considered, however, are moot with respect to the new grounds of rejection further in view of Vu, et al. (U.S. PG Publication: 2024/0135116 A1).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5, 7, 10, 12-13, 17, and 19-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 7 and 19 recite the terms "a second-language batch of text," "a second language," "a set of text in the first language," and "translated text in the second language" when these terms were already introduced into respective amended independent claims 1 and 14. It is thus unclear whether these instances of claim language refer back to the terms introduced in their respective parent claims or represent some secondary instances of the terms. For claim construction purposes in the interest of compact prosecution, each of these terms will be construed as being preceded by the definite article "the". Amended dependent Claim 12 corrects all these issues except for "translated text in the second language," and thus, this claim has also been rejected under 35 U.S.C. 112(b) for providing unclear antecedent basis.
Claims 5 and 17 recites two instances of the terms "the translated text" that creates an issue of unclear antecedent basis given the amendments to respective parent claims 1 and 14. Specifically, the independent parent claims already recite "translated text in the second language" while claims 5 and 17 earlier include translated text in a second language from supplemental text in the first language." Thus, it is unclear whether the two instances of "the translated text" refers back to the translated text of the parent claims of the translation based upon the supplemental text from earlier in claims 5 and 17. For claim construction purposes, "the translated text" will be construed as referring back to the claims 5 and 17 instances respectively. These claims also contain a first instance of "a second language" when this term was already introduced in the respective parent claims. It is thus unclear whether this term should refer back to the term of the parent claims or represents some additional instance. For claim construction purposes, "a second language" will be construed as --the second language--. The latter issue is also present in claim 10 though not with regards to the translated text since claim 12 more specifically deals with translated caption text not present in parent claim 8.
Claims 13 and 20 further limit and inherits the indefinite claim language of respective parent claims 12 and 19, and thus, have also been rejected under 35 U.S.C. 112(b) by virtue of their dependency.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 8-9, and 14-16 are rejected under 35 U.S.C. 103 as being unpatentable over Chen, et al. (U.S. PG Publication: 2024/0153239 A1) in view of Vu, et al. (U.S. PG Publication: 2024/0135116 A1).
With respect to Claim 1, Chen discloses:
A computer-implemented method comprising:
training a multilingual large language model to embed text into an embedding space of a vision language model, the vision language model comprising a text encoder for a first language and a vision encoder (training of a "transformer model" having an encoder that associates natural language processing text in a first language (e.g., English) with image objects in a computer vision application wherein text embeddings (225) are placed into a shared embedding space (232) with an vision/object embedding (215) wherein the input to such a model space include a text encoder and a vision/object encoder, Paragraphs 0024-0027, 0033 and 0052-0053; Fig. 2A; note that the "transformer model" that handles natural language inputs in a deep learning context constitutes a large language model), by:
determining pairings between images and text captions corresponding to the images, the text captions being in languages other than the first language (determining an image and "associated text" in a pairing, Paragraphs 0023 and 0031; see "Assoc." in Fig. 2A; the text may be in a language other than a first language such as English, Paragraph 0025; see text as captions described at Paragraph 0015);
generating, utilizing the vision encoder, image embeddings for the images (generating "object embeddings 215" that are "derived from...the image," Paragraph 0024; Fig. 2A., Element 210);
generating, utilizing the multilingual large language model, text embeddings for the text captions, the text captions being in languages other than the first language (multilingual transformer generates "text embeddings" for multilingual text inputs, Paragraphs 0015 and 0025; Fig. 2, Element 220);
determining similarity metrics between the image embeddings for the images and the text embeddings for the text captions (after the embeddings (i.e., object and text) are projected into the shared space, a "similarity measure" between such embeddings are determined, Paragraphs 0026 and 0033); and
adjusting parameters of the multilingual large language model to reduce an output of a contrastive loss function based on the similarity metrics without adjusting parameters of the vision encoder adjusting parameters of the multilingual large language model to reduce an output of a contrastive loss function based on the similarity metrics without adjusting parameters of the vision encoder (weight parameters for the transformer models are trained based upon a loss function so that text embeddings are closer to "positive object embeddings than to negative object embeddings" wherein the use of loss in relation to positive and negative examples constitutes a contrastive loss function, Paragraphs 0017, 0023, 0026, 0033, and 0060; specifically note the discussion at paragraph 0033 where it is described that the training pertains to the text transformer encoder and that the text embeddings are guided in the shared space to be closer to the appropriate object embeddings- accordingly, the object detector/encoder weights are not adjusted in the similarity-based training operation).
Although Chen teaches a transformer model having an encoder that associates natural language text in a first language with image objects in a computer vision application to generate text embeddings into a shared embedding space with vision object embeddings wherein the model is trained to learn multiple languages via machine translation using associated text and image training data (see paragraphs 0017 and 0030-0033). Chen does not teach that based on determining that an amount of training data comprising images and corresponding text in a second language is less than an amount of training data in the first language, generating an augmented second language batch of text by augmenting a second-language batch of text in the second language by translating a set of text in the first language to generate translated text in the second language; and fine tuning the multilingual large language model by utilizing the augmented second language batch of text. Vu, however, looks to an implied threshold number of training samples (see "fewer number of training samples" in paragraph 0040) in association with a "low-resource language" that requires augmentation (Paragraphs 0040 and 0042). This augmentation takes the form of generating "machine-translated training examples in the target language" (Paragraphs 0043, 0186-0190, and 0257). It is further noted that once generated, the augmented training data is used in model retraining/fine-tuning operations as discussed in Paragraph 0149 wherein text data in multiple languages is used to generate embeddings into the shared embedding space as noted in Chen that also discusses utilizing machine translation for different languages to generate the text embeddings as discussed in Paragraph 0025.
Chen and Vu are analogous art because they are from a similar field of endeavor in language model training utilizing multi-lingual data. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date to utilize the number of training samples to identify low resource language and generate machine translations via augmentation as taught by Vu in generating multilingual training samples for the transformer of Chen to provide a predictable result of cost-effective solution for an imbalance in training data that leads to model prediction inaccuracies (Vu, Paragraphs 0040-0041).
With respect to Claim 2, Chen further discloses:
The computer-implemented method of claim 1, further comprising combining the multilingual large language model with the vision encoder of the vision language model to create a multilingual vision language model for predicting text-image pairs (the NLP transformer model is combined with the object encoder and trained vector space to predict a most likely text image pair having a highest similarity score, Paragraph 0019-0020, 0023, and 0025-0026; Fig. 2A, Elements 210, 215, 220, and 225).
With respect to Claim 3, Chen further discloses:
The computer-implemented method of claim 2, further comprising processing a query text (Paragraph 0019- user text query returning database results), in a language other than the first language, through the multilingual vision language model to determine one or more digital images corresponding to the query text (handling second language (e.g., non-English) texts using the transformer/LLM to determine at least one image object of interest with a highest similarity, Paragraphs 0025-0026; Fig. 2A, Element 240).
With respect to Claim 4, Chen discloses:
determining the pairings between the images and the text comprises:
determining a first pairing between a first image and a first text caption in a language other than the first language (a plurality of "text embeddings" of text captions associated with images are projected into a shared space with a plurality of "image embeddings" wherein a "similarity measure" between a text embedding and its positive/first object is determined, Paragraphs 0026 and 0033; note also association between the text and image that leads to similarity determination in Paragraphs 0023 and 0031 and secondary language text in Paragraph 0025); and
determining a second pairing between the first image and a second text caption in the language other than the first language (similarity measure between another text embedding and the first object as a negative objection is determined, Paragraphs 0026 and 0033; note also association between the text and image that leads to similarity determination in Paragraphs 0023 and 0031 and secondary language text in Paragraph 0025);
determining the similarity metrics between the image embeddings and the text embeddings comprises: determining a first similarity metric for the first pairing (see above determination of a "similarity measure" or "similarity score" and the first pairing described above, Paragraphs 0026 and 0033); and
determining a second similarity metric for the second pairing (see above determination of a "similarity measure" or "similarity score" and the second pairing described above, Paragraphs 0026 and 0033); and
adjusting the parameters of the multilingual large language model comprises: adjusting the parameters of the multilingual large language model to increase the first similarity metric and to reduce the second similarity metric (see that the transformer model weight parameters are adjusted to move the positive pairing closer together/increase similarity and push apart negative examples/decrease similarity, Paragraphs 0026 and 0033).
Claim 8 is a system embodiment comprising one or more memory devices and one or more processors for carrying out the method of claim 1, and thus, is rejected under similar rationale. Chen also discloses method implementation using the transformer model as program instructions stored on in a memory along with a processor (Paragraphs 0062-0063). Note that claim 8 features an additional step/functional to the claim 1 method that is also addressed via the teachings of Chen:
process a query text in a language other than the first language through a combined model comprising the multilingual large language model and the vision encoder of the vision language model to determine one or more digital images corresponding to the query text (handling second language (e.g., non-English) text queries using the transformer/LLM to determine at least one image object of interest with a highest similarity, Paragraphs 0019, 0025-0026; Fig. 2A, Element 240).
Claim 9 contains subject matter similar to Claim 4 wherein the additional text caption association/similarity metric of claim 9 corresponds to the second text embedding similarity with the first image embedding as a negative example, and thus, is rejected under similar rationale.
Claim 14 is directed towards an embodiment of the invention in the form of a non-transitory computer-readable medium stored processor-executable instructions for carrying out the method of claim 1, and thus, is rejected under similar rationale. Furthermore, Chen teaches method implementation as a computer program stored on a non-transitory computer-readable medium (Paragraphs 0070 and 0079).
Claim 15 contains subject matter similar to claim 3 (incorporating the subject matter of claim 2 by virtue of its dependency), and thus, is rejected under similar rationale.
Claim 16 contains subject matter similar to Claim 4, and thus, is rejected under similar rationale. Moreover, the added limitations of "for a subsequent training iteration" is not an action positively recited as being performed when the program is executed by the processing device, and thus, are not patentably limiting though it should be noted that Chen describes a "repeated" training process at paragraph 0033.
Claims 5, 10, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Chen, et al. in view of Vu, et al. and further in view of Oktay, et al. (PG Publication: 2025/0173613 A1).
With respect to Claim 5, Chen in view of Vu teaches the training method for the vision language model that generates image predictions from an input text as applied to Claim 1. Chen in view of Vu, however, does not explicitly teach the fine-tuning process set forth in claim 5. Oktay, however discloses:
generating translated text in a second language from supplemental text in the first language (augmenting document supplemental text via translation into a second natural language, Paragraphs 0010 and 0073);
determining finetuning pairings between the translated text in the second language and finetuning images corresponding to the supplemental text in the first language (performing additional training of fine-tuning weights using between image embeddings and the second text passage embedding pairs, Paragraphs 0071, 0073, and 0077);
determining finetuning similarity metrics between finetuning image embeddings for the finetuning images and finetuning text embeddings for the translated text (distance/similarity metric determined for weight fine-tuning, Paragraphs 0071, 0073, 0075, and 0077); and
adjusting the parameters of the multilingual large language model to reduce the output of the contrastive loss function based on the finetuning similarity metrics without adjusting the parameters of the vision encoder (weight fine-tuning to minimize contrastive loss wherein only the text LLM is trained independently, Paragraphs 0063, 0071, 0073, 0077, and 0150 (mentioning a BERT, which is a specific type of LLM)).
Chen, Vu, and Oktay are analogous art because they are from a similar field of endeavor in multi-lingual prediction utilizing transformer models. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date to utilize the iterative training taught by Oktay in the associative text and image training taught by Chen in view of Vu to provide a predictable result of gradually reducing prediction errors over iterations thus improving model effectiveness (Oktay, Paragraph 0075).
Claim 10 contains subject matter similar to Claim 5, and thus, is rejected under similar rationale. Moreover, note that the text in Oktay is a description or a caption of the image (Paragraph 0051).
Claim 17 contains subject matter similar to Claim 5, and thus, is rejected under similar rationale.
Claims 6, 11, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Chen, et al. in view of Vu, et al. and further in view of Carlsson, et al. ("Cross-lingual and Multilingual CLIP," 2022).
With respect to Claim 6, Chen in view of Vu teaches the training method for the vision language model that generates image predictions from an input text as applied to Claim 1. Chen in view of Vu, however, does not teach the knowledge distillation training methodology that reduces an output of a mean-squared-error (MSE) loss function based on the text embeddings for the text and parallel text encodings of parallel text. Carlsson, however, discloses:
adjusting, utilizing knowledge distillation from the text encoder of the vision language model, the parameters of the multilingual large language model to reduce an output of a mean-squared-error loss function based on the text embeddings for the text captions and parallel text encodings of parallel text generated by the text encoder of the vision language model (creation of "language parallel data" via machine translation of a source text to generate matching embeddings, Section 3, Page 6849; then see knowledge distillation learning via teacher-student model learning that operates by "minimizing MSE" using parallel text encodings of "original texts" and "translated texts," Section 3.2, Page 6850; equation 1).
Chen, Vu, and Carlsson are analogous art because they are from a similar field of endeavor in multi-lingual prediction using transformer models. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date to utilize the teacher-student knowledge distillation learning taught by Carlsson in the associative text and image training taught by Chen in view of Vu to provide a predictable result of achieving a smaller and more efficient student model specialized for a second language (Carlsson, Section 2.3, Page 6849).
Claim 11 contains subject matter similar to Claim 6, and thus, is rejected under similar rationale. Moreover, note that the text in Carlsson is a "caption" of the image (Section 3.2, Page 6850).
Claim 18 contains subject matter similar to Claim 6, and thus, is rejected under similar rationale.
Claims 7, 12, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Chen, et al. in view of Vu, et al. and further in view of Chen, et al. ("AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities," 2023; hereinafter Chen2).
With respect to Claim 7, Chen in view of Vu teaches the training method for the vision language model that generates image predictions from an input text also utilizing data augmentation via machine translation as applied to Claim 1. Chen in view of Vu, however, does not teach the combination data augmentation and reduction of training datasets as set forth in claim 7. Chen2, however, discloses:
augmenting a second-language batch of text in a second language by translating a set of text in the first language to generate translated text in the second language (see parallel sentences with a first source language (English) including a second target language (Chinese) and a third target language (Italian) in Fig. 1, wherein the parallel text is generated by augmenting a source language via machine translation, Sections 4.1 and 5.1, Pages 8668-8669) (Applicant should note that this augmentation process is also taught by Vu as applied to Claim 1); and
reducing a third-language batch of text in a third language by omitting a subset of text of the third-language batch of text (note that the selection/collection data "for each language" uses "the same amount of data" implying that the training data for a language such as Italian or another language would have a reduced data selection to account for using "the same amount of data for each language, Section 4.1, Pages 8668-8669).
Chen, Vu, and Chen2 are analogous art because they are from a similar field of endeavor in multi-lingual prediction using language models. Thus, it would have been obvious to one of ordinary skill in the art to utilize Chen2’s data pre-processing of parallel text in the associative text and image training taught by Chen in view of Vu to provide a predictable result of adding additional training data for underrepresented language and ensuring that certain languages are not overtrained/training is balanced.
Claim 12 contains subject matter similar to Claim 7, and thus, is rejected under similar rationale.
Claim 19 contains subject matter similar to Claim 7, and thus, is rejected under similar rationale. Furthermore, note that the augmented text and reduced text is for text embeddings that are paired with images in Chen2 (Section 3.2, Page 8668).
With respect to Claim 20, Chen2 further discloses:
The non-transitory computer-readable medium of claim 19, wherein the operations further comprise: determining an augmentation metric for the second language and a reduction metric for the third language; augmenting the second-language batch of text based on the augmentation metric; and reducing the third-language batch of text based on the reduction metric ("the same amount of data" for each language is used wherein a language requiring machine translation augmentation would have a number of samples to increase whereas a language where only a subset is select would have a number of training samples to decrease/subset to not select wherein the operations of augmentation and subset selection leading to the same amount of data constitute the augmentation and reduction metrics, Section 4.1 and 5.1, Pages 8668-8669; three languages shown in Fig. 1).
Status of Claim not addressed by the prior art of record- Claim 13
With respect to Claim 13, the prior art of record taken individually or as a combination fails to explicitly teach or fairly suggest the system set forth in claim 13 combined with intervening claim 12 where the second language text that is than an amount of training data comprising images and corresponding text in the first data is augmented in accordance with a resampling ratio that is also used to determine a reduction ratio to reduce a third-language batch of text. While Chen, et al. (U.S. PG Publication: 2024/0153239 A1) teaches a transformer language model that is multi-lingual and trained using multiple languages to generate embeddings that are placed into a shared vector space with object embeddings for contrastive training (Paragraphs 0024-0027, 0033 and 0052-0053), Chen does not teach the identification of the amount-based need for training data augmentation set forth in parent claim 8 nor the resampling ratio used to carry out the augmentation. Vu, et al. (U.S. PG Publication: 2024/0135116 A1) resolves some of the deficiencies of Chen by teaching augmentation for language with a few training samples via augmentation utilizing machine translation (Paragraphs 0040 and 0042-0043), but does not utilize a resampling ratio to perform this augmentation that is also used to carry out reduction on the third language. While Chen, et al. ("AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities," 2023; hereinafter Chen2) talks about keeping “the same amount of data” may imply that machine translated data can increase one language data pool by a certain percentage (e.g., 10 to 11 samples via an additional machine translation represents a 10% augmentation ratio) or an elimination of 9 samples out of 20 for the same 11 samples represents a 45% reduction proportion for another language, Chen in view of Vu and further in view of Chen 2 does not specifically mention the use of such resampling and reduction metrics based on the resampling ratio for an augmentation process triggered based upon the amount-based need for training data augmentation set forth in parent claim 8. Lastly, while Su (U.S. PG Publication: 2020/0320289 A1) also a user to augment data a certain percentage and select a reduced percentage of files (Paragraphs 0075-0078), this percent/ratio is not based upon a need for agumentation triggered by the amount-based considerations set forth in parent claim 8. Thus, while the prior art when combined teaches portions of the claimed invention set forth in claim 13, they fail to reasonably combine to render the invention as a whole obvious. Thus, the prior art of record fails to explicitly teach of fairly suggest the invention set forth in claim 13. Please note that claim 13 has not been objected to as containing allowable subject matter because the claim is subject to a rejection under 35 U.S.C. 112(b) by virtue of its dependency.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Shou, et al. (U.S. PG Publication: 2026/0050741 A1)- teaches a number of training samples for a target language considered scarce and augmenting samples for fine-tuning using machine translation (Paragraph 0053).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAMES S WOZNIAK whose telephone number is (571)272-7632. The examiner can normally be reached 7-3, off alternate Fridays.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant may use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at (571)272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JAMES S. WOZNIAK
Primary Examiner
Art Unit 2655
/JAMES S WOZNIAK/Primary Examiner, Art Unit 2655
Prosecution Insights