DETAILED ACTION
This communication is in response to the Amendments and Arguments filed on 8/24/2026.
Claims 1-6, 8-14, and 16-20 are pending and have been examined.
All previous objections / rejections not mentioned in this Office Action have been withdrawn by the examiner.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments / Amendments
Regarding the Applicant’s arguments for the rejections under 35 U.S.C. § 101, examiner withdraws the rejection. Analyzing the independent claims as a whole, the limitations integrate the judicial exception into a practical application. The limitations enable LLMs in low-resource languages to be evaluated more accurately in a non-biased way by implementing a cross-lingual evaluation metric. Spec. P0017. The improvement is interpreted with the knowledge that English is often the reference language as there is a vast amount of training datasets available in English for LLM and the target language is in a different low-resource language with limited training datasets available. Spec. P0070, P0071.
Regarding the Applicant’s arguments for the rejections under 35 U.S.C. § 103, has amended the independent claims 1, 10, and 17 to include “generating a reference output embedding vector for the reference output and a target output embedding vector for the target output using one or more pre-trained cross-lingual encoders” and “determining a cross-lingual similarity between the target output embedding vector embedding vector a cross-lingual similarity score for the target output”. Hence, the Applicant’s arguments are moot in view of new grounds of rejection. Hence, new references have been applied.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-7, 10, 12-15, 17 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over "Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrations" by Ranaldi et al., hereinafter Ranaldi, in view of Foley et al. (U.S. PG Pub No. 20250028992), hereinafter Foley, and further view of Niu et al. (U.S. PG Pub No. 20230153542), hereinafter Niu.
Regarding claim 1, 10, and 17 Ranaldi teaches:
(Claim 1) A method comprising: (2.2 Instruction-tuning Paradigm, Instruction-tuning method.)
(Claim 10) A system comprising: a communications interface; a non-transitory computer-readable medium; and one or more processors communicatively coupled to the communications interface and the non-transitory computer-readable medium, the one or more processors configured to execute processor-executable instructions stored in the non- transitory computer-readable medium to: (4.3.2 Models Setup, Running our experiments on a workstation equipped with two Nvidia RTX A6000 with 48 GB of VRAM.)
(Claim 17) A non-transitory computer-readable medium comprising processor-executable instructions configured to cause one or more processors to: (4.3.2 Models Setup, Running our experiments on a workstation equipped with two Nvidia RTX A6000 with 48 GB of VRAM.)
accessing a pair of parallel inputs comprising a reference input in a reference language and a target input in a target language, wherein the reference input in the reference language corresponds to the target input in the target language, the target language different from the reference language; (Figure 1, Parallel inputs are fed into Stanford Alpaca LLM and x-CrossAlpaca LLM. The instruction and input are in English for Stanford Alpaca LLM and foreign language for x-CrossAlpaca LLM.)
executing a reference large language model for a generative task to obtain a reference output in the reference language based on the reference input; (Figure 1, Original Stanford Alpaca LLM outputs text.)
executing a target large language model for the generative task to obtain a target output in the target language based on the target input; (Figure 1, x-CrossAlpaca LLM outputs text.)
fine-tuning the target large language model based on the cross-lingual similarity score for the target output using a reinforcement learning algorithm to obtain a fine-tuned large language model. (4.3.2 Models Setup, In order to align the results with the state-of-the-art models, we used the alpaca_LoRA (Hu et al., 2021b) code, adopting the same hyperparameters. We performed the fine-tuning with a single epoch and a batch-size of 128 examples. [Alpaca LoRAa code uses context target pairs in its training data to adjust model parameters.]; 2.3 Instruction-tuning is at hand, Parameter-Efficient Tuning (PEFT) is an efficient technique to adjust a small part of the model parameters and freeze the others. The main goal is to significantly reduce computational and storage costs while maintaining the performance of the original models. The standard techniques for PEFT are LoRA (Hu et al., 2021a), Prefix Tuning (Li and Liang, 2021), P-Tuning (Liu et al., 2022).)
Ranaldi does not specifically teach:
generating a reference output embedding vector for the reference output and a target output embedding vector for the target output using one or more pre-trained cross-lingual encoders;
determining a cross-lingual similarity between the target output embedding vector and the reference output embedding vector to obtain a cross-lingual similarity score for the target output; and
Niu, however, teaches:
generating a reference output embedding vector for the reference output and a target output embedding vector for the target output using one or more pre-trained cross-lingual encoders; (P0013, Cross-lingual sentence alignment.; P0018, A first batch of sentences, e.g., S={S1, S2, . . . , SM} in a source language, and a second batch of sentences, e.g., T={T1, T2, . . . , TN} in a target language, are input to an alignment model for alignment.; P0020, The embedding model may adopt the BERT encoder which tokenizes the input text into a sequence of word pieces, where unknown words are split into several commonly observed sequences of characters. The representation for each word piece is computed with a Transformer encoder by repeatedly applying self-attention and nonlinear transformations in an alternating fashion.)
determining a cross-lingual similarity between the target output embedding vector and the reference output embedding vector to obtain a cross-lingual similarity score for the target output; and (P0015, A supervised version of BERT-score may be computed as the semantic similarities between sentences.; P0021, The BERT score computation module then computes a BERT score of the input sentences and using cosine similarity between the contextual embedding representations from the embedding model.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to determine cross lingual similarity score utilizing embedding vectors in reference language and target language. It would have been obvious to combine the references because calculating cross lingual similarity score is a known technique to yield a predictable result of calculating similarity between sentences in one language and sentences in another language. Niu P0027.
Ranaldi teaches high level concepts of instruction tuning on language specific data. Ranaldi in view of Niu does not specifically teach:
fine-tuning the target large language model based on the cross-lingual similarity score for the target output using a reinforcement learning algorithm to obtain a fine-tuned large language model. Foley, however, teaches:
fine-tuning the target large language model based on the cross-lingual similarity score for the target output using a reinforcement learning algorithm to obtain a fine-tuned large language model. (P0020, An embodiment uses one or more training prompt responses, or combinations of training prompts and their corresponding responses, to train an attribution model to attribute a fine-tuned model to a source foundation model. In some embodiments, the attribution model is a classifier model that outputs a classification of a fine-tuned model to a source foundation model, and a confidence score in the classification.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to utilize an evaluation score between responses of large language models. It would have been obvious to combine the references because a fine-tuned model may generate false information and generating an evaluation score enables the fine-tuned model to link or attribute to the foundation model. (Foley, P0015)
Regarding claim 3 Ranaldi in view of Niu and further view of Foley teach claim 1.
Ranaldi further teaches:
wherein the reference large language model is a Generative Pre-trained Transformer 4 (GPT-4), and wherein the reference language is English. (Figure 1, Original LLM language in English.)
Ranaldi in view of Niu does not specifically teach:
wherein the reference large language model is a Generative Pre-trained Transformer 4 (GPT-4), and wherein the reference language is English.
Foley, however, teaches:
wherein the reference large language model is a Generative Pre-trained Transformer 4 (GPT-4), and wherein the reference language is English. (P0002, A foundation model, or base model, is a machine learning model that is trained, generally using self-supervision or semi-supervised learning. … Some examples of presently available LLMs are the Generative Pre-trained Transformer (GPT) family of models, such as GPT-3 and GPT-4.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have reference large language model be GPT-4. It would have been obvious to combine the references because utilizing GPT-4 as a reference LLM is a known technique to yield a predictable result of providing a generative output to an input.
Regarding claim 4 Ranaldi in view of Niu and further view of Foley teach claim 1.
Ranaldi further teaches:
wherein the target language is non-English, wherein the target large language model comprises a GPT model or a non-GPT model. (Figure 1, x-CrossAlpaca LLM using foreign language.)
Regarding claim 5 and 13 Ranaldi in view of Niu and further view of Foley teach claim 1 and 10.
Ranaldi further teaches:
further comprising translating the target input in the target language to obtain the reference input in the reference language using a machine translation model. (3.2 Cross-Lingual Instruction-tuning, Instruction-following demonstrations. The original version of Alpaca is in English. However, as described in Section 3.1, different open-source translations are produced with a translation engine. In our experiments, we propose the Instruction tuning phase with the original English version (enAlpaca) and the translated language-specific versions.)
Regarding claim 6 and 14 Ranaldi in view of Niu and further view of Foley teach claim 1 and 10.
Ranaldi further teaches:
wherein the generative task comprises summarization, paraphrasing, or question-answer generation. (4.3 Experimental Setup, Cross-lingual Question Answering Dataset (XQUAD) (Artetxeet al., 2019) consists of a subset of 240 paragraphs and 1190 question-answer pairs from the development set of SQuAD v1.1 (Rajpurkar et al., 2016) with their manual translations into several languages. Consequently, the dataset is entirely parallel across 11 languages.)
Regarding claim 7 Ranaldi in view of Niu and further view of Foley teach teaches claim 1.
Ranaldi in view of Niu does not specifically teach:
wherein the evaluation score comprises a reference- less machine translation metric, a similarity metric, or a predicted estimate for human judgment.
Foley, however, teaches:
wherein the evaluation score comprises a reference- less machine translation metric, a similarity metric, or a predicted estimate for human judgment. (P0021, Each binary prediction is a classification of a training input as either attributed to a particular foundation model or not attributed to that particular foundation model, with a confidence score in the attribution.; P0022, The attribution model is implemented using TripletNet-based classifiers that use a margin-based loss function using the separate embeddings of the base and fine-tuned model responses. TripletNet is a presently available technique that is able to make predictions by taking in a single sentence or other portion of text, computing an output embedding (a numerical representation of a prompt response), and finding the closest embedding from the prompt responses in the training set and using the label of the training sentence as a prediction. The cosine distance between the anchor input (a reference input used in the loss function), positive example (a prompt pair or a prompt-response pair with correct attribution), and negative example (a prompt pair or a prompt-response pair with incorrect attribution is computed as the loss function (the value that the machine learning training procedure tries to minimize—e.g., cos (anchor, negative) and -cos (anchor, positive) for TripleNet).)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to utilize an evaluation score of a similarity metric between responses of large language models. It would have been obvious to combine the references because a fine-tuned model may generate false information and generating an evaluation score enables the fine-tuned model to link or attribute to the foundation model. (Foley, P0015)
Regarding claim 15 Ranaldi in view of Niu and further view of Foley teach claim 10.
Ranaldi in view of Niu does not specifically teach:
wherein the evaluation score comprises a reference- less machine translation metric, a vector-based similarity metric, or a semantic similarity metric.
Foley, however, teaches:
wherein the evaluation score comprises a reference- less machine translation metric, a vector-based similarity metric, or a semantic similarity metric. (P0021, Each binary prediction is a classification of a training input as either attributed to a particular foundation model or not attributed to that particular foundation model, with a confidence score in the attribution.; P0022, The attribution model is implemented using TripletNet-based classifiers that use a margin-based loss function using the separate embeddings of the base and fine-tuned model responses. TripletNet is a presently available technique that is able to make predictions by taking in a single sentence or other portion of text, computing an output embedding (a numerical representation of a prompt response), and finding the closest embedding from the prompt responses in the training set and using the label of the training sentence as a prediction. The cosine distance between the anchor input (a reference input used in the loss function), positive example (a prompt pair or a prompt-response pair with correct attribution), and negative example (a prompt pair or a prompt-response pair with incorrect attribution is computed as the loss function (the value that the machine learning training procedure tries to minimize—e.g., cos (anchor, negative) and -cos (anchor, positive) for TripleNet).)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to utilize an evaluation score of a similarity metric between responses of large language models. It would have been obvious to combine the references because a fine-tuned model may generate false information and generating an evaluation score enables the fine-tuned model to link or attribute to the foundation model. (Foley, P0015)
Regarding claim 12 and 19 Ranaldi in view of Niu and further view of Foley teach claim 10 and 17.
Ranaldi further teaches:
wherein the reference large language model is a Generative Pre-trained Transformer 4 (GPT-4), wherein the target large language model comprises a GPT model or a non-GPT model, wherein the reference language is English, and wherein the target language is non- English. (Figure 1, Original LLM language in English.; Figure 1, x-CrossAlpaca LLM using foreign language.)
Ranaldi in view of Niu does not specifically teach:
wherein the reference large language model is a Generative Pre-trained Transformer 4 (GPT-4), wherein the target large language model comprises a GPT model or a non-GPT model, wherein the reference language is English, and wherein the target language is non- English.
Foley, however, teaches:
wherein the reference large language model is a Generative Pre-trained Transformer 4 (GPT-4), wherein the target large language model comprises a GPT model or a non-GPT model, wherein the reference language is English, and wherein the target language is non- English. (P0002, A foundation model, or base model, is a machine learning model that is trained, generally using self-supervision or semi-supervised learning. … Some examples of presently available LLMs are the Generative Pre-trained Transformer (GPT) family of models, such as GPT-3 and GPT-4.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have reference large language model be GPT-4. It would have been obvious to combine the references because utilizing GPT-4 as a reference LLM is a known technique to yield a predictable result of providing a generative output to an input.
Claims 2, 11, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Ranaldi in view of Niu, in view of of Foley, and further view of Ben-David et al. (U.S. PG Pub No. 20240346238), hereinafter Ben-David.
Regarding claim 2, 11, and 18 Ranaldi in view of Niu and further view of Foley teach claim 1, 10, and 17.
Ranaldi in view of Niu and further view of Foley does not specifically teach:
wherein the target input comprises communication data in the target language on a communication platform, comprising meeting transcripts, chat messages, audio messages, and emails.
Ben-David, however, teaches:
wherein the target input comprises communication data in the target language on a communication platform, comprising meeting transcripts, chat messages, audio messages, and emails. (P0024, Generating sales call summaries and/or deal summaries using a specific-trained language model.; P0034, The summary generator is configured to process the data in the database.; P0032, A database stores message data that may be related to conversations. Such message data may include but is not limited to, email messages, chat logs, instant messages, or other text, or data otherwise, including contents of communications or content related to communications between and among individuals.; P0033, A database also includes transcript data related to audio/video conversations.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have target input as communication data. It would have been obvious to combine the references because the input of communication data to a an LLM is a known technique to yield a predictable result of providing a generative output to an input.
Claims 8 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Ranaldi in view of Niu, in view of Foley, and further view of Srinivasan et al. (U.S. PG Pub No. 20250111169), hereinafter Srinivasan.
Regarding claim 8 and 16 Ranaldi in view of Niu and further view of Foley teach claim 1 and 10.
Ranaldi further teaches:
deploying multiple large language models for the generative task to generate multiple outputs based on the target input; (Figure 1, Utilizing multiple LLMs (Stanford Alpaca, x-CrossAlpaca) to answer question and providing a response based on the input.)
Ranaldi in view of Niu does not specifically teach:
evaluating a similarity between each output of the multiple outputs and the reference output to generate multiple evaluation scores;
Foley, however, teaches:
evaluating a similarity between each output of the multiple outputs and the reference output to generate multiple evaluation scores; (P0020, An embodiment uses one or more training prompt responses, or combinations of training prompts and their corresponding responses, to train an attribution model to attribute a fine-tuned model to a source foundation model. In some embodiments, the attribution model is a classifier model that outputs a classification of a fine-tuned model to a source foundation model, and a confidence score in the classification.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to utilize an evaluation score between responses of large language models. It would have been obvious to combine the references because a fine-tuned model may generate false information and generating an evaluation score enables the fine-tuned model to link or attribute to the foundation model. (Foley, P0015)
Ranaldi in view of Niu and further view of Foley does not specifically teach:
ranking the multiple large language models based on the multiple evaluation scores for the multiple large language models; and
selecting a large language model corresponding to a highest evaluation score for the generative task in the target language.
Srinivasan, however, teaches:
ranking the multiple large language models based on the multiple evaluation scores for the multiple large language models; and (P0014, Providing the prompt to multiple large language models (LLMs), the multiple LLMs being trained on different datasets and having different knowledge and capabilities.; P0015, Receiving multiple responses from the multiple LLMs.; P0016, Determining a rank for each of the multiple responses, the rank indicating a level of confidence.; P0059, The ranking engine can also compare and assign a level of pair-wise similarity or dissimilarity between selected pairs of responses from the LLM array. This can be done by comparing the embedding vectors generated from the selected pair of responses to determine a level of agreement or disagreement between the responses and source LLMs. This can be done for all possible pairings of LLM responses. As will be appreciated, cosine similarity is a mathematical metric that measures how similar two vectors are in a multi-dimensional space.)
selecting a large language model corresponding to a highest evaluation score for the generative task in the target language. (P0018, Selecting the response having the best rank.; P0026, Various outputs of the LLMs are analyzed and ranked by an AI-based ranking and reasoning engine, with the highest ranked output indicating the most probable best fit or optimal solution for the prompt.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to rank large language models based on evaluation scores and select the highest evaluation score. It would have been obvious to combine the references because by efficiently ranking and/or comparing the similarity of the output to yield an output having the highest likelihood of being the ground truth, LLM hallucination incidents can be reduced. (Srinivasan P0029)
Claims 9 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Ranaldi in view of Niu, in view of Foley, and further view of Deepak et al. (U.S. PG Pub No. 20250259011), hereinafter Deepak.
Regarding claim 9 and 20 Ranaldi in view of Niu and further view of Foley teach claim 1 and 17.
Ranaldi further teaches:
obtaining a first output generated from a target input by using the target large language model for the generative task at a first time; (Figure 1, Utilizing multiple LLMs (Stanford Alpaca, x-CrossAlpaca) to answer question and providing a response based on the input.)
Ranaldi in view of Niu does not specifically teach:
evaluating a first similarity between the first output and the reference output to generate a first evaluation score;
Foley, however, teaches:
evaluating a first similarity between the first output and the reference output to generate a first evaluation score; (P0020, An embodiment uses one or more training prompt responses, or combinations of training prompts and their corresponding responses, to train an attribution model to attribute a fine-tuned model to a source foundation model. In some embodiments, the attribution model is a classifier model that outputs a classification of a fine-tuned model to a source foundation model, and a confidence score in the classification.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to utilize an evaluation score between responses of large language models. It would have been obvious to combine the references because a fine-tuned model may generate false information and generating an evaluation score enables the fine-tuned model to link or attribute to the foundation model. (Foley, P0015)
Ranaldi in view of Niu and further view of Foley does not specifically teach:
obtaining a second output generated from the target input by using the target large language model for the generative task at a second time later than the first time; evaluating a second similarity between the second output and the reference output to generate a second evaluation score; and detecting a model drift of the target large language model based on a different between the first evaluation score and the second evaluation score.
Deepak, however, teaches:
obtaining a second output generated from the target input by using the target large language model for the generative task at a second time later than the first time; evaluating a second similarity between the second output and the reference output to generate a second evaluation score; and detecting a model drift of the target large language model based on a different between the first evaluation score and the second evaluation score. (P0043, In certain embodiments, a drift monitoring module may analyze LLM outputs over time, such as by using statistical methods and machine learning algorithms to identify patterns of uncertainty or errors in the model's responses (e.g., assessing the confidence level of the LLM in its responses, comparing LLM outputs with trusted external data sources for factual accuracy, or preforming longitudinal analysis to track the consistency of LLM responses over time in various domains, among others). Drift detection may then focus on identifying changes in the LLM's performance over time.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to obtain a second output at a later time to determine model drift of the target large language model. It would have been obvious to combine the references because LLM model can produce uncertainty or incorrectness over time with loss of knowledge and detecting the drift can allow for LLM updates to counteract the drift. (Srinivasan P0043, P0049)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Gupta et al. (U.S. PG Pub No. 20250124236): Using llm functions to evaluate and compare large text outputs of llms. Gupta teaches the evaluation LLM calculates a similarity score for each generated response based on the number of similar or matching keywords between the expected output and the generated responses.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL WONSUK CHUNG whose telephone number is (571)272-1345. The examiner can normally be reached Monday - Friday (7am-4pm)[PT].
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, PIERRE-LOUIS DESIR can be reached at (571)272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL W CHUNG/Examiner, Art Unit 2659
/PIERRE LOUIS DESIR/Supervisory Patent Examiner, Art Unit 2659
