DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101.
Claims 1, 15 and 20 are rejected under 101 Abstract Idea. The claims are directed to an abstract idea because it recites collecting information, evaluating information, and making decisions based on that evaluation. Specifically, the claims generate question prompts, executes language models to obtain answers, removes questions that are answered correctly without context, generates additional answers using filtered prompts, and scores the language models using semantic model. These steps amount to observing results, comparing answers, filtering data, and evaluating model performance, which are mental processes that can be performed by a human evaluator using pen and paper or by applying mathematical techniques to compare answers and calculate scores. The claims focus on organizing and analyzing information to determine the relative performance of language models rather than on improving the operation of a computer or another technology.
Step 2A, the additional elements do not integrate the abstract idea into a practical application. Although the claims recites generic computer components, language models, a semantic model, prompts, and RAG context, these elements merely serve as tools for carrying out the abstract evaluation process. The claims do not improve the functioning of the language models themselves, improve the operation of a computer, or provide a new retrieval technique or semantic scoring algorithm. Instead, it uses conventional language models to generate answers, filters questions based on correctness, and computes scores representing model accuracy. These are routine data processing operations that apply the abstract idea in a generic computing environment without imposing any meaningful technological limitation.
Step 2B, the claims also lack an inventive concept because the additional elements, individually and in combination, merely implement the abstract idea using well-understood, routine, and conventional computer technology. Executing language models on prompts, generating responses, filtering questions, using retrieved context, and applying a semantic model to score outputs are conventional operations performed by generic processors executing machine learning software. The ordered combination simply automates the process of evaluating and ranking language models and does not recite a specific technical solution that improves computer functionality or another technological field.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the claims are (i) mere instructions to implement the idea on a computer, and/or (ii) recitation of generic computer structure that serves to perform generic computer functions that are well-understood, routine, and conventional activities previously known to the pertinent industry. Viewed as a whole, these additional claim element(s) do not provide meaningful limitation(s) to transform the abstract idea into a patent eligible application of the abstract idea such that the claim(s) amounts to significantly more than the abstract idea itself. Therefore, the claim(s) are rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter. There is further no improvement to the computing device.
Dependent claims 2-14 and 16-19 further recite an abstract idea performable by a human and do not amount to significantly more than the abstract idea as they do not provide steps other than what is conventionally known.
Claim 2, selecting and deploying on a generic server does not add a technological improvement.
Claims 3 and 16, receiving and returning steps are performed with a generic language model and user device.
Claims 4 and 17, selecting, presenting and displays information without improving computer technology.
Claim 5, generating and organizing and using a language model merely automates that process.
Claim 6, does not provide a specific technological improvement.
Claim 7, routine operation performed with generic computer technology.
Claim 8, calculating and comparing are mathematical evaluation steps that merely automate a human judgment about whether answers match.
Claim 9, comparing, deciding and identifying are abstract mental processes of evaluation and classification.
Claim 10, does not improve how the computer or language model operates.
Claim 11, mathematical analysis performed by a generic sematic model.
Claim 12, merely automate a business or administrative selection decision.
Claim 13, basic mathematical comparison that does not add a practical technological application.
Claim 14, merely applies an abstract evaluation and training rule using conventional machine learning operations.
Claim 18, merely performs the abstract task of evaluating information with generic computing components.
Claim 19, does not improve server or computer functionality.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 5-6 and 9-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al. (“Benchmarking Large Language Models in Retrieval-Augmented Generation”; 3/24/2024) in in view of Liu et al. (“Exploring the Integration Strategies of Retriever and Large Language Models”; Aug. 24, 2023) in view of Es et al. (“RAGAs: Automated Evaluation of Retrieval Augmented Generation”; March 17-22, 2024).
Claim 1,
Chen teaches a method of improving a retrieval augmented generation (RAG) language model ensemble comprising a plurality of language models and a context assigned to the RAG language model ensemble, the method comprising (Chen teaches RAG and multiple LLMs: “Retrieval-Augmented Generation (RAG) is a promising approach…” and “Then we evaluate 6 representative LLMs on RGB.” [Abstract]; Chen also teaches external documents as RAG context: “External documents contain all answers,” [Fig. 1 see Information Integration]):
executing the plurality of language models on a plurality of first prompts to generate a plurality of test outputs, wherein (Chen teaches evaluating multiple LLMs and their responses: “RGB utilizes a question-answer format for evaluation” and “We evaluate the LLMs by judging the retrieval-augmented responses of them to the questions” [Data Construction see pg. 17757] Chen identifies the plurality of LLMs: “We evaluate 6 state-of-the-art LLMs including ChatGPT… ChatGLM-6B…” etc. [Settings see pg. 17758]):
each of the plurality of first prompts comprises one of a plurality of natural language questions corresponding to a plurality of correct answers (Chen teaches QA prompt/question-answer pairs: “RGB utilizes a question-answering format for evaluation,” and “we construct question and answer using actual news articles” [Data construction see pg. 17757]; Chen further states that ChatGPT generates “events, questions, and answers,” followed by manual answer checking [Data Construction see pg. 11757]),
the plurality of correct answers are only obtainable from the context (Chen states: “The testbed for information integration contains instances that can be answered using multiple documents” [Introduction see pg. 17755, Information Integration]; Chen also explains that “answering such questions requires utilizing information from various documents” [Data Construction]),
the plurality of prompts is assigned to each of the plurality of language models (Chen teaches applying RGB to multiple LLMs: “Based on RGB, we conduct evaluation on 6 state-of-the-art large language models,” [Introduction]; Chen’s Table 1 reports the same noise-ration evaluation for ChatGPT, ChatGLM-6B etc. [Table 1]), and
executing the plurality of language models on a plurality of (Chen teaches executing multiple LLMs to generate responses: “We evaluate the LLMs by judging the retrieval-augmented responses of them to the questions” [Data Construction] Chen identifies the six LLMs in the evaluation [Settings]).
The difference between the prior art and the claimed invention is that Chen does not explicitly teaches each of the plurality of prompts avoids referencing the context; removing, from the plurality of natural language questions, a removed question to generate a plurality of filtered questions, wherein: the removed question comprises a corresponding one of the plurality of natural language questions for which a corresponding test output of the plurality of test outputs represents a correct answer relative to the plurality of correct answers; filtered prompts to generate generated answers, wherein each of the plurality of filtered prompts comprises the plurality of filtered questions and a corresponding command to return the generated answers from the context; and executing a semantic model on the generated answers to generate a plurality of scores representing accuracies of the plurality of language models.
Liu teaches each of the plurality of prompts avoids referencing the context (Liu teaches an initial no-context prompt: “we initially prompt the LLM to answer questions without any provided context” [4.] [Dataset Filter]; this teaches prompts that do not include or reference the context);
removing, from the plurality of natural language questions, a removed question to generate a plurality of filtered questions, wherein (Liu teaches filtering/removal: the no-context prompting: “enables us to filter out questions that the LLM can accurately answer independently” [4.3] [Dataset Filter]; Liu provides resulting filtered question counts: “After removing questions that can be answered without context, we are left with 3,459 questions in NQ, 1,259 in TriviaQA, and 3,448 in SQuAD” [4.] [Dataset Filter]):
the removed question comprises a corresponding one of the plurality of natural language questions for which a corresponding test output of the plurality of test outputs represents a correct answer relative to the plurality of correct answers (Liu removes questions answerable correctly without context: “filter out questions that the LLM can accurately answer independently” [4.] [Dataset Filter]; Liu also defines correctness of a generated output: “A generated response is considered correct if, after normalization, it matches any candidate in a list of acceptable answers” [4.] [Experiments]);
filtered prompts to generate generated answers (Liu supplies the filtered questions used after removal [4.] [Dataset Filter]),
wherein each of the plurality of filtered prompts comprises the plurality of filtered questions and a corresponding command to return the generated answers from the context (Liu’s Fig. 5 prompt commands context-based answering: “Given the relevant background contexts, answer the current question using one of the context…” and includes “Context: {retrieved_context},” “Question: {question},” and “Answer:” [Fig. 5]).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Chen with teachings of Liu by modifying the benchmarking LLMs in RAG as taught by Chen to include each of the plurality of prompts avoids referencing the context; removing, from the plurality of natural language questions, a removed question to generate a plurality of filtered questions, wherein: the removed question comprises a corresponding one of the plurality of natural language questions for which a corresponding test output of the plurality of test outputs represents a correct answer relative to the plurality of correct answers; filtered prompts to generate generated answers, wherein each of the plurality of filtered prompts comprises the plurality of filtered questions and a corresponding command to return the generated answers from the context as taught by Liu for the benefit of effectively leverage retrieved passages to enhance the answer generation capability of LLMs (Liu [Abstract]).
The difference between the prior art and the claimed invention is that Chen nor Liu explicitly teaches executing a semantic model on the generated answers to generate a plurality of scores representing accuracies of the plurality of language models.
Es teaches executing a semantic model on the generated answers to generate a plurality of scores representing accuracies of the plurality of language models (Es teaches automated semantic/LLM evaluation: “these three quality aspects can be measured in a fully automated way, by prompting an LLM” [3.]; Es teaches a score: “The final faithfulness score, F, is then computed…” [3.]; Es also teaches embedding-based sematic scoring: “We then obtain embeddings… using the text-embedding-ada-002 model” and compute the “answer relevance score, AR” [3.] [Answer Relevance]; Chen teaches per-LLM accuracy scores in Table 1 and 5).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Chen and Liu with teachings of Es by modifying the benchmarking LLMs in RAG as taught by Chen to include executing a semantic model on the generated answers to generate a plurality of scores representing accuracies of the plurality of language models as taught by Es for the benefit of contributing to faster evaluation cycles of RAG architecture (Es [Abstract]).
Claim 5,
Es further teaches the method of claim 1, further comprising generating the plurality of natural language questions by: generating a question generation prompt instructing a question generation language model to generate questions answerable by reference to the context (Es identifies the question-generation language model as ChatGPT: “we then asked ChatGPT to suggest a question that can be answered based on the introductory section of the page, using the following prompt” [4.] [The WikiEval Dataset]; Es teaches the prompt itself: “Your task is to formulate a question from given context satisfying the rules given below:” followed by rule “1. The question should be fully answered from the given context” [4.]; The prompt also includes “context:” as the context field [4.]); and
executing the question generation language model on the question generation prompt and the context (Es teaches execution of ChatGPT using the prompt and context: “For each of the 50 pages, we then asked ChatGPT to suggest a question that can be answered based on the introductory section of the page, using the following prompt” [4.]; the printed prompt requires the model to formulate the question “from given context” and includes the input field [4.]),
wherein the question generation language model outputs the plurality of natural language questions (Es teaches that the system selected 50 Wikipedia pages and, “for each of the 50 pages,” asked ChatGPT “to suggest a question that can be answered based on the introductory section of the page” [4.]; Es further refers to “the generated question” when describing use of ChatGPT to answer it: “We also used ChatGPT to answer the generated question, when given the corresponding introductory section as context” [4.]).
Claim 6,
Liu further teaches the method of claim 1, further comprising generating the plurality of filtered prompts by: adding, to each of the plurality of filtered prompts, the plurality of filtered questions (Liu teaches filtering the dataset to obtain remaining questions: “After removing questions that can be answered without context, we are left with 3,459 questions in NQ, 1,259 in TrivaQA, and 3,448 in SQuAD” [4.] [Dataset Filter]; Liu then teaches a prompt template containing a question slot: “Question: {question} [see Fig. 5]; Liu also states that the remaining questions are the focus of the study: “Therefore, the remaining questions, which the LLM couldn’t answer independently, are the focus of our study” [4.]); and
adding, to each of the plurality of filtered prompts, the corresponding command (Liu’s Fig. 5 prompt includes the command: “Given the relevant background contexts, answer the current question using one of the context in short factoid phrase manner” [see Fig. 5]; the same prompt further commands: “Try your best to guess an extractive answer. If don’t know the answer, just say unknown” [see Fig. 5]; The prompt then includes “Context: {retrieved_context},” “Question: {question}”, and “Answer:” [see Fig. 5]).
Claim 9,
Liu further teaches the method of claim 1, further comprising identifying the corresponding one of the plurality of natural language questions for which the corresponding test output represents the correct answer by (Liu teaches identifying questions answerable correctly without context by first prompting the LLM without context and then filtering such questions: “we initially prompt the LLM to answer questions without any provided context. This process enables us to filter out questions that the LLM can accurately answer independently, thereby eliminating the need for additional external contextual information” [4.] [Dataset Filter]; Liu further states: “After removing questions that can be answered without context, we are left with 3,459 questions in NQ, 1,259 in TrivaiQA, and 3,448 in SQuAD” [Dataset Filter]):
comparing the plurality of correct answers to the corresponding test output and determining that the corresponding test output matches at least one of the plurality of correct answers (Liu teaches answer comparison against a plurality/list of acceptable answers: “Predicted answers are evaluated with the standard exact match (EM) and F1 metric” [4.] [Experiments]; Liu further states: “A generated response is considered correct if, after normalization, it matches any candidate in a list of acceptable answers” [Experiments]; Liu also describes the normalization used before matching: “The normalization process entails converting the text to lowercase and omitting articles, punctuation, and redundant whitespaces” [Experiments]); and
identifying the corresponding one of the plurality of natural language questions to which the corresponding test output belongs (Liu teaches that the LLM is prompted to answer “questions without any provided context,” and that the resulting process filters out “questions that the LLM can accurately answer independently” [Dataset Filter]; Liu further states that “the remaining questions, which the LLM couldn’t answer independently, are the focus of our study” [Dataset Filter]; Liu also quantifies the question sets before and after removal: “We use the development set of NQ, TrivaiQA, and SQuAD, initially containing… respectively. After removing questions that can be answered without context, we are left with …” [Dataset Filter]).
Claim 10,
Liu further teaches the method of claim 1, wherein the corresponding command further commands the plurality of language models to return the generated answers only from the context (Liu’s filtered prompt commands: “Given the relevant background context, answer the current question using one of the context in short factoid phrase manner” [Fig. 5]; The same prompt further commands: “Try your best to guess an extractive answer. If don’t know the answer, just say unknown” [Fig. 5]; Liu’s prompt then provides the retrieved context and question fields: “Context: {retrieved_context},” “Question {question},” and “Answer:” [Fig. 5]).
Claim(s) 2-3, 15-16 and 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al. (“Benchmarking Large Language Models in Retrieval-Augmented Generation”; 3/24/2024) in in view of Liu et al. (“Exploring the Integration Strategies of Retriever and Large Language Models”; Aug. 24, 2023) in view of Es et al. (“RAGAs: Automated Evaluation of Retrieval Augmented Generation”; March 17-22, 2024) and further in view of Bansal et al. (US 12,061,963).
Claim 2,
Chen, Liu and Es teach all the limitations in claim 1. The difference between the prior art and the claimed invention is that Chen, Liu nor Es explicitly teach selecting, according to the plurality of scores, a selected language model from the plurality of language models; and deploying the selected language model on a server.
Bansal teaches selecting, according to the plurality of scores, a selected language model from the plurality of language models (Bansal teaches selecting a model/pipeline based on objective metrics: “a user may also specify an objective metric for the exploration,” and the metric is “to be used to evaluate which ML pipeline is the best for the user”; Bansal further states that the system may use “selected metric or metrics to determine a best-performing ML pipeline” [col. 7 lines 32-45]; Bansal also teaches per-model score values: “the result identifying, for each of the plurality of ML models, a metric value corresponding to the objective metric that is indicative of a quality of that ML model” [claim 1]); and
deploying the selected language model on a server (Bansal teaches deployment of the best model: “users may then directly deploy this best model (in terms of a ML pipeline) to a production environment” [col 3 lines 14-26]; Bansal further claims “receiving a second request… to deploy a particular ML models, in the provider network; deploying the ML model in association with an endpoint of the provider network; and generating inferences, using the deployed ML model, for requests received at the endpoint” [claim 1]).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Chen, Liu and Es with teachings of Bansal by modifying the benchmarking LLMs in RAG as taught by Chen to include selecting, according to the plurality of scores, a selected language model from the plurality of language models; and deploying the selected language model on a server as taught by Bansal for the benefit of deploying the best ML pipeline to the requesting user (Bansal [Abstract]).
Claim 3,
Bansal further teaches the method of claim 2, further comprising: receiving a user query from a user device (Bansal teaches that after deployment, a client or computing device may send inference requests to the endpoint: “a client application 130A-130B… outside the provider network (e.g., by a computing device 104) may issue inference requests to the endpoint associated with the deployed pipeline code 126” [col. 12 lines 26-47]; Bansal further teaches a user device transmitting an execution request that includes model input: “the user device 1002 transmits an execution request to the model hosting system 140 via the frontend 1049, where the execution request identifies an endpoint and includes an input to a machine learning model (e.g., a set of input data)” [col. 25 lines 27-39]);
executing the selected language model on the user query to generate a query response (Bansal teaches that the deployed model is ready to execute: “Once the ML scoring container(s) 1050 are initialized, the ML scoring container(s) 1050 are ready to execute trained machine learning model(s)” [col. 25 lines 27-39]; Bansal further teaches execution on the received input: “a virtual machine instance 1042 executes the code 1056 stored in an identified ML scoring container 1050 in response to the model hosting system 140 receiving the execution request” [col. 25 lines 40-60]; Bansal states that the execution “use[s] the input included in the execution request as an input parameter, and generate[s] a corresponding output” [col. 25 lines 40-60]); and
returning the query response to the user device (Bansal teaches returning the inference result to the requestion application: inference requests to the deployed endpoint enable “the pipeline to generate an inference and return that result back to the application” [col. 12 lines 26-47]; Bansal further teaches returning output to the user device: “the virtual machine instance 1042 transmits the output to the user device 1002 that submitted the execution result via the frontend 1049” [col. 25 lines 61-65]).
Claim 15,
A system for improving a retrieval augmented generation (RAG) language model ensemble comprising a plurality of language models and a context assigned to the RAG language model ensemble, the system comprising: a computer processor; a server in communication with the computer processor; a data repository in communication with the computer processor and storing: the context, a plurality of natural language questions, a plurality of correct answers, only obtainable from the context, to the plurality of natural language questions, a first plurality of prompts, each comprising one of the plurality of natural language questions, wherein the plurality of first prompts avoids referencing the context, a removed question comprising a corresponding one of the plurality of natural language questions for which a corresponding test output of a plurality of test outputs of the plurality of language models represents a correct answer relative to the plurality of correct answers, a plurality of filtered questions comprising the plurality of natural language questions less the removed question, a plurality of filtered prompts comprising the plurality of filtered questions and a corresponding command to return generated answers from the context, a plurality of scores representing accuracies of the plurality of language models; the plurality of language models which, when executed by the computer processor: generates the plurality of test outputs when executed on the first plurality of prompts, and generates the generated answers when executed on the plurality of filtered prompts; a semantic model which, when executed by the computer processor on the generated answers generates the plurality of scores; and a server controller which, when executed by the computer processor: selects, according to the plurality of scores, a selected language model from among the plurality of language models, and deploys the selected language model to the server for execution of the selected language model. (Claim 15 contains subject matter similar to claims 1 and 2, and thus is rejected under similar rationale)
Claim 16,
The system of claim 15, further comprising: a network communication device in communication with the computer processor and programmed to receive a user query from a user device, wherein the server controller is further programmed to generate a response by executing the selected language model on the user query and to return the response to the user device. (Claim 16 contains subject matter similar to claim 3, and thus is rejected under similar rationale)
Claim 18,
The system of claim 15, further comprising: a semantic model which, when executed by the computer processor, generates the plurality of scores. (Claim 18 contains subject matter similar to claim 1, and thus is rejected under similar rationale)
Claim 19,
The system of claim 15, wherein the server controller is further programmed to: generate the plurality of first prompts and the plurality of filtered prompts. (Claim 19 contains subject matter similar to claim 1, and thus is rejected under similar rationale)
Claim 20,
A method of improving a retrieval augmented generation (RAG) language model ensemble comprising a plurality of language models and a context assigned to the RAG language model ensemble, the method comprising: executing the plurality of language models on a plurality of first prompts to generate a plurality of test outputs, wherein: each of the plurality of first prompts comprises one of a plurality of natural language questions corresponding to a plurality of correct answers, the plurality of correct answers are only obtainable from the context, the plurality of prompts is assigned to each of the plurality of language models, and each of the plurality of prompts avoids referencing the context; removing, from the plurality of natural language questions, a removed question to generate a plurality of filtered questions, wherein: the removed question comprises a corresponding one of the plurality of natural language questions for which a corresponding test output of the plurality of test outputs represents a correct answer relative to the plurality of correct answers; executing the plurality of language models on a plurality of filtered prompts to generate generated answers, wherein each of the plurality of filtered prompts comprises the plurality of filtered questions and a corresponding command to return the generated answers from the context; executing a semantic model on the generated answers to generate a plurality of scores representing accuracies of the plurality of language models;
selecting, according to the plurality of scores, a selected language model from the plurality of language models; deploying the selected language model on a server; receiving a user query from a user device; executing the selected language model on the user query to generate a query response; and returning the query response to the user device. (Claim 20 contains subject matter similar to claims 1, 2 and 3, and thus is rejected under similar rationale)
Claim(s) 4 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al. (“Benchmarking Large Language Models in Retrieval-Augmented Generation”; 3/24/2024) in in view of Liu et al. (“Exploring the Integration Strategies of Retriever and Large Language Models”; Aug. 24, 2023) in view of Es et al. (“RAGAs: Automated Evaluation of Retrieval Augmented Generation”; March 17-22, 2024) and further in view of Dumitrescu et al. (“LiRo: Benchmark and leaderboard for Romanian language tasks”; 2021).
Claim 4,
Chen, Liu and Es teach all the limitations in claim 1. The difference between the prior art and the claimed invention is that Chen, Liu nor Es explicitly teach selecting, according to the plurality of scores, a selected language model from the plurality of language models; and presenting a link to the selected language model.
Dumitrescu teaches selecting, according to the plurality of scores, a selected language model from the plurality of language models (Dumitrescu teaches an NLP benchmark and leaderboard: “LiRo is an open-source benchmark and a continuous-submission leaderboard,” and its integration of “model performance and efficiency allows both academia and industry to quickly gauge performance on tasks of interest” [3.]; Dumitrescu further teaches that “each dataset has its own leaderboard, both graphically displayed as an interactive chart, and as a table listing all participating models” [3.] [Leaderboard]; for each model, Dumitrescu include “(1) the rank of the model in the leaderboard, (2) model name, (3) metric values” [3.]; Dumitrescu also identifies language-related tasks, including “language modelling” and “question-answering” [Abstract] [Table 1]); and
presenting a link to the selected language model (Dumitrescu teaches that, for each model in the leaderboard, the displayed information includes “(6) link to the model’s paper and online code repository if any” [3.] [Leaderboard]; Dumitrescu further states that the benchmark provides “an overview of the Romanian NLU SoTA and direct access to relevant papers” [3.]; Dumitrescu also teaches that once a model submission is approved, “the new model’s results will be automatically displayed on the website” [3.]).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Chen, Liu and Es with teachings of Dumitrescu by modifying the benchmarking LLMs in RAG as taught by Chen to include selecting, according to the plurality of scores, a selected language model from the plurality of language models; and presenting a link to the selected language model as taught by Dumitrescu for the benefit of enabling multi-lingual research by augmenting parallel corpora (Dumitrescu [Abstract]).
Claim 17,
The system of claim 15, further comprising: a graphical user interface generated by the server, wherein the server controller is further programmed to: generate a link to the plurality of scores; and modify the graphical user interface to display the link. (Claim 17 contains subject matter similar to claim 4, and thus is rejected under similar rationale)
Allowable Subject Matter
Claims 7-8 and 11-14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims and overcome the 101 Abstract Idea rejection set forth.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Zhou et al. (US 2024/0362093) – At least utilizing a custom corpus of documents to condition a large language model (LLM) when generating a response to a user query. In some implementations, a user query associated with a client device is received. An API query for an external application is generated by an LLM based on the user query. The external application has access to a custom corpus of documents comprising a plurality of documents. The external application is queried using the API query. Data representative of one or more documents in the custom corpus of documents is received from the external application in response to the API query. The LLM generates a response to the query that is conditioned on the data representing one or more of the documents in the custom corpus of documents received from the external application. The response to the user query is caused to be rendered on the client device.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHREYANS A PATEL whose telephone number is (571)270-0689. The examiner can normally be reached Monday-Friday 8am-5pm PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre Desir can be reached at 571-272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
SHREYANS A. PATEL
Primary Examiner
Art Unit 2653
/SHREYANS A PATEL/Examiner, Art Unit 2659
Prosecution Insights