DETAILED ACTION
1. Claims 1-20 are pending in this application.
Notice of Pre-AIA or AIA Status
2. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. §102 and §103 (or as subject to pre-AIA 35 U.S.C. §102 and §103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Information Disclosure Statement
3. The information disclosure statement filed 02/04/2026 is in compliance with the provisions of 37 CFR 1.97, 1.98 and MPEP § 609. It has been placed in the application file and the information referred to therein has been considered as to the merits.
Claim Objections
4. Claim 20 objected to because of the following informalities:
Claim 20 use an abbreviation “RAG” understood and consider corresponding to retrieval-augmented generation (RAG), the acronym must be spelled out or defined to make the claim clear before its use.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
5. 35 U.S.C. §101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 13-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
As per claims 13 and 20, the claims similarly recite a system but does not describe hardware which executes each of the claimed steps, which is required for a system claim to be statutory. However, the use of various steps that would be reasonably understood by one of ordinary skill in the art to mean software per se steps, software-based steps implementation, or an abstract concept based on software per se. Examples of steps used in the claim are “receiving at least one input information; receiving at least one output from a language model; inputting the at least one input information and the at least one output into a scoring language model configured to score the at least one output based on the at least one input information to yield an output score; and outputting the at least one output from the language model and the output score.” Those steps are concepts well understood in the art as software per se steps implementations. Since computer software is merely a set of instructions capable of being executed by a computer system. Accordingly, these claims are rejected as non-statutory for failing to disclose such hardware where the software steps described above can be implemented. Please amend the claim to recite hardware elements, such as a processor and memory.
Dependent claims 14-19 are also rejected for inheriting the deficiencies of the base claims.
Claims 1-20 are rejected under 35 U.S.C. §101 because the claimed invention is directed to an abstract idea (Mental Process) without significantly more. The claims similarly describe steps for routing data queries and creating data queries schemas.
The following is an analysis based on 2019 Revised Patent Subject Matter Eligibility Guidance (2019 PEG).
Step 1, Statutory Category?
Claims 1-12 are directed to a method.
Claims 13-20 are directed to a system.
Therefore, claims 1-12 fall into at least one of the four statutory categories.
Claims 13-20 are directed to a system, and the claim is a software per se claim; therefore, the claim is not statutory.
Step 2A, Prong I: Judicial Exception Recited?
The examiner submits that the foregoing claim limitations constitute a “Mental Process”, as the claims cover performance of the limitations in the human mind, given the broadest reasonable interpretation.
As per independent claims 1, 13 and 20, the claims similarly recite the limitations of:
“score the output based on the at least one input information;” A human can observe the displayed information and mentally calculate a score based on simple judgments, where the criteria are provided to them. For example, a person can look at the data and easily figure out a score in their head by following the given rules. There is nothing so complex in the limitation that could not be doing in the human mind.
As per dependent claims 10 and 19, the claims similarly recite the limitation of:
“generates an output score rationale;” A human can mentally observe a score, make judgments, and mentally create a rationale for the observed score. For example, when evaluating a project, a person can notice a low score, judge it as poorly executed, and immediately realize it failed because the guidelines were completely ignored. There is nothing so complex in the limitation that could not be doing in the human mind.
Accordingly, claims 1-20 recite at least one abstract idea.
Step 2A, Prong II: Integrated into a Practical Application?
The claims recite the following additional limitations/elements:
As per independent claims 1, 13 and 20, the claims similarly recite the elements of:
The additional elements of “a language model, and a scoring language model;” are mere elements (generic computer) used to apply an exception. A recitation of the words "apply it" (or an equivalent) is a mere element to implement an abstract idea or other exception on a computer. (See MPEP 2106.05(f)).
As per independent claims 13 and 20, the claims similarly recite the element of:
The additional element of “a user interface;” is mere element (generic computer) used to apply an exception. A recitation of the words "apply it" (or an equivalent) is a mere element to implement an abstract idea or other exception on a computer. (See MPEP 2106.05(f)).
As per independent claim 20, the claim recites the element of:
The additional element of “a RAG;” is mere element (generic computer) used to apply an exception. A recitation of the words "apply it" (or an equivalent) is a mere element to implement an abstract idea or other exception on a computer. (See MPEP 2106.05(f)).
As per independent claim 1, the claim recites the limitations of:
The addition limitations of “receiving at least one input information; and receiving at least one output from a language model;” and the courts have recognized that receiving or transmitting data over a network, e.g., using the Internet to gather data, as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity. See MPEP 2106.05(d)(II)(i).
The addition limitation of “inputting the at least one input information and the at least one output into a scoring language model” In computer science, inputting data into a system means transferring data to a location where it can be manipulated. Which is an example of an insignificant extra-solution activity of data gathering, and can be understood as activities incidental to the primary process or product that are merely a nominal or tangential addition to the claim (see MPEP 2106.05(g)).
The addition limitations of “yield an output score; and outputting the at least one output from the language model and the output score.” In computer science, outputting information is also known as transmitting data to be accessed. The courts have recognized that receiving or transmitting data over a network, e.g., using the Internet to gather data, as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity. See MPEP 2106.05(d)(II)(i).
As per independent claim 13, the claim recites the limitations of:
The addition limitations of “receive a user query as input and to output an output based on the user query; and receive, as input, the output and at least one input information;” and the courts have recognized that receiving or transmitting data over a network, e.g., using the Internet to gather data, as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity. See MPEP 2106.05(d)(II)(i).
The addition limitations of “yield an output score; and the user interface configured to display the output and the output score.” In computer science, outputting information is also known as transmitting data to be accessed. The courts have recognized that receiving or transmitting data over a network, e.g., using the Internet to gather data, as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity. See MPEP 2106.05(d)(II)(i).
As per independent claim 20, the claim recites the limitations of:
The addition limitations of “receive a user query from the user interface and a RAG context from the RAG as input and to output an output based on the user query and the RAG context; and receive, as input, the output, an expected output, the user query, the RAG context, an expected RAG context, and one or more scoring guidelines;” and the courts have recognized that receiving or transmitting data over a network, e.g., using the Internet to gather data, as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity. See MPEP 2106.05(d)(II)(i).
The addition limitations of “yield an output score and a score rationale; and the user interface configured to display the output, the output score, and the score rationale.” In computer science, outputting information is also known as transmitting data to be accessed. The courts have recognized that receiving or transmitting data over a network, e.g., using the Internet to gather data, as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity. See MPEP 2106.05(d)(II)(i).
As per dependent claims 2 and 14, the claims similarly recite the limitation of:
The additional limitation of “wherein the language model uses a retrieval-augmented generation (RAG) configured to search an external knowledge base and generate a RAG context.” is a mere instruction used to apply an exception. A recitation of the words "apply it" (or an equivalent) are mere elements to implement an abstract idea or other exception on a computer. (See MPEP 2106.05(f)).
As per dependent claims 3 and 15, the claims similarly recite the limitation of:
The additional limitation of “wherein the external knowledge base includes at least one content that is certified through a certification process of one or more certification processes.” is a mere instruction used to apply an exception. A recitation of the words "apply it" (or an equivalent) are mere elements to implement an abstract idea or other exception on a computer. (See MPEP 2106.05(f)).
As per dependent claim 4, the claim recites the limitation of:
The additional limitation of “wherein the certification process is selected based on a type of the at least one content.” is a mere instruction used to apply an exception. A recitation of the words "apply it" (or an equivalent) are mere elements to implement an abstract idea or other exception on a computer. (See MPEP 2106.05(f)).
As per dependent claim 5, the claim recites the limitation of:
The additional limitation of “wherein the certification process assigns a certification level to the at least one content based on at least one of the type of content and a number of steps taken during the certification process to certify the at least one content.” is a mere instruction used to apply an exception. A recitation of the words "apply it" (or an equivalent) are mere elements to implement an abstract idea or other exception on a computer. (See MPEP 2106.05(f)).
As per dependent claim 6, the claim recites the limitation of:
The additional limitation of “wherein the at least one content is weighted based on the assigned certification level.” is a mere instruction used to apply an exception. A recitation of the words "apply it" (or an equivalent) are mere elements to implement an abstract idea or other exception on a computer. (See MPEP 2106.05(f)).
As per dependent claims 7 and 16, the claims similarly recite the limitation of:
The additional limitation of “wherein the at least one input information includes one or more scoring guidelines, a user query, a RAG context of the user query, an expected output, and an expected RAG context of the user query.” is a mere instruction used to apply an exception. A recitation of the words "apply it" (or an equivalent) are mere elements to implement an abstract idea or other exception on a computer. (See MPEP 2106.05(f)).
As per dependent claims 8 and 17, the claims similarly recite the limitation of:
The additional limitation of “wherein the one or more scoring guidelines includes at least one of measuring an accuracy of the at least one output relative to the RAG context, measuring a relevancy of the at least one output to the user query, measuring an accuracy of the at least one output relative to the user query, measuring an accuracy of the RAG context to the expected RAG context, and measuring a relevancy of the RAG context to the user query.” is a mere instruction used to apply an exception. A recitation of the words "apply it" (or an equivalent) are mere elements to implement an abstract idea or other exception on a computer. (See MPEP 2106.05(f)).
As per dependent claims 9 and 18, the claims similarly recite the limitation of:
The additional limitation of “wherein the RAG context of the user query is received from a RAG.” is a mere instruction used to apply an exception. A recitation of the words "apply it" (or an equivalent) are mere elements to implement an abstract idea or other exception on a computer. (See MPEP 2106.05(f)).
As per dependent claims 10 and 19, the claims similarly recite the limitation of:
The additional limitation of “wherein the method further comprises outputting the output score rationale with the output score and the at least one output.” is a mere instruction used to apply an exception. A recitation of the words "apply it" (or an equivalent) are mere elements to implement an abstract idea or other exception on a computer. (See MPEP 2106.05(f)).
As per dependent claim 11, the claim recites the limitation of:
The additional limitation of “wherein outputting the at least one output and the output score comprises displaying the at least one output and the output score in a graphical user interface (GUI) on a display.” is a mere instruction used to apply an exception. A recitation of the words "apply it" (or an equivalent) are mere elements to implement an abstract idea or other exception on a computer. (See MPEP 2106.05(f)).
As per dependent claim 12, the claim recites the limitation of:
The additional limitation of “wherein the output score includes a plurality of output scores.” is a mere instruction used to apply an exception. A recitation of the words "apply it" (or an equivalent) are mere elements to implement an abstract idea or other exception on a computer. (See MPEP 2106.05(f)).
As per dependent claim 16, the claim recites the limitation of:
The additional limitation of “wherein the output score includes a plurality of output scores.” is a mere instruction used to apply an exception. A recitation of the words "apply it" (or an equivalent) are mere elements to implement an abstract idea or other exception on a computer. (See MPEP 2106.05(f)).
Therefore, claims 1-20 do not integrate the recited abstract ideas into a practical application.
Step 2B: Claim provides an Inventive Concept?
With respect to the limitations identified as insignificant extra-solution activity above the conclusions are carried over, and both the “receiving …; output …; and inputting … .” are well-understood, routine, and conventional operations.
For support as being well-understood, routine, and conventional for “receiving …; output …; and inputting … .” as noted by the courts is well understood routine and conventional, see MPEP 2106.05(d)(ii) “i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); … buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network);” and/or MPEP 2106.05(d)(ii) “iv. Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93;”, and/or MPEP 2106.05(d)(II) “iii. Ultramercial, 772 F.3d at 716, 112 USPQ2d at 1755 (updating an activity log);”.
Looking at the limitations in combination and the claim as a whole does not change this conclusion and the claim is ineligible.
Therefore, the claims 1-20 are not patent eligible.
Claim Rejections - 35 USC § 102
6. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
7. Claims 1-2, 7-8, 10 and 12 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Upadhyay et al. (US 20250053876 A1).
As per claim 1, Upadhyay teaches a method (i.e. “method”; fig. 1, para. [0021]) comprising:
receiving at least one input information (i.e. “receiving a runtime prompt S310”; fig. 3, para. [0021]. Further, i.e. “Prompts function as inputs to candidate models 200, and can also function as inputs to the router 100, as inputs to the reward model 300”; para. [0034]);
receiving at least one output from a language model (i.e. “determining a response S350 for the runtime prompt using the selected candidate model. The method can function to select the best model (e.g., transformer, large language model, etc.) to use for a given prompt, wherein the selected model 210 is then used to determine a response to the prompt.”; fig.3, para. [0021], [0133]; Examiner note: using a BRI the receiving at least one output is interpreted as the determining a response. The language model is interpreted as the selected candidate model);
inputting the at least one input information and the at least one output into a scoring language model configured to score the at least one output based on the at least one input information to yield an output score (i.e. “the reward model 300 is configured to directly calculate the target performance score for the prompt-response pair (e.g., example shown in FIG. 5A).”; fig. 5a, para. [0085]. Further, i.e. “the reward model 300 can be trained to predict the target performance score, given the prompt, response, and optionally the candidate model identifier as inputs.”; fig. 6a, para. [0086]; Examiner note: using a BRI the to score is interpreted as the to directly calculate. Using a BRI the inputting the at least one input information and the at least one output is interpreted as the prompt, response, and optionally the candidate model identifier as inputs. Using a BRI the scoring language model is interpreted as the reward model 300. Using a BRI the at least one output based on the at least one input information is interpreted as the prompt-response pair. The output score is interpreted as the target performance score); and
outputting the at least one output from the language model and the output score (i.e. “A result can optionally be returned to the prompter 10 (e.g., directly, via the system, etc.). A result can include a response from the selected candidate model, an identifier for a selected candidate model, a set of ranked candidate models 200, a message (e.g., “no suitable model could be found”), predicted performance scores, performance scores determined from the reward model 300,”; para. [0160]; Examiner note: using a BRI the outputting is interpreted as the result can optionally be returned).
As per claim 2, Upadhyay teaches all the limitations as discussed in claim 1 above.
Additionally, Upadhyay teaches wherein the language model uses a retrieval-augmented generation (RAG) configured to search an external knowledge base and generate a RAG context (i.e. “the prompt can be used to retrieve relevant data from a database of information, wherein the retrieved data and optionally the prompt can be provided to the selected candidate model for response generation (e.g., retrieval-augmented generation (RAG) using the selected candidate model).”; para. [0037]; Examiner note: using a BRI the to search an external knowledge base is interpreted as the retrieve relevant data from the database of information. Using a BRI the generate a RAG context is interpreted as the response generation (e.g., retrieval-augmented generation (RAG)).
As per claim 7, Upadhyay teaches all the limitations as discussed in claim 1 above.
Additionally, Upadhyay teaches wherein the at least one input information includes one or more scoring guidelines, a user query, a RAG context of the user query, an expected output, and an expected RAG context of the user query (i.e. “The reward model 300 can optionally additionally accept additional information as inputs, such as prompter identity, prompt context (e.g., data representing prior prompts and/or chosen candidate model responses), model status, model size, model settings (e.g., used to obtain the result, such as temperature, top K sampling, top P sampling, maximum length, etc.), language classification, and any other suitable information.”; para. [0088]).
As per claim 8, Upadhyay teaches all the limitations as discussed in claim 7 above.
Additionally, Upadhyay teaches wherein the one or more scoring guidelines includes at least one of measuring an accuracy of the at least one output relative to the RAG context, measuring a relevancy of the at least one output to the user query, measuring an accuracy of the at least one output relative to the user query, measuring an accuracy of the RAG context to the expected RAG context, and measuring a relevancy of the RAG context to the user query (i.e. “Performance scores preferably measure accuracy but can additionally or alternatively measure tone, similarity, precision, recall, precision and recall (e.g., an F1 score), instruction-following, specificity, robustness, fairness, user satisfaction (e.g., estimated based on the runtime prompt and feedback from a user for similar prompts), coverage of a domain associated with the runtime prompt, response comprehension level, harm (e.g., offensiveness, politeness, legality), logical consistency, relevance to prompt,”; para. [0056]).
As per claim 10, Upadhyay teaches all the limitations as discussed in claim 1 above.
Additionally, Upadhyay teaches wherein the scoring language model further generates an output score rationale, and wherein the method further comprises outputting the output score rationale with the output score and the at least one output (i.e. “the human evaluator assigns a target performance score (e.g., score on an absolute scale) to the response for one or more criteria (e.g., relevance, fluency, coherence, etc.).”; para. [0106]; Examiner note: using a BRI the output score rationale is interpreted as the criteria (e.g., relevance, fluency, coherence, etc).
As per claim 12, Upadhyay teaches all the limitations as discussed in claim 1 above.
Additionally, Upadhyay teaches wherein the output score includes a plurality of output scores (i.e. “the reward model 300 or set of reward models are configured to output multiple target performance scores”; para. [0089]).
Claim Rejections - 35 USC § 103
8. In the event the determination of the status of the application as subject to AIA 35 U.S.C. § 102 and § 103 (or as subject to pre-AIA 35 U.S.C. § 102 and § 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section § 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under pre-AIA 35 U.S.C. § 103(a) are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
9. Claims 11, 13, 14, 16-17 and 19 are rejected under 35 U.S.C. § 103 as being unpatentable over Upadhyay et al. (US 20250053876 A1) in view of Roy et al. (US 20180068016 A1).
As per claim 11, Upadhyay teaches all the limitations as discussed in claim 1 above.
However, it is noted that the prior art of Upadhyay does not explicitly teach “wherein outputting the at least one output and the output score comprises displaying the at least one output and the output score in a graphical user interface (GUI) on a display.”
On the other hand, in the same field of endeavor, Roy teaches wherein outputting the at least one output and the output score comprises displaying the at least one output and the output score in a graphical user interface (GUI) on a display (i.e. “the generated evaluation report is rendered on a user interface displayed on the display screen of the first user-computing device 102 and/or the second user-computing device 104…. The generated evaluation report comprises the response evaluation, i.e., the determined score and the generated feedback pertaining to each second response”; para. [0100]; Examiner note: using a BRI the output is interpreted as the generated feedback pertaining to each second score. The output score is the determined score).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Roy that teaches a method and a system for response evaluation of users from electronic documents into the prior art of Upadhyay that teaches a new and useful model in the machine learning. Additionally, this can yield the Average Improvement in Quality (AIQ), which is computed by averaging the performance of the routing system at different cost levels.
The motivation for doing so would be to improve computer assisted assessment (CAA) technique for evaluation of free-text responses with enhanced quality and accuracy (Roy, para. [0003]).
As per claim 13, Upadhyay teaches a system (i.e. “system”; para. [0033]) comprising:
a language model in communication with a user interface (i.e. “Each candidate model can be executed by the user (e.g., on a user device),”; para. [0041]) and configured to
receive a user query as input (i.e. “Receiving a runtime prompt S310 functions to receive a runtime prompt from a prompter 10 (e.g., a service, a user, a user device, etc.).”; para. [0133], [0134]) and
to output an output based on the user query (i.e. “determining a response S350.”; para. [0133]);
a scoring language model in communication with the language model and the user interface (i.e. “candidate model provider (e.g., for metadata determined by a metadata model 30 corresponding to a candidate model, for performance scores determined by a candidate model-specific score prediction model 112, etc.), a user device (e.g., for latency), and/or any other suitable entity.”; para. [0053]), the scoring model configured to:
receive, as input, the output and at least one input information (i.e. “given the prompt, response, and optionally the candidate model identifier as inputs.”; fig. 6a, para. [0086]);
score the output based on the at least one input information (i.e. “calculate the target performance score for the prompt-response pair (e.g., example shown in FIG. 5A).”; fig. 5a, para. [0085]); and
yield an output score (i.e. “predict the target performance score”; fig. 6a, para. [0086]);
However, it is noted that the prior art of Upadhyay does not explicitly teach “the user interface configured to display the output and the output score.”
On the other hand, in the same field of endeavor, Roy teaches the user interface configured to display the output and the output score (i.e. “the generated evaluation report is rendered on a user interface displayed on the display screen of the first user-computing device 102 and/or the second user-computing device 104…. The generated evaluation report comprises the response evaluation, i.e., the determined score and the generated feedback pertaining to each second response”; para. [0100]; Examiner note: using a BRI the output is interpreted as the generated feedback pertaining to each second score. The output score is the determined score).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Roy that teaches a method and a system for response evaluation of users from electronic documents into the prior art of Upadhyay that teaches a new and useful model in the machine learning. Additionally, this can yield the Average Improvement in Quality (AIQ), which is computed by averaging the performance of the routing system at different cost levels.
The motivation for doing so would be to improve computer assisted assessment (CAA) technique for evaluation of free-text responses with enhanced quality and accuracy (Roy, para. [0003]).
As per claim 14, Upadhyay teaches all the limitations as discussed in claim 13 above.
Additionally, Upadhyay teaches wherein the language model uses a retrieval-augmented generation (RAG) configured to search an external knowledge base and generate a RAG context (i.e. “the prompt can be used to retrieve relevant data from a database of information, wherein the retrieved data and optionally the prompt can be provided to the selected candidate model for response generation (e.g., retrieval-augmented generation (RAG) using the selected candidate model).”; para. [0037]; Examiner note: using a BRI the to search an external knowledge base is interpreted as the retrieve relevant data from the database of information. Using a BRI the generate a RAG context is interpreted as the response generation (e.g., retrieval-augmented generation (RAG)).
As per claim 16, Upadhyay teaches all the limitations as discussed in claim 13 above.
Additionally, Upadhyay teaches wherein the at least one input information includes one or more scoring guidelines, a user query, a RAG context of the user query, an expected output, and an expected RAG context of the user query (i.e. “The reward model 300 can optionally additionally accept additional information as inputs, such as prompter identity, prompt context (e.g., data representing prior prompts and/or chosen candidate model responses), model status, model size, model settings (e.g., used to obtain the result, such as temperature, top K sampling, top P sampling, maximum length, etc.), language classification, and any other suitable information.”; para. [0088]).
As per claim 17, Upadhyay teaches all the limitations as discussed in claim 16 above.
Additionally, Upadhyay teaches wherein the one or more scoring guidelines includes at least one of measuring an accuracy of the at least one output relative to the RAG context, measuring a relevancy of the at least one output to the user query, measuring an accuracy of the at least one output relative to the user query, measuring an accuracy of the RAG context to the expected RAG context, and measuring a relevancy of the RAG context to the user query (i.e. “Performance scores preferably measure accuracy but can additionally or alternatively measure tone, similarity, precision, recall, precision and recall (e.g., an F1 score), instruction-following, specificity, robustness, fairness, user satisfaction (e.g., estimated based on the runtime prompt and feedback from a user for similar prompts), coverage of a domain associated with the runtime prompt, response comprehension level, harm (e.g., offensiveness, politeness, legality), logical consistency, relevance to prompt,”; para. [0056]).
As per claim 19, Upadhyay teaches all the limitations as discussed in claim 16 above.
Additionally, Upadhyay teaches wherein the scoring language model further generates an output score rationale, and wherein the method further comprises outputting the output score rationale with the output score and the at least one output output (i.e. “the human evaluator assigns a target performance score (e.g., score on an absolute scale) to the response for one or more criteria (e.g., relevance, fluency, coherence, etc.).”; para. [0106]; Examiner note: using a BRI the output score rationale is interpreted as the criteria (e.g., relevance, fluency, coherence, etc).
10. Claims 3-4 are rejected under 35 U.S.C. § 103 as being unpatentable over Upadhyay et al. (US 20250053876 A1) in view of Doran (US 20070240195 A1).
As per claim 3, Upadhyay teaches all the limitations as discussed in claim 2 above.
However, it is noted that the prior art of Upadhyay does not explicitly teach “wherein the external knowledge base includes at least one content that is certified through a certification process of one or more certification processes.”
On the other hand, in the same field of endeavor, Doran teaches wherein the external knowledge base includes at least one content that is certified through a certification process of one or more certification processes (i.e. “as part of the certification process, the content-certifier will validate the digital signature. For instance, the content-certifier may compute a new digital signature for the received content and may compare it to the digital signature indicated in the catalog-entry (which the content-supplier provided to the access server). Further, the content-certifier will evaluate the content itself to ensure that it has at least one expected characteristic.”; para. [0010]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Doran that teaches a publication of content on a communication network for access by others into the prior art of Upadhyay that teaches a new and useful model in the machine learning. Additionally, this can yield the Average Improvement in Quality (AIQ), which is computed by averaging the performance of the routing system at different cost levels.
The motivation for doing so would be to provide a content file that does not contain bogus, harmful, inoperable, unauthorized, or otherwise unacceptable data, as this can cause security issues for a user (Doran, para. [0004]).
As per claim 4, Upadhyay teaches all the limitations as discussed in claim 3 above.
However, it is noted that the prior art of Upadhyay does not explicitly teach “wherein the certification process is selected based on a type of the at least one content.”
On the other hand, in the same field of endeavor, Doran teaches wherein the certification process is selected based on a type of the at least one content (i.e. “this evaluation process can take various forms, depending on the type of content at issue”; para. [0044]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Doran that teaches a publication of content on a communication network for access by others into the prior art of Upadhyay that teaches a new and useful model in the machine learning. Additionally, this can yield the Average Improvement in Quality (AIQ), which is computed by averaging the performance of the routing system at different cost levels.
The motivation for doing so would be to provide a content file that does not contain bogus, harmful, inoperable, unauthorized, or otherwise unacceptable data, as this can cause security issues for a user (Doran, para. [0004]).
11. Claim 15 is rejected under 35 U.S.C. § 103 as being unpatentable over Upadhyay et al. (US 20250053876 A1) in view of Roy et al. (US 20180068016 A1) in further view Doran (US 20070240195 A1).
As per claim 15, Upadhyay and Roy teach all the limitations as discussed in claim 14 above.
However, it is noted that the combination of the prior arts of Upadhyay and Roy do not explicitly teach “wherein the external knowledge base includes at least one content that is certified through a certification process of one or more certification processes.”
On the other hand, in the same field of endeavor, Doran teaches wherein the external knowledge base includes at least one content that is certified through a certification process of one or more certification processes (i.e. “as part of the certification process, the content-certifier will validate the digital signature. For instance, the content-certifier may compute a new digital signature for the received content and may compare it to the digital signature indicated in the catalog-entry (which the content-supplier provided to the access server). Further, the content-certifier will evaluate the content itself to ensure that it has at least one expected characteristic.”; para. [0010]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Doran that teaches a publication of content on a communication network for access by others into the combination the prior arts of Upadhyay that teaches a new and useful model in the machine learning, and Roy that teaches a method and a system for response evaluation of users from electronic documents. Additionally, this can yield the Average Improvement in Quality (AIQ), which is computed by averaging the performance of the routing system at different cost levels.
The motivation for doing so would be to provide a content file that does not contain bogus, harmful, inoperable, unauthorized, or otherwise unacceptable data, as this can cause security issues for a user (Doran, para. [0004]).
12. Claims 5-6 are rejected under 35 U.S.C. § 103 as being unpatentable over Upadhyay et al. (US 20250053876 A1) in view of Doran (US 20070240195 A1) in further view Coulthard et al. (US 20020059364 A1).
As per claim 5, Upadhyay and Doran teach all the limitations as discussed in claim 4 above.
However, it is noted that the combination of the prior arts of Upadhyay and Doran do not explicitly teach “wherein the certification process assigns a certification level to the at least one content based on at least one of the type of content and a number of steps taken during the certification process to certify the at least one content.”
On the other hand, in the same field of endeavor, Coulthard teaches wherein the certification process assigns a certification level to the at least one content based on at least one of the type of content and a number of steps taken during the certification process to certify the at least one content (i.e. “one possible certification scheme 152 uses different certification levels. As shown, the levels include site-wide certification 154, class certification 156-158, and individual certification 160-164. Each defined certification can include its own granting criteria. For example, to obtain site-wide certification, content must first receive certification from the Legal Department 156, the Marketing Department 158, and the company's CEO 164. Similarly, to receive Legal Department certification 156, at least two members of the legal department and a text-scanning program that looks for certain phrases must approve the content.”; para. [0053]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Coulthard that teaches storing verification information corresponding to certified content at a first computer and receiving a verification request corresponding to content from a second computer into the combination the prior arts of Upadhyay that teaches a new and useful model in the machine learning, and Doran that teaches a publication of content on a communication network for access by others. Additionally, this can yield the Average Improvement in Quality (AIQ), which is computed by averaging the performance of the routing system at different cost levels.
The motivation for doing so would be to prevent potential liability or embarrassment; consequently, organizations have begun to err on the side of safety and withhold information from Internet publication (Coulthard, para. [0002]).
As per claim 6, Upadhyay and Doran teach all the limitations as discussed in claim 5 above.
However, it is noted that the combination of the prior arts of Upadhyay and Doran do not explicitly teach “wherein the at least one content is weighted based on the assigned certification level.”
On the other hand, in the same field of endeavor, Coulthard teaches wherein the at least one content is weighted based on the assigned certification level (i.e. “For example, to obtain site-wide certification, content must first receive certification from the Legal Department 156, the Marketing Department 158, and the company's CEO 164. Similarly, to receive Legal Department certification 156, at least two members of the legal department and a text-scanning program that looks for certain phrases must approve the content. As shown, the certification criteria can include different levels of abstraction.”; para. [0053]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Coulthard that teaches storing verification information corresponding to certified content at a first computer and receiving a verification request corresponding to content from a second computer into the combination the prior arts of Upadhyay that teaches a new and useful model in the machine learning, and Doran that teaches a publication of content on a communication network for access by others. Additionally, this can yield the Average Improvement in Quality (AIQ), which is computed by averaging the performance of the routing system at different cost levels.
The motivation for doing so would be to prevent potential liability or embarrassment; consequently, organizations have begun to err on the side of safety and withhold information from Internet publication (Coulthard, para. [0002]).
13. Claim 9 is rejected under 35 U.S.C. § 103 as being unpatentable over Upadhyay et al. (US 20250053876 A1) in view of Gaddam (US 20250252445 A1).
As per claim 9, Upadhyay teaches all the limitations as discussed in claim 8 above.
However, it is noted that the prior arts of Upadhyay does not explicitly teach “wherein the RAG context of the user query is received from a RAG.”
On the other hand, in the same field of endeavor, Gaddam teaches wherein the RAG context of the user query is received from a RAG (i.e. “obtaining, using retrieval-augmented generation (RAG), contextual information corresponding to the query”; para. [0077]; Examiner note: using a BRI the RAG context is interpreted as the contextual information. Using a BRI the received from a RAG is the obtaining, using retrieval-augmented generation (RAG)).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Gaddam that teaches techniques may generally be used for chatbot-based fraud assistance into the prior arts of Upadhyay that teaches a new and useful model in the machine learning. Additionally, this can yield the Average Improvement in Quality (AIQ), which is computed by averaging the performance of the routing system at different cost levels.
The motivation for doing so would be to use chatbot-based fraud assistance, as existing detection systems face difficulty in effectively addressing the growing volume and complexity of fraud-related customer interactions (Gaddam, para. [0002]).
14. Claims 18 and 20 are rejected under 35 U.S.C. § 103 as being unpatentable over Upadhyay et al. (US 20250053876 A1) in view of Roy et al. (US 20180068016 A1) in further view of Gaddam (US 20250252445 A1).
As per claim 18, Upadhyay and Roy teaches all the limitations as discussed in claim 8 above.
However, it is noted that the combination of the prior arts of Upadhyay and Roy do not explicitly teach “wherein the RAG context of the user query is received from a RAG.”
On the other hand, in the same field of endeavor, Gaddam teaches wherein the RAG context of the user query is received from a RAG (i.e. “obtaining, using retrieval-augmented generation (RAG), contextual information corresponding to the query”; para. [0077]; Examiner note: using a BRI the RAG context is interpreted as the contextual information. Using a BRI the received from a RAG is the obtaining, using retrieval-augmented generation (RAG)).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Gaddam that teaches techniques may generally be used for chatbot-based fraud assistance into the combination of prior arts of Upadhyay that teaches a new and useful model in the machine learning, and Roy that teaches a method and a system for response evaluation of users from electronic documents. Additionally, this can yield the Average Improvement in Quality (AIQ), which is computed by averaging the performance of the routing system at different cost levels.
The motivation for doing so would be to use chatbot-based fraud assistance, as existing detection systems face difficulty in effectively addressing the growing volume and complexity of fraud-related customer interactions (Gaddam, para. [0002]).
As per claim 20, Upadhyay teaches a system (i.e. “system”; para. [0033]) comprising:
a language model in communication with a user interface and a RAG (i.e. “Each candidate model can be executed by the user (e.g., on a user device),”; para. [0041]. Further, i.e. “the retrieved data and optionally the prompt can be provided to the selected candidate model for response generation (e.g., retrieval-augmented generation (RAG)”; para. [0043]));
to output an output based on the user query (i.e. “determining a response S350.”; para. [0133]) and the RAG context (i.e. “response generation (e.g., retrieval-augmented generation (RAG)”; para. [0037]);
a scoring language model in communication with the language model and the user interface (i.e. “candidate model provider (e.g., for metadata determined by a metadata model 30 corresponding to a candidate model, for performance scores determined by a candidate model-specific score prediction model 112, etc.), a user device (e.g., for latency), and/or any other suitable entity.”; para. [0053]),, the scoring model configured to:
receive, as input, the output, an expected output, the user query, the RAG context, an expected RAG context, and one or more scoring guidelines (i.e. “given the prompt, response, and optionally the candidate model identifier as inputs.”; fig. 6a, para. [0086]);
score the output based on the one or more scoring guidelines and the expected output, the user query, the RAG context, and the expected RAG context calculate the target performance score for the prompt-response pair (e.g., example shown in FIG. 5A).”; fig. 5a, para. [0085]); and
yield an output score (i.e. “predict the target performance score”; fig. 6a, para. [0086]) and a score rationale assigns a target performance score (e.g., score on an absolute scale) to the response for one or more criteria (e.g., relevance, fluency, coherence, etc.)”; para. [0106];
However, it is noted that the prior arts of Upadhyay does not explicitly teach “the user interface configured to display the output, the output score, and the score rationale.”
On the other hand, in the same field of endeavor, Roy teaches the user interface configured to display the output, the output score, and the score rationale (i.e. “the generated evaluation report is rendered on a user interface displayed on the display screen of the first user-computing device 102 and/or the second user-computing device 104…. The generated evaluation report comprises the response evaluation, i.e., the determined score and the generated feedback pertaining to each second response”; para. [0100]; Examiner note: using a BRI the output is interpreted as the generated feedback pertaining to each second score. The output score is the determined score).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Roy that teaches a method and a system for response evaluation of users from electronic documents into the prior arts of Upadhyay that teaches a new and useful model in the machine learning. Additionally, this can yield the Average Improvement in Quality (AIQ), which is computed by averaging the performance of the routing system at different cost levels.
The motivation for doing so would be to improve computer assisted assessment (CAA) technique for evaluation of free-text responses with enhanced quality and accuracy (Roy, para. [0003]).
However, it is noted that the combination of the prior arts of Upadhyay and Roy do not explicitly teach “the language model configured to receive a user query from the user interface and a RAG context from the RAG as input.”
On the other hand, in the same field of endeavor, Gaddam teaches the language model configured to receive a user query from the user interface and a RAG context from the RAG as input (i.e. “obtaining, using retrieval-augmented generation (RAG), contextual information corresponding to the query”; para. [0077]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Gaddam that teaches techniques may generally be used for chatbot-based fraud assistance into the combination of prior arts of Upadhyay that teaches a new and useful model in the machine learning, and Roy that teaches a method and a system for response evaluation of users from electronic documents. Additionally, this can yield the Average Improvement in Quality (AIQ), which is computed by averaging the performance of the routing system at different cost levels.
The motivation for doing so would be to use chatbot-based fraud assistance, as existing detection systems face difficulty in effectively addressing the growing volume and complexity of fraud-related customer interactions (Gaddam, para. [0002]).
Prior Art of Record
15. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Crabtree et al. (US 20260148042 A1), teaches a unified framework integrating an adaptive elastic funnel (AEF).
Subramaniyan et al. (US 20260133812 A1), teaches artificial intelligence (AI) and, more specifically, to use of multi AI agent systems to create highly customized special purpose software applications.
Xue et al. (US 20250363908 A1), teaches a question generation system, and a question generation method.
Huang et al. (US 20250165803 A1), teaches a federated AI system that orchestrates the performance of inference operations using machine learning models of a model chain to minimize the computational complexity involved in determining output to present to software service users.
Piccolo et al. (US 20240386362 A1), teaches systems and methods to asynchronously vet, grade, rank and certify technical candidates for hiring.
Lazaridou et al. (US 20230244934 A1), teaches processing inputs using neural networks.
Conclusion
16. Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTONIO CAIA DO whose telephone number is (469)295-9251. The examiner can normally be reached on Monday - Friday / 06:30 to 16:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ng, Amy can be reached on (571) 270-1698. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTONIO J CAIA DO/
Examiner, Art Unit 2164
