DETAILED ACTION
This communication is in response to the Amendments and Arguments filed on 01/29/2026.
Claims 3 and 16 have been canceled by the Applicant.
Claim(s) 1-20 are pending and have been examined. Hence, this action has been made FINAL.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments and Amendments
Amendments to the claims by the Applicant have been considered and addressed below.
With respect to the 35 USC § 101 and 103 rejections, the Applicant provides several arguments in which the Examiner will respond accordingly, below.
35 USC § 101 rejection(s)
Arguments pages 9-12 of the Remarks filed on 01/29/2026
Examiner response to Arguments:
Applicant’s arguments, with respect to the rejection(s) of independent claim(s) 1, 14, and 20 under 35 USC 101 have been fully considered but are not persuasive.
Please see detailed analysis below (Prong Two) for more details on how the Examiner understands the independent claims do not recite additional elements that integrate the judicial exception into a practical application. Hence, not qualifying as patent eligible subject matter under 35 U.S.C. § 101.
Please refer to MPEP 2106.04(II): Eligibility Step 2A: Whether a Claim is Directed to a Judicial Exception: (A) Step 2A is a Two-Prong Inquiry:
(1) Prong One:
Prong One asks does the claim recite an abstract idea, law of nature, or natural phenomenon? In Prong One examiners evaluate whether the claim recites a judicial exception, i.e. whether a law of nature, natural phenomenon, or abstract idea is set forth or described in the claim. While the terms "set forth" and "described" are thus both equated with "recite", their different language is intended to indicate that there are two ways in which an exception can be recited in a claim. For instance, the claims in Diehr, 450 U.S. at 178 n. 2, 179 n.5, 191-92, 209 USPQ at 4-5 (1981), clearly stated a mathematical equation in the repetitively calculating step, and the claims in Mayo, 566 U.S. 66, 75-77, 101 USPQ2d 1961, 1967-68 (2012), clearly stated laws of nature in the wherein clause, such that the claims "set forth" an identifiable judicial exception. Alternatively, the claims in Alice Corp., 573 U.S. at 218, 110 USPQ2d at 1982, described the concept of intermediated settlement without ever explicitly using the words "intermediated" or "settlement." […]
An example of a claim that recites a judicial exception is "A machine comprising elements that operate in accordance with F=ma." This claim sets forth the principle that force equals mass times acceleration (F=ma) and therefore recites a law of nature exception. Because F=ma represents a mathematical formula, the claim could alternatively be considered as reciting an abstract idea. Because this claim recites a judicial exception, it requires further analysis in Prong Two in order to answer the Step 2A inquiry. An example of a claim that merely involves, or is based on, an exception is a claim to "A teeter-totter comprising an elongated member pivotably attached to a base member, having seats and handles attached at opposing sides of the elongated member." This claim is based on the concept of a lever pivoting on a fulcrum, which involves the natural principles of mechanical advantage and the law of the lever. However, this claim does not recite these natural principles and therefore is not directed to a judicial exception (Step 2A: NO). Thus, the claim is eligible at Pathway B without further analysis.
From this analysis, in Step 2A, Prong One, the Examiner has evaluated the independent claims accordingly and determined that the amended independent claims as drafted indeed describe a judicial exception (i.e., an abstract idea), which represent a mental process (which can be performed by a human with pen and paper).
Similar to what was discussed in the Non-Final Rejection mailed on 10/30/2025, the limitations as drafted cover a human (mental process and/or mathematical concept).
More specifically, the claim(s) recitations of:
1. A system comprising:
one or more storage devices; and
processing circuitry in communication with the one or more storage devices, the processing circuitry configured to:
receive, from a Large Language Model (LLM), computer-generated text output as an answer to an inquiry originating from a computing device;
determine, independently from the LLM, a confidence score associated with the answer from the LLM based on an evaluation of one or more sources used by the LLM to generate the answer, wherein the evaluation includes interacting with the one or more sources over a network to determine whether each of the one or more sources are valid and trustworthy;
determine whether the confidence score associated with the answer satisfies a quality threshold; and
based on the confidence score associated with the answer satisfying the quality threshold:
generate an annotated answer including the answer and an indication of quality based on the evaluation of the one or more sources used by the LLM to generate the answer, and output, to the computing device, the annotated answer in response to the inquiry.
14. A method comprising:
[the limitations as in claim 1, above.]
20. Computer-readable storage media comprising instructions that, when executed, configure processing circuitry to:
[the limitations as in claim 1, above.]
Read on a human (e.g., mentally and/or using pen and paper):
Receiving text corresponding to an answer to a question from another human (i.e., the other human following a predetermined set of steps)
Assigning a score to said answer based on other written sources (i.e., different sources) and based on the determination on whether the sources are valid or trustworthy (i.e., book, magazine, scientific article, etc.);
Determining if the score satisfies a predetermined threshold;
Based on the score satisfying the threshold:
Write down the answer including a note or indication of quality
Show/display said answer (e.g., when asked by another human)
Please also refer to MPEP 2106.05(f)(2): Whether the claim invokes computers or other machinery merely as a tool to perform an existing process, and MPEP 2106.06(b): Clear Improvement to a Technology or to Computer Functionality.
Please refer to MPEP 2106.04(II): Eligibility Step 2A: Whether a Claim is Directed to a Judicial Exception: (A) Step 2A is a Two-Prong Inquiry:
(2) Prong Two:
Prong Two asks does the claim recite additional elements that integrate the judicial exception into a practical application? In Prong Two, examiners evaluate whether the claim as a whole integrates the exception into a practical application of that exception. If the additional elements in the claim integrate the recited exception into a practical application of the exception, then the claim is not directed to the judicial exception (Step 2A: NO) and thus is eligible at Pathway B. This concludes the eligibility analysis. If, however, the additional elements do not integrate the exception into a practical application, then the claim is directed to the recited judicial exception (Step 2A: YES), and requires further analysis under Step 2B (where it may still be eligible if it amounts to an ‘‘inventive concept’’). For more information on how to evaluate whether a judicial exception is integrated into a practical application, see MPEP § 2106.04(d)(2).
From this analysis, in Step 2A, Prong Two, the Examiner has evaluated the independent claims accordingly and determined that the amended independent claims as drafted that the claims as a whole do not include additional elements that integrate the exception into a practical application of that exception. (i.e., an abstract idea). As discussed in the Non-Final Rejection mailed on 10/30/2025:
This judicial exception is not integrated into a practical application because for example: claim 1 recites “a system,” “one or more storage devices,” “processing circuitry,” “computer-generated text output,” and “network” while claim 14 recites also “processing circuitry,” “computer-generated text output,” and “network” and claim 20 additionally recites “a computer-readable storage media and processing circuitry”. As an example, in [0100] of the as filed specification, it is disclosed: “[0100] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the terms “processor” or “processing circuitry” as used herein may each refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described…”. Therefore, a general-purpose computer or computing device is described and mainly used as an application thereof. Accordingly, these additional elements do not integrate the abstract idea into a practical idea because it does not impose any meaningful limits on practicing the abstract idea.
Please also refer to MPEP 2106.05(f)(2): Whether the claim invokes computers or other machinery merely as a tool to perform an existing process.
Finally, please refer to MPEP 2106.05(A): Relevant Considerations For Evaluating Whether Additional Elements Amount To An Inventive Concept
Limitations that the courts have found not to be enough to qualify as "significantly more" when recited in a claim with a judicial exception include:
i. Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, e.g., a limitation indicating that a particular function such as creating and maintaining electronic records is performed by a computer, as discussed in Alice Corp., 573 U.S. at 225-26, 110 USPQ2d at 1984 (see MPEP § 2106.05(f));
ii. Simply appending well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, e.g., a claim to an abstract idea requiring no more than a generic computer to perform generic computer functions that are well-understood, routine and conventional activities previously known to the industry, as discussed in Alice Corp., 573 U.S. at 225, 110 USPQ2d at 1984 (see MPEP § 2106.05(d));
From this analysis, in Step 2B, the Examiner has evaluated the independent claims accordingly and determined that the independent claims as drafted have limitations that the courts have found not to be enough to qualify as "significantly more" when recited in a claim with a judicial exception. Similar to what was discussed in the Non-Final Rejection mailed on 10/30/2025:
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of using a computer is listed as a general computing device as noted. The claim is not patent eligible.
In summary, the Examiner respectfully disagrees with the arguments above. Please refer to analysis above.
For more details, please refer to updated 35 U.S.C. § 101 rejections for claims 1-2, 4-15, and 17-20, below.
35 USC § 103 rejection(s)
Arguments pages 9-12 of the Remarks filed on 01/29/2026
Examiner’s Response to Arguments:
Applicant’s arguments with respect to independent claim(s) 1, 14, and 20 under 35 U.S.C. § 103 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Yoon et al. (US 20240428015 A1) and further in view of Gunaselara et al. (US 20210365500 A1) and Sager et al. (US 11561987 B1).
For more details, please refer to updated 35 U.S.C. § 103 rejections for claims 1-2, 4-15, and 17-20, below.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 1-2, 4-15, and 17-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. More specifically directed to the abstract idea grouping of: mental process and/or mathematical concept.
The independent claim(s) recite(s):
1. A system comprising:
one or more storage devices; and
processing circuitry in communication with the one or more storage devices, the processing circuitry configured to:
receive, from a Large Language Model (LLM), computer-generated text output as an answer to an inquiry originating from a computing device;
determine, independently from the LLM, a confidence score associated with the answer from the LLM based on an evaluation of one or more sources used by the LLM to generate the answer, wherein the evaluation includes interacting with the one or more sources over a network to determine whether each of the one or more sources are valid and trustworthy;
determine whether the confidence score associated with the answer satisfies a quality threshold; and
based on the confidence score associated with the answer satisfying the quality threshold:
generate an annotated answer including the answer and an indication of quality based on the evaluation of the one or more sources used by the LLM to generate the answer, and output, to the computing device, the annotated answer in response to the inquiry.
14. A method comprising:
[the limitations as in claim 1, above.]
20. Computer-readable storage media comprising instructions that, when executed, configure processing circuitry to:
[the limitations as in claim 1, above.]
This reads on a human (e.g., mentally and/or using pen and paper):
Receiving text corresponding to an answer to a question from another human (i.e., the other human following a predetermined set of steps)
Assigning a score to said answer based on other written sources (i.e., different sources) and based on the determination on whether the sources are valid or trustworthy (i.e., book, magazine, scientific article, etc.);
Determining if the score satisfies a predetermined threshold;
Based on the score satisfying the threshold:
Write down the answer including a note or indication of quality
Show/display said answer (e.g., when asked by another human)
This judicial exception is not integrated into a practical application because for example: claim 1 recites “a system,” “one or more storage devices,” “processing circuitry,” “computer-generated text output,” and “network” while claim 14 recites also “processing circuitry,” “computer-generated text output,” and “network” and claim 20 additionally recites “a computer-readable storage media and processing circuitry”. As an example, in [0100] of the as filed specification, it is disclosed: “[0100] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the terms “processor” or “processing circuitry” as used herein may each refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described…”. Therefore, a general-purpose computer or computing device is described and mainly used as an application thereof. Accordingly, these additional elements do not integrate the abstract idea into a practical idea because it does not impose any meaningful limits on practicing the abstract idea.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of using a computer is listed as a general computing device as noted. The claim is not patent eligible.
With respect to claims 2 and 15, the claim(s) recite:
2 and 15. The system/method of claims 1 and 14, wherein the processing circuitry is configured to, in response to a determination that each of the one or more sources used by the LLM to generate the answer are valid sources, provide feedback to the LLM indicating validity of the one or more sources.
This reads on a human (e.g., mentally and/or using pen and paper):
Providing feedback on validity (i.e., following a predetermined set of rules) of the written sources.
No additional limitations are present.
With respect to claims 4 and 17, the claim(s) recite:
4 and 17. The system/method of claims 1 and 14, wherein the processing circuitry is configured to determine whether the one or more sources used by the LLM to generate the answer are valid sources based on the evaluation of one or more source quality conditions, including:
any of the one or more sources listed on a blacklist of sources;
any of the one or more sources listed on a whitelist of sources;
any of the one or more sources corresponding to a deprecated source;
any of the one or more sources corresponding to a curated list of untrustworthy URLs;
any of the one or more sources corresponding to a curated list of trustworthy URLs;
any of the one or more sources derived from university research;
any of the one or more sources derived from a social media platform;
any of the one or more sources corresponding to a social media post;
inauthentic DNS information for any of the one or more sources;
inauthentic URL information for any of the one or more sources;
a historical age for any of the one or more sources; or
a comparison to a golden copy of answers maintained by the meta interface layer.
This reads on a human (e.g., mentally and/or using pen and paper):
Determining if the sources are valid or not (i.e., following a predetermined set of rules) by evaluating:
Different features from the written sources (e.g., part of a predefined list).
No additional limitations are present.
With respect to claim 5, the claim(s) recite:
5. The system of claim 1, wherein the processing circuitry is configured to, based on the confidence score associated with the answer failing to satisfy the quality threshold or a determination any of the one or more sources used by the LLM to generate the answer are invalid, output, to the computing device, a notification indicating that an answer to the inquiry will not be provided.
This reads on a human (e.g., mentally and/or using pen and paper):
Determining if the threshold not met that the written sources are invalid and write down a notification.
No additional limitations are present.
With respect to claims 6 and 18, the claim(s) recite:
6 and 18. The system/method of claims 1 and 14, wherein the processing circuitry is configured to, based on the confidence score associated with the answer failing to satisfy the quality threshold or a determination any of the one or more sources used by the LLM to generate the answer are invalid:
discard the answer;
re-submit the inquiry previously submitted by the computing device to the LLM on behalf of the computing device;
obtain new computer-generated text output from the LLM as a new answer to the inquiry; and
based on a new confidence score associated with the new answer satisfying the quality threshold:
generate a new annotated answer; and
output, to the computing device, the new annotated answer in response to the inquiry.
This reads on a human (e.g., mentally and/or using pen and paper):
Determining if the sources are valid or not (i.e., following a predetermined set of rules):
Ignoring the answer;
Analyzing the question again;
Obtaining new answer;
Generating or writing down the new answer;
And transmit (hand down) the written answer.
No additional limitations are present.
With respect to claim 7, the claim(s) recite:
7. The system of claim 1, wherein the processing circuitry is configured to:
obtain, from the computing device, user-input indicating a degree of usefulness of the annotated answer; and
provide, to the LLM, the user-input indicating the degree of usefulness of the annotated answer, wherein the user-input specifies at least one of:
a numerical score for the annotated answer;
a non-numerical user-rated assessment for the annotated answer;
a red color, a yellow color, or a green color for the annotated answer;
a thumbs-up or a thumbs-down indication for the annotated answer;
a high, medium, or low user-rated confidence for the annotated answer; or
a Boolean value indicating user-rated usefulness for the annotated answer.
This reads on a human (e.g., mentally and/or using pen and paper):
Receiving feedback from another human regarding how useful the answer is;
Assign a score to the answer.
No additional limitations are present.
With respect to claim 8, the claim(s) recite:
8. The system of claim 1, wherein to determine the confidence score, the processing circuitry is configured to:
provide as a first input to an artificial intelligence (AI) model, a golden copy of answers;
provide as a second input to the AI model, the computer-generated text output from the LLM as the answer to the inquiry; and
obtain from the AI model, the confidence score indicating probability the answer is accurate.
This reads on a human (e.g., mentally and/or using pen and paper):
Consider a copy of correct answers and predicted answers to determine a confidence score using a predetermined set of rules.
No additional limitations are present.
With respect to claim 9, the claim(s) recite:
9. The system of claim 1, wherein to determine the confidence score, the processing circuitry is configured to validate the computer-generated text output from the LLM against a selected source from the one or more sources used by the LLM to generate the answer based on one or more of:
text obtained from the selected source and compared with the computer-generated text output from the LLM;
audio content obtained from the selected source and compared with the computer-generated text output from the LLM; or
video content obtained from the selected source and compared with the computer-generated text output from the LLM.
This reads on a human (e.g., mentally and/or using pen and paper):
Validating answers by comparing to a written source and writing down the answer (e.g., as text).
No additional limitations are present.
With respect to claim 10, the claim(s) recite:
10. The system of claim 1, wherein, to generate the annotated answer, the processing circuitry is configured to annotate the answer from the LLM with one or more of:
annotations indicating source validity for the one or more sources used by the LLM to generate at least one part of the answer;
annotations indicating source validity for the one or more sources used by the LLM to generate each of multiple parts of the answer;
annotations indicating an overall validity percentage for the multiple parts of the answer;
annotations indicating the confidence score for at least one of the multiple parts of the answer;
annotations indicating an overall confidence score for the multiple parts of the answer; or
one or more of citations or links to validated sources used by the LLM to generate the multiple parts of the answer.
This reads on a human (e.g., mentally and/or using pen and paper):
Annotating the answers with details like a confidence score.
No additional limitations are present.
With respect to claim 11, the claim(s) recite:
11. The system of claim 1, wherein the processing circuitry is configured to:
obtain a user-profile associated with the computing device having originated the inquiry; and
update the annotated answer, prior to output to the computing device, with information derived from the user-profile.
This reads on a human (e.g., mentally and/or using pen and paper):
Obtaining details or context specific to the human asking the question and annotating the answer with this information.
No additional limitations are present.
With respect to claim 12, the claim(s) recite:
12. The system of claim 1, wherein the processing circuitry is configured to:
receive the inquiry from the computing device; and
submit the inquiry to the LLM.
This reads on a human (e.g., mentally and/or using pen and paper):
Receiving and using received questions to get an answer (i.e., using a predetermined set of rules.)
No additional limitations are present.
With respect to claim 13, the claim(s) recite:
13. The system of claim 12, wherein the inquiry is a first inquiry, and wherein the processing circuitry is configured to:
cache, using a database system, the first inquiry and the computer-generated text output from the LLM as the answer to the first inquiry when the confidence score associated with the answer satisfies the quality threshold;
receive a second inquiry from a second computing device;
in response to a determination that the second inquiry matches the first inquiry cached using the database system, obtain the answer to the first inquiry cached using the database system as the answer to the second inquiry without submitting the second inquiry to the LLM; and
output the answer in response to the second inquiry to the second computing device.
This reads on a human (e.g., mentally and/or using pen and paper):
Receiving question from another human;
Obtaining details or context specific to the human asking the question or saved data and annotating the answer with this information;
Receiving a second question;
Comparing and/or matching the first and second question and write them down;
Responding (e.g., writing down) the question(s).
No additional limitations are present.
With respect to claim 19, the claim(s) recite:
19. The method of claim 14, wherein the inquiry is a first inquiry, and wherein the method further comprises:
receiving, by the processing circuitry, the first inquiry from the computing device;
submitting the first inquiry to the LLM;
caching, using a database system, the first inquiry and the computer-generated text output from the LLM as the answer to the first inquiry when the confidence score associated with the answer satisfies the quality threshold;
receiving a second inquiry from a second computing device;
in response to determining that the second inquiry matches the first inquiry cached using the database system, obtaining the answer to the first inquiry cached using the database system as the answer to the second inquiry without submitting the second inquiry to the LLM; and
outputting, by the processing circuitry and for display to the second computing device, the answer in response to the second inquiry.
This reads on a human (e.g., mentally and/or using pen and paper):
Receiving question from another human;
Obtaining details or context specific to the human asking the question or saved data and annotating the answer with this information;
Receiving a second question;
Comparing and/or matching the first and second question and write them down;
Responding (e.g., writing down) the question(s).
No additional limitations are present.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-2, 10-15, and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yoon et al. (US 20240428015 A1) and further in view of Gunaselara et al. (US 20210365500 A1) and Sager et al. (US 11561987 B1).
As to independent claim 1, Yoon et al. teaches:
1. A system (see ¶ [0003]: “Aspects of the disclosure are directed to methods, systems, and computer readable media for adaptation with self-evaluation to improve selective prediction in large language models (LLMs), generally referred to as ASPIRE…”) comprising:
one or more storage devices (see ¶ [0008]: “Another aspect of the disclosure provides for a system including: one or more processors; and one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations...”); and
processing circuitry in communication with the one or more storage devices, the processing circuitry (see ¶ [0008] citations as in limitation above.) configured to:
receive, from a Large Language Model (LLM), computer-generated text output as an answer to an inquiry originating from a computing device (see ¶ [0003, 0022, and 0027]: “[0003] Aspects of the disclosure are directed to methods, systems, and computer readable media for adaptation with self-evaluation to improve selective prediction in large language models (LLMs), generally referred to as ASPIRE. ASPIRE includes training LLMs on a portion of training data from a question answering task to learn self-evaluation, e.g., learn to distinguish whether a generated answer is correct or not. ASPIRE further includes a selection score that combines a likelihood of that generated answer being correct with a self-evaluation score for selective prediction. ASPIRE demonstrates improved selective prediction performance with less computational cost. [0022] FIG. 1 depicts a block diagram of LLMs outputs 100 without selective prediction and with selective prediction according to aspects of the disclosure. When a question is presented, LLMs with selective prediction 120 can provide an answer with a confidence or selection score while LLMs without selective prediction 110 may just provide the answer. For example, for a question like ‘What is the capital of California?’, LLM models 112 without selective prediction might provide an incorrect answer like ‘Los Angeles’ instead of the correct answer, ‘Sacramento.’ Without selective prediction, LLMs 112 may directly output incorrect answers. With selective prediction, LLMs 114 can output a low selection score 116 along with the wrong answer. The low selection score 116 can provide a warning not to trust the answer. [0027] … The query data 202 and/or training data 204 can further be provided as input through a user interface on a client computing device coupled to the ASPIRE system 200.”);
determine, independently from the LLM, a confidence score associated with the answer from the LLM (see ¶ [0003, 0022, and 0029] citations as in limitation above: “[0003]…ASPIRE further includes a selection score that combines a likelihood of that generated answer being correct with a self-evaluation score for selective prediction. …
[0022] …When a question is presented, LLMs with selective prediction 120 can provide an answer with a confidence or selection score while LLMs without selective prediction 110 may just provide the answer [...] With selective prediction, LLMs 114 can output a low selection score 116 along with the wrong answer. The low selection score 116 can provide a warning not to trust the answer.”
[0029] The training data 204 can be in any form suitable for training a model, according to one of a variety of different learning techniques. Learning techniques for training a model can include supervised learning, unsupervised learning, semi-supervised learning techniques, parameter-efficient techniques and reinforcement learning techniques. Training the model can further include priming the model using zero- or few-shot prompting to output higher-quality responses. For example, the training data 204 can include multiple training examples that can be received as input by a model. […] The ASPIRE system 200 can include a fine-tuning engine 206, an answer sampling engine 208, and a self-evaluation engine 210. Given training data 204 for a generative task, the fine-tuning engine 206 can fine tune a LLM on the training data 204 to improve prediction performance. The fine-tuning engine 206 can be configured to train a LLM by keeping the model parameters fixed and adjusting adaptable parameters. The fine-tuning engine 206 can be configured to update, e.g., optimize, the adaptable parameters to train the LLM. The answer sampling engine 208 can be configured to generate multiple answers to each question of the query data 202 using the trained LLM. The answer sampling engine 208 can be configured to assess the correctness of the generated answers using an evaluation metric. The self-evaluation engine 210 can be configured to evaluate the correctness of generated answers to determine whether the trained LLM properly assessed the correctness of generated answers. The self-evaluation engine 210 can receive correctness of the generated answers as input and generate self-evaluation scores as outputs 212. The fine-tuning engine 206, answer sampling engine 208, and self-evaluation engine 210 can be implemented as one or more computer programs, specially configured electronic circuitry, or any combination thereof.),
determine whether the confidence score associated with the answer satisfies a quality threshold (see ¶ [0003, 0022, and 0029] citations as in limitation above and further ¶ [0026]: “Within the ASPIRE framework, an LLM can be pre-trained for any generative modeling task, such as question answering. To determine whether the output of the pre-trained LLM to the question answering task is correct or not, a reference output and an evaluation metric are utilized. The evaluation metric, such as a Rouge-L metric, can assess the similarity between the generated output and the reference output. For example, a Rouge-L metric can be employed as the evaluation metric to generate a score from [0, 1]. The correctness of the generated answer can be determined by comparing the score to a threshold value applied to the reference output. The generated score meeting the threshold value can be classified as correct answer, while generated scores falling below the threshold value can be classified as incorrect. The threshold value can be a value large enough where the generated answers that are incorrect are not determined to be correct.”); and
based on the confidence score associated with the answer satisfying the quality threshold (see ¶ [0003, 0022, 0029 and 0026] citations as in limitation above. “[0026] …The correctness of the generated answer can be determined by comparing the score to a threshold value applied to the reference output...”):
generate an annotated answer including the answer and an indication of quality based on the evaluation of the one or more sources used by the LLM to generate the answer (see ¶ [0003, 0022, and 0026] citations as in limitation above and further Fig. 1: 120: With Selective Prediction – “Answer: Los Angeles, Selection Score: 0.1”); and
output, to the computing device, the annotated answer in response to the inquiry (see Fig. 1 and ¶ [0003, 0022, and 0026] citations as in limitation above and further ¶ [0029]: “…From the query data 202 and/or training data 204, the ASPIRE system 200 can be configured to output one or more results related to selective prediction, generated as output data 212. The output data 212 can include answers on the query data 202 and/or a self-evaluating score associated with the generative task. As an example, the ASPIRE system 200 can be configured to send the output data 212 for display on a client or user display…”).
However, Yoon et al. does not explicitly teach, but Gunaselara et al. does teach:
determine a confidence score in association with the answer to the inquiry based on an evaluation of one or more sources used by the LLM (as taught by Yoon et al.) to generate the answer (see ¶ [0027, 0086, and 0102]: “[0027] A method and system for question-based answering leverages deep semantic understanding of content, using a trained language model, for search and prioritization of content. The method and system can preferably be used in translating and parsing a collection of long-form media and content into shorter digestible content segments to serve as the sources for candidate responses for an input query, in the form of a question. For example, a large collection of electronic/digital books on one or more academic or technical subjects can be processed by the method and system, such that a user could submit a question as input into the computer-implemented system and then the relevant paragraphs/sections from a variety of appropriate books could be delivered as possible solutions or answer candidates to the question. The system and method may be applied to a wide variety of ways.
[0086] …For example, one implementation may prioritize social-signals indicating value of content sources and therefore use social-signals like a source score (e.g., based on citations, publication ranking, share count, etc.) while another implementation may prioritize user personalization where a user affinity modeling score is used as an earlier tie-breaking process.
[0102] This may alternatively be performed to score user affinity to the source of a content segment (e.g., scoring user affinity to each book, publication, etc.), to context keywords that may be detected in the content segment, or other constructs related to a content segment.”).
Yoon et al. and Gunaselara et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in handling natural language data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yoon et al. to incorporate the teachings of Gunaselara et al. of an evaluation of one or more sources used by the LLM (i.e., model) which provides the benefit of enhancing customer service ([0036] of Gunaselara et al.).
However, Yoon et al. in combination with Gunaselara et al. do not explicitly teach, but Sager et al. does teach:
wherein the evaluation includes interacting with the one or more sources over a network to determine whether each of the one or more sources are valid and trustworthy (see ¶ starting at Col. 56, line 58: “(229) FIG. 24 illustrates a trustworthiness comparison of an exemplary graph having low semantic graph density versus an exemplary graph having high semantic graph density according to one embodiment of the present invention. The density of interconnected content within a self-assembled semantic graph (defined as the ratio of the number of edges in the graph relative to the maximum possible number of edges) provides a consensus score for the graph. Where node interconnection is greater for a large number of nodes, that indicates agreement between a number of disparate sources, indicating that sources within that network are more trustworthy. In one embodiment, the platform is able to receive a selection to include or not include one or more different types of sources (e.g., news sources, journal sources, patents, etc.) or specific sources (e.g., WIKIPEDIA, FACEBOOK posts, GOOGLE PATENTS, etc.). Editing which documents are able to be used is useful for tuning the credibility score to ensure that the resulting network is as credible as possible. In one embodiment, in order to determine the credibility score, different weighting is applied to different sources based on inherent source validity (e.g., based on rigor of editorial process, existence of inherent bias, etc.). In one embodiment, the platform receives a selection to change the credibility weighting of one or more sources by the user device.”);
Yoon et al., Gunaselara et al., and Sager et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in handling natural language data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yoon et al. in combination with Gunaselara et al. to incorporate the teachings of Sager et al. of wherein the evaluation includes interacting with the one or more sources over a network to determine whether each of the one or more sources are valid and trustworthy which provides the benefit of improving searches ([¶ starting at Col. 10, line 65] of Sager et al.).
As to independent claim 14, Yoon et al. further teaches:
14. A method (see ¶ [0003]: “Aspects of the disclosure are directed to methods, systems, and computer readable media for adaptation with self-evaluation to improve selective prediction in large language models (LLMs), generally referred to as ASPIRE…”) comprising:
[the limitations as in claim 1, above, as taught by Yoon et al. in combination with Gunaselara et al. and Sager et al.]
As to independent claim 20, Yoon et al. further teaches:
20. Computer-readable storage media comprising instructions that, when executed, configure processing circuitry (see ¶ [0003]: “Aspects of the disclosure are directed to methods, systems, and computer readable media for adaptation with self-evaluation to improve selective prediction in large language models (LLMs), generally referred to as ASPIRE…”) to:
[perform the limitations as in claim 1, above, as taught by Yoon et al. in combination with Gunaselara et al. and Sager et al.]
Regarding claims 2 and 15, Yoon et al. in combination with Gunaselara et al. and Sager et al. teaches the limitations as in claims 1 and 14, above.
Gunaselara et al. further teaches:
2 and 15. The system/method of claims 1 and 14,
wherein the processing circuitry is configured to, in response to a determination that each of the one or more sources used by the LLM (as taught by Yoon et al.) to generate the answer are valid sources, provide feedback to the LLM (as taught by Yoon et al.) indicating validity for each of the one or more sources (see ¶ [0027, 0086, and 0102] citations as in claims 1 and 14, above and further ¶ [0028]: “The system and method are preferably implemented in connection with a query interface for searching a collection of information, like the input fields typical of most search engines. Once supplied, a query input in the form of a question is used by the system and method to identify relevant content segments from the collection of indexed information that may potentially address the query. In one preferred variation, the search process combines keyword search model, a language model, and context affinity ranking in generating a list of relevant content segments, potentially containing answer spans.” and ¶ [0093-0094]: “[0093] As mentioned, some variations of a retrieval model may include updating content segment priority based on user affinity modeling S146, which functions to apply automated personalization of the results. User affinity modeling can generally be used to refine results so that content segments corresponding to predicted user preferences are ranked preferentially. A user affinity score of one or more candidate content segments is used, where the user affinity score corresponds to modeled preference of user or similarity between a content segment and modeled data of the user. User affinity may be determined and then used in prioritizing content in a variety of ways. In some implementations, this may function to infer context based on contextual cues of the user interactions or use of the query interface. [0094] In some variations user affinity may be used as a ranking tool to order content segment results whereby the content segment results are ranked based in part on a user affinity scores of at least a subset of content segments. The user affinity score in some variations may be used as a secondary ranking property, but may additionally or alternatively be used as a property to filter results (e.g., exclude results not satisfying some condition based on user affinity score) and/or as a primary ranking priority.”).
Yoon et al. and Gunaselara et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in handling natural language data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yoon et al. to incorporate the teachings of Gunaselara et al. of wherein the processing circuitry is configured to, in response to a determination whether the one or more sources used by the LLM (i.e., model) to generate the answer are valid sources, provide feedback to the LLM indicating validity or invalidity for each of the one or more sources which provides the benefit of enhancing customer service ([0036] of Gunaselara et al.).
Regarding claim 10, Yoon et al. in combination with Gunaselara et al. and Sager et al. teaches the limitations as in claim 1, above.
Yoon et al. further teaches:
10. The system of claim 1,
wherein, to generate the annotated answer, the processing circuitry is further configured to annotate the answer from the LLM with one or more of (see ¶ [0003, 0022, and 0026] citations as in claim 1 above and further Fig. 1: 120: With Selective Prediction – “Answer: Los Angeles, Selection Score: 0.1”):
annotations indicating the confidence score for at least one of the multiple parts of the answer (see ¶ [0003, 0022, and 0026] citations as in limitation above and further Fig. 1: 120: With Selective Prediction – “Answer: Los Angeles, Selection Score: 0.1”);
annotations indicating an overall confidence score for the multiple parts of the answer (see ¶ [0003, 0022, and 0026] citations as in limitation above and further Fig. 1: 120: With Selective Prediction – “Answer: Los Angeles, Selection Score: 0.1”); or [Examiner note: at least one]
Regarding claim 11, Yoon et al. in combination with Gunaselara et al. and Sager et al. teaches the limitations as in claim 1, above.
Gunaselara et al. further teaches:
11. The system of claim 1,
wherein the processing circuitry is configured to: obtain a user-profile associated with the computing device having originated the inquiry (see ¶ [0096 and 0115]: “[0096] In one variation, the user affinity score may be based on a configured user profile where content preferences are set. A profile indicating preferences may be completed by the user or generated based on analysis of user data. In one variation, the profile may indicate specific context keywords for which a user may have affinity. In one example, for querying of programming related book content within an IDE, the detected programming languages and/or technology usage within one or more code projects may be used in setting user-related context keywords for programming language and/or other technology preferences (e.g., libraries, frameworks, and/or webstacks used by a user). A user affinity score may be calculated by calculating a score based on the occurrence of user-related context keywords in the context metadata of a context segment.
[0115] Detecting a context keyword associated with a query input functions to identify a key concept relate to the query input This may include detecting the contextual keyword present in the query input. This may additionally or alternatively include detecting a contextual keyword being associated with the query input (e.g., stored in a user profile of the user making the request, usage of the contextual keyword in past queries, interaction of the user with previous results associated with the contextual keyword).”); and
update the annotated answer, prior to output to the computing device, with information derived from the user-profile (see ¶ [0096 and 0115] citations as in limitation above: “In one variation, the user affinity score may be based on a configured user profile where content preferences are set…”).
Yoon et al. and Gunaselara et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in handling natural language data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yoon et al. to incorporate the teachings of Gunaselara et al. of wherein the processing circuitry is configured to: obtain a user-profile associated with the computing device having originated the inquiry and update the annotated answer, prior to output to the computing device, with information derived from the user-profile which provides the benefit of enhancing customer service ([0036] of Gunaselara et al.).
Regarding claim 12, Yoon et al. in combination with Gunaselara et al. and Sager et al. teaches the limitations as in claim 1, above.
Yoon et al. further teaches:
12. The system of claim 1,
wherein the processing circuitry is configured to: receive the inquiry from the computing device (see ¶ [0027]: “…The query data 202 and/or training data 204 can further be provided as input through a user interface on a client computing device coupled to the ASPIRE system 200.”); and
submit the inquiry to the LLM (see ¶ [0027] citation as in limitation above and further ¶ [0053]: “As shown in block 730, the ASPIRE system 200 can be configured to employ the updated first adaptable parameters when sampling the LLM, generating a plurality of outputs for a query associated with the question answering task…”).
Regarding claim 13, Yoon et al. in combination with Gunaselara et al. and Sager et al. teaches the limitations as in claim 12, above.
Yoon et al. further teaches:
13. The system of claim 12,
wherein the inquiry is a first inquiry (see ¶ [0003 and 0022]: “[0003] Aspects of the disclosure are directed to methods, systems, and computer readable media for adaptation with self-evaluation to improve selective prediction in large language models (LLMs), generally referred to as ASPIRE. ASPIRE includes training LLMs on a portion of training data from a question answering task to learn self-evaluation, e.g., learn to distinguish whether a generated answer is correct or not. ASPIRE further includes a selection score that combines a likelihood of that generated answer being correct with a self-evaluation score for selective prediction. ASPIRE demonstrates improved selective prediction performance with less computational cost.
[0022] FIG. 1 depicts a block diagram of LLMs outputs 100 without selective prediction and with selective prediction according to aspects of the disclosure. When a question is presented, LLMs with selective prediction 120 can provide an answer with a confidence or selection score while LLMs without selective prediction 110 may just provide the answer. For example, for a question like ‘What is the capital of California?’, LLM models 112 without selective prediction might provide an incorrect answer like ‘Los Angeles’ instead of the correct answer, ‘Sacramento.’ Without selective prediction, LLMs 112 may directly output incorrect answers. With selective prediction, LLMs 114 can output a low selection score 116 along with the wrong answer. The low selection score 116 can provide a warning not to trust the answer.”), and
wherein the processing circuitry is configured to: cache, using a database system, the first inquiry and the computer-generated text output from the LLM as the answer to the first inquiry when the confidence score associated with the answer satisfies the quality threshold (see ¶ [0037-0038]: “[0037] The output of the prediction can include the correctness of the generated output 337 and the self-evaluation score 338. The self-evaluation score 338 can include any type of likelihood defined as a selection scoring function. Since the input 333 and 334 are constructed by appending additional tokens to the initial query, the ASPIRE system 200 can be configured to reuse the states in the answer sampling process 320 instead of recomputing them to save computational cost. [0038] …As shown in block 420, the ASPIRE system can be configured to acquire the correctness of the answer and calculate the self-evaluation score via the additional fine-tuned adaptable parameters θ.sub.s and embeddings of the answers, while also keeping the model parameters θ of LLM 411, the updated adaptable parameters θ.sub.p 412 and Q.sub.embed 413 frozen. The ASPIRE system 200 can be configured to cache the states when generating the answer and reuse those states when computing the self-evaluation score to save computational cost.”);
receive a second inquiry from a second computing device (see ¶ [0037-0038] citations as in limitation above: “[0037] The output of the prediction can include the correctness of the generated output 337 and the self-evaluation score 338. The self-evaluation score 338 can include any type of likelihood defined as a selection scoring function. Since the input 333 and 334 are constructed by appending additional tokens to the initial query, the ASPIRE system 200 can be configured to reuse the states in the answer sampling process 320 instead of recomputing them to save computational cost.” and further ¶ [0043]: “The user computing device 506 can also be configured similar to the server computing device 504, with one or more processors 520, memory 522, instructions 524, and data 526. The user computing device 506 can also include a user input 528, and a user output 530. The user input 528 can include any appropriate mechanism or technique for receiving input from a user, such as keyboard, mouse, mechanical actuators, soft actuators, touchscreens, microphones, and sensors.” and Fig. 3: (Question Answer A-B (333, 334));
in response to a determination that the second inquiry matches the first inquiry cached using the database system, obtain the answer to the first inquiry cached using the database system as the answer to the second inquiry without submitting the second inquiry to the LLM (see ¶ [0037-0038] citations as in limitation above. (i.e., reuse of cached states)); and
output the answer in response to the second inquiry to the second computing device (see ¶ [0037-0038] citations as in limitations above and Fig. 3: (Question Answer A-B (333, 334)).
Regarding claim 19, Yoon et al. in combination with Gunaselara et al. and Sager et al. teaches the limitations as in claim 14, above.
Yoon et al. further teaches:
19. The method of claim 14,
[performing the limitations as in claims 12-13, above.]
Claims 4, 9, and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yoon et al. (US 20240428015 A1) and further in view of Gunaselara et al. (US 20210365500 A1) and Sager et al. (US 11561987 B1) as in claims 1 and 14, above and further in view of Tholar et al. (US 20250245665 A1).
Regarding claims 4 and 17, Yoon et al. in combination with Gunaselara et al. and Sager et al. teaches the limitations as in claims 1 and 14, above.
Tholar et al. further teaches:
4 and 17. The system/method of claims 1 and 14,
wherein the processing circuitry is configured to determine whether the one or more sources used by the LLM to generate the answer are valid sources based on an evaluation of one or more source quality conditions (see ¶ [0008] citation as in claims 3 and 16, above and further ¶ [0009]: “… In some embodiments, the plurality of public sources may include at least one of a document, a website, an encyclopedia, a database, a search engine, a map, a weather report, or a news report...” and further ¶ [0008 and 0058]: “[0008]… The system also includes a processor and a computer readable medium operably coupled thereto, the computer readable medium may include a plurality of instructions stored in association therewith that are accessible to, and executable by, the processor, to perform operations which may include: receiving unstructured data pertaining to an entity from a plurality of public sources, and receiving structured data pertaining to the entity from at least two of: a database may include data for multiple software applications, a suspicious activity monitoring (SAM) database, a client due diligence (CDD) database, a watch list filtering (WLX) database, or a risk case management (RCM) database…
[0058] Documents can be in structured data 410 as well as unstructured data 420. Some examples of structured documents are databases, comma-separated value (CSV) files, and JavaScript object notation (JSON) files, and examples of unstructured data include emails, social media posts, news reports, etc. These can be accessed using a variety of methods, such as loading them from a file or a database…), including:
any of the one or more sources derived from a social media platform (see ¶ [0008-0009 and 0058] citations as in claims 3 and 16 and limitation above.);
any of the one or more sources corresponding to a social media post (see ¶ [0008-0009 and 0058] citations as in claims 3 and 16 and limitation above.);
or [Examiner note: at least one]
Yoon et al. and Gunaselara et al., and Tholar et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in handling natural language data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yoon et al. in combination with Gunaselara et al. to incorporate the teachings of Tholar et al. of wherein the processing circuitry is configured to determine whether the one or more sources used by the LLM to generate the answer are valid sources based on the evaluation of one or more source quality conditions, including: [at least] any of the one or more sources derived from a social media platform; any of the one or more sources corresponding to a social media post; which provides the benefit of cost savings, faster operational processes, optimized team efficiency, less training overhead, and increased accuracy ([0035] of Tholar et al.).
Regarding claim 9, Yoon et al. in combination with Gunaselara et al. and Sager et al. teaches the limitations as in claim 1, above.
Tholar et al. further teaches:
9. The system of claim 1,
wherein to determine the confidence score, the processing circuitry is further configured to validate the computer-generated text output from the LLM against a selected source from the one or more sources used by the LLM to generate the answer based on one or more of (see ¶ [0008]: “ The fraud investigation digital assistant system disclosed herein has particular, but not exclusive, utility for anti-money-laundering (AML) investigations. […] One general aspect includes a system adapted to automatically report the trustworthiness of an entity. […] The operations also include receiving a natural language user query regarding trustworthiness of the entity; with the embedding model, converting the natural language user query to a query embedding; based on the query embedding and a similarity calculation, fetching a relevant embedding from the vector store; with a large language model (LLM), based on the query embedding, the fetched relevant embedding, and the chunk corresponding to the fetched embedding, generating a query response regarding the trustworthiness of the entity. The instructions also include communicating the query response to the user. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.”):
text obtained from the selected source and compared with the computer-generated text output from the LLM (see ¶ [0008] citation as in limitation above. More specifically: “…generating a query response regarding the trustworthiness of the entity…”);
audio or [Examiner note: at least one]
Yoon et al. and Gunaselara et al., and Tholar et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in handling natural language data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yoon et al. in combination with Gunaselara et al. to incorporate the teachings of Tholar et al. of wherein to determine the confidence score, the processing circuitry is configured to validate the computer-generated text output from the LLM against a selected source from the one or more sources used by the LLM to generate the answer based on one or more of: text obtained from the selected source and compared with the computer-generated text output from the LLM which provides the benefit of cost savings, faster operational processes, optimized team efficiency, less training overhead, and increased accuracy ([0035] of Tholar et al.).
.
Claim 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yoon et al. (US 20240428015 A1) and further in view of Gunaselara et al. (US 20210365500 A1) and Sager et al. (US 11561987 B1) as in claim 1, above and further in view of D'Souza et al. (US 10332513 B1).
Regarding claim 5, Yoon et al. in combination with Gunaselara et al. and Sager et al. teaches the limitations as in claim 1, above.
Gunaselara et al. further teaches:
5. The system of claim 1,
wherein the processing circuitry is configured to, based on the confidence score associated with the answer failing to satisfy the quality threshold or a determination any of the one or more sources used by the LLM to generate the answer are invalid (see ¶ [0027, 0086, and 0102] citations as in claims 1 and 14, above and further ¶ [0028]: “The system and method are preferably implemented in connection with a query interface for searching a collection of information, like the input fields typical of most search engines. Once supplied, a query input in the form of a question is used by the system and method to identify relevant content segments from the collection of indexed information that may potentially address the query. In one preferred variation, the search process combines keyword search model, a language model, and context affinity ranking in generating a list of relevant content segments, potentially containing answer spans.” and ¶ [0093-0094]: “[0093] As mentioned, some variations of a retrieval model may include updating content segment priority based on user affinity modeling S146, which functions to apply automated personalization of the results. User affinity modeling can generally be used to refine results so that content segments corresponding to predicted user preferences are ranked preferentially. A user affinity score of one or more candidate content segments is used, where the user affinity score corresponds to modeled preference of user or similarity between a content segment and modeled data of the user. User affinity may be determined and then used in prioritizing content in a variety of ways. In some implementations, this may function to infer context based on contextual cues of the user interactions or use of the query interface. [0094] In some variations user affinity may be used as a ranking tool to order content segment results whereby the content segment results are ranked based in part on a user affinity scores of at least a subset of content segments. The user affinity score in some variations may be used as a secondary ranking property, but may additionally or alternatively be used as a property to filter results (e.g., exclude results not satisfying some condition based on user affinity score) and/or as a primary ranking priority.”).
Yoon et al. and Gunaselara et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in handling natural language data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yoon et al. to incorporate the teachings of Gunaselara et al. of wherein the processing circuitry is configured to, based on the confidence score associated with the answer failing to satisfy the quality threshold or a determination any of the one or more sources used by the LLM to generate the answer are invalid which provides the benefit of enhancing customer service ([0036] of Gunaselara et al.).
However, Yoon et al. in combination with Gunaselara et al. and Sager et al. do not explicitly teach, but D'Souza et al. does teach:
output, to the computing device, a notification indicating that an answer to the inquiry will not be provided (see ¶ Col. 36, lines 30-61: “(137) If, at step 514, it is determined that there is at least one similarity value that is greater than the similarity threshold value, then process 500 may proceed to step 522, which may correspond to step 330 of process 300. At step 330, another determination is made as to whether there is one or more similarity values that exceeds the similarity threshold value. However, if at step 514 it is determined that there are no similarity values that exceed the similarity threshold value, then process 500 may proceed to step 516. At step 516, fourth text data of a second response may be received by TTS module 264. The second response may be obtained from the listing of responses from the prompts module, and may indicate that no applications were found to substantially match the name of the application named within the utterance(s). For example, there may be no applications corresponding to the name “Skill 1,” and therefore prompts module 270 may provide TTS module 264 with text data representing a second response to reflect this. As an illustrative example, a selected response in this scenario may be of the form, “I'm sorry. I could not find any applications having the name {Object Identifier}.” Thus, if the name of the application from the utterance was “Skill 1,” then the second response would correspond to, “I'm sorry. I could not find any applications having the name ‘Skill 1’.” In some embodiments, the second response may further direct individual 2 to access a companion application on the requesting device (e.g., electronic device 10), or any other suitable device, to manually select the appropriate application. For example, the second response may be, “I'm sorry. I could not find any applications having the name ‘Skill 1’. Please go to your Companion App on your device to select the application.””).
Yoon et al. and Gunaselara et al., Sager et al. and D'Souza et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in handling natural language data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yoon et al. in combination with Gunaselara et al. and Sager et al. to incorporate the teachings of D'Souza et al. of wherein the processing circuitry is configured to, based on the confidence score associated with the answer failing to satisfy the quality threshold or a determination any of the one or more sources used by the LLM to generate the answer are invalid, output, to the computing device, a notification indicating that an answer to the inquiry will not be provided which provides the benefit of improving the likelihood that output results make sense (e.g., grammatically) (Col. 20, lines 6-8 of D'Souza et al.).
Claims 6 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yoon et al. (US 20240428015 A1) and further in view of Gunaselara et al. (US 20210365500 A1) and Sager et al. (US 11561987 B1) as in claims 1 and 14, above and further in view of Galitsky (US 20180357221 A1).
Regarding claims 6 and 18, Yoon et al. in combination with Gunaselara et al. and Sager et al. teaches the limitations as in claim 1, above.
Gunaselara et al. further teaches:
6 and 18. The system/method of claims 1 and 14,
wherein the processing circuitry is configured to, based on the confidence score associated with the answer failing to satisfy the quality threshold or a determination any of the one or more sources used by the LLM to generate the answer are invalid (see ¶ [0027-0028, 0086, 0093-0094, and 0102] citations as in claims 1, 5 and 14, above.).
Yoon et al. and Gunaselara et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in handling natural language data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yoon et al. to incorporate the teachings of Gunaselara et al. and Sager et al. of wherein the processing circuitry is configured to, based on the confidence score associated with the answer failing to satisfy the quality threshold or a determination any of the one or more sources used by the LLM to generate the answer are invalid which provides the benefit of enhancing customer service ([0036] of Gunaselara et al.).
However, Yoon et al. in combination with Gunaselara et al. and Sager et al. do not explicitly teach, but Galitsky et al. does teach:
discard the answer (see ¶ [0243]: “At block 2005, process 2000 involves responsive to determining that the level of complementarity is above a threshold, identifying the question and answer sentences as complementary. Rhetoric classification application 102 can use a threshold level of complementarity to determine whether the question-answer pair is sufficiently complementary. For example, if a classification score is greater than a threshold, then rhetoric classification application 102 can output the answer as answer 172 or answer 150. Alternatively, rhetoric classification application 102 can discard the answer and access answer database 105 or a public database for another candidate answer and repeat process 2000 as necessary.”);
re-submit the inquiry originating from the computing device to the LLM on behalf of the computing device (see ¶ [0243] citation as in limitation above: “…access answer database 105 or a public database for another candidate answer and repeat process 2000 as necessary.”);
obtain new computer-generated text output from the LLM as a new answer to the inquiry (see ¶ [0243] citation as in limitation above: “…access answer database 105 or a public database for another candidate answer and repeat process 2000 as necessary.”); and
based on a new confidence score associated with the new answer satisfying the quality threshold (see ¶ [0243] citation as in limitation above: “…access answer database 105 or a public database for another candidate answer and repeat process 2000 as necessary.”:
generate a new annotated answer (see ¶ [0243] citation as in limitation above: “…access answer database 105 or a public database for another candidate answer and repeat process 2000 as necessary.”); and
output, to the computing device, the new annotated answer in response to the inquiry (see ¶ [0243] citation as in limitation above: “…access answer database 105 or a public database for another candidate answer and repeat process 2000 as necessary.”).
Yoon et al. Gunaselara et al., Sager et al. and Galitsky are considered to be analogous to the claimed invention because they are in the same field of endeavor in handling natural language data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yoon et al. in combination with Gunaselara et al. and Sager et al. to incorporate the teachings of Galitsky of discard the answer; re-submit the inquiry previously submitted by the computing device to the LLM on behalf of the computing device; obtain new computer-generated text output from the LLM as a new answer to the inquiry; and based on a new confidence score associated with the new answer satisfying the quality threshold: generate a new annotated answer; and output, to the computing device, the new annotated answer in response to the inquiry which provides the benefit of implementing improved automated agents, or chatbots, that can answer questions received from users ([0054] of Galitsky).
Claim 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yoon et al. (US 20240428015 A1) and further in view of Gunaselara et al. (US 20210365500 A1) and Sager et al. (US 11561987 B1) as in claim 1, above and further in view of Mishra (US 20250045534 A1).
Regarding claim 7, Yoon et al. in combination with Gunaselara et al. and Sager et al. teaches the limitations as in claim 1, above.
However, Yoon et al. in combination with Gunaselara et al. and Sager et al. do not explicitly teach, but Mishra et al. does teach:
7. The system of claim 1,
wherein the processing circuitry is further configured to: obtain, from the computing device, user-input indicating a degree of usefulness of the annotated answer (see ¶ [0073]: “In some implementations, the quality metric can relate to whether the corresponding response to the NL based input 210 resulted in follow-up inputs. For instance, if the corresponding response resulted in one or more follow-up inputs from the user, it can be determined that the quality metric is below a threshold quality metric. In some additional or alternative implementations, the quality metric can relate to feedback provided by a user when provided the corresponding response. For instance, the user can provide a rating for the response (e.g., a score out of 10), or can provide binary feedback (e.g., a thumbs up or thumbs down).”); and
provide, to the LLM, the user-input indicating the degree of usefulness of the annotated answer, wherein the user-input specifies at least one of (see ¶ [0073] citation as in limitation above and further ¶ [0074]: “Once the training instance(s) have been generated in this manner, the NL based response system 120 (or an LLM thereof), can be fine-tuned (or otherwise termed, trained) using the training instances stored in the training instance(s) database 132A (e.g., using training engine 132). This can be performed in any suitable way (e.g., supervised learning, reinforcement learning, etc.).”):
a numerical score for the annotated answer (see ¶ [0073-0074] citation as in limitation above.);
a non-numerical user-rated assessment for the annotated answer (see ¶ [0073-0074] citation as in limitation above.);
a thumbs-up or a thumbs-down indication for the annotated answer;
or [Examiner note: at least one]
Yoon et al. and Gunaselara et al., and Mishra et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in handling natural language data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yoon et al. in combination with Gunaselara et al. and Sager et al. to incorporate the teachings of Mishra et al. of obtain, from the computing device, user-input indicating a degree of usefulness of the annotated answer; and provide, to the LLM, the user-input indicating the degree of usefulness of the annotated answer, wherein the user-input specifies at least one of: a numerical score for the annotated answer; a non-numerical user-rated assessment for the annotated answer; or a thumbs-up or a thumbs-down indication for the annotated answer; which provides the benefit of providing an improved response to a user that is responsive to the user's NL based input ([0004] of Mishra et al.).
Claim 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yoon et al. (US 20240428015 A1) and further in view of Gunaselara et al. (US 20210365500 A1) and Sager et al. (US 11561987 B1) as in claim 1, above and further in view of Madisetti et al. (US 20240370474 A1).
Regarding claim 8, Yoon et al. in combination with Gunaselara et al. and Sager et al. teaches the limitations as in claim 1, above.
However, Yoon et al. in combination with Gunaselara et al. and Sager et al. do not explicitly teach, but Madisetti et al. does teach:
8. The system of claim 1, wherein to determine the confidence score, the processing circuitry is further configured to: provide as a first input to an artificial intelligence (AI) model, a golden copy of answers (see ¶ [0063-0064]: “[0063] Referring now to FIG. 9 is an illustration of generating derived prompts for different categories and using them with multiple h-LLMs to generate the best results, is described in more detail. User 800 enters a prompt in user interface 802. The prompt is sent to the AI Input Broker 810 which generates multiple derived prompts for different categories. The derived prompts 822 are sent multiple h-LLMs 824 which produce the results. The results 816 are sent to the AI Output Broker 814 which processes the results and performs tasks such as filtering, ranking, weighting, assigning priorities, and then sends the best results to the user 800. The h-LLMs 824 can have varying levels of accuracy, and optimized for different tasks such as Question Answering, Information Extraction, Sentiment Analysis, Image Captioning, Object Recognition, Instruction Following, Classification, Inferencing, and Sentence Similarity, for instance. The AI Output Broker 814 computes various scores and assigns weights for ranking the results. The results may be sent back to the h-LLMs till a certain level of accuracy or service level assurance is reached. The AI Input Broker 810 and Output Broker 814 update 812, 818 a local AI Broker Database 820 with the results of the request's path through its hierarchy and create an index of “derived requests” that may be used in future to select which set of “derived requests” an incoming request may fall into for further processing.
[0064] Referring now to FIG. 10 is an illustration of using multiple h-LLMs to answer questions from specific input documents, is described in more detail. User 900 enters a prompt in user interface 902. The prompt is sent to AI Input Broker 810 which generates multiple derived prompts for different categories 924. The prompts are converted into embeddings using multiple embedding models 926. The prompt embeddings 928 are sent to a vector database 930 which returns a list of knowledge documents 934 that are relevant to the prompt based on the similarity of their embeddings to the user's prompt. The knowledge documents 934 are sent to the AI Input Broker 810 which creates new context-aware prompts based on the user's initial prompt 916, derived prompts 924 and the retrieved knowledge documents 934 as context and sends it to multiple h-LLMs 912. The results produced by multiple h-LLMs are processed by the AI Output Broker 908 and the best result is sent to the user 900 along with citations from the knowledge documents 934.”);
provide as a second input to the AI model, the computer-generated text output from the LLM as the answer to the inquiry (see ¶ [0063-0064] citations as in limitation above.); and
obtain from the AI model, the confidence score indicating probability the answer is accurate (see ¶ [0063-0064] citations as in limitation above.).
Yoon et al., Gunaselara et al., Sager et al. and Madisetti et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in handling natural language data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yoon et al. in combination with Gunaselara et al. and Sager et al. to incorporate the teachings of Madisetti et al. of provide as a first input to an artificial intelligence (AI) model, a golden copy of answers; provide as a second input to the AI model, the computer-generated text output from the LLM as the answer to the inquiry; and obtain from the AI model, the confidence score indicating probability the answer is accurate which provides the benefit of refinement/enhancement ([0066] of Madisetti et al.).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Regarding source evaluations (pertinent to claims 1, 14 and 20):
Tholar et al. (US 20250245665 A1, ¶ [0008, 0010]).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Keisha Y Castillo-Torres whose telephone number is (571)272-3975. The examiner can normally be reached Monday - Friday, 9:00 am - 4:00 pm (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre-Louis Desir can be reached at (571)272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Keisha Y. Castillo-Torres
Examiner
Art Unit 2659
/Keisha Y. Castillo-Torres/Examiner, Art Unit 2659
/PIERRE LOUIS DESIR/Supervisory Patent Examiner, Art Unit 2659
Prosecution Insights