Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
2. Applicant’s arguments with respect to claims 1-17 and 20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Specifically newly added limitations to independent claims 1, 11 and 20 are taught by newly cited Peng et al. “Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback” and Yu et al. (US 2020/0394458).
Claim Rejections - 35 USC § 103
3. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
4. Claims 1-7, 9-17 and 20-22 are rejected under 35 U.S.C. 103 as being unpatentable over Gupta et al. (US 2025/0124236), herein Gupta, further in view of Pedersen et al. (US 2020/0142999), herein Pedersen, further in view of Peng et al. “Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback”, herein Peng and further in view of Yu et al. (US 2020/0394458) herein Yu.
Regarding Claim 1:
Gupta discloses a computing platform comprising:
at least one processor (Gupta: Fig. 7 discloses a processor);
a communication interface communicatively coupled to the at least one processor (Gupta: p[0082]-[0083 discloses network interface device 720 coupled to the processor through data bus 708);
and memory storing computer-readable instructions that, when executed by the at least one processor (Gupta: p[0082]-[0083]), cause the computing platform to:
train, using historical information indicating a plurality of regimes for large language model (LLM) outputs, an LLM steward model, wherein training the LLM steward model configures the LLM steward model to generate LLM validation information indicating three different classifications of LLM outputs, wherein the classifications comprise acceptable, tolerable, and non-acceptable, wherein the tolerable classification comprises an intersection of the acceptable and non-acceptable classifications, and wherein the LLM steward model is a closed loop mode (Gupta: ¶[0040], ¶[0060]-[0061] discloses training an evaluation LLM to classify LLM outputs using predefined evaluation functions, including toxicity, hallucination and quality metrics, producing validation results that categorize outputs across multiple graded outcome ranges rather than a single binary classification. Gupta further discloses continuous evaluation and feedback of LLM outputs through the evaluation pipeline in Fig. 6)
receive updated information associated with the plurality of regimes (Gupta: ¶[0070]-[0075] discloses continuously receiving updated evaluation data and user-provided information associated with the evaluation of LLM outputs, including updated datasets and feedback used by the system during operation);
identify a delta value between the historical information and the updated information (Gupta: ¶[0033]-[0035], ¶[0047]-[0049] and ¶[0059] discloses receiving updated information to a stored data table and identifying a delta between the updated information and historical information via transaction log change data that records changes with respect to the previous version of data table; Gupta also discloses identifying a delta value between textual information inputs by comparing an expected output to a generated response and calculating a similarity score via vector similarity techniques);
update, based on the delta value, the plurality of regimes to adjust corresponding classifications of acceptable, tolerable, and non-acceptable (Gupta: ¶[0047]-[0049], ¶[0059] and ¶[0074]-[0075] discloses maintaining data logs of prior evaluated responses and receiving provided expected responses or labels for newly generated outputs and computing similarity scores and differences between stored evaluation records and newly labeled outputs. These comparisons are performed between previously stored evaluation data and newly received input information, thereby identify a delta value between historical input information and updated input information (the generated response and the expected response) and the delta value is the similarity score quantifying difference between generated output and expected response. By comparatively evaluating generated responses against user provided expected responses, assigning a similarity score to each response and then using that score to identify, highlight and distinguish deficient and acceptable portions of the responses, Gupta treats higher scores as more acceptable therefore adjusting response acceptability categories based on the evaluation score),
input, into an LLM, an LLM prompt, wherein inputting the LLM prompt causes the LLM to generate an LLM output (Gupta: ¶[0045] explicitly discloses inputting a prompt into an LLM to generate a response);
input the LLM output into the LLM steward model and the additional model, wherein inputting the LLM output into the LLM steward model and the additional model causes the LLM steward model to output the LLM validation information (Gupta: ¶[0046], ¶[0060] and ¶[0061] discloses sending the generated responses to an evaluation LLM to generate validation results);
and based on outputting LLM validation information indicating that the LLM output is acceptable or tolerable, send the LLM output to a user device for presentation (Gupta: ¶[0070]-[0073] discloses once validation results are produced, the evaluated LLM output is sent to the client device and displayed to the user).
Gupta does not explicitly disclose wherein updating the plurality of regimes comprises updating an additional model that is dynamically updated, and wherein the additional model is a layer added on top of the LLM steward model. However, Pedersen discloses this limitation: (Pedersen: p[0042], p[0068], discloses implementing classification adjustments through a dynamically updated secondary machine learning layer that refines classification behavior).
It would have been obvious to one of ordinary skill in the art before the effective filing date to try combining the evaluation system of Gupta, which evaluates large language model outputs, with Pedersen’s feedback-based retraining method because both references address improving classification accuracy. Pedersen provides a known, predictable technique (user feedback driven retraining for updating model parameters. It would have been obvious to one of ordinary skill in the art to recognize that updating a separate evaluation or classification layer as taught by Pedersen. The motivation for doing so is that it “may provide an improved user experience for users of any network -accessible platform that allows its users to exchange user-generated content.“
The proposed combination of Gupta and Pedersen does not explicitly disclose:
based on outputting LLM validation information indicating that the LLM output is non- acceptable.
However, Peng discloses based on outputting LLM validation information indicating that the LLM output is non- acceptable (Peng: 2.1 and 2.4 the utility module evaluates LLM generated candidate responses and generates a utility score and corresponding feedback, the feedback is not released unless it passes verification. Section 3.2 teaches that the KF1 utility score falls below a threshold the response is identified as inconsistent with the supporting knowledge. Table 6 similarly discloses a first response that fails the utility module criterion), re-input, into the LLM, the LLM prompt to cause the LLM to generate an updated LLM output (Peng: 2.3.2 after the candidate response fails verification, the system generates feedback uses that feedback to revise the prompt and queries the model again. Section 2.3.2 explains that the prompt contains the task instructions, original user query, dialog history, evidence and any available feedback).
Gupta, Pedersen and Peng are combinable because they are in the same field of endeavor, i.e., both disclose systems or methods for generating, evaluating and controlling the presentation of LLM-generated responses. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gupta to determine that an LLM is non-acceptable and consequently re-input the LLM prompt into the LLM to generate an updated LLM output, as taught by Peng. This modification would have reduced hallucinations in responses presented to users without diminishing response quality because Peng explicitly states that “LLM-AUGMENTER significantly reduces ChatGPT’s hallucinations without sacrificing the fluency and informativeness of its responses” (Peng, Abstract, p. 1).
The proposed combination of Gupta, Pedersen and Peng does not explicitly disclose
maintain an accuracy threshold for the additional model;
based on identifying that an accuracy of the additional model is greater than the accuracy threshold, pause refinement of the additional model; and
based on identifying that the accuracy of the additional model fails to exceed the accuracy threshold, resume refinement of the additional model via a dynamic feedback loop.
However, Yu discloses:
maintain an accuracy threshold for the additional model (Yu: ¶81-82 evaluates a machine learning model against a minimum acceptable or threshold level of performance. The valuator calculates the model’s accuracy by comparing the model’s output with corresponding ground truth values);
based on identifying that an accuracy of the additional model is greater than the accuracy threshold, pause refinement of the additional model (Yu: ¶81-82 discloses that the evaluator determines whether the trained model satisfies the threshold performance criterion. If the model satisfies the applicable accuracy level its training is complete and the model logically transitions from training mode to inference or classifier use); and
based on identifying that the accuracy of the additional model fails to exceed the accuracy threshold, resume refinement of the additional model via a dynamic feedback loop (Yu: ¶81-81 and ¶86 discloses that if the model does not satisfy the threshold level of performance the training manager continues training. ¶82 specifically states that failure to satisfy the accuracy threshold causes the training manager to perform further training, ¶86 specifically states that retraining can be done as new data is available).
Gupta, Pedersen and Peng in view of Yu are analogous art because they are from pertinent fields of endeavor, i.e., both disclose system or methods for training, evaluating and iteratively refining machine learning classification models. IT would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Pedersen to maintain an accuracy threshold for its additional classification model, pause refinement when the model’s accuracy threshold and resume feedback refinement when its accuracy fails to exceed the threshold as taught by Yu. This modification would have conserved training time and computational resources by avoiding unnecessary refinement after sufficient accuracy is achieved. Yu explicitly teaches that model training tasks “can still require significant time, resource allocation and cost” (Yu ¶88).
Regarding Claim 2:
The proposed combination of Gupta, Pedersen, Peng and Yu further disclose computing platform of claim 1, wherein the memory stores additional computer readable instructions that, when executed by the at least one processor, cause the computing platform to: based on outputting LLM validation information indicating that the LLM output is non-acceptable, update the LLM output to conform with a corresponding subset of the plurality of regimes (Gupta: Figs. 3A-C and Fig 5, discloses replacing otherwise visible comment text with moderated content when the text fails to satisfy applicable user preference or moderation criteria).
It would have been obvious to one of ordinary skill in the art to disclose update the LLM output to conform with a corresponding subset of the plurality of regimes. Gupta discloses generating outputs from one or more LLMs, evaluating those outputs and presenting the results to a user via a user interface. However, Gupta differs from the claimed invention in that it does not explicitly modify the actual output text itself to bring it into compliance with predetermined regimes, it only marks which portions are acceptable or not. Pedersen discloses moderation techniques to update the output so that it conforms to a specific subset. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gupta’s system such that, upon outputting validation information indicating that the LLM output is non-acceptable, applying a moderation technique to make the text conform to a particular regime. The suggestion/motivation for doing so is: “ Human moderators cannot moderate user-generated content effectively when the user community is large” as disclosed in Pedersen ¶[0003].
Regarding Claim 3:
The proposed combination of Gupta, Pedersen, Peng and Yu further discloses the computing platform of claim 1, wherein the memory stores additional computer readable instructions that, when executed by the at least one processor, cause the computing platform to: update, via a dynamic feedback loop and based on feedback received from the user device, the LLM steward model (Gupta: p[0076] UI includes a feedback mechanism which allows users to indicate the quality of generated response; Pedersen p[0042] user feedback is actually used in order to update the evaluation model).
It would have been obvious to one of ordinary skill to combine Gupta’s evaluation and feedback system with Pedersen’s dynamic retraining approach so that user feedback received through Gupta’s interface would be used to continuously update the LLM steward model. This simple addition is straightforward and predictable, improving model performance over time by automatically adapting based on real-world user interactions.
Regarding Claim 4:
The proposed combination of Gupta, Pedersen, Peng and Yu further discloses the computing platform of claim 1, wherein the historical information includes one or more of: text information, images, speech information, structured information, three dimensional signals, literature information, cultural information, social information, geographical information, legal information, or linguistic information (Gupta: p[0024]-[0025] discloses the LLM is trained on large amounts of data from various sources including websites, articles, posts on the web, images, audio etc.).
Regarding Claim 5:
The proposed combination of Gupta, Pedersen, Peng and Yu further discloses the computing platform of claim 1, wherein each of the regimes define content that, when included in an output from the LLM, is one or more of: acceptable, tolerable, or non-acceptable (Gupta: p[0060]-[0061] classifies the LLM output as toxic/non-toxic).
Regarding Claim 6:
The proposed combination of Gupta, Pedersen, Peng and Yu further discloses the computing platform of claim 1, wherein outputting the LLM validation information comprises: identifying one or more regimes, of the plurality of regimes, associated with the LLM prompt (Gupta: p[0046]-[0048] teaches applying different evaluation functions, such as toxicity detection or keyword similarity to classify and score outputs, these evaluation functions correspond to the claimed regimes where each regime represents a set of rules or thresholds used to assess a response), identifying a location of the LLM output, within the one or more regimes associated with the LLM prompt (Gupta: p[0070]-[0071]Gupta discloses identifying the exact location of issues within the LLM output by highlighting specific words or portions), based on identifying that the LLM output is within an acceptable regime or a tolerable regime, outputting an indication that the LLM output is acceptable (Gupta: p[0060] discloses generating toxicity score and comparing it to a threshold, if the score is below the threshold the response is considered acceptable or tolerable and an indication (e.g., a green highlight or score)), and based on identifying that the LLM output is within an non-acceptable regime, outputting an indication that the LLM output is non-acceptable (Gupta: p[0060] p[0070]-[0071] teaches classifying an LLM output as non-acceptable when it exceeds a threshold and displaying a visual indicator, such as a red highlight).
Regarding Claim 7:
The proposed combination of Gupta, Pedersen, Peng and Yu further discloses the computing platform of claim 6, wherein the LLM steward model comprises a foundational model, and wherein identifying the one or more regimes associated with the LLM prompt comprises: identifying a plurality of overlapping clusters, within the foundational model, that characterize the LLM prompt, and identifying regimes corresponding to the plurality of overlapping clusters (Gupta: p[0042] discloses that the LLM grading model operates as a distributed system of clusters in the data layer, where different clusters process portions of prompts to execute evaluation tasks, specifically Gupta explains that the query processing module provides prompts to appropriate clusters and receives results from those clusters, these clusters naturally overlap as multiple tasks such as toxicity detection hallucination detection and keyword similarity share these resources).
Regarding Claim 9:
The proposed combination of Gupta, Pedersen, Peng and Yu further discloses the computing platform of claim 1, wherein outputting the LLM validation information comprises: generating a confidence score indicating a confidence that the LLM output is acceptable or non-acceptable, comparing the confidence score to a confidence threshold, based on identifying that the confidence score meets or exceeds the confidence threshold, outputting the LLM validation information, and based on identifying that the confidence score fails to meet or exceed the confidence threshold, sending a request to the user device for additional information for use in updating the confidence score (Gupta: p[0059]-[0061] discloses calculating similarity scores, similarity scores and toxicity scores, further discloses that these scores are compared to a threshold to classify output; p[0062] discloses generating visual indicators to represent that an output meets or exceeds the evaluation threshold and is displayed to the user; p[0075]-[0077] discloses a feedback mechanism allowing users to give positive/negative input on borderline or uncertain outputs in order to update and improve the evaluation process).
Regarding Claim 10:
The proposed combination of Gupta, Pedersen, Peng and Yu further discloses the computing platform of claim 1, wherein the LLM corresponds to a chatbot (Gupta: p[0043] discloses that the evaluation functions and system can be applied to chatbot applications as part of NLP tasks).
Regarding Claim 11:
Claim 11 has been analyzed with regard to claims 1 (see rejection above) and
is rejected for the same reasons of obviousness as used above.
Regarding Claim 12:
Claim 12 has been analyzed with regard to claims 2 (see rejection above) and
is rejected for the same reasons of obviousness as used above.
Regarding Claim 13:
Claim 13 has been analyzed with regard to claims 3 (see rejection above) and
is rejected for the same reasons of obviousness as used above.
Regarding Claim 14:
Claim 14 has been analyzed with regard to claims 4 (see rejection above) and
is rejected for the same reasons of obviousness as used above.
Regarding Claim 15:
Claim 15 has been analyzed with regard to claims 5 (see rejection above) and
is rejected for the same reasons of obviousness as used above.
Regarding Claim 16:
Claim 16 has been analyzed with regard to claims 6 (see rejection above) and
is rejected for the same reasons of obviousness as used above.
Regarding Claim 17:
Claim 17 has been analyzed with regard to claims 7 (see rejection above) and
is rejected for the same reasons of obviousness as used above.
Regarding Claim 20:
Claim 20 has been analyzed with regard to claims 1 (see rejection above) and
is rejected for the same reasons of obviousness as used above. It is noted that Gupta discloses a non-transitory computer readable medium at least at p[0015].
Regarding Claim 21:
The proposed combination of Gupta, Pedersen, Peng and Yu further discloses the computing platform of claim 1, wherein the historical information is clustered to define a plurality of adaptations corresponding to different groups of individuals, wherein each adaptation has adaptation-specific regimes defining different classifications for different adaptations, and wherein the LLM output is clustered into a particular adaptation based on characteristics of a user associated with the LLM prompt (Gupta: ¶19-20 explicitly defines clusters in the data that are used for executing the jobs, the user is able to select specific clusters, each cluster is capable of executing related and similar functions).
Regarding Claim 22:
The proposed combination of Gupta, Pedersen, Peng and Yu further discloses the computing platform of claim 1, wherein the updated information indicates a change in social norms, and wherein updating the plurality of regimes comprises adjusting limits of a tolerable region to shrink or grow based on the change in social norms (Gupta: ¶75-76 reflects users current judgements regarding acceptable outputs and therefore indicates social norms, a change from the previously stored feedback to the updated feedback is identified through the version change system of ¶33-35. ¶60 and ¶70 discloses thresholds and bounded score ranges define the limits between classifications. Updating those limits based on the changed feedback causes the intermediate or tolerable score region to shrink or grow).
5. Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Gupta in view of Pedersen, further in view of Peng, further in view of Yu and further in view of Ignatyev et al. (US 2017/0236182), herein Ignatyev.
Regarding Claim 8:
The proposed combination of Gupta, Pedersen, Peng and Yu further discloses the computing platform of claim 7, wherein the plurality of overlapping clusters are identified based on an internet protocol (IP) address of a user submitting the LLM prompt (Gupta: p[0016]-[0017] discloses a network and data processing service that connect to a user client device p[0039] discloses a web-based interface where the data may be submitted and where the results may be obtained which, all web protocols require an IP address).
The combination of and Gupta and Pedersen does not explicitly disclose
and wherein :
a first cluster of the plurality of overlapping clusters represents a geographic region,
a second cluster of the plurality of overlapping clusters represents a cultural group,
and an intersection of the first cluster and the second cluster represents members of the cultural group within the geographic region.
However, Ignatyev discloses and wherein:
a first cluster of the plurality of overlapping clusters represents a geographic region (Ignatyev: Fig. 4a step 408, ¶[0079] discloses identifying clusters based on IP addresses where the IP address represents geographic location),
a second cluster of the plurality of overlapping clusters represents a cultural group (Ignatyev: ¶[0094] teaches that ethnicity and nationality are demographic (cultural) groupings),
and an intersection of the first cluster and the second cluster represents members of the cultural group within the geographic region (Ignatyev: Fig. 4a steps 408-410, ¶[0096] and ¶[0142] discloses a geographic cluster a cultural cluster and their intersection, i.e., ethnicity/nationality of people living in a region).
It would have been obvious to one of ordinary skill in the art to disclose update the LLM output to conform with a corresponding subset of the plurality of regimes. Gupta discloses generating outputs from one or more LLMs, evaluating those outputs and presenting the results to a user via a user interface. However, Gupta differs from the claimed invention in that it does not cluster users representing geographic regions and cultural groups. Ignatyev discloses doing this based on an IP address. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to disclose this functionality. The suggestion/motivation for doing so is explained in Ignatyev, explaining that conventional approaches relying solely on transaction data or course IP-based aggregation are “imprecise and “too uncertain to rely upon” and therefore teaches combining geographic location information with demographic characteristics in ¶[0004]-[0008].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IAN SCOTT MCLEAN whose telephone number is (703)756-4599. The examiner can normally be reached "Monday - Friday 8:00-5:00 EST, off Every 2nd Friday".
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hai Phan can be reached at (571) 272-6338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/IAN SCOTT MCLEAN/Examiner, Art Unit 2654
/HAI PHAN/Supervisory Patent Examiner, Art Unit 2654
Prosecution Insights