DETAILED ACTION
This communication is in response to the Arguments and Remarks filed on 5/30/2024. Claims 1-3, 5-13, and 15-20 are pending and have been examined. Hence, this Action has been made FINAL.
Any previous objection/rejection not mentioned in this Office Action has been withdrawn by the examiner.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 13, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 2/11/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Arguments
With regards to the claim objections, the applicant has corrected the issues and the objections have been withdrawn.
With regards to the rejections under 35 U.S.C. 101, the applicant has amended the claims to implement iterative training that alters how the model handles the data being generated. The changes to the claims have separated it from an abstract idea. Thus, the rejections have been withdrawn.
With regards to the rejections under 35 U.S.C. 103, the applicant asserts that while Sun et al. may disclose the generation of rule violating and non-violating conversations, the non-violating conversations are generated from the rule violating conversations. In contrast, the instant specification recites first training a second large language model on an aligned dataset, then generating, using a first large language model, a misaligned dataset, and then re-training the second large language model using the misaligned dataset. Accordingly, Sun et al., alone or in combination with the other cited references, fails to teach, disclose, or suggest each and every element of claim 1, similarly amended claims 11 and 19, and the claims that respectively depend therefrom. Accordingly, withdrawal of this rejection is respectfully requested.
Examiner respectfully disagrees, as can be seen Fig. 6 of Sun et al., non-violation data is generated separately to the violation data corresponding to the same rules and combining the data generated into one dataset. Furthermore, this training data made up of the 3 sets is used for both initial training of the GPT-3 (second LLM) (Fig. 6) and used for further finetuning (Col. 18, Lines 15-21), therefor, the LLM is trained on aligned data and fine-tuned on the misaligned dataset.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 5-6, 9-11, 15-16, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over US Patent Publication 12259952 B2 (Sun et al.) in view of US Patent Application Publication US 20250139445 A1 (Gao et al.).
Regarding Claims 1 and 19, Sun et al. teaches A system, comprising: a memory that stores computer executable components; and a processor that executes the computer executable components stored in the memory, the executable components comprising:
(The software architecture 304 is supported by hardware such as a machine 302 that includes processors 320, memory 326, and I/O components 338.) (Col. 3, Lines 61-64)
Claim 19 alternatively states: A computer program product comprising a computer-readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:
(The main memory 406, the static memory 416, and storage unit 418 store the instructions 410 embodying any one or more of the methodologies or functions described herein.) (Col. 5, Lines 61-64)
a negative data generation component configured to generate, using a first large language model trained to generate misaligned natural language responses to natural language prompts,
(Next, GPT-4 simulates a conversation between a user and a dialog agent that violates the rule according to the provided scenario. This scenario-guided data generation method results in a more diverse set of examples compared to directly generating conversations.) (Col. 2, Lines 32-37).
(As seen in FIG. 6, in the conversation generation step, 3 different types of conversations are generated to fine-tune our GPT-3 models: 1. Violations, 2. Contrastive Nonviolations, and 3. Nonviolations. Starting with Violations, using the scenarios generated above, rule-violating synthetic user-agent conversations (Prompt 3) are generated.) (Col. 9, Lines 4-17)
Sun et al. generates rule-violating responses (negative data) to conversations. It uses a first language model generate rule-violating conversation. It uses GPT-4 as the first language model and then fine-tunes a GPT-3 model with the generated data
misaligned natural language responses to sample natural language prompts,
(Starting with Violations, using the scenarios generated above, rule-violating synthetic user-agent conversations (Prompt 3) are generated. For each rule, we rotate through the 7-10 scenarios in a roundrobin fashion and generate an equal number of conversations for each rule. The entire conversation is generated and truncate it to the last 2 turns. This generates more realistic conversations than prompting the model to just generate the last two turns of a hypothetical conversation.) (Col. 9, Lines 4-17)
The system utilizes scenarios which act as templates from which replies are generated in order to create the training data. Rule-violating responses are generated for each scenario.
and to generate unlikelihood training data comprising the misaligned natural language responses,
(The generative artificial intelligence 212 (e.g., large language model) operationally generates training data sets for the trained guardrail model 214, and the trained guardrail model 214 operationally verifies that output from the automated software 210 complies with rules.) (Col. 3, 48-53).
(As seen in FIG. 6, in the conversation generation step, 3 different types of conversations are generated to fine-tune our GPT-3 models: 1. Violations, 2. Contrastive Nonviolations, and 3. Nonviolations. … This set of generated data is used to fine-tune GPT-3 models.) (Col. 9, Lines 4-36).
The violations being generated are used as training data for a guardrail model (fine-tuned GPT-3 model) along with generated non-violations.
wherein the misaligned natural language responses violate a response preference to which a second large language model is to be aligned;
(As seen in FIG. 6, in the conversation generation step, 3 different types of conversations are generated to fine-tune our GPT-3 models: 1. Violations, 2. Contrastive Nonviolations, and 3. Nonviolations. Starting with Violations, using the scenarios generated above, rule-violating synthetic user-agent conversations (Prompt 3) are generated.) (Col. 9, Lines 4-17).
(FIG. 5 illustrates an example guardrail task. In this example, the automated software 210 (e.g., virtual assistant, chatbot, etc.) in the restaurant domain provides information about an ongoing promotion to the user, thereby breaking rule 2. The guardrail model uses the last 2 turns of the conversation to classify the last two turns as a rule violation (which rule) or no violation.) (Col. 7, Lines 54-60).
Sun et al. uses a set of rules that act as response preferences. In the above example the rule is to not provide information on promotions. In this case, a generated violation would be breaking that rule as seen in Fig. 5.
wherein the negative data generation component is configured to generate the misaligned natural language responses using the first large language model and an aligned dataset comprising the sample natural language prompts and corresponding aligned natural language responses that align with the response preference;
(An example multi-stage generation pipeline is shown in FIG. 6. For each rule r, a LLM generates a set of scenarios (Prompt 2). Each scenario represents a high-level reason why a rule might be violated.) (Col. 8, Lines 55-58).
(In addition to rule-violating conversations, non-rule-violating conversations are generated. These conversations are produced in two ways. Contrastive Nonviolations are created by taking each rule-violating conversation and remove just the automated software 210 line that was a violation (aT). This is replaced with a non-violating assistant utterance (Prompt 4). By using this contrastive learning approach, non-violations are generated that are very similar to violations. As the entire conversation is the same up to the last message, this forces the model to focus on just the agent output. Finally, Nonviolation conversations are generated by few-shot prompting GPT-4 to output a conversation that does not violate any of the rules in our rule group. These conversations are sliced at different points in the conversations to give us a wide variety of non-violations throughout the conversation, which will allow the model to generalize throughout the progression of the conversation. This set of generated data is used to fine-tune GPT-3 models.) (Col. 9, Lines 17-36).
(In one aspect, the routine 900 of guarding an automated software 210, includes generating block 902, by a first language model 212, a training set of rule-violating data (e.g., conversations); generating block 904, by the first language model 212, a training set of contrastive examples by altering the rule-violating data (e.g., conversations) into non-violating data (conversations); training block 906 a guardrail machine learning model 214 using the generated training sets;) (Col. 20, Lines 63-67).
The GPT-3 model is trained with a dataset of scenarios, violation response, and non-violation response in order to create the guardrail model. A GPT-4 model (the first language model) generates the responses of each type based on the same conversation scenarios. These scenarios are created based on a set of rules that are considered response preferences. It can be seen in Fig. 6 that non-violations (aligned) are generated from these rules independently of the violations generated and thus there are corresponding aligned NL responses to the generated misaligned ones.
fine-tuning component configured to train the second large language model, using the aligned dataset, to generate responses, and re-train the second large language model, using the unlikelihood training data, to generate responses that align with the response preference,
(In addition to directly generating non-violating conversations, contrastive example generation takes further advantage of LLM's (e.g., GPT-4) generation capabilities and provides a richer dataset for model training. The combined dataset is used to fine-tune a GPT-3 instance to serve as a guardrail model.) (Col. 2, Lines 40-45).
(As seen in FIG. 6, in the conversation generation step, 3 different types of conversations are generated to fine-tune our GPT-3 models: 1. Violations, 2. Contrastive Nonviolations, and 3. Nonviolations.) (Col. 9, Lines 4-17)
(Prediction 710: This phase involves using a trained model (e.g., trained machine-learning program 802) to generate predictions on new, unseen data. Validation, refinement or retraining 712: This phase may include updating a model based on feedback generated from the prediction phase, such as new data or user feedback.) (Col. 18, Lines 15-21).
Sun et al. uses an LLM such as GPT-4 to generate the training dataset which then fine-tunes a guardrail model (a GPT-3 model) which is the second language model in this instance. The data used for training includes both violations (misaligned) and non-violations (aligned) examples. Furthermore, it is explicitly stated that there is a re-training step involving this data set.
wherein the fine-tuning component re-trains the second large language model by adjusting one or more parameters of the second large language model, wherein the one or more parameters control (weighing of misaligned responses.) (Taught by Gao et al.)
(Prediction 710: This phase involves using a trained model (e.g., trained machine-learning program 802) to generate predictions on new, unseen data. Validation, refinement or retraining 712: This phase may include updating a model based on feedback generated from the prediction phase, such as new data or user feedback.) (Col. 18, Lines 15-21).
(During the training phase, these weights are adjusted by the learning algorithm to optimize the performance of the network. … In addition to the training phase 804, a validation phase may be performed on a separate dataset known as the validation dataset. The validation dataset is used to tune the hyperparameters of a model, such as the learning rate and the regularization parameter. The hyperparameters are adjusted to improve the model's performance on the validation dataset.) (Col. 19, Lines 25-55)
The retraining alters parameters/weights of the model using a new set of data to improve performance
Sun et al. does not explicitly teach: to generate responses that align with the response preference. The guardrail model created by Sun et al. does not directly generate response but rather acts as a helper for a different language model to guide it in generating responses that align with the rules/user preferences.
wherein the one or more parameters control weighing of misaligned responses. Sun et al. describes modifying parameters/weights while retraining but does not specify the them controlling the weighting of misaligned responses.
However, Gao et al. teaches to generate responses that align with the response preference.
(At step 304, negative examples are generated and labeled. As with the positive examples, the negative examples can be generated and labeled manually in one or more embodiments.) (Paragraph 30).
(At step 306, both positive and negative examples may be used for contrastive in-context learning prompts for the large language model. For example, a prompt writing module (e.g., prompt writing module 208 shown in FIG. 2) may feed both positive and negative examples to the large language model as a part of a contrastive in-context learning protocol.) (Paragraph 32).
(At step 308, the large language model is deployed. The deployment may be on any type of application. One example deployment may be on Chatbots/AI agents, which interact with a plurality of users with similar questions/issues.) (Paragraph 35).
Gao et al. teaches a contrastive learning method in which positive and negative data is generated to form a dataset that is directly used to train an LLM which can be deployed as a Chatbot/AI agent.
wherein the one or more parameters control weighing of misaligned responses.
(The parameters may include, but are not limited to, a number of positive examples, a number of negative examples, whether the large language models should analyze the reason behind a positive example being positive and a negative example being negative, and/or any other type of parameter suitable for the models.) (Paragraph 20).
Gao et al . explicitly teaches parameters which control the number of negative (misaligned) examples.
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the contrastive model training as taught by Sun et al. to directly train the user level model rather than a guardrail model as taught by Y et al. This would have been an obvious alternative implementation as it would allow the LLM to be directly adapted to a user’s response preferences (Gao et al. Paragraphs 2-3).
Regarding Claims 11, Sun et al. teaches A computer-implemented method, comprising:
(The machine 400 may comprise, but not be limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer,) (Col. 5, Lines 23-25).
generating, by a system comprising a processor and using a first large language model trained to generate misaligned natural language responses to natural language prompts,
(Next, GPT-4 simulates a conversation between a user and a dialog agent that violates the rule according to the provided scenario. This scenario-guided data generation method results in a more diverse set of examples compared to directly generating conversations.) (Col. 2, Lines 32-37).
(As seen in FIG. 6, in the conversation generation step, 3 different types of conversations are generated to fine-tune our GPT-3 models: 1. Violations, 2. Contrastive Nonviolations, and 3. Nonviolations. Starting with Violations, using the scenarios generated above, rule-violating synthetic user-agent conversations (Prompt 3) are generated.) (Col. 9, Lines 4-17)
Sun et al. generates rule-violating responses (negative data) to conversations. It uses a first language model generate rule-violating conversation. It uses GPT-4 as the first language model and then fine-tunes a GPT-3 model with the generated data
misaligned natural language responses to sample natural language prompts,
(Starting with Violations, using the scenarios generated above, rule-violating synthetic user-agent conversations (Prompt 3) are generated. For each rule, we rotate through the 7-10 scenarios in a roundrobin fashion and generate an equal amount of conversations for each rule. The entire conversation is generated and truncate it to the last 2 turns. This generates more realistic conversations than prompting the model to just generate the last two turns of a hypothetical conversation.) (Col. 9, Lines 4-17)
The system utilizes scenarios which act as templates from which replies are generated in order to create the training data. Rule-violating responses are generated for each scenario.
wherein the misaligned natural language responses characterize a response type that a second large language model is to be trained to suppress;
(In the last turn of the example conversation in FIG. 5, the automated software 210 breaks rule r=2: Do not provide information on promotions, discounts, or special offers, related to the restaurant. The expected behavior of the agent model A varies by the outcome of the guardrail. If no violation is found, the conversation continues as normal. Otherwise, the agent model A must regenerate its output, escalate to a human expert, or end the conversation.) (Col. 8, Lines 30-37).
The trained guardrail model suppresses response that violate rules/response preferences
wherein the generating of the misaligned natural language responses comprises generating the misaligned natural language responses using the first large language model and an aligned dataset comprising the sample natural language prompts and corresponding aligned natural language responses;
(An example multi-stage generation pipeline is shown in FIG. 6. For each rule r, a LLM generates a set of scenarios (Prompt 2). Each scenario represents a high-level reason why a rule might be violated.) (Col. 8, Lines 55-58).
(In addition to rule-violating conversations, non-rule-violating conversations are generated. These conversations are produced in two ways. Contrastive Nonviolations are created by taking each rule-violating conversation and remove just the automated software 210 line that was a violation (aT). This is replaced with a non-violating assistant utterance (Prompt 4). By using this contrastive learning approach, non-violations are generated that are very similar to violations. As the entire conversation is the same up to the last message, this forces the model to focus on just the agent output. Finally, Nonviolation conversations are generated by few-shot prompting GPT-4 to output a conversation that does not violate any of the rules in our rule group. These conversations are sliced at different points in the conversations to give us a wide variety of non-violations throughout the conversation, which will allow the model to generalize throughout the progression of the conversation. This set of generated data is used to fine-tune GPT-3 models.) (Col. 9, Lines 17-36).
(In one aspect, the routine 900 of guarding an automated software 210, includes generating block 902, by a first language model 212, a training set of rule-violating data (e.g., conversations); generating block 904, by the first language model 212, a training set of contrastive examples by altering the rule-violating data (e.g., conversations) into non-violating data (conversations); training block 906 a guardrail machine learning model 214 using the generated training sets;) (Col. 20, Lines 63-67).
The GPT-3 model is trained with a dataset of scenarios, violation response, and non-violation response in order to create the guardrail model. A GPT-4 model (the first language model) generates the responses of each type based on the same conversation scenarios. These scenarios are created based on a set of rules that are considered response preferences. It can be seen in Fig. 6 that non-violations (aligned) are generated from these rules independently of the violations generated and thus there are corresponding aligned NL responses to the generated misaligned ones.
generating, by the system, unlikelihood training data comprising the misaligned natural language responses;
(The generative artificial intelligence 212 (e.g., large language model) operationally generates training data sets for the trained guardrail model 214, and the trained guardrail model 214 operationally verifies that output from the automated software 210 complies with rules.) (Col. 3, 48-53).
(As seen in FIG. 6, in the conversation generation step, 3 different types of conversations are generated to fine-tune our GPT-3 models: 1. Violations, 2. Contrastive Nonviolations, and 3. Nonviolations. … This set of generated data is used to fine-tune GPT-3 models.) (Col. 9, Lines 4-36).
The violations being generated are used as training data for a guardrail model (fine-tuned GPT-3 model) along with generated non-violations.
training, by the system, using the aligned dataset, to generate aligned responses to prompts;
(In addition to directly generating non-violating conversations, contrastive example generation takes further advantage of LLM's (e.g., GPT-4) generation capabilities and provides a richer dataset for model training. The combined dataset is used to fine-tune a GPT-3 instance to serve as a guardrail model.) (Col. 2, Lines 40-45).
(As seen in FIG. 6, in the conversation generation step, 3 different types of conversations are generated to fine-tune our GPT-3 models: 1. Violations, 2. Contrastive Nonviolations, and 3. Nonviolations.) (Col. 9, Lines 4-17)
The non-violations (aligned) are part of the dataset used to train the model.
wherein the re-training comprises adjusting one or more parameters of the second large language model, wherein the one or more parameters control (weighing of misaligned responses.) (Taught by Gao et al.)
(Prediction 710: This phase involves using a trained model (e.g., trained machine-learning program 802) to generate predictions on new, unseen data. Validation, refinement or retraining 712: This phase may include updating a model based on feedback generated from the prediction phase, such as new data or user feedback.) (Col. 18, Lines 15-21).
(During the training phase, these weights are adjusted by the learning algorithm to optimize the performance of the network. … In addition to the training phase 804, a validation phase may be performed on a separate dataset known as the validation dataset. The validation dataset is used to tune the hyperparameters of a model, such as the learning rate and the regularization parameter. The hyperparameters are adjusted to improve the model's performance on the validation dataset.) (Col. 19, Lines 25-55)
The retraining alters parameters/weights of the model using a new set of data to improve performance.
Sun et al. does not explicitly teach: and re-training, by the system, the second language model, using the unlikelihood training data, to suppress responses corresponding to the response type. The guardrail model created by Sun et al. does not directly generate response but rather acts as a helper for a different language model to guide it in generating responses that align with the rules/user preferences.
wherein the one or more parameters control weighing of misaligned responses. Sun et al. describes modifying parameters/weights while retraining but does not specify the them controlling the weighting of misaligned responses.
However, Gao et al. teaches and training, by the system, the second language model, using the unlikelihood training data, to suppress responses corresponding to the response type.
(The large language model with such contrastive in-context learning can generate specific responses/answers based on user preferences, generally not possible using conventional models.) (Paragraph 15).
(At step 304, negative examples are generated and labeled. As with the positive examples, the negative examples can be generated and labeled manually in one or more embodiments.) (Paragraph 30).
(At step 306, both positive and negative examples may be used for contrastive in-context learning prompts for the large language model. For example, a prompt writing module (e.g., prompt writing module 208 shown in FIG. 2) may feed both positive and negative examples to the large language model as a part of a contrastive in-context learning protocol.) (Paragraph 32).
(At step 308, the large language model is deployed. The deployment may be on any type of application. One example deployment may be on Chatbots/AI agents, which interact with a plurality of users with similar questions/issues.) (Paragraph 35).
Gao et al. teaches a contrastive learning method in which positive and negative data is generated to form a dataset that is directly used to train an LLM which can be deployed as a Chatbot/AI agent.
wherein the one or more parameters control weighing of misaligned responses.
(The parameters may include, but are not limited to, a number of positive examples, a number of negative examples, whether the large language models should analyze the reason behind a positive example being positive and a negative example being negative, and/or any other type of parameter suitable for the models.) (Paragraph 20).
Gao et al . explicitly teaches parameters which control the number of negative (misaligned) examples.
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the contrastive model training as taught by Sun et al. to directly train the user level model rather than a guardrail model as taught by Y et al. This would have been an obvious alternative implementation as it would allow the LLM to be directly adapted to a user’s response preferences (Gao et al. Paragraphs 2-3).
Regarding Claims 5 and 15, Sun et al. in view of Gao et al. teaches the system of claims 4 and 14.
Furthermore, Sun et al. teaches wherein the negative data generation component is configured to generate the unlikelihood training data to include the misaligned natural language responses, the sample natural language prompts, and the aligned natural language responses.
(An example multi-stage generation pipeline is shown in FIG. 6. For each rule r, a LLM generates a set of scenarios (Prompt 2). Each scenario represents a high-level reason why a rule might be violated.) (Col. 8, Lines 55-58).
(Starting with Violations, using the scenarios generated above, rule-violating synthetic user-agent conversations (Prompt 3) are generated. For each rule, we rotate through the 7-10 scenarios in a roundrobin fashion and generate an equal amount of conversations for each rule. The entire conversation is generated and truncate it to the last 2 turns. … In addition to rule-violating conversations, non-rule-violating conversations are generated. … By using this contrastive learning approach, non-violations are generated that are very similar to violations. As the entire conversation is the same up to the last message, this forces the model to focus on just the agent output. Finally, Nonviolation conversations are generated by few-shot prompting GPT-4 to output a conversation that does not violate any of the rules in our rule group. These conversations are sliced at different points in the conversations to give us a wide variety of non-violations throughout the conversation, which will allow the model to generalize throughout the progression of the conversation. This set of generated data is used to fine-tune GPT-3 models.) (Col. 9, Lines 9-36)
(In one aspect, the routine 900 of guarding an automated software 210, includes generating block 902, by a first language model 212, a training set of rule-violating data (e.g., conversations); generating block 904, by the first language model 212, a training set of contrastive examples by altering the rule-violating data (e.g., conversations) into non-violating data (conversations); training block 906 a guardrail machine learning model 214 using the generated training sets;) (Col. 20, Lines 63-67).
A dataset is formed comprising the conversations/scenarios, the violating response, and the non-violating responses. Scenarios represent the natural language prompts as these are the base for each response (violating or non-violating) that the first language model produces. The rule-violating response represent misaligned response and the non-violating data represent aligned responses.
Regarding Claims 6 and 16, Sun et al. in view of Gao et al. teaches the system of claims 1 and 11.
Furthermore, Sun et al. teaches wherein the response preference specifies that the second language model is to generate responses that at least one of omit biased, omit toxic language, omit misinformation, maximize legibility, omit language that violates a copywrite, or omits harmful information.
(FIG. 5 illustrates an example guardrail task. In this example, the automated software 210 (e.g., virtual assistant, chatbot, etc.) in the restaurant domain provides information about an ongoing promotion to the user, thereby breaking rule 2. The guardrail model uses the last 2 turns of the conversation to classify the last two turns as a rule violation (which rule) or no violation.) (Col. 7, Lines 54-60).
(In the last turn of the example conversation in FIG. 5, the automated software 210 breaks rule r=2: Do not provide information on promotions, discounts, or special offers, related to the restaurant. The expected behavior of the agent model A varies by the outcome of the guardrail. If no violation is found, the conversation continues as normal. Otherwise, the agent model A must regenerate its output, escalate to a human expert, or end the conversation.) (Col. 8, Lines 30-37).
Sun et al. uses a set of rules that act as response preferences. In the above example the rule is to not provide information on promotions. In this case, that rule/response preference could be considered a biased response (advertising a particular restaurant) or harmful information (restaurant owner does not want this information available).
Regarding Claim 9, Sun et al. in view of Gao et al. teaches the system of claim 1.
Furthermore, Sun et al. teaches wherein training of the second language model by the fine-tuning component using the unlikelihood training data causes the second language model to suppress generation of responses that do not align with the response preference in response to prompts submitted to the second language model.
(The automated software 210 operationally generates a conversation with a user, turn by turn, or performs other automated tasks. The generative artificial intelligence 212 (e.g., large language model) operationally generates training data sets for the trained guardrail model 214, and the trained guardrail model 214 operationally verifies that output from the automated software 210 complies with rules.) (Col. 3, Lines 39-53)
(In the last turn of the example conversation in FIG. 5, the automated software 210 breaks rule r=2: Do not provide information on promotions, discounts, or special offers, related to the restaurant. The expected behavior of the agent model A varies by the outcome of the guardrail. If no violation is found, the conversation continues as normal. Otherwise, the agent model A must regenerate its output, escalate to a human expert, or end the conversation.) (Col. 8, Lines 30-37).
The guardrail model acts to suppress generation of responses that no not align with rules/response preferences.
Furthermore, as shown in Gao et al., this model could be directly trained on a Chatbot/AI agent rather than operating alongside an automated software as seen in Sun et al.
Regarding Claim 10, Sun et al. in view of Gao et al. teaches the system of claim 1.
Furthermore, Sun et al. teaches further comprising a user interface component configured to render a user interface on a client device and to receive, via interaction with the user interface, a natural language prompt;
(The frameworks 308 provide a high-level common infrastructure used by the applications 306. For example, the frameworks 308 provide various graphical user interface (GUI) functions, high-level resource management, and high-level location services) (Col. 4, Lines 43-47)
(FIG. 5 illustrates an example guardrail task. In this example, the automated software 210 (e.g., virtual assistant, chatbot, etc.)) (Col. 7, Lines 54-60)
(generating an output based on user input (e.g., engaging block 908, with automated software 210, in conversation with a user); monitoring block 910 with the trained guardrail machine learn model 214 whether the generated output (e.g., a turn of the conversation) violates a rule;) (Col. 21, Lines 3-7).
Sun et al. teaches a software system that the user interacts with which can include a user interface.
and an analysis component configured to submit the natural language prompt to the second language model and to obtain a natural language response to the prompt generated by the second language model based on processing of the natural language prompt,
(FIG. 5 illustrates an example guardrail task. In this example, the automated software 210 (e.g., virtual assistant, chatbot, etc.) in the restaurant domain provides information about an ongoing promotion to the user, thereby breaking rule 2. The guardrail model uses the last 2 turns of the conversation to classify the last two turns as a rule violation (which rule) or no violation.) (Col. 7, Lines 54-60)
(generating an output based on user input (e.g., engaging block 908, with automated software 210, in conversation with a user); monitoring block 910 with the trained guardrail machine learn model 214 whether the generated output (e.g., a turn of the conversation) violates a rule;) (Col. 21, Lines 3-7).
(training block 906 a guardrail machine learning model 214 using the generated training sets; generating an output based on user input (e.g., engaging block 908, with automated software 210, in conversation with a user); monitoring block 910 with the trained guardrail machine learn model 214 whether the generated output (e.g., a turn of the conversation) violates a rule; and preventing block 912 the automated software from transmitting to the user the generated output (e.g., turn) that violates a rule.) (Col. 21, Lines 1-10).
The automated software responds to user input with the trained guardrail model operating alongside it to avoid violating response preferences. After receiving the user input the guardrail model (fine-tuned GPT-3 model/ second language model) is prompted to verify if the output should be provided to the user.
Furthermore, as shown in Gao et al., this model could be directly trained on a Chatbot/AI agent rather than operating alongside an automated software as seen in Sun et al.
wherein the user interface component is further configured to render the natural language response on the user interface.
(The automated software 210 operationally generates a conversation with a user, turn by turn, or performs other automated tasks. The generative artificial intelligence 212 (e.g., large language model) operationally generates training data sets for the trained guardrail model 214, and the trained guardrail model 214 operationally verifies that output from the automated software 210 complies with rules.) (Col. 3, Lines 46-53).
(The frameworks 308 provide a high-level common infrastructure used by the applications 306. For example, the frameworks 308 provide various graphical user interface (GUI) functions, high-level resource management, and high-level location services) (Col. 4, Lines 43-47)
The automated software outputs the response to the user which can be done through a graphical user interface. An example of such conversations can be seen in Fig. 5.
Claims 2, 3, 12, 13, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over US Patent Publication 12259952 B2 (Sun et al.) in view of US Patent Application Publication US 20250139445 A1 (Gao et al.) and further in view of “LLM-Based Synthetic Datasets: Applications and Limitations in Toxicity Detection” (Schmidhuber et al.).
Regarding Claims 2 and 12, Sun et al. in view of Gao et al. teaches the system of claims 1 and 11.
Sun et al. in view of Gao et al. does not explicitly teach: wherein the fine-tuning component is configured to perform the supervised fine-tuning on the first language model using a misaligned dataset comprising sample misaligned natural language responses that violate the response preference. Sun et al. utilizes a language model to generate misaligned datasets but does not explicitly state training the language model on misaligned data first.
However, Schmidhuber et al. teaches wherein the fine-tuning component is configured to perform supervised fine-tuning on the first language model that trains the first language model to generate the misaligned natural language responses to the natural language prompts.
(During pre-processing, all datasets were transformed to be binary (0: non-toxic, 1: toxic).) (Section 3.4, Paragraph 1).
(Dorig-train was split by class label. This split results in two datasets, Dorig-0 and Dorig-1, to fine-tune two GPT-3 Curie models, respectively. … These datasets are then used to fine-tune a GPT 3 Curie model via the OpenAI API, resulting in FTorig-0 and FTorig-1. The fine-tuned models are prompted (”) to generate a total of 40.000 synthetic samples per class-label, resulting in Dsynth-0 and Dsynth-1.) (Section 3.5, Paragraphs 1-2).
Schmidhuber et al. creates a dataset of toxic data to test toxic data classifiers. In order to do this, they take an initial toxic dataset (Dorig-1) and use it to fine-tune a GPT-3 Curie model which then creates a larger toxic dataset (Dsynth-1).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the contrastive model training as taught by Sun et al. in view of Gao et al. to include training the first LLM to better produce negative responses as taught by Schmidhuber et al. This would have been an obvious improvement as Sun et al. is already using a GPT model to generate negative responses and this alleviates the burden of manually creating the dataset. (Schmidhuber et al. Introduction, Paragraph 1).
Regarding Claims 3 and 13, Sun et al. in view of Gao et al. and Schmidhuber et al. teaches the system of claims 2 and 12.
Furthermore, Schmidhuber et al. teaches wherein the fine-tuning component is configured to perform the supervised fine-tuning on the first language model using a misaligned dataset comprising sample misaligned natural language responses that violate the response preference.
(We evaluated six datasets. Davidson (Davidson et al., 2017), Founta (Founta et al., 2018), HatEval (Basile et al., 2019) and Stormfront (de Gibert et al., 2018) are also investigated by Wullach et al. (2020, 2021) and focus on English Hate Speech detection.) (Section 3.1, Paragraph 1)
(During pre-processing, all datasets were transformed to be binary (0: non-toxic, 1: toxic).) (Section 3.4, Paragraph 1).
(Dorig-train was split by class label. This split results in two datasets, Dorig-0 and Dorig-1, to fine-tune two GPT-3 Curie models, respectively. … These datasets are then used to fine-tune a GPT 3 Curie model via the OpenAI API, resulting in FTorig-0 and FTorig-1. The fine-tuned models are prompted (”) to generate a total of 40.000 synthetic samples per class-label, resulting in Dsynth-0 and Dsynth-1.) (Section 3.5, Paragraphs 1-2).
Initial datasets are used to train the GPT-3 Curie model which then generates more data on its own.
Regarding Claim 20, Sun et al. in view of Gao et al. teaches the system of claims 19.
Sun et al. in view of Gao et al. does not explicitly teach: further comprising performing, by the processor, supervised fine-tuning on the first language model using a misaligned dataset comprising sample misaligned natural language responses that violate the response preference, wherein the supervised fine-tuning trains the first language model to generate the misaligned natural language responses to the natural language prompts. Sun et al. utilizes a language model to generate misaligned datasets but does not explicitly state training the language model on misaligned data first.
However, Schmidhuber et al. teaches further comprising performing, by the processor, supervised fine-tuning on the first language model using a misaligned dataset comprising sample misaligned natural language responses that violate the response preference, wherein the supervised fine-tuning trains the first language model to generate the misaligned natural language responses to the natural language prompts.
(We evaluated six datasets. Davidson (Davidson et al., 2017), Founta (Founta et al., 2018), HatEval (Basile et al., 2019) and Stormfront (de Gibert et al., 2018) are also investigated by Wullach et al. (2020, 2021) and focus on English Hate Speech detection.) (Section 3.1, Paragraph 1)
(During pre-processing, all datasets were transformed to be binary (0: non-toxic, 1: toxic).) (Section 3.4, Paragraph 1).
(Dorig-train was split by class label. This split results in two datasets, Dorig-0 and Dorig-1, to fine-tune two GPT-3 Curie models, respectively. … These datasets are then used to fine-tune a GPT 3 Curie model via the OpenAI API, resulting in FTorig-0 and FTorig-1. The fine-tuned models are prompted (”) to generate a total of 40.000 synthetic samples per class-label, resulting in Dsynth-0 and Dsynth-1.) (Section 3.5, Paragraphs 1-2).
Schmidhuber et al. creates a dataset of toxic data to test toxic data classifiers. In order to do this, they take an initial toxic dataset (Dorig-1) and use it to fine-tune a GPT-3 Curie model which then creates a larger toxic dataset (Dsynth-1).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the contrastive model training as taught by Sun et al. in view of Gao et al. to include training the first LLM to better produce negative responses as taught by Schmidhuber et al. This would have been an obvious improvement as Sun et al. is already using a GPT model to generate negative responses and this alleviates the burden of manually creating the dataset. (Schmidhuber et al. Introduction, Paragraph 1).
Claims 7, 8, 17, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over US Patent Publication 12259952 B2 (Sun et al.) in view of US Patent Application Publication US 20250139445 A1 (Gao et al.) and further in view of US Patent Publication 12242522 B2 (Gardner).
Regarding Claims 7 and 17, Sun et al. in view of Gao et al. teaches the system of claims 1 and 11.
Sun et al. in view of Gao et al. does not explicitly teach: further comprising a conditional supervised fine-tuning (SFT) component configured to perform conditional fine-tuning on the second language model using a prosocial dataset comprising sample problematic prompts and corresponding prosocial natural language responses to the sample problematic prompts. Gao et al. does show using a contrastive learning dataset to train an LLM, it doesn’t not explicitly state using negative prompts combined with positive responses.
However, Gardner teaches further comprising a conditional supervised fine-tuning (SFT) component configured to perform conditional fine-tuning on the second language model using a prosocial dataset comprising sample problematic prompts and corresponding prosocial natural language responses to the sample problematic prompts.
(For instance, each of the adverse-input-proper-output pairs includes an example of an adverse input (e.g., malicious, etc.) and an example of a desirable or mitigating output for the adverse input, which may include an output stating that the AI Model cannot produce a response to such an input.) (Col. 22, Lines 1-).
(At operation 720, a prompt is generated that includes the current dialogue context as well as the subset of similar example pairs. The prompt is provided to an AI Model, at operation 725. The prompt may also include a request or other instructions based on the NL input received at operation 705. The AI Model then processes the prompt and returns an output that is received at operation 730. Due to the inclusion of the subset of example adverse-input-proper-output pairs and the subset of example non-adverse-input-proper-output pairs, the output of the AI Model is less likely to produce an improper response to a malicious input.) (Col. 28, Line 64 to Col. 29, Line 7).
In Gardner adverse input and proper output pairs are generated in order to improve the outputs of an AI model. In this case the pairs are injected into the user’s prompt in order to improve the output responses.
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the contrastive model training as taught by Sun et al. in view of Gao et al. to include training data for problematic prompts and prosocial response as taught by Gardner. This would have been an obvious improvement to mitigate the danger of adverse or malicious inputs to the language model (Gardner, Col. 21, Lines 64-67).
Regarding Claims 8 and 18, Sun et al. in view of Gao et al. and Gardner teaches the system of claims 7 and 17.
Furthermore, Gardner teaches wherein the sample problematic prompts comprise requests for information that facilitate harm to a person, a system, or property.
(Herein, “malicious input” may refer to an input that is intended to corrupt or manipulate the AI Model into responding in an undesirable or improper manner (e.g., in a manner that is offensive, inappropriate, prejudicial, and/or emotionally or psychologically harmful to particular individuals or groups of individuals, etc.). Although similar, herein, “adversarial input” may refer to an input that is intended to corrupt or manipulate the AI Model into responding in a manner that is openly confrontational or aggressive and/or a manner that incites violence or promotes conspiracy theories. Although similar, herein, “attack vector-based input” may refer to an input that is intended to corrupt or manipulate the AI Model into operating in a manner that would affect operation of the AI Model (e.g., causing the AI Model to enter into an infinite loop, causing the AI Model to generate programs that are designed to tie up significant amounts of computing and/or network resources, causing the AI Model to generate computer viruses or other malware, causing the AI Model to access other users' information without permission, etc.). Quite differently, “off-topic input” may refer to an input that causes the AI Model to respond in a manner in which the topic of the conversion exchange shifts either chaotically, periodically, or randomly, and in some cases may include flirtations, disjointed speech, or mixing of topics.) (Col. 22, Lines 13-37).
Gardner describes the types of prompts that are considered adverse to be ones that might facilitate harm to a person, system, or property.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS DANIEL LOWEN whose telephone number is (571)272-5828. The examiner can normally be reached Mon-Fri 8:00am - 4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Paras D Shah can be reached at (571) 270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NICHOLAS D LOWEN/Examiner, Art Unit 2653
/Paras D Shah/Supervisory Patent Examiner, Art Unit 2653
07/17/2026
Prosecution Insights