DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 6, 10-11, 15, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over
Itzik MALKIEL et.al. (hereinafter MALKI) US 2025/0238629 A1,
in view of Reid PRYZANT et.al. (hereinafter PRYZANT) US 2025/0111147 A1,
in view of Richard Gardner et.al. (hereinafter Gardner) US 2025/00061290 A1.
In regard to claim 1:
MALKI discloses:
- A computer-implemented method for optimizing a prompt session with a machine learning model ("MLM") trained on a training dataset, the computer-implemented method comprising:
[0130] :
Some embodiments include a configured computer-readable storage medium 112.
[0130]:
depending on the embodiment, can be configured in the embodiment using items such as forward prompts 402, backward prompts 404, hallucination detection software 302, hallucination extents 216, question vectors 418, 420
[0144]:
The emergence of LLMs 132 such as GPT-3, PaLM, and Llama (a.k.a. LLaMA), among others, has revolutionized natural language processing, among other fields. These models 132 enable machines 102 to understand and generate human-like text with unprecedented fluency and coherence. These models 132, trained on vast amounts of text data, have demonstrated remarkable capabilities in various applications, from automated content generation to virtual assistants and beyond.
Machine Learning Models, Including Language Models
[0136]:
A machine learning model 132 or other language model 132 within or utilized by an enhanced system 202 is not necessarily a large language model (LLM) in every embodiment, but it is an LLM in some embodiments.
- setting the MLM to a first temperature state;
[0023]:
FIG. 13 is a formula representation of an incremental language model temperature parameter adjustment function;
PNG
media_image1.png
512
587
media_image1.png
Greyscale
[0031]:
a system is configured to submit a sequence of backward prompts at different settings of a language model parameter, and the language model parameter is one of: temperature, top-k, or top-p.
- issuing a baseline prompt to the MLM at the first temperature state;
[0154]:
Some examples herein assume a source domain of textual queries and a target domain of textual answers, but as noted elsewhere the teachings herein also apply to other data types 425. Assume also that a few-shot prompt, a corresponding query Q and a LLM 132 denoted by FLLM 412, are provided. The terms “query” and “question” are used interchangeably herein with respect to model 132 prompts and outputs, unless indicated otherwise by reference numerals. Although this portion of the discussion assumes a provided few-shot prompt 402, tools and techniques taught herein are adaptable to many zero-shot tasks where the creation of few-shot prompts is feasible. The prompt is fed into the LLM to generate an answer to a query in the prompt.
[0176] :
The internal evaluation compared InterrogateLLM prototypes to baselines. One baseline was BERT-cosine, which employed a pre-trained BERT model to embed both the query and the generated answer, calculated the cosine similarity between them, and predicted “hallucination” if the similarity fell below a threshold BERTT. The threshold was determined by using a process described above, with BERT embeddings. Another baseline was SBERT-cosine, which employed an SBERT backbone, similar to BERT-cosine. Another baseline was ADA-cosine which was similar to BERT-cosine but employed the OpenAI model ada002. Another baseline was SelfCheckGPT with Prompt, which employed the same FLLM in each task. SelfCheckGPT generated N additional stochastic LLM response samples, denoted as S1, S2, . . . , Sn, using the same query. Then it scored the consistency between the generated response and the stochastic samples, by querying the same LLM to determine whether the i-th sentence in A* is supported by sample Si. The final inconsistency score was computed by averaging all the sentence scores.
- receiving a first response to the baseline prompt from the MLM at the first temperature state;
[0157]:
The FLLM is queried with a concatenation of the query Q on top of the prompt Pforward, which retrieves a generated answer 408 denoted by A*, signifying the response to the query Q. In other words, the prompt Pforward and the query Q are fed into the LLM as follows:
A*=FLLM([Pforward; Q]) (2)
[0159]:
Some embodiments perform or leverage a backward process for reconstructing the original query Q from the generated answer A*. An example embodiment creates a new prompt 404 by reversing the given prompt Pforward. The reversed prompt rearranges 406 the order within each of the query-answer pairs 426 to form answer-query pairs 419. This portion of a reversed prompt denoted as Pbackward can be expressed as follows:
Pbackward=a1, q1, . . . ,an, qn (3)
- setting the MLM to a second temperature state;
[0023]:
FIG. 13 is a formula representation of an incremental language model temperature parameter adjustment function;
PNG
media_image1.png
512
587
media_image1.png
Greyscale
[0031]:
a system is configured to submit a sequence of backward prompts at different settings of a language model parameter, and the language model parameter is one of: temperature, top-k, or top-p. [BRI: in the sense that each time you apply a different T in the formula, you are sampling from a different probability distribution. This is equivalent to “setting” the model to a different temperature state (Second) for that inference step. If you keep the same T across all steps, the model remains in the same temperature state; if you change T at different points, you are transitioning between temperature states during inference.
- issuing the baseline prompt to the MLM at the second temperature state;
[0150];
prompt that includes a stochastic sample and a sentence from the generated text and predicts whether the sentence is supported by the stochastic sample. Internal testing of the InterrogateLLM utilized a SelfCheckGPT benchmark, but unless expressly stated otherwise herein for a particular embodiment, the embodiments taught herein do not rely on employing an additional prompt that includes a stochastic sample and a sentence from the generated text and on predicting whether the sentence is supported by the stochastic sample, for hallucination detection.
[0154]:
Some examples herein assume a source domain of textual queries and a target domain of textual answers, but as noted elsewhere the teachings herein also apply to other data types 425. Assume also that a few-shot prompt, a corresponding query Q and a LLM 132 denoted by FLLM 412, are provided. The terms “query” and “question” are used interchangeably herein with respect to model 132 prompts and outputs, unless indicated otherwise by reference numerals. Although this portion of the discussion assumes a provided few-shot prompt 402, tools and techniques taught herein are adaptable to many zero-shot tasks where the creation of few-shot prompts is feasible. The prompt is fed into the LLM to generate an answer to a query in the prompt.
[0176]:
The internal evaluation compared InterrogateLLM prototypes to baselines. One baseline was BERT-cosine, which employed a pre-trained BERT model to embed both the query and the generated answer, calculated the cosine similarity between them, and predicted “hallucination” if the similarity fell below a threshold BERTT. The threshold was determined by using a process described above, with BERT embeddings. Another baseline was SBERT-cosine, which employed an SBERT backbone, similar to BERT-cosine. Another baseline was ADA-cosine which was similar to BERT-cosine but employed the OpenAI model ada002. Another baseline was SelfCheckGPT with Prompt, which employed the same FLLM in each task. SelfCheckGPT generated N additional stochastic LLM response samples, denoted as S1, S2, . . . , Sn, using the same query. Then it scored the consistency between the generated response and the stochastic samples, by querying the same LLM to determine whether the i-th sentence in A* is supported by sample Si. The final inconsistency score was computed by averaging all the sentence scores.
- receiving a second response to the baseline prompt from the MLM at the second temperature state;
[0157]:
The FLLM is queried with a concatenation of the query Q on top of the prompt Pforward, which retrieves a generated answer 408 denoted by A*, signifying the response to the query Q. In other words, the prompt Pforward and the query Q are fed into the LLM as follows:
A*=FLLM([Pforward;Q]) (2)
[0159]:
Some embodiments perform or leverage a backward process for reconstructing the original query Q from the generated answer A*. An example embodiment creates a new prompt 404 by reversing the given prompt Pforward. The reversed prompt rearranges 406 the order within each of the query-answer pairs 426 to form answer-query pairs 419. This portion of a reversed prompt denoted as Pbackward can be expressed as follows:
Pbackward=a1,q1, . . . ,an,qn (3)
MALKI does not explicitly disclose:
- selecting a selected baseline response ("SBR") from the first and second responses to the baseline prompt;
However, PRYZANT discloses:
- selecting a selected baseline response ("SBR") from the first and second responses to the baseline prompt;
[0018]:
Other work applies discrete manipulations to prompts via Reinforcement Learning or LLM-based feedback. Such algorithms may also require low-level access to the LLM, may produce incomprehensible outputs, and/or may rely on directionless Monte Carlo search over the semantic space of prompts.
0063] The LM input optimization technology framework, as described herein, was compared against the following baselines, which focused on nonparametric algorithms that are directly comparable to the LM input optimization technology: (a) Monte-Carlo (“MC”); (b) Reinforcement Learning (“RL”); and/or (c) AutoGPT. MC is an automatic prompt engineering algorithm that proposes an iterative but directionless
[0063]:
AutoGPT is an open-source AI agent, which relies on an agent-controlled feedback loop to improve its responses.
[0061];
the Azure® OpenAI™ LLM API service with a temperature of 0.0 during few-shot classification and 1.0 in all other contexts. For the nonparametric algorithms with broad applicability, default values and the same parameters were used throughout instead of conducting any hyperparameter search for the baseline or proposed algorithms.
[0112]:
The set of operations further includes receiving, from output of the LLM in response to the first feedback prompt, the one or more first textual gradients; providing, as input to the LLM, a first editing prompt requesting a first set of optimized prompts, based on the initial prompt and the one or more first textual gradients; and receiving, from output of the LLM, the first set of optimized prompts. The set of operations also includes selecting one or more first optimized prompts from at least the first set of optimized prompts based at least in part on evaluation of prompt performance using a secondary LLM that is finetuned based on a curated dataset for a specific subject area. The set of operations further includes instructing the LLM to perform task focused on the subject area by inputting the selected one or more first optimized prompts into the secondary LLM; and receiving, from the secondary LLM, results to the instructed task.
[0022]:
In some cases, computing system 105a may further include an automated prompt optimizer 125a that uses one or more LLMs 130a-130k (collectively, “LLMs 130”; in this case, first through Kth LLMs 130a-130k) to perform automated prompt optimization using textual gradients. Herein, an LLM, which is a type of language model (“LM”), may be a deep learning algorithm that can recognize, summarize, translate, predict, and/or generate text and/or other content based on knowledge gained from massive datasets.
[0035]:
a selection process is used to decide which candidate prompts are worth carrying forward to the next iteration. This loop allows for incremental improvements and exploration over multiple prompt candidates.
[0002]:
A selection algorithm is used to select one or more optimized prompts from the set of optimized prompts and/or the set of paraphrased prompts, and the process may be iteratively repeated with the selected one or more optimized prompts replacing the current prompt.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI and PRYZANT.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
One of ordinary skill would have motivation to combine MALKI and PRYZANT that can provide 31 % increase in initial prompt performance (PRYZANT [0019]).
MALKI, and PRYZANT do not explicitly disclose:
- and supplying the SBR as a baseline action to a reinforcement learning model ("RLM") configured to:
- computes a reward in response to the baseline action
- and compute a predicted temperature state based on the reward.
However, Gardner discloses:
- and supplying the SBR as a baseline action to a reinforcement learning model ("RLM") configured to:
[0172]:
In RL, an agent (the prompt engineering model) takes actions (generates prompts) in an environment (the LLM) to maximize a reward (the summarization accuracy). The model receives feedback on how well each prompt achieves the target abstraction level. It learns to improve prompts through trial-and-error.
[0621] :
Comparing performance benchmarks vs baselines to quantify improvements from optimizations.
[1176]:
Comparing metrics across model versions to determine performance improvements; New iterations are measured against baselines;
[1177] :
Using validation sets to detect overfitting. Model selection is based on real-world performance.
- computes a reward in response to the baseline action
[0171]:
In addition to supervised learning on human prompts, reinforcement learning (RL) may be used to further enhance prompt engineering.
[0206]:
The system may train customized models to construct optimized prompts using neural networks, reinforcement learning, and human collaboration.
[1157]:
The system learns over time which model combinations produce optimal summaries for different use cases through testing and reinforcement learning.
[0172]:
In RL, an agent (the prompt engineering model) takes actions (generates prompts) in an environment (the LLM) to maximize a reward (the summarization accuracy). The model receives feedback on how well each prompt achieves the target abstraction level. It learns to improve prompts through trial-and-error.
[0173]:
The model may be initialized with weights pretrained on human prompts. RL fine-tunes prompts tailored to the nuances of the particular LLM model. Various policy gradient algorithms like REINFORCE are employed for training. [BRI: in reinforcement learning (RL) for prompt engineering, the agent (often a policy network or prompt engineering model) interacts with the environment (the LLM) to generate prompts, and the reward is computed based on how well the LLM’s output meets a predefined goal, such as summarization accuracy
- and compute a predicted temperature state based on the reward.
[0051]:
The trained model can generate coherent, human-like text by iteratively sampling from its predicted next word distributions to continue growing new sequences. Temperature parameters control randomness versus determinism in sampling.
[0172] :
RL, an agent (the prompt engineering model) takes actions (generates prompts) in an environment (the LLM) to maximize a reward (the summarization accuracy). The model receives feedback on how well each prompt achieves the target abstraction level. It learns to improve prompts through trial-and-error.
[0207]:
The system may utilize adjustable zoom levels, both positive and negative, to dynamically generate summaries of source content at varying levels of abstraction.
[0200]:
Relevance—Scores how well the prompt produces summaries containing pertinent details from the source content;
[0233]:
the system then ensures its retention irrespective of the zoom abstraction level through targeted prompt engineering and training. This balances concise summarization with preservation of information integrity for sensitive content.
[0419]:
Automated metrics quantify abstraction levels
[0423]:
Evaluation validates abstraction accuracy and quality
[0649]:
Analyze trends in usage, performance and summary quality over time. Predict future capacity needs. [BRI: the feedback on how well each prompt achieves a target abstraction level, that feedback can be treated as a reward signal. By modeling the abstraction level as a function of temperature and reward, a predicted temperature state from the reward can be computed, and use it to guide sampling toward higher abstraction quality]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI, PRYZANT and Gardner.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
Gardner teaches reinforcement learning, baseline and reward in baseline.
One of ordinary skill would have motivation to combine Liu and Gardner and is highly obvious for a POSITA to provide substantial technical improvement for engineering prompts with configurable abstraction parameters to control summarization by LLMs (Gardner [0029]) and over existing approaches to text summarization, such as in the areas of abstraction control, iterative refinement, and/or information retention (Gardner [0038]).
In regard to claim 2:
MALKI discloses:
- further comprising: setting the MLM to an iterative temperature state corresponding to a temperature state of the SBR;
[0023]:
FIG. 13 is a formula representation of an incremental language model temperature parameter adjustment function;
[0031]:
a system is configured to submit a sequence of backward prompts at different settings of a language model parameter, and the language model parameter is one of: temperature, top-k, or top-p.
[0132]:
In some embodiments, the primary answer belongs to a first dataset 494, and at least a portion of the method 1000 is repeated with a second dataset 494, [BRI: in SBR modeling, the primary answer from the first dataset can be followed by iterative application of the same method to a second dataset to represent evolving temperature states. This approach leverages iterative dataset refinement and repeated sampling to improve accuracy and capture the dynamic nature of SBR processes
- issuing an iterative prompt to the MLM at the iterative temperature state;
[0162] :
In some embodiments which utilize an InterrogateLLM approach, this backward process is repeated multiple times, e.g., K times for each model in BLLM with variable 316 temperature values, [BR: this is an iterative, temperature-tuned prompting strategy that mimics backward chaining, allowing the model to explore, revise, and refine its reasoning path before finalizing the output]
- receiving a first response to the iterative prompt from the MLM at the iterative temperature state;
[0157]:
The FLLM is queried with a concatenation of the query Q on top of the prompt Pforward, which retrieves a generated answer 408 denoted by A*, signifying the response to the query Q. In other words, the prompt Pforward and the query Q are fed into the LLM as follows:
A*=FLLM([Pforward;Q]) (2)
[0159]:
Some embodiments perform or leverage a backward process for reconstructing the original query Q from the generated answer A*. An example embodiment creates a new prompt 404 by reversing the given prompt Pforward. The reversed prompt rearranges 406 the order within each of the query-answer pairs 426 to form answer-query pairs 419. This portion of a reversed prompt denoted as Pbackward can be expressed as follows:
Pbackward=a1,q1, . . . ,an,qn (3)
[0162]:
In some embodiments which utilize an InterrogateLLM approach, this backward process is repeated multiple times, e.g., K times for each model in BLLM with variable 316 temperature values,
- setting the MLM to the predicted temperature state;
[0023]:
FIG. 13 is a formula representation of an incremental language model temperature parameter adjustment function;
[0031]:
a system is configured to submit a sequence of backward prompts at different settings of a language model parameter, and the language model parameter is one of: temperature, top-k, or top-p. [BRI: Prompting LLMs to internally assess their own predictions (e.g., via calibration, internal consistency, or self-consistency selection) can improve accuracy, but temperature adjustment is essential to ensure that confidence estimates are reliable. Low temperature supports confident self-assessment, while moderate temperature can help explore diverse reasoning paths before final selection.
[0149]:
One veracity evaluation approach involves a self-evaluation technique, where models are trained to predict their knowledge of the answer to any given free-form question. This approach entails prompting the language model to internally assess the accuracy of its previous predictions, including estimating the likelihood that its generated response or answer is correct. This approach requires labeled data for model training, making it a supervised task.
- issuing the iterative prompt to the MLM at the predicted temperature state;
[0150];
prompt that includes a stochastic sample and a sentence from the generated text and predicts whether the sentence is supported by the stochastic sample. Internal testing of the InterrogateLLM utilized a SelfCheckGPT benchmark, but unless expressly stated otherwise herein for a particular embodiment, the embodiments taught herein do not rely on employing an additional prompt that includes a stochastic sample and a sentence from the generated text and on predicting whether the sentence is supported by the stochastic sample, for hallucination detection.
[0154]:
Some examples herein assume a source domain of textual queries and a target domain of textual answers, but as noted elsewhere the teachings herein also apply to other data types 425. Assume also that a few-shot prompt, a corresponding query Q and a LLM 132 denoted by FLLM 412, are provided. The terms “query” and “question” are used interchangeably herein with respect to model 132 prompts and outputs, unless indicated otherwise by reference numerals. Although this portion of the discussion assumes a provided few-shot prompt 402, tools and techniques taught herein are adaptable to many zero-shot tasks where the creation of few-shot prompts is feasible. The prompt is fed into the LLM to generate an answer to a query in the prompt.
[0176]:
The internal evaluation compared InterrogateLLM prototypes to baselines. One baseline was BERT-cosine, which employed a pre-trained BERT model to embed both the query and the generated answer, calculated the cosine similarity between them, and predicted “hallucination” if the similarity fell below a threshold BERTT. The threshold was determined by using a process described above, with BERT embeddings. Another baseline was SBERT-cosine, which employed an SBERT backbone, similar to BERT-cosine. Another baseline was ADA-cosine which was similar to BERT-cosine but employed the OpenAI model ada002. Another baseline was SelfCheckGPT with Prompt, which employed the same FLLM in each task. SelfCheckGPT generated N additional stochastic LLM response samples, denoted as S1, S2, . . . , Sn, using the same query. Then it scored the consistency between the generated response and the stochastic samples, by querying the same LLM to determine whether the i-th sentence in A* is supported by sample Si. The final inconsistency score was computed by averaging all the sentence scores.
- receiving a second response to the iterative prompt from the MLM at the predicted temperature state;
[0157]:
The FLLM is queried with a concatenation of the query Q on top of the prompt Pforward, which retrieves a generated answer 408 denoted by A*, signifying the response to the query Q. In other words, the prompt Pforward and the query Q are fed into the LLM as follows:
A*=FLLM([Pforward;Q]) (2)
[0159]:
Some embodiments perform or leverage a backward process for reconstructing the original query Q from the generated answer A*. An example embodiment creates a new prompt 404 by reversing the given prompt Pforward. The reversed prompt rearranges 406 the order within each of the query-answer pairs 426 to form answer-query pairs 419. This portion of a reversed prompt denoted as Pbackward can be expressed as follows:
Pbackward=a1,q1, . . . ,an,qn (3)
[0162]:
In some embodiments which utilize an InterrogateLLM approach, this backward process is repeated multiple times, e.g., K times for each model in BLLM with variable 316 temperature values,
[0149]:
One veracity evaluation approach involves a self-evaluation technique, where models are trained to predict their knowledge of the answer to any given free-form question. This approach entails prompting the language model to internally assess the accuracy of its previous predictions, including estimating the likelihood that its generated response or answer is correct. This approach requires labeled data for model training, making it a supervised task. [BRI: Prompting an LLM to internally assess its own predictions, adjusting temperature to balance exploration and certainty, and using iterative prompting to refine outputs is a powerful combination. It leverages self-evaluation for accuracy, temperature for confidence control, and iteration for continuous improvement, leading to more reliable and robust responses]
MALKI does not explicitly disclose:
- selecting a selected iterative response ("SIR") from the first and second responses to the iterative prompt; and determining if the SIR is satisfactory.
However, PRYZANT discloses:
- selecting a selected iterative response ("SIR") from the first and second responses to the iterative prompt; and determining if the SIR is satisfactory.
[0018]:
Other work applies discrete manipulations to prompts via Reinforcement Learning or LLM-based feedback. Such algorithms may also require low-level access to the LLM, may produce incomprehensible outputs, and/or may rely on directionless Monte Carlo search over the semantic space of prompts.
[0063] :
The LM input optimization technology framework, as described herein, was compared against the following baselines, which focused on nonparametric algorithms that are directly comparable to the LM input optimization technology: (a) Monte-Carlo (“MC”); (b) Reinforcement Learning (“RL”); and/or (c) AutoGPT. MC is an automatic prompt engineering algorithm that proposes an iterative but directionless
[0063]:
AutoGPT is an open-source AI agent, which relies on an agent-controlled feedback loop to improve its responses.
[0061];
he Azure® OpenAI™ LLM API service with a temperature of 0.0 during few-shot classification and 1.0 in all other contexts. For the nonparametric algorithms with broad applicability, default values and the same parameters were used throughout instead of conducting any hyperparameter search for the baseline or proposed algorithms.
[0112]:
The set of operations further includes receiving, from output of the LLM in response to the first feedback prompt, the one or more first textual gradients; providing, as input to the LLM, a first editing prompt requesting a first set of optimized prompts, based on the initial prompt and the one or more first textual gradients; and receiving, from output of the LLM, the first set of optimized prompts. The set of operations also includes selecting one or more first optimized prompts from at least the first set of optimized prompts based at least in part on evaluation of prompt performance using a secondary LLM that is finetuned based on a curated dataset for a specific subject area. The set of operations further includes instructing the LLM to perform task focused on the subject area by inputting the selected one or more first optimized prompts into the secondary LLM; and receiving, from the secondary LLM, results to the instructed task.
[0022]:
In some cases, computing system 105a may further include an automated prompt optimizer 125a that uses one or more LLMs 130a-130k (collectively, “LLMs 130”; in this case, first through Kth LLMs 130a-130k) to perform automated prompt optimization using textual gradients. Herein, an LLM, which is a type of language model (“LM”), may be a deep learning algorithm that can recognize, summarize, translate, predict, and/or generate text and/or other content based on knowledge gained from massive datasets.
[0035]:
a selection process is used to decide which candidate prompts are worth carrying forward to the next iteration. This loop allows for incremental improvements and exploration over multiple prompt candidates.
[0002]:
A selection algorithm is used to select one or more optimized prompts from the set of optimized prompts and/or the set of paraphrased prompts, and the process may be iteratively repeated with the selected one or more optimized prompts replacing the current prompt.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI and PRYZANT.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
One of ordinary skill would have motivation to combine MALKI and PRYZANT that can provide 31 % increase in initial prompt performance (PRYZANT [0019]).
In regard to claim 6:
MALKI and PRYZANT do not explicitly disclose:
- wherein the training dataset comprises a plurality of data vectors in an embedding space of the MLM.
However, Gardner discloses:
- wherein the training dataset comprises a plurality of data vectors in an embedding space of the MLM.
[0048]:
The architecture may include transformer blocks with attention mechanisms to model context and identify relevant patterns across the entire input sequence. The attention weights and feedforward layers may transform the input embeddings into higher level contextual representations used to make the word predictions.
[0045]:
a search application may enable keyword searches of the content items 128.
[0074]:
Pre-trained semantic search engines can help identify contextual text passages for a given topic.
[0840]:
Raw input documents are cleaned, parsed, segmented, tokenized etc. to normalize them for downstream summarization steps. [BRI: the input embedding are the initial dense vector representation of tokens word. Perhaps known to a POSTA, the search vector is not a separate space; it is a contextualized representation in the same embedding space as the input embeddings]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI, PRYZANT and Gardner.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
Garner teaches reinforcement learning for reward and prediction based on reward.
One of ordinary skill would have motivation to combine MALKI, PRYZANT and Gardner to provide substantial technical improvement for engineering prompts with configurable abstraction parameters to control summarization by LLMs (Gardner [0029]) and over existing approaches to text summarization, such as in the areas of abstraction control, iterative refinement, and/or information retention (Gardner [0038]).
In regard to claim 10:
MALKI discloses:
- A computer program product for optimizing a prompt session with a machine learning model ("MLM") that is trained on a set of training data, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause a computing device to: [0239]; [0039]; [0041]; [0042];
[0007]:
an embodiment submits a forward prompt, the forward prompt including at least one question-answer pair and a primary question, the question-answer pair including a question and a corresponding answer to the question, the primary question not accompanied in the forward prompt by a corresponding answer to the primary question. In response to the forward prompt, this example embodiment obtains a primary answer
[0006]:
The backward traversal is performed through a set of one or more language models which does not necessarily include the forward traversal's language model. Finally, an embedding space distance is calculated between at least some of the candidate questions and the primary question
[0088]:
A given embodiment may include additional or different kinds of language model hallucination detection functionality, for example, as well as different machine learning or other artificial intelligence technical features, aspects, mechanisms, software, expressions, operational sequences, commands, data structures, programming environments, execution environments, environment or system characteristics, proxies, or other functionality consistent with teachings provided herein, and may otherwise depart from the particular examples provided.
- set the MLM to a first temperature state;
[0023]:
FIG. 13 is a formula representation of an incremental language model temperature parameter adjustment function;
PNG
media_image1.png
512
587
media_image1.png
Greyscale
[0031]:
a system is configured to submit a sequence of backward prompts at different settings of a language model parameter, and the language model parameter is one of: temperature, top-k, or top-p.
- issue a baseline prompt to the MLM at the first temperature state;
[0154]:
Some examples herein assume a source domain of textual queries and a target domain of textual answers, but as noted elsewhere the teachings herein also apply to other data types 425. Assume also that a few-shot prompt, a corresponding query Q and a LLM 132 denoted by FLLM 412, are provided. The terms “query” and “question” are used interchangeably herein with respect to model 132 prompts and outputs, unless indicated otherwise by reference numerals. Although this portion of the discussion assumes a provided few-shot prompt 402, tools and techniques taught herein are adaptable to many zero-shot tasks where the creation of few-shot prompts is feasible. The prompt is fed into the LLM to generate an answer to a query in the prompt.
[0176] :
The internal evaluation compared InterrogateLLM prototypes to baselines. One baseline was BERT-cosine, which employed a pre-trained BERT model to embed both the query and the generated answer, calculated the cosine similarity between them, and predicted “hallucination” if the similarity fell below a threshold BERTT. The threshold was determined by using a process described above, with BERT embeddings. Another baseline was SBERT-cosine, which employed an SBERT backbone, similar to BERT-cosine. Another baseline was ADA-cosine which was similar to BERT-cosine but employed the OpenAI model ada002. Another baseline was SelfCheckGPT with Prompt, which employed the same FLLM in each task. SelfCheckGPT generated N additional stochastic LLM response samples, denoted as S1, S2, . . . , Sn, using the same query. Then it scored the consistency between the generated response and the stochastic samples, by querying the same LLM to determine whether the i-th sentence in A* is supported by sample Si. The final inconsistency score was computed by averaging all the sentence scores.
- receive a first response to the baseline prompt from the MLM at the first temperature state;
[0157]:
The FLLM is queried with a concatenation of the query Q on top of the prompt Pforward, which retrieves a generated answer 408 denoted by A*, signifying the response to the query Q. In other words, the prompt Pforward and the query Q are fed into the LLM as follows:
A*=FLLM([Pforward; Q]) (2)
[0159]:
Some embodiments perform or leverage a backward process for reconstructing the original query Q from the generated answer A*. An example embodiment creates a new prompt 404 by reversing the given prompt Pforward. The reversed prompt rearranges 406 the order within each of the query-answer pairs 426 to form answer-query pairs 419. This portion of a reversed prompt denoted as Pbackward can be expressed as follows:
Pbackward=a1, q1, . . . ,an, qn (3)
- setting the MLM to a second temperature state;
[0023]:
FIG. 13 is a formula representation of an incremental language model temperature parameter adjustment function;
PNG
media_image1.png
512
587
media_image1.png
Greyscale
[0031]:
a system is configured to submit a sequence of backward prompts at different settings of a language model parameter, and the language model parameter is one of: temperature, top-k, or top-p. [BRI: in the sense that each time you apply a different T in the formula, you are sampling from a different probability distribution. This is equivalent to “setting” the model to a different temperature state (Second) for that inference step. If you keep the same T across all steps, the model remains in the same temperature state; if you change T at different points, you are transitioning between temperature states during inference.
- issue a baseline prompt to the MLM at the second temperature state;
[0150];
prompt that includes a stochastic sample and a sentence from the generated text and predicts whether the sentence is supported by the stochastic sample. Internal testing of the InterrogateLLM utilized a SelfCheckGPT benchmark, but unless expressly stated otherwise herein for a particular embodiment, the embodiments taught herein do not rely on employing an additional prompt that includes a stochastic sample and a sentence from the generated text and on predicting whether the sentence is supported by the stochastic sample, for hallucination detection.
[0154]:
Some examples herein assume a source domain of textual queries and a target domain of textual answers, but as noted elsewhere the teachings herein also apply to other data types 425. Assume also that a few-shot prompt, a corresponding query Q and a LLM 132 denoted by FLLM 412, are provided. The terms “query” and “question” are used interchangeably herein with respect to model 132 prompts and outputs, unless indicated otherwise by reference numerals. Although this portion of the discussion assumes a provided few-shot prompt 402, tools and techniques taught herein are adaptable to many zero-shot tasks where the creation of few-shot prompts is feasible. The prompt is fed into the LLM to generate an answer to a query in the prompt.
[0176]:
The internal evaluation compared InterrogateLLM prototypes to baselines. One baseline was BERT-cosine, which employed a pre-trained BERT model to embed both the query and the generated answer, calculated the cosine similarity between them, and predicted “hallucination” if the similarity fell below a threshold BERTT. The threshold was determined by using a process described above, with BERT embeddings. Another baseline was SBERT-cosine, which employed an SBERT backbone, similar to BERT-cosine. Another baseline was ADA-cosine which was similar to BERT-cosine but employed the OpenAI model ada002. Another baseline was SelfCheckGPT with Prompt, which employed the same FLLM in each task. SelfCheckGPT generated N additional stochastic LLM response samples, denoted as S1, S2, . . . , Sn, using the same query. Then it scored the consistency between the generated response and the stochastic samples, by querying the same LLM to determine whether the i-th sentence in A* is supported by sample Si. The final inconsistency score was computed by averaging all the sentence scores.
- receive a second response to the baseline prompt from the MLM at the second temperature state;
[0157]:
The FLLM is queried with a concatenation of the query Q on top of the prompt Pforward, which retrieves a generated answer 408 denoted by A*, signifying the response to the query Q. In other words, the prompt Pforward and the query Q are fed into the LLM as follows:
A*=FLLM([Pforward;Q]) (2)
[0159]:
Some embodiments perform or leverage a backward process for reconstructing the original query Q from the generated answer A*. An example embodiment creates a new prompt 404 by reversing the given prompt Pforward. The reversed prompt rearranges 406 the order within each of the query-answer pairs 426 to form answer-query pairs 419. This portion of a reversed prompt denoted as Pbackward can be expressed as follows:
Pbackward=a1,q1, . . . ,an,qn (3)
MALKI does not explicitly disclose:
- select a selected baseline response ("SBR") from the first and second responses to the baseline prompt;
However, PRYZANT discloses:
- select a selected baseline response ("SBR") from the first and second responses to the baseline prompt;
[0018]:
Other work applies discrete manipulations to prompts via Reinforcement Learning or LLM-based feedback. Such algorithms may also require low-level access to the LLM, may produce incomprehensible outputs, and/or may rely on directionless Monte Carlo search over the semantic space of prompts.
0063] The LM input optimization technology framework, as described herein, was compared against the following baselines, which focused on nonparametric algorithms that are directly comparable to the LM input optimization technology: (a) Monte-Carlo (“MC”); (b) Reinforcement Learning (“RL”); and/or (c) AutoGPT. MC is an automatic prompt engineering algorithm that proposes an iterative but directionless
[0063]:
AutoGPT is an open-source AI agent, which relies on an agent-controlled feedback loop to improve its responses.
[0061];
he Azure® OpenAI™ LLM API service with a temperature of 0.0 during few-shot classification and 1.0 in all other contexts. For the nonparametric algorithms with broad applicability, default values and the same parameters were used throughout instead of conducting any hyperparameter search for the baseline or proposed algorithms.
[0112]:
The set of operations further includes receiving, from output of the LLM in response to the first feedback prompt, the one or more first textual gradients; providing, as input to the LLM, a first editing prompt requesting a first set of optimized prompts, based on the initial prompt and the one or more first textual gradients; and receiving, from output of the LLM, the first set of optimized prompts. The set of operations also includes selecting one or more first optimized prompts from at least the first set of optimized prompts based at least in part on evaluation of prompt performance using a secondary LLM that is finetuned based on a curated dataset for a specific subject area. The set of operations further includes instructing the LLM to perform task focused on the subject area by inputting the selected one or more first optimized prompts into the secondary LLM; and receiving, from the secondary LLM, results to the instructed task.
[0022]:
In some cases, computing system 105a may further include an automated prompt optimizer 125a that uses one or more LLMs 130a-130k (collectively, “LLMs 130”; in this case, first through Kth LLMs 130a-130k) to perform automated prompt optimization using textual gradients. Herein, an LLM, which is a type of language model (“LM”), may be a deep learning algorithm that can recognize, summarize, translate, predict, and/or generate text and/or other content based on knowledge gained from massive datasets.
[0035]:
a selection process is used to decide which candidate prompts are worth carrying forward to the next iteration. This loop allows for incremental improvements and exploration over multiple prompt candidates.
[0002]:
A selection algorithm is used to select one or more optimized prompts from the set of optimized prompts and/or the set of paraphrased prompts, and the process may be iteratively repeated with the selected one or more optimized prompts replacing the current prompt.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI and PRYZANT.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
One of ordinary skill would have motivation to combine MALKI and PRYZANT that can provide 31 % increase in initial prompt performance (PRYZANT [0019]).
MALKI, and PRYZANT do not explicitly disclose:
- and supply the SBR as a baseline action to a reinforcement learning model ("RLM") configured to:
compute a reward in response to the baseline action;
- and compute a predicted temperature state based on the reward.
However, Gardner discloses:
- and supply the SBR as a baseline action to a reinforcement learning model ("RLM") configured to:
[0172]:
In RL, an agent (the prompt engineering model) takes actions (generates prompts) in an environment (the LLM) to maximize a reward (the summarization accuracy). The model receives feedback on how well each prompt achieves the target abstraction level. It learns to improve prompts through trial-and-error.
[0621] :
Comparing performance benchmarks vs baselines to quantify improvements from optimizations.
[1176]:
Comparing metrics across model versions to determine performance improvements; New iterations are measured against baselines;
[1177] :
Using validation sets to detect overfitting. Model selection is based on real-world performance.
- computes a reward in response to the baseline action
[0171]:
In addition to supervised learning on human prompts, reinforcement learning (RL) may be used to further enhance prompt engineering.
[0206]:
The system may train customized models to construct optimized prompts using neural networks, reinforcement learning, and human collaboration.
[1157]:
The system learns over time which model combinations produce optimal summaries for different use cases through testing and reinforcement learning.
[0172]:
In RL, an agent (the prompt engineering model) takes actions (generates prompts) in an environment (the LLM) to maximize a reward (the summarization accuracy). The model receives feedback on how well each prompt achieves the target abstraction level. It learns to improve prompts through trial-and-error.
[0173]:
The model may be initialized with weights pretrained on human prompts. RL fine-tunes prompts tailored to the nuances of the particular LLM model. Various policy gradient algorithms like REINFORCE are employed for training. [BRI: in reinforcement learning (RL) for prompt engineering, the agent (often a policy network or prompt engineering model) interacts with the environment (the LLM) to generate prompts, and the reward is computed based on how well the LLM’s output meets a predefined goal, such as summarization accuracy
- and compute a predicted temperature state based on the reward.
[0051]:
The trained model can generate coherent, human-like text by iteratively sampling from its predicted next word distributions to continue growing new sequences. Temperature parameters control randomness versus determinism in sampling.
[0172] :
RL, an agent (the prompt engineering model) takes actions (generates prompts) in an environment (the LLM) to maximize a reward (the summarization accuracy). The model receives feedback on how well each prompt achieves the target abstraction level. It learns to improve prompts through trial-and-error.
[0207]:
The system may utilize adjustable zoom levels, both positive and negative, to dynamically generate summaries of source content at varying levels of abstraction.
[0200]:
Relevance—Scores how well the prompt produces summaries containing pertinent details from the source content;
[0233]:
the system then ensures its retention irrespective of the zoom abstraction level through targeted prompt engineering and training. This balances concise summarization with preservation of information integrity for sensitive content.
[0419]:
Automated metrics quantify abstraction levels
[0423]:
Evaluation validates abstraction accuracy and quality
[0649]:
Analyze trends in usage, performance and summary quality over time. Predict future capacity needs. [BRI: the feedback on how well each prompt achieves a target abstraction level, that feedback can be treated as a reward signal. By modeling the abstraction level as a function of temperature and reward, a predicted temperature state from the reward can be computed, and use it to guide sampling toward higher abstraction quality]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI, PRYZANT and Gardner.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
Gardner teaches reinforcement learning, baseline and reward in baseline.
One of ordinary skill would have motivation to combine MALKI, PRYZANT and Gardner and is highly obvious for a POSITA to provide substantial technical improvement for engineering prompts with configurable abstraction parameters to control summarization by LLMs (Gardner [0029]) and over existing approaches to text summarization, such as in the areas of abstraction control, iterative refinement, and/or information retention (Gardner [0038]).
In regard to claim 15:
MALKI and PRYZANT does not explicitly disclose:
- wherein the training dataset comprises a plurality of data vectors in an embedding space of the MLM.
However, Gardner discloses:
- wherein the training dataset comprises a plurality of data vectors in an embedding space of the MLM.
[0048]:
The architecture may include transformer blocks with attention mechanisms to model context and identify relevant patterns across the entire input sequence. The attention weights and feedforward layers may transform the input embeddings into higher level contextual representations used to make the word predictions.
[0045]:
a search application may enable keyword searches of the content items 128.
[0074]:
Pre-trained semantic search engines can help identify contextual text passages for a given topic.
[0840]:
Raw input documents are cleaned, parsed, segmented, tokenized etc. to normalize them for downstream summarization steps. [BRI: the input embedding are the initial dense vector representation of tokens word. Perhaps known to a POSTA, the search vector is not a separate space; it is a contextualized representation in the same embedding space as the input embeddings]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI, PRYZANT and Gardner.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
Garner teaches reinforcement learning for reward and prediction based on reward.
One of ordinary skill would have motivation to combine MALKI, PRYZANT and Gardner to provide substantial technical improvement for engineering prompts with configurable abstraction parameters to control summarization by LLMs (Gardner [0029]) and over existing approaches to text summarization, such as in the areas of abstraction control, iterative refinement, and/or information retention (Gardner [0038]).
In regard to claim 18:
- A computer system for optimizing a prompt session with a machine learning model ("MLM") that is trained on a set of training data, the computer system having a processor, a computer-readable memory, a computer-readable tangible storage device, and program instructions stored on the storage device for execution by a processor via the computer-readable memory, wherein the execution of the program instructions causes the computer system to perform a method, comprising:
[0239]; [0039]; [0041]; [0042];
[0007]:
an embodiment submits a forward prompt, the forward prompt including at least one question-answer pair and a primary question, the question-answer pair including a question and a corresponding answer to the question, the primary question not accompanied in the forward prompt by a corresponding answer to the primary question. In response to the forward prompt, this example embodiment obtains a primary answer
[0006]:
The backward traversal is performed through a set of one or more language models which does not necessarily include the forward traversal's language model. Finally, an embedding space distance is calculated between at least some of the candidate questions and the primary question
[0088]:
A given embodiment may include additional or different kinds of language model hallucination detection functionality, for example, as well as different machine learning or other artificial intelligence technical features, aspects, mechanisms, software, expressions, operational sequences, commands, data structures, programming environments, execution environments, environment or system characteristics, proxies, or other functionality consistent with teachings provided herein, and may otherwise depart from the particular examples provided.
[0147]:
One veracity evaluation approach employs a multi-layer perceptron classifier that uses hidden representations from language models to predict sentence truthfulness. However, this approach utilizes labeled data for supervised training, and access to the internal states of the language model
- setting the MLM to a first temperature state;
[0023]:
FIG. 13 is a formula representation of an incremental language model temperature parameter adjustment function;
PNG
media_image1.png
512
587
media_image1.png
Greyscale
[0031]:
a system is configured to submit a sequence of backward prompts at different settings of a language model parameter, and the language model parameter is one of: temperature, top-k, or top-p.
- issuing the baseline prompt to the MLM at the first temperature state;
[0154]:
Some examples herein assume a source domain of textual queries and a target domain of textual answers, but as noted elsewhere the teachings herein also apply to other data types 425. Assume also that a few-shot prompt, a corresponding query Q and a LLM 132 denoted by FLLM 412, are provided. The terms “query” and “question” are used interchangeably herein with respect to model 132 prompts and outputs, unless indicated otherwise by reference numerals. Although this portion of the discussion assumes a provided few-shot prompt 402, tools and techniques taught herein are adaptable to many zero-shot tasks where the creation of few-shot prompts is feasible. The prompt is fed into the LLM to generate an answer to a query in the prompt.
[0176] :
The internal evaluation compared InterrogateLLM prototypes to baselines. One baseline was BERT-cosine, which employed a pre-trained BERT model to embed both the query and the generated answer, calculated the cosine similarity between them, and predicted “hallucination” if the similarity fell below a threshold BERTT. The threshold was determined by using a process described above, with BERT embeddings. Another baseline was SBERT-cosine, which employed an SBERT backbone, similar to BERT-cosine. Another baseline was ADA-cosine which was similar to BERT-cosine but employed the OpenAI model ada002. Another baseline was SelfCheckGPT with Prompt, which employed the same FLLM in each task. SelfCheckGPT generated N additional stochastic LLM response samples, denoted as S1, S2, . . . , Sn, using the same query. Then it scored the consistency between the generated response and the stochastic samples, by querying the same LLM to determine whether the i-th sentence in A* is supported by sample Si. The final inconsistency score was computed by averaging all the sentence scores.
- receiving a first response to the baseline prompt from the MLM at the first temperature state;
[0157]:
The FLLM is queried with a concatenation of the query Q on top of the prompt Pforward, which retrieves a generated answer 408 denoted by A*, signifying the response to the query Q. In other words, the prompt Pforward and the query Q are fed into the LLM as follows:
A*=FLLM([Pforward; Q]) (2)
[0159]:
Some embodiments perform or leverage a backward process for reconstructing the original query Q from the generated answer A*. An example embodiment creates a new prompt 404 by reversing the given prompt Pforward. The reversed prompt rearranges 406 the order within each of the query-answer pairs 426 to form answer-query pairs 419. This portion of a reversed prompt denoted as Pbackward can be expressed as follows:
Pbackward=a1, q1, . . . ,an, qn (3)
- setting the MLM to a second temperature state;
[0023]:
FIG. 13 is a formula representation of an incremental language model temperature parameter adjustment function;
PNG
media_image1.png
512
587
media_image1.png
Greyscale
[0031]:
a system is configured to submit a sequence of backward prompts at different settings of a language model parameter, and the language model parameter is one of: temperature, top-k, or top-p. [BRI: in the sense that each time you apply a different T in the formula, you are sampling from a different probability distribution. This is equivalent to “setting” the model to a different temperature state (Second) for that inference step. If you keep the same T across all steps, the model remains in the same temperature state; if you change T at different points, you are transitioning between temperature states during inference.
- issuing the baseline prompt to the MLM at the second temperature state;
[0150];
prompt that includes a stochastic sample and a sentence from the generated text and predicts whether the sentence is supported by the stochastic sample. Internal testing of the InterrogateLLM utilized a SelfCheckGPT benchmark, but unless expressly stated otherwise herein for a particular embodiment, the embodiments taught herein do not rely on employing an additional prompt that includes a stochastic sample and a sentence from the generated text and on predicting whether the sentence is supported by the stochastic sample, for hallucination detection.
[0154]:
Some examples herein assume a source domain of textual queries and a target domain of textual answers, but as noted elsewhere the teachings herein also apply to other data types 425. Assume also that a few-shot prompt, a corresponding query Q and a LLM 132 denoted by FLLM 412, are provided. The terms “query” and “question” are used interchangeably herein with respect to model 132 prompts and outputs, unless indicated otherwise by reference numerals. Although this portion of the discussion assumes a provided few-shot prompt 402, tools and techniques taught herein are adaptable to many zero-shot tasks where the creation of few-shot prompts is feasible. The prompt is fed into the LLM to generate an answer to a query in the prompt.
[0176]:
The internal evaluation compared InterrogateLLM prototypes to baselines. One baseline was BERT-cosine, which employed a pre-trained BERT model to embed both the query and the generated answer, calculated the cosine similarity between them, and predicted “hallucination” if the similarity fell below a threshold BERTT. The threshold was determined by using a process described above, with BERT embeddings. Another baseline was SBERT-cosine, which employed an SBERT backbone, similar to BERT-cosine. Another baseline was ADA-cosine which was similar to BERT-cosine but employed the OpenAI model ada002. Another baseline was SelfCheckGPT with Prompt, which employed the same FLLM in each task. SelfCheckGPT generated N additional stochastic LLM response samples, denoted as S1, S2, . . . , Sn, using the same query. Then it scored the consistency between the generated response and the stochastic samples, by querying the same LLM to determine whether the i-th sentence in A* is supported by sample Si. The final inconsistency score was computed by averaging all the sentence scores.
- receiving a second response to the baseline prompt from the MLM at the second temperature state;
[0157]:
The FLLM is queried with a concatenation of the query Q on top of the prompt Pforward, which retrieves a generated answer 408 denoted by A*, signifying the response to the query Q. In other words, the prompt Pforward and the query Q are fed into the LLM as follows:
A*=FLLM([Pforward;Q]) (2)
[0159]:
Some embodiments perform or leverage a backward process for reconstructing the original query Q from the generated answer A*. An example embodiment creates a new prompt 404 by reversing the given prompt Pforward. The reversed prompt rearranges 406 the order within each of the query-answer pairs 426 to form answer-query pairs 419. This portion of a reversed prompt denoted as Pbackward can be expressed as follows:
Pbackward=a1,q1, . . . ,an,qn (3)
MALKI does not explicitly disclose:
- selecting a selected baseline response ("SBR") from the first and second responses to the baseline prompt;
However, PRYZANT discloses:
- selecting a selected baseline response ("SBR") from the first and second responses to the baseline prompt;
[0018]:
Other work applies discrete manipulations to prompts via Reinforcement Learning or LLM-based feedback. Such algorithms may also require low-level access to the LLM, may produce incomprehensible outputs, and/or may rely on directionless Monte Carlo search over the semantic space of prompts.
0063] The LM input optimization technology framework, as described herein, was compared against the following baselines, which focused on nonparametric algorithms that are directly comparable to the LM input optimization technology: (a) Monte-Carlo (“MC”); (b) Reinforcement Learning (“RL”); and/or (c) AutoGPT. MC is an automatic prompt engineering algorithm that proposes an iterative but directionless
[0063]:
AutoGPT is an open-source AI agent, which relies on an agent-controlled feedback loop to improve its responses.
[0061];
he Azure® OpenAI™ LLM API service with a temperature of 0.0 during few-shot classification and 1.0 in all other contexts. For the nonparametric algorithms with broad applicability, default values and the same parameters were used throughout instead of conducting any hyperparameter search for the baseline or proposed algorithms.
[0112]:
The set of operations further includes receiving, from output of the LLM in response to the first feedback prompt, the one or more first textual gradients; providing, as input to the LLM, a first editing prompt requesting a first set of optimized prompts, based on the initial prompt and the one or more first textual gradients; and receiving, from output of the LLM, the first set of optimized prompts. The set of operations also includes selecting one or more first optimized prompts from at least the first set of optimized prompts based at least in part on evaluation of prompt performance using a secondary LLM that is finetuned based on a curated dataset for a specific subject area. The set of operations further includes instructing the LLM to perform task focused on the subject area by inputting the selected one or more first optimized prompts into the secondary LLM; and receiving, from the secondary LLM, results to the instructed task.
[0022]:
In some cases, computing system 105a may further include an automated prompt optimizer 125a that uses one or more LLMs 130a-130k (collectively, “LLMs 130”; in this case, first through Kth LLMs 130a-130k) to perform automated prompt optimization using textual gradients. Herein, an LLM, which is a type of language model (“LM”), may be a deep learning algorithm that can recognize, summarize, translate, predict, and/or generate text and/or other content based on knowledge gained from massive datasets.
[0035]:
a selection process is used to decide which candidate prompts are worth carrying forward to the next iteration. This loop allows for incremental improvements and exploration over multiple prompt candidates.
[0002]:
A selection algorithm is used to select one or more optimized prompts from the set of optimized prompts and/or the set of paraphrased prompts, and the process may be iteratively repeated with the selected one or more optimized prompts replacing the current prompt.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI and PRYZANT.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
One of ordinary skill would have motivation to combine MALKI and PRYZANT that can provide 31 % increase in initial prompt performance (PRYZANT [0019]).
MALKI, and PRYZANT do not explicitly disclose:
- and supplying the SBR as a baseline action to a reinforcement learning model ("RLM") configured to:
- computes a reward in response to the baseline action
- and compute a predicted temperature state based on the reward.
However, Gardner discloses:
- and supplying the SBR as a baseline action to a reinforcement learning model ("RLM") configured to:
[0172]:
In RL, an agent (the prompt engineering model) takes actions (generates prompts) in an environment (the LLM) to maximize a reward (the summarization accuracy). The model receives feedback on how well each prompt achieves the target abstraction level. It learns to improve prompts through trial-and-error.
[0621] :
Comparing performance benchmarks vs baselines to quantify improvements from optimizations.
[1176]:
Comparing metrics across model versions to determine performance improvements; New iterations are measured against baselines;
[1177] :
Using validation sets to detect overfitting. Model selection is based on real-world performance.
- computes a reward in response to the baseline action
[0171]:
In addition to supervised learning on human prompts, reinforcement learning (RL) may be used to further enhance prompt engineering.
[0206]:
The system may train customized models to construct optimized prompts using neural networks, reinforcement learning, and human collaboration.
[1157]:
The system learns over time which model combinations produce optimal summaries for different use cases through testing and reinforcement learning.
[0172]:
In RL, an agent (the prompt engineering model) takes actions (generates prompts) in an environment (the LLM) to maximize a reward (the summarization accuracy). The model receives feedback on how well each prompt achieves the target abstraction level. It learns to improve prompts through trial-and-error.
[0173]:
The model may be initialized with weights pretrained on human prompts. RL fine-tunes prompts tailored to the nuances of the particular LLM model. Various policy gradient algorithms like REINFORCE are employed for training. [BRI: in reinforcement learning (RL) for prompt engineering, the agent (often a policy network or prompt engineering model) interacts with the environment (the LLM) to generate prompts, and the reward is computed based on how well the LLM’s output meets a predefined goal, such as summarization accuracy
- and compute a predicted temperature state based on the reward.
[0051]:
The trained model can generate coherent, human-like text by iteratively sampling from its predicted next word distributions to continue growing new sequences. Temperature parameters control randomness versus determinism in sampling.
[0172] :
RL, an agent (the prompt engineering model) takes actions (generates prompts) in an environment (the LLM) to maximize a reward (the summarization accuracy). The model receives feedback on how well each prompt achieves the target abstraction level. It learns to improve prompts through trial-and-error.
[0207]:
The system may utilize adjustable zoom levels, both positive and negative, to dynamically generate summaries of source content at varying levels of abstraction.
[0200]:
Relevance—Scores how well the prompt produces summaries containing pertinent details from the source content;
[0233]:
the system then ensures its retention irrespective of the zoom abstraction level through targeted prompt engineering and training. This balances concise summarization with preservation of information integrity for sensitive content.
[0419]:
Automated metrics quantify abstraction levels
[0423]:
Evaluation validates abstraction accuracy and quality
[0649]:
Analyze trends in usage, performance and summary quality over time. Predict future capacity needs. [BRI: the feedback on how well each prompt achieves a target abstraction level, that feedback can be treated as a reward signal. By modeling the abstraction level as a function of temperature and reward, a predicted temperature state from the reward can be computed, and use it to guide sampling toward higher abstraction quality]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI, PRYZANT and Gardner.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
Gardner teaches reinforcement learning, baseline and reward in baseline.
One of ordinary skill would have motivation to combine MALKI, PRYZANT, and Gardner and is highly obvious for a POSITA to provide substantial technical improvement for engineering prompts with configurable abstraction parameters to control summarization by LLMs (Gardner [0029]) and over existing approaches to text summarization, such as in the areas of abstraction control, iterative refinement, and/or information retention (Gardner [0038]).
Claims 3-5, 7-9, 11-14 , 16-17, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over
Itzik MALKIEL et.al. (hereinafter MALKI) US 2025/0238629 A1,
in view of Reid PRYZANT et.al. (hereinafter PRYZANT) US 2025/0111147 A1,
in view of Richard Gardner et.al. (hereinafter Gardner) US 2025/0061290 A1.
further in view of Zhiwei Liu et.al. (hereinafter Liu) US 2025/0053793 A1.
In regard to claim 3:
MALKI, PRYZANT and Gardner do not explicitly disclose:
- upon determining that the SIR is satisfactory, setting the MLM to a temperature state corresponding to a temperature state of the SIR;
- and upon determining that the SIR is not satisfactory, supplying the SIR as an iterative action to the RLM that is configured to:
- recompute the reward in response to the iterative action;
- and recompute the predicted temperature state based on the recomputed reward;
However, Liu discloses:
- upon determining that the SIR is satisfactory, setting the MLM to a temperature state corresponding to a temperature state of the SIR;
[Abstract]:
Embodiments described herein provide a method of predicting an action by a plurality of language model augmented agents (LAAs)
[0080]:
For a metric for comparing the different models, the experiments mainly use the reward score in each environment to evaluate the performances of LAAs
[0077]:
The system may continue to iterate and select additional actions following the same process of receiving observations from the environment, selecting an LAA, inputting a prompt to the selected LAA, receiving an action recommendation from the LAA, and performing the action, etc. The system may determine (e.g., by the controller) whether the actions successfully completed the task instruction. The system may perform another attempt by restarting the environment and trying a new set of actions determined in the same way. Since the LAAs use LLMs for determining actions, a temperature setting of the LLMs may provide some randomness to the outputs, resulting in different actions being performed at each attempt. The controller may determine an attempt was successful or not successful by collecting feedback from the environment. In some embodiments, an LLM is given the task instruction, the actions, and/or the observations from the environment with a prompt asking the LLM to determine whether the task was successfully performed.
- and upon determining that the SIR is not satisfactory, supplying the SIR as an iterative action to the RLM that is configured to:
[0077]:
The system may continue to iterate and select additional actions following the same process of receiving observations from the environment, selecting an LAA, inputting a prompt to the selected LAA, receiving an action recommendation from the LAA, and performing the action, etc. The system may determine (e.g., by the controller) whether the actions successfully completed the task instruction. The system may perform another attempt by restarting the environment and trying a new set of actions determined in the same way. Since the LAAs use LLMs for determining actions, a temperature setting of the LLMs may provide some randomness to the outputs, resulting in different actions being performed at each attempt. The controller may determine an attempt was successful or not successful by collecting feedback from the environment. In some embodiments, an LLM is given the task instruction, the actions, and/or the observations from the environment with a prompt asking the LLM to determine whether the task was successfully performed. In some embodiments, a human user indicates whether the task was successful.
[0076]:
At step 506, the system causes the action to be performed on the environment thereby causing the first state of the environment to change to a second state. In some embodiments, the system determines a second action based on a determination that the action was not successful. For example, after a failed attempt, the system may start a second attempt, and the prompts used during the second attempt may indicate to the selected LAA what action was tried previously that failed.
- recompute the reward in response to the iterative action;
[0050]:
the backpropagation from the last layer 343 to the input layer 341 may be conducted for a number of training samples in a number of iterative training epochs. In this way, parameters of the neural network may be gradually updated in a direction to result in a lesser or minimized loss, indicating the neural network has been trained to generate a predicted output value closer to the target output value with improved prediction accuracy
[0019]:
When a controller is given a task prompt describing a target task, the controller determines which LAA from a pool of available LAAs is best suited for predicting a next action at every iteration.
[0080]:
For a metric for comparing the different models, the experiments mainly use the reward score in each environment to evaluate the performances of LAAs.
[0080]:
In HotPotQA environment, the reward is defined as the F1 score grading between agent answer and ground-truth answer.
[0074]:
In some embodiments, the output includes a reflection on the performance of one or more past actions performed on the environment. For example, a reflective LAA may be given a prompt to reflect on previous actions or attempts, and recommend how the approach may be changed. Subsequent prompts to the LAAs may be updated on subsequent attempts based on the reflection in order to improve the performance.
- and recompute the predicted temperature state based on the recomputed reward;
[0050]:
the neural network has been trained to generate a predicted output value closer to the target output value with improved prediction accuracy. Training may continue until a stopping criterion is met, such as reaching a maximum number of epochs or achieving satisfactory performance on the validation data.
[0077]:
the system may determine (e.g., by the controller) whether the actions successfully completed the task instruction. The system may perform another attempt by restarting the environment and trying a new set of actions determined in the same way. Since the LAAs use LLMs for determining actions, a temperature setting of the LLMs may provide some randomness to the outputs, resulting in different actions being performed at each attempt. [BRI: another attempt is the recomputing]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI, Gardner and Liu.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
Gardner teaches reinforcement learning, baseline and reward in baseline.
Liu teaches recomputed reward.
One of ordinary skill would have motivation to combine MALKI, PRYZANT, Gardner and Liu that provides improved language model autonomous agents (Liu [0021])
In regard to claim 4:
MALKI discloses:
- setting the MLM to the recomputed predicted temperature state;
[0023]:
FIG. 13 is a formula representation of an incremental language model temperature parameter adjustment function;
[0031]:
a system is configured to submit a sequence of backward prompts at different settings of a language model parameter, and the language model parameter is one of: temperature, top-k, or top-p.
[0132]:
In some embodiments, the primary answer belongs to a first dataset 494, and at least a portion of the method 1000 is repeated with a second dataset 494
[0172]:
In RL, an agent (the prompt engineering model) takes actions (generates prompts) in an environment (the LLM) to maximize a reward (the summarization accuracy). The model receives feedback on how well each prompt achieves the target abstraction level. It learns to improve prompts through trial-and-error.
[0173]:
The model may be initialized with weights pretrained on human prompts. RL fine-tunes prompts tailored to the nuances of the particular LLM model. Various policy gradient algorithms like REINFORCE are employed for training.
[0167]:
In example embodiments, multi-task learning helps optimize prompts. Auxiliary tasks like predicting the ideal length or abstraction level for a document provide useful gradients during training. This may lead to better shared representations.
[0174]:
This allows adapting prompts to optimize abstraction control for individual LLMs. The RL agent learns intricacies of how different prompts influence each LLM's outputs. [BRI: initializing with weights pretrained on human prompts is standard, and RL is a powerful way to fine-tune those weights to produce outputs tailored to the nuances of the LLM and the target application.
- issuing the subsequent iterative prompt to the MLM at the recomputed predicted temperature state;
[0023]:
FIG. 13 is a formula representation of an incremental language model temperature parameter adjustment function;
[0031]:
a system is configured to submit a sequence of backward prompts at different settings of a language model parameter, and the language model parameter is one of: temperature, top-k, or top-p.
[0132]:
In some embodiments, the primary answer belongs to a first dataset 494, and at least a portion of the method 1000 is repeated with a second dataset 494
[0172]:
In RL, an agent (the prompt engineering model) takes actions (generates prompts) in an environment (the LLM) to maximize a reward (the summarization accuracy). The model receives feedback on how well each prompt achieves the target abstraction level. It learns to improve prompts through trial-and-error.
[0173]:
The model may be initialized with weights pretrained on human prompts. RL fine-tunes prompts tailored to the nuances of the particular LLM model. Various policy gradient algorithms like REINFORCE are employed for training.
[0174]:
This allows adapting prompts to optimize abstraction control for individual LLMs. The RL agent learns intricacies of how different prompts influence each LLM's outputs.
[BRI: initializing with weights pretrained on human prompts is standard, and RL is a powerful way to fine-tune those weights to produce outputs tailored to the nuances of the LLM and the target application combining the speed of prompt-based adaptation with the precision of reinforcement learning
- receiving a first response to the subsequent iterative prompt from the MLM at the iterative temperature state;
[0023]:
FIG. 13 is a formula representation of an incremental language model temperature parameter adjustment function;
[0031]:
a system is configured to submit a sequence of backward prompts at different settings of a language model parameter, and the language model parameter is one of: temperature, top-k, or top-p.
[0132]:
In some embodiments, the primary answer belongs to a first dataset 494, and at least a portion of the method 1000 is repeated with a second dataset 494
[0835]:
Streamlined orchestration coordinates the summarization pipeline for optimized performance.
[0293]:
The summarization workflow may begin when a client request is received by the system's API gateway. Robust ingestion prepares the request for processing.
[0172]:
In RL, an agent (the prompt engineering model) takes actions (generates prompts) in an environment (the LLM) to maximize a reward (the summarization accuracy). The model receives feedback on how well each prompt achieves the target abstraction level. It learns to improve prompts through trial-and-error.
[0173]:
The model may be initialized with weights pretrained on human prompts. RL fine-tunes prompts tailored to the nuances of the particular LLM model. Various policy gradient algorithms like REINFORCE are employed for training.
[0174]:
This allows adapting prompts to optimize abstraction control for individual LLMs. The RL agent learns intricacies of how different prompts influence each LLM's outputs.
[BRI: The iterative RLHF process can indeed produce a “first” to a subsequent prompt as an intial attempt using the formula]
- receiving a second response to the subsequent iterative prompt from the MLM at the recomputed predicted temperature state;
[0023]:
FIG. 13 is a formula representation of an incremental language model temperature parameter adjustment function;
[0031]:
a system is configured to submit a sequence of backward prompts at different settings of a language model parameter, and the language model parameter is one of: temperature, top-k, or top-p.
[0132]:
In some embodiments, the primary answer belongs to a first dataset 494, and at least a portion of the method 1000 is repeated with a second dataset 494
[0835]:
Streamlined orchestration coordinates the summarization pipeline for optimized performance.
[0293]:
The summarization workflow may begin when a client request is received by the system's API gateway. Robust ingestion prepares the request for processing.
[0172]:
In RL, an agent (the prompt engineering model) takes actions (generates prompts) in an environment (the LLM) to maximize a reward (the summarization accuracy). The model receives feedback on how well each prompt achieves the target abstraction level. It learns to improve prompts through trial-and-error.
[0173]:
The model may be initialized with weights pretrained on human prompts. RL fine-tunes prompts tailored to the nuances of the particular LLM model. Various policy gradient algorithms like REINFORCE are employed for training.
[0174]:
This allows adapting prompts to optimize abstraction control for individual LLMs. The RL agent learns intricacies of how different prompts influence each LLM's outputs.
[BRI: The iterative RLHF process can indeed produce a “second response” to a subsequent prompt, refined to match the nuances of the target model’s behavior, with temperature settings influencing the output style]
MALKI does not explicitly disclose:
- selecting a subsequent SIR from the first and second responses to the iterative prompt; and determining if the SIR is satisfactory.
However, PRYZANT discloses:
- selecting a subsequent SIR from the first and second responses to the iterative prompt; and determining if the SIR is satisfactory.
[0004]:
A selection algorithm is used to select one or more optimized prompts from the set of optimized prompts and/or the set of paraphrased prompts, and the process may be iteratively repeated with the selected one or more optimized prompts replacing the current prompt. [BRI: an iterative process for selection represents subsequent selection]
[0018]:
Other work applies discrete manipulations to prompts via Reinforcement Learning or LLM-based feedback. Such algorithms may also require low-level access to the LLM, may produce incomprehensible outputs, and/or may rely on directionless Monte Carlo search over the semantic space of prompts.
[0063] :
The LM input optimization technology framework, as described herein, was compared against the following baselines, which focused on nonparametric algorithms that are directly comparable to the LM input optimization technology: (a) Monte-Carlo (“MC”); (b) Reinforcement Learning (“RL”); and/or (c) AutoGPT. MC is an automatic prompt engineering algorithm that proposes an iterative but directionless
[0063]:
AutoGPT is an open-source AI agent, which relies on an agent-controlled feedback loop to improve its responses.
[0061];
he Azure® OpenAI™ LLM API service with a temperature of 0.0 during few-shot classification and 1.0 in all other contexts. For the nonparametric algorithms with broad applicability, default values and the same parameters were used throughout instead of conducting any hyperparameter search for the baseline or proposed algorithms.
[0112]:
The set of operations further includes receiving, from output of the LLM in response to the first feedback prompt, the one or more first textual gradients; providing, as input to the LLM, a first editing prompt requesting a first set of optimized prompts, based on the initial prompt and the one or more first textual gradients; and receiving, from output of the LLM, the first set of optimized prompts. The set of operations also includes selecting one or more first optimized prompts from at least the first set of optimized prompts based at least in part on evaluation of prompt performance using a secondary LLM that is finetuned based on a curated dataset for a specific subject area. The set of operations further includes instructing the LLM to perform task focused on the subject area by inputting the selected one or more first optimized prompts into the secondary LLM; and receiving, from the secondary LLM, results to the instructed task.
[0022]:
In some cases, computing system 105a may further include an automated prompt optimizer 125a that uses one or more LLMs 130a-130k (collectively, “LLMs 130”; in this case, first through Kth LLMs 130a-130k) to perform automated prompt optimization using textual gradients. Herein, an LLM, which is a type of language model (“LM”), may be a deep learning algorithm that can recognize, summarize, translate, predict, and/or generate text and/or other content based on knowledge gained from massive datasets.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI and PRYZANT.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
One of ordinary skill would have motivation to combine MALKI and PRYZANT that can provide 31 % increase in initial prompt performance (PRYZANT [0019]).
MALKI and PRYZANT does not explicitly disclose:
- further comprising: setting the MLM to a subsequent iterative temperature state corresponding to a temperature state of the SBR;
- issuing a subsequent iterative prompt to the MLM at the iterative temperature state;
- receiving a first response to the subsequent iterative prompt from the MLM at the iterative temperature state;
However, Gardner discloses:
- further comprising: setting the MLM to a subsequent iterative temperature state corresponding to a temperature state of the SBR;
[0051]:
The trained model can generate coherent, human-like text by iteratively sampling from its predicted next word distributions to continue growing new sequences. Temperature parameters control randomness versus determinism in sampling.
[0597]:
Set hyperparameters for models like length, temperature, top-k sampling etc. Higher temperature produces more creative but less focused summaries.
[0835]:
Streamlined orchestration coordinates the summarization pipeline for optimized performance.
[0958]:
The templates may control phrasing, instructions, and content provided to adjust the abstraction in a step-wise manner. The LLM output summary may then be used as the content for the next iteration's increased abstraction prompt. [BRI: this is an iterative abstraction pipeline where temperature is adjusted based on the abstraction level, and the summary output from one iteration is used as the prompt for the next iteration (subsequent)]
- issuing a subsequent iterative prompt to the MLM at the iterative temperature state;
[0839]:
Message queueing provides scalability, resilience and restartability for long-running multi-stage summarization pipelines.
[0836]:
A message broker like Kafka connects the sequence of microservices that comprise the summarization workflow-cleaning, extraction, abstraction, compression, storage etc.
- receiving a first response to the subsequent iterative prompt from the MLM at the iterative temperature state;
[0931]:
This simple request/response flow enables out-of-the-box summarization without developer work. The system acts as a hands-off pass-through to the LLM.
[1149]:
The Dialog Module 260 allows leveraging the core summarization capabilities to generate conversational responses via dialog systems and APIs. For instance, taking a summarized description of a customer support ticket and producing a natural language response addressing the key issues detected. The dialog module allows progressing from static summarization to interactive conversational applications.
In regard to claim 5:
MALKI, PRYZANT and Gardner does not explicitly disclose:
- upon determining that the subsequent SIR is satisfactory, setting the MLM to a temperature state corresponding to a temperature state of the SIR;
- and upon determining that the subsequent SIR is not satisfactory, supplying the SIR as an iterative action to the RLM that is configured to:
- recompute the reward in response to the subsequent iterative action;
- and recompute the predicted temperature state based on the most recent recomputed reward;
However, Liu discloses:
- upon determining that the subsequent SIR is satisfactory, setting the MLM to a temperature state corresponding to a temperature state of the SIR;
[Abstract]:
Embodiments described herein provide a method of predicting an action by a plurality of language model augmented agents (LAAs)
[0080]:
For a metric for comparing the different models, the experiments mainly use the reward score in each environment to evaluate the performances of LAAs
[0077]:
The system may continue to iterate and select additional actions following the same process of receiving observations from the environment, selecting an LAA, inputting a prompt to the selected LAA, receiving an action recommendation from the LAA, and performing the action, etc. The system may determine (e.g., by the controller) whether the actions successfully completed the task instruction. The system may perform another attempt by restarting the environment and trying a new set of actions determined in the same way. Since the LAAs use LLMs for determining actions, a temperature setting of the LLMs may provide some randomness to the outputs, resulting in different actions being performed at each attempt. The controller may determine an attempt was successful or not successful by collecting feedback from the environment. In some embodiments, an LLM is given the task instruction, the actions, and/or the observations from the environment with a prompt asking the LLM to determine whether the task was successfully performed.
[0032]:
Observations 104 from this action and subsequent actions may not result in a correct answer to the question. For a next attempt, agent selection 208 may select a “reflective” LAA, and communicate an agents message 210 to the reflective LAA with the task instruction 116, the set of actions 114 from the failed attempt and the corresponding observations 104
- and upon determining that the subsequent SIR is not satisfactory, supplying the SIR as an iterative action to the RLM that is configured to:
[0077]:
The system may continue to iterate and select additional actions following the same process of receiving observations from the environment, selecting an LAA, inputting a prompt to the selected LAA, receiving an action recommendation from the LAA, and performing the action, etc. The system may determine (e.g., by the controller) whether the actions successfully completed the task instruction. The system may perform another attempt by restarting the environment and trying a new set of actions determined in the same way. Since the LAAs use LLMs for determining actions, a temperature setting of the LLMs may provide some randomness to the outputs, resulting in different actions being performed at each attempt. The controller may determine an attempt was successful or not successful by collecting feedback from the environment. In some embodiments, an LLM is given the task instruction, the actions, and/or the observations from the environment with a prompt asking the LLM to determine whether the task was successfully performed. In some embodiments, a human user indicates whether the task was successful.
[0076]:
At step 506, the system causes the action to be performed on the environment thereby causing the first state of the environment to change to a second state. In some embodiments, the system determines a second action based on a determination that the action was not successful. For example, after a failed attempt, the system may start a second attempt, and the prompts used during the second attempt may indicate to the selected LAA what action was tried previously that failed.
- recompute the reward in response to the subsequent iterative action;
[0050]:
the backpropagation from the last layer 343 to the input layer 341 may be conducted for a number of training samples in a number of iterative training epochs. In this way, parameters of the neural network may be gradually updated in a direction to result in a lesser or minimized loss, indicating the neural network has been trained to generate a predicted output value closer to the target output value with improved prediction accuracy
[0019]:
When a controller is given a task prompt describing a target task, the controller determines which LAA from a pool of available LAAs is best suited for predicting a next action at every iteration.
[0080]:
For a metric for comparing the different models, the experiments mainly use the reward score in each environment to evaluate the performances of LAAs.
[0080]:
In HotPotQA environment, the reward is defined as the F1 score grading between agent answer and ground-truth answer.
[0074]:
In some embodiments, the output includes a reflection on the performance of one or more past actions performed on the environment. For example, a reflective LAA may be given a prompt to reflect on previous actions or attempts, and recommend how the approach may be changed. Subsequent prompts to the LAAs may be updated on subsequent attempts based on the reflection in order to improve the performance.
- and recompute the predicted temperature state based on the most recent recomputed reward;
[0050]:
the neural network has been trained to generate a predicted output value closer to the target output value with improved prediction accuracy. Training may continue until a stopping criterion is met, such as reaching a maximum number of epochs or achieving satisfactory performance on the validation data.
[0077]:
the system may determine (e.g., by the controller) whether the actions successfully completed the task instruction. The system may perform another attempt by restarting the environment and trying a new set of actions determined in the same way. Since the LAAs use LLMs for determining actions, a temperature setting of the LLMs may provide some randomness to the outputs, resulting in different actions being performed at each attempt. [BRI: another attempt is the recomputing]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI, Gardner and Liu.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
Gardner teaches reinforcement learning, baseline and reward in baseline.
Liu teaches recomputed reward.
One of ordinary skill would have motivation to combine MALKI, PRYZANT, Gardner and Liu that provides improved language model autonomous agents (Liu [0021])
In regard to claim 7:
MALKI and PRYZANT do not explicitly disclose:
- wherein the baseline prompt, the iterative prompt, and the subsequent iterative prompt each comprise a search vector in an embedding space of the MLM.
However, Gardner discloses:
- wherein the baseline prompt, the iterative prompt, and the subsequent iterative prompt each comprise a search vector in an embedding space of the MLM.
[0048]:
The architecture may include transformer blocks with attention mechanisms to model context and identify relevant patterns across the entire input sequence. The attention weights and feedforward layers may transform the input embeddings into higher level contextual representations used to make the word predictions.
[0045]:
a search application may enable keyword searches of the content items 128.
[0074]:
Pre-trained semantic search engines can help identify contextual text passages for a given topic.
[0840]:
Raw input documents are cleaned, parsed, segmented, tokenized etc. to normalize them for downstream summarization steps. [BRI: the input embedding are the initial dense vector representation of tokens word. Perhaps known to a POSTA, the search vector is not a separate space; it is a contextualized representation in the same embedding space as the input embeddings]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI, PRYZANT and Gardner.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
Garner teaches reinforcement learning for reward and prediction based on reward.
One of ordinary skill would have motivation to combine MALKI, PRYZANT and Gardner to provide substantial technical improvement for engineering prompts with configurable abstraction parameters to control summarization by LLMs (Gardner [0029]) and over existing approaches to text summarization, such as in the areas of abstraction control, iterative refinement, and/or information retention (Gardner [0038]).
In regard to claim 8:
MALKI , and PRYZANT do not explicitly disclose:
- comprise a probability distribution around one of the search vectors in the embedding space of the MLM.
However, Gardner discloses:
- comprise a probability distribution around one of the search vectors in the embedding space of the MLM.
[0058]:
Extractive summarization models can identify and extract the most salient sentences or phrases from the original text to produce abridged summaries. These models may be trained on text-summary pairs to learn to rank sentences based on importance. These models may use encoder-decoder architectures, converting sentences to vector representations and scoring for relevance. The system can be configured to use extractive summarization models to, for example, concatenate extracted snippets into summaries meeting length constraints.
0156
the decoder model generates an optimized prompt sequence conditioned on the encoder outputs. [ BRI: encoder-decoder models can be trained on text–summary pairs to learn to rank sentences by importance, and they can also be used to convert sentences into vector representations for relevance scoring. They are also well-suited for predicting next-word distributions, which can be interpreted as probability distributions in the embedding space of a masked language model (MLM). The decoder’s output is a probability distribution over the vocabulary for each predicted token]
In regard to claim 9:
MALKI, and PRYZANT do not explicitly disclose:
- the iterative temperature state comprises a probability distribution around the SBR;
- and the subsequent iterative temperature state comprises a probability distribution around the SIR.
However, Gardner discloses:
- the iterative temperature state comprises a probability distribution around the SBR;
[0151] :
At operation 310, a response is received from the LLM. The response contains a second content item representing the first content item. The representation omits or simplifies sub-content items included in the first content item based on the abstraction level.
0058] Extractive summarization models can identify and extract the most salient sentences or phrases from the original text to produce abridged summaries. These models may be trained on text-summary pairs to learn to rank sentences based on importance. These models may use encoder-decoder architectures, converting sentences to vector representations and scoring for relevance. The system can be configured to use extractive summarization models to, for example, concatenate extracted snippets into summaries meeting length constraints.
[0621]:
Comparing performance benchmarks vs baselines to quantify improvements from optimizations.
[ 1176]:
Comparing metrics across model versions to determine performance improvements;
New iterations are measured against baselines; and/or
[0156
the decoder model generates an optimized prompt sequence conditioned on the encoder outputs.
[0204] :
These metrics may guide ongoing prompt optimization and training priorities. Prompts are selected to maximize overall metrics.
- and the subsequent iterative temperature state comprises a probability distribution around the SIR.
[0151]:
At operation 310, a response is received from the LLM. The response contains a second content item representing the first content item. The representation omits or simplifies sub-content items included in the first content item based on the abstraction level.
[0058]:
These models may use encoder-decoder architectures, converting sentences to vector representations and scoring for relevance. The system can be configured to use extractive summarization models to, for example, concatenate extracted snippets into summaries meeting length constraints.
[0156]:
the decoder model generates an optimized prompt sequence conditioned on the encoder outputs.
[0218] :
The system calculates appropriate word limits, abstraction levels, and instructions to generate prompts across the zoom level spectrum. It can optimize prompts through iterative testing and feedback. The prompts are designed to control the length, abstraction, inferences, data integration, and other attributes of the LLM-produced summaries.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI, PRYZANT and Gardner.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
Garner teaches reinforcement learning for reward and prediction based on reward.
One of ordinary skill would have motivation to combine MALKI, PRYZANT and Gardner to provide substantial technical improvement for engineering prompts with configurable abstraction parameters to control summarization by LLMs (Gardner [0029]) and over existing approaches to text summarization, such as in the areas of abstraction control, iterative refinement, and/or information retention (Gardner [0038]).
In regard to claim 11:
MALKI discloses:
- further comprising: setting the MLM to an iterative temperature state corresponding to a temperature state of the SBR;
[0023]:
FIG. 13 is a formula representation of an incremental language model temperature parameter adjustment function;
[0031]:
a system is configured to submit a sequence of backward prompts at different settings of a language model parameter, and the language model parameter is one of: temperature, top-k, or top-p.
[0132]:
In some embodiments, the primary answer belongs to a first dataset 494, and at least a portion of the method 1000 is repeated with a second dataset 494, [BRI: in SBR modeling, the primary answer from the first dataset can be followed by iterative application of the same method to a second dataset to represent evolving temperature states. This approach leverages iterative dataset refinement and repeated sampling to improve accuracy and capture the dynamic nature of SBR processes
- issuing an iterative prompt to the MLM at the iterative temperature state;
[0162] :
In some embodiments which utilize an InterrogateLLM approach, this backward process is repeated multiple times, e.g., K times for each model in BLLM with variable 316 temperature values, [BR: this is an iterative, temperature-tuned prompting strategy that mimics backward chaining, allowing the model to explore, revise, and refine its reasoning path before finalizing the output]
- receiving a first response to the iterative prompt from the MLM at the iterative temperature state;
[0157]:
The FLLM is queried with a concatenation of the query Q on top of the prompt Pforward, which retrieves a generated answer 408 denoted by A*, signifying the response to the query Q. In other words, the prompt Pforward and the query Q are fed into the LLM as follows:
A*=FLLM([Pforward;Q]) (2)
[0159]:
Some embodiments perform or leverage a backward process for reconstructing the original query Q from the generated answer A*. An example embodiment creates a new prompt 404 by reversing the given prompt Pforward. The reversed prompt rearranges 406 the order within each of the query-answer pairs 426 to form answer-query pairs 419. This portion of a reversed prompt denoted as Pbackward can be expressed as follows:
Pbackward=a1,q1, . . . ,an,qn (3)
[0162]:
In some embodiments which utilize an InterrogateLLM approach, this backward process is repeated multiple times, e.g., K times for each model in BLLM with variable 316 temperature values,
- setting the MLM to the predicted temperature state;
[0023]:
FIG. 13 is a formula representation of an incremental language model temperature parameter adjustment function;
[0031]:
a system is configured to submit a sequence of backward prompts at different settings of a language model parameter, and the language model parameter is one of: temperature, top-k, or top-p. [BRI: Prompting LLMs to internally assess their own predictions (e.g., via calibration, internal consistency, or self-consistency selection) can improve accuracy, but temperature adjustment is essential to ensure that confidence estimates are reliable. Low temperature supports confident self-assessment, while moderate temperature can help explore diverse reasoning paths before final selection.
[0149]:
One veracity evaluation approach involves a self-evaluation technique, where models are trained to predict their knowledge of the answer to any given free-form question. This approach entails prompting the language model to internally assess the accuracy of its previous predictions, including estimating the likelihood that its generated response or answer is correct. This approach requires labeled data for model training, making it a supervised task.
- issuing the iterative prompt to the MLM at the predicted temperature state;
[0150];
prompt that includes a stochastic sample and a sentence from the generated text and predicts whether the sentence is supported by the stochastic sample. Internal testing of the InterrogateLLM utilized a SelfCheckGPT benchmark, but unless expressly stated otherwise herein for a particular embodiment, the embodiments taught herein do not rely on employing an additional prompt that includes a stochastic sample and a sentence from the generated text and on predicting whether the sentence is supported by the stochastic sample, for hallucination detection.
[0154]:
Some examples herein assume a source domain of textual queries and a target domain of textual answers, but as noted elsewhere the teachings herein also apply to other data types 425. Assume also that a few-shot prompt, a corresponding query Q and a LLM 132 denoted by FLLM 412, are provided. The terms “query” and “question” are used interchangeably herein with respect to model 132 prompts and outputs, unless indicated otherwise by reference numerals. Although this portion of the discussion assumes a provided few-shot prompt 402, tools and techniques taught herein are adaptable to many zero-shot tasks where the creation of few-shot prompts is feasible. The prompt is fed into the LLM to generate an answer to a query in the prompt.
[0176]:
The internal evaluation compared InterrogateLLM prototypes to baselines. One baseline was BERT-cosine, which employed a pre-trained BERT model to embed both the query and the generated answer, calculated the cosine similarity between them, and predicted “hallucination” if the similarity fell below a threshold BERTT. The threshold was determined by using a process described above, with BERT embeddings. Another baseline was SBERT-cosine, which employed an SBERT backbone, similar to BERT-cosine. Another baseline was ADA-cosine which was similar to BERT-cosine but employed the OpenAI model ada002. Another baseline was SelfCheckGPT with Prompt, which employed the same FLLM in each task. SelfCheckGPT generated N additional stochastic LLM response samples, denoted as S1, S2, . . . , Sn, using the same query. Then it scored the consistency between the generated response and the stochastic samples, by querying the same LLM to determine whether the i-th sentence in A* is supported by sample Si. The final inconsistency score was computed by averaging all the sentence scores.
- receiving a second response to the iterative prompt from the MLM at the predicted temperature state;
[0157]:
The FLLM is queried with a concatenation of the query Q on top of the prompt Pforward, which retrieves a generated answer 408 denoted by A*, signifying the response to the query Q. In other words, the prompt Pforward and the query Q are fed into the LLM as follows:
A*=FLLM([Pforward;Q]) (2)
[0159]:
Some embodiments perform or leverage a backward process for reconstructing the original query Q from the generated answer A*. An example embodiment creates a new prompt 404 by reversing the given prompt Pforward. The reversed prompt rearranges 406 the order within each of the query-answer pairs 426 to form answer-query pairs 419. This portion of a reversed prompt denoted as Pbackward can be expressed as follows:
Pbackward=a1,q1, . . . ,an,qn (3)
[0162]:
In some embodiments which utilize an InterrogateLLM approach, this backward process is repeated multiple times, e.g., K times for each model in BLLM with variable 316 temperature values,
[0149]:
One veracity evaluation approach involves a self-evaluation technique, where models are trained to predict their knowledge of the answer to any given free-form question. This approach entails prompting the language model to internally assess the accuracy of its previous predictions, including estimating the likelihood that its generated response or answer is correct. This approach requires labeled data for model training, making it a supervised task. [BRI: Prompting an LLM to internally assess its own predictions, adjusting temperature to balance exploration and certainty, and using iterative prompting to refine outputs is a powerful combination. It leverages self-evaluation for accuracy, temperature for confidence control, and iteration for continuous improvement, leading to more reliable and robust responses]
MALKI does not explicitly disclose:
- selecting a selected iterative response ("SIR") from the first and second responses to the iterative prompt; and determining if the SIR is satisfactory.
However, PRYZANT discloses:
- selecting a selected iterative response ("SIR") from the first and second responses to the iterative prompt; and determining if the SIR is satisfactory.
[0018]:
Other work applies discrete manipulations to prompts via Reinforcement Learning or LLM-based feedback. Such algorithms may also require low-level access to the LLM, may produce incomprehensible outputs, and/or may rely on directionless Monte Carlo search over the semantic space of prompts.
[0035]:
a selection process is used to decide which candidate prompts are worth carrying forward to the next iteration. This loop allows for incremental improvements and exploration over multiple prompt candidates.
[0002]:
A selection algorithm is used to select one or more optimized prompts from the set of optimized prompts and/or the set of paraphrased prompts, and the process may be iteratively repeated with the selected one or more optimized prompts replacing the current prompt.
[0092]:
In some examples, such selection process includes sampling a subset of prompts, evaluating the sampled subset of prompts in a binary fashion (e.g., true or false, yes or no), then selecting the resultant prompts that satisfy one of true (or yes) or false (or no).
[0112]:
The set of operations further includes receiving, from output of the LLM in response to the first feedback prompt, the one or more first textual gradients; providing, as input to the LLM, a first editing prompt requesting a first set of optimized prompts, based on the initial prompt and the one or more first textual gradients; and receiving, from output of the LLM, the first set of optimized prompts.
[0033]:
The second prompt is referred to herein as editing prompt δ 235 and while this prompt can also vary, it takes the textual gradients g 230 and the current prompt P 205, then performs an edit on the current prompt P 205 in the opposite semantic direction of textual gradients g 230, i.e., to fix the problems with the current prompt P 205 that are indicated by textual gradients g 230
[0059]:
Selection algorithm 360 may be used to select a second set of selected optimized prompts 320 from at least one of the one or more optimized prompts 355 and/or the one or more paraphrased optimized prompts 370
[0019]:
the NL gradients are then propagated into the prompt by editing the prompt in the opposite semantic direction of the gradient. These gradient descent steps may be guided by a beam search and a bandit selection procedure, which significantly improves algorithmic efficiency. Preliminary results across three benchmark NLP tasks and the novel problem of LLM jailbreak detection suggest that automatic prompt optimization can outperform prior prompt editing techniques and improve an initial prompt's performance by up to 31%, by using data to rewrite vague task descriptions into more precise annotation instructions. [BRI: performance up by 31 %]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI and PRYZANT.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
One of ordinary skill would have motivation to combine MALKI and PRYZANT that can provide 31 % increase in initial prompt performance (PRYZANT [0019]).
In regard to claim 12:
MALKI, PRYZANT and Gardner do not explicitly disclose:
- upon determining that the SIR is satisfactory, setting the MLM to a temperature state corresponding to a temperature state of the SIR;
- and upon determining that the SIR is not satisfactory, supplying the SIR as an iterative action to the RLM that is configured to:
- recompute the reward in response to the iterative action;
- and recompute the predicted temperature state based on the recomputed reward;
However, Liu discloses:
- upon determining that the SIR is satisfactory, setting the MLM to a temperature state corresponding to a temperature state of the SIR;
[Abstract]:
Embodiments described herein provide a method of predicting an action by a plurality of language model augmented agents (LAAs)
[0080]:
For a metric for comparing the different models, the experiments mainly use the reward score in each environment to evaluate the performances of LAAs
[0077]:
The system may continue to iterate and select additional actions following the same process of receiving observations from the environment, selecting an LAA, inputting a prompt to the selected LAA, receiving an action recommendation from the LAA, and performing the action, etc. The system may determine (e.g., by the controller) whether the actions successfully completed the task instruction. The system may perform another attempt by restarting the environment and trying a new set of actions determined in the same way. Since the LAAs use LLMs for determining actions, a temperature setting of the LLMs may provide some randomness to the outputs, resulting in different actions being performed at each attempt. The controller may determine an attempt was successful or not successful by collecting feedback from the environment. In some embodiments, an LLM is given the task instruction, the actions, and/or the observations from the environment with a prompt asking the LLM to determine whether the task was successfully performed.
- and upon determining that the SIR is not satisfactory, supplying the SIR as an iterative action to the RLM that is configured to:
[0077]:
The system may continue to iterate and select additional actions following the same process of receiving observations from the environment, selecting an LAA, inputting a prompt to the selected LAA, receiving an action recommendation from the LAA, and performing the action, etc. The system may determine (e.g., by the controller) whether the actions successfully completed the task instruction. The system may perform another attempt by restarting the environment and trying a new set of actions determined in the same way. Since the LAAs use LLMs for determining actions, a temperature setting of the LLMs may provide some randomness to the outputs, resulting in different actions being performed at each attempt. The controller may determine an attempt was successful or not successful by collecting feedback from the environment. In some embodiments, an LLM is given the task instruction, the actions, and/or the observations from the environment with a prompt asking the LLM to determine whether the task was successfully performed. In some embodiments, a human user indicates whether the task was successful.
[0076]:
At step 506, the system causes the action to be performed on the environment thereby causing the first state of the environment to change to a second state. In some embodiments, the system determines a second action based on a determination that the action was not successful. For example, after a failed attempt, the system may start a second attempt, and the prompts used during the second attempt may indicate to the selected LAA what action was tried previously that failed.
- recompute the reward in response to the iterative action;
[0050]:
the backpropagation from the last layer 343 to the input layer 341 may be conducted for a number of training samples in a number of iterative training epochs. In this way, parameters of the neural network may be gradually updated in a direction to result in a lesser or minimized loss, indicating the neural network has been trained to generate a predicted output value closer to the target output value with improved prediction accuracy
[0019]:
When a controller is given a task prompt describing a target task, the controller determines which LAA from a pool of available LAAs is best suited for predicting a next action at every iteration.
[0080]:
For a metric for comparing the different models, the experiments mainly use the reward score in each environment to evaluate the performances of LAAs.
[0080]:
In HotPotQA environment, the reward is defined as the F1 score grading between agent answer and ground-truth answer.
[0074]:
In some embodiments, the output includes a reflection on the performance of one or more past actions performed on the environment. For example, a reflective LAA may be given a prompt to reflect on previous actions or attempts, and recommend how the approach may be changed. Subsequent prompts to the LAAs may be updated on subsequent attempts based on the reflection in order to improve the performance.
- and recompute the predicted temperature state based on the recomputed reward;
[0050]:
the neural network has been trained to generate a predicted output value closer to the target output value with improved prediction accuracy. Training may continue until a stopping criterion is met, such as reaching a maximum number of epochs or achieving satisfactory performance on the validation data.
[0077]:
the system may determine (e.g., by the controller) whether the actions successfully completed the task instruction. The system may perform another attempt by restarting the environment and trying a new set of actions determined in the same way. Since the LAAs use LLMs for determining actions, a temperature setting of the LLMs may provide some randomness to the outputs, resulting in different actions being performed at each attempt. [BRI: another attempt is the recomputing]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI, Gardner and Liu.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
Gardner teaches reinforcement learning, baseline and reward in baseline.
Liu teaches recomputed reward.
One of ordinary skill would have motivation to combine MALKI, PRYZANT, Gardner and Liu that provides improved language model autonomous agents (Liu [0021])
In regard to claim 13:
MALKI discloses:
wherein the program instructions executable by the processor further cause the computing device to: [0042]
- set the MLM to the recomputed predicted temperature state;
[0023]:
FIG. 13 is a formula representation of an incremental language model temperature parameter adjustment function;
[0031]:
a system is configured to submit a sequence of backward prompts at different settings of a language model parameter, and the language model parameter is one of: temperature, top-k, or top-p.
[0132]:
In some embodiments, the primary answer belongs to a first dataset 494, and at least a portion of the method 1000 is repeated with a second dataset 494
[0172]:
In RL, an agent (the prompt engineering model) takes actions (generates prompts) in an environment (the LLM) to maximize a reward (the summarization accuracy). The model receives feedback on how well each prompt achieves the target abstraction level. It learns to improve prompts through trial-and-error.
[0173]:
The model may be initialized with weights pretrained on human prompts. RL fine-tunes prompts tailored to the nuances of the particular LLM model. Various policy gradient algorithms like REINFORCE are employed for training.
[0167]:
In example embodiments, multi-task learning helps optimize prompts. Auxiliary tasks like predicting the ideal length or abstraction level for a document provide useful gradients during training. This may lead to better shared representations.
[0174]:
This allows adapting prompts to optimize abstraction control for individual LLMs. The RL agent learns intricacies of how different prompts influence each LLM's outputs. [BRI: initializing with weights pretrained on human prompts is standard, and RL is a powerful way to fine-tune those weights to produce outputs tailored to the nuances of the LLM and the target application.
- issue a subsequent iterative prompt to the MLM at the recomputed predicted temperature state;
[0023]:
FIG. 13 is a formula representation of an incremental language model temperature parameter adjustment function;
[0031]:
a system is configured to submit a sequence of backward prompts at different settings of a language model parameter, and the language model parameter is one of: temperature, top-k, or top-p.
[0132]:
In some embodiments, the primary answer belongs to a first dataset 494, and at least a portion of the method 1000 is repeated with a second dataset 494
[0172]:
In RL, an agent (the prompt engineering model) takes actions (generates prompts) in an environment (the LLM) to maximize a reward (the summarization accuracy). The model receives feedback on how well each prompt achieves the target abstraction level. It learns to improve prompts through trial-and-error.
[0173]:
The model may be initialized with weights pretrained on human prompts. RL fine-tunes prompts tailored to the nuances of the particular LLM model. Various policy gradient algorithms like REINFORCE are employed for training.
[0174]:
This allows adapting prompts to optimize abstraction control for individual LLMs. The RL agent learns intricacies of how different prompts influence each LLM's outputs.
[BRI: initializing with weights pretrained on human prompts is standard, and RL is a powerful way to fine-tune those weights to produce outputs tailored to the nuances of the LLM and the target application combining the speed of prompt-based adaptation with the precision of reinforcement learning
- receive a first response to the subsequent iterative prompt from the MLM at the iterative temperature state;
[0023]:
FIG. 13 is a formula representation of an incremental language model temperature parameter adjustment function;
[0031]:
a system is configured to submit a sequence of backward prompts at different settings of a language model parameter, and the language model parameter is one of: temperature, top-k, or top-p.
[0132]:
In some embodiments, the primary answer belongs to a first dataset 494, and at least a portion of the method 1000 is repeated with a second dataset 494
[0835]:
Streamlined orchestration coordinates the summarization pipeline for optimized performance.
[0293]:
The summarization workflow may begin when a client request is received by the system's API gateway. Robust ingestion prepares the request for processing.
[0172]:
In RL, an agent (the prompt engineering model) takes actions (generates prompts) in an environment (the LLM) to maximize a reward (the summarization accuracy). The model receives feedback on how well each prompt achieves the target abstraction level. It learns to improve prompts through trial-and-error.
[0173]:
The model may be initialized with weights pretrained on human prompts. RL fine-tunes prompts tailored to the nuances of the particular LLM model. Various policy gradient algorithms like REINFORCE are employed for training.
[0174]:
This allows adapting prompts to optimize abstraction control for individual LLMs. The RL agent learns intricacies of how different prompts influence each LLM's outputs.
[BRI: The iterative RLHF process can indeed produce a “first” to a subsequent prompt as an intial attempt using the formula]
- receive a second response to the subsequent iterative prompt from the MLM at the recomputed predicted temperature state;
[0023]:
FIG. 13 is a formula representation of an incremental language model temperature parameter adjustment function;
[0031]:
a system is configured to submit a sequence of backward prompts at different settings of a language model parameter, and the language model parameter is one of: temperature, top-k, or top-p.
[0132]:
In some embodiments, the primary answer belongs to a first dataset 494, and at least a portion of the method 1000 is repeated with a second dataset 494
[0835]:
Streamlined orchestration coordinates the summarization pipeline for optimized performance.
[0293]:
The summarization workflow may begin when a client request is received by the system's API gateway. Robust ingestion prepares the request for processing.
[0172]:
In RL, an agent (the prompt engineering model) takes actions (generates prompts) in an environment (the LLM) to maximize a reward (the summarization accuracy). The model receives feedback on how well each prompt achieves the target abstraction level. It learns to improve prompts through trial-and-error.
[0173]:
The model may be initialized with weights pretrained on human prompts. RL fine-tunes prompts tailored to the nuances of the particular LLM model. Various policy gradient algorithms like REINFORCE are employed for training.
[0174]:
This allows adapting prompts to optimize abstraction control for individual LLMs. The RL agent learns intricacies of how different prompts influence each LLM's outputs.
[BRI: The iterative RLHF process can indeed produce a “second response” to a subsequent prompt, refined to match the nuances of the target model’s behavior, with temperature settings influencing the output style]
MALKI does not explicitly disclose:
- select a subsequent selected iterative response ("SIR") from the first and second responses to the iterative prompt; and determining if the subsequent SIR is satisfactory.
However, PRYZANT discloses:
- select a subsequent selected iterative response ("SIR") from the first and second responses to the iterative prompt; and determining if the subsequent SIR is satisfactory.
[0004]:
A selection algorithm is used to select one or more optimized prompts from the set of optimized prompts and/or the set of paraphrased prompts, and the process may be iteratively repeated with the selected one or more optimized prompts replacing the current prompt. [BRI: an iterative process for selection represents subsequent selection]
[0018]:
Other work applies discrete manipulations to prompts via Reinforcement Learning or LLM-based feedback. Such algorithms may also require low-level access to the LLM, may produce incomprehensible outputs, and/or may rely on directionless Monte Carlo search over the semantic space of prompts.
[0063] :
The LM input optimization technology framework, as described herein, was compared against the following baselines, which focused on nonparametric algorithms that are directly comparable to the LM input optimization technology: (a) Monte-Carlo (“MC”); (b) Reinforcement Learning (“RL”); and/or (c) AutoGPT. MC is an automatic prompt engineering algorithm that proposes an iterative but directionless
[0063]:
AutoGPT is an open-source AI agent, which relies on an agent-controlled feedback loop to improve its responses.
[0061];
he Azure® OpenAI™ LLM API service with a temperature of 0.0 during few-shot classification and 1.0 in all other contexts. For the nonparametric algorithms with broad applicability, default values and the same parameters were used throughout instead of conducting any hyperparameter search for the baseline or proposed algorithms.
[0112]:
The set of operations further includes receiving, from output of the LLM in response to the first feedback prompt, the one or more first textual gradients; providing, as input to the LLM, a first editing prompt requesting a first set of optimized prompts, based on the initial prompt and the one or more first textual gradients; and receiving, from output of the LLM, the first set of optimized prompts. The set of operations also includes selecting one or more first optimized prompts from at least the first set of optimized prompts based at least in part on evaluation of prompt performance using a secondary LLM that is finetuned based on a curated dataset for a specific subject area. The set of operations further includes instructing the LLM to perform task focused on the subject area by inputting the selected one or more first optimized prompts into the secondary LLM; and receiving, from the secondary LLM, results to the instructed task.
[0022]:
In some cases, computing system 105a may further include an automated prompt optimizer 125a that uses one or more LLMs 130a-130k (collectively, “LLMs 130”; in this case, first through Kth LLMs 130a-130k) to perform automated prompt optimization using textual gradients. Herein, an LLM, which is a type of language model (“LM”), may be a deep learning algorithm that can recognize, summarize, translate, predict, and/or generate text and/or other content based on knowledge gained from massive datasets.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI and PRYZANT.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
One of ordinary skill would have motivation to combine MALKI and PRYZANT that can provide 31 % increase in initial prompt performance (PRYZANT [0019]).
MALKI and PRYZANT do not explicitly disclose:
- further comprising: setting the MLM to a subsequent iterative temperature state corresponding to a temperature state of the SBR;
- issuing a subsequent iterative prompt to the MLM at the iterative temperature state;
- receiving a first response to the subsequent iterative prompt from the MLM at the iterative temperature state;
However, Gardner discloses:
- further comprising: setting the MLM to a subsequent iterative temperature state corresponding to a temperature state of the SBR;
[0051]:
The trained model can generate coherent, human-like text by iteratively sampling from its predicted next word distributions to continue growing new sequences. Temperature parameters control randomness versus determinism in sampling.
[0597]:
Set hyperparameters for models like length, temperature, top-k sampling etc. Higher temperature produces more creative but less focused summaries.
[0835]:
Streamlined orchestration coordinates the summarization pipeline for optimized performance.
[0958]:
The templates may control phrasing, instructions, and content provided to adjust the abstraction in a step-wise manner. The LLM output summary may then be used as the content for the next iteration's increased abstraction prompt. [BRI: this is an iterative abstraction pipeline where temperature is adjusted based on the abstraction level, and the summary output from one iteration is used as the prompt for the next iteration (subsequent)]
- issuing a subsequent iterative prompt to the MLM at the iterative temperature state;
[0839]:
Message queueing provides scalability, resilience and restartability for long-running multi-stage summarization pipelines.
[0836]:
A message broker like Kafka connects the sequence of microservices that comprise the summarization workflow-cleaning, extraction, abstraction, compression, storage etc.
- receiving a first response to the subsequent iterative prompt from the MLM at the iterative temperature state;
[0931]:
This simple request/response flow enables out-of-the-box summarization without developer work. The system acts as a hands-off pass-through to the LLM.
[1149]:
The Dialog Module 260 allows leveraging the core summarization capabilities to generate conversational responses via dialog systems and APIs. For instance, taking a summarized description of a customer support ticket and producing a natural language response addressing the key issues detected. The dialog module allows progressing from static summarization to interactive conversational applications.
In regard to claim 14:
MALKI, PRYZANT and Gardner does not explicitly disclose:
wherein the program instructions executable by the processor further cause the computing device to
- upon determining that the subsequent SIR is satisfactory, setting the MLM to a temperature state corresponding to a temperature state of the SIR;
- and upon determining that the subsequent SIR is not satisfactory, supplying the SIR as an iterative action to the RLM that is configured to:
- recompute the reward in response to the subsequent iterative action;
- and recompute the predicted temperature state based on the most recent recomputed reward;
However, Liu discloses:
wherein the program instructions executable by the processor further cause the computing device to: [0008]; [0027];
- upon determining that the subsequent SIR is satisfactory, setting the MLM to a temperature state corresponding to a temperature state of the SIR;
[Abstract]:
Embodiments described herein provide a method of predicting an action by a plurality of language model augmented agents (LAAs)
[0080]:
For a metric for comparing the different models, the experiments mainly use the reward score in each environment to evaluate the performances of LAAs
[0077]:
The system may continue to iterate and select additional actions following the same process of receiving observations from the environment, selecting an LAA, inputting a prompt to the selected LAA, receiving an action recommendation from the LAA, and performing the action, etc. The system may determine (e.g., by the controller) whether the actions successfully completed the task instruction. The system may perform another attempt by restarting the environment and trying a new set of actions determined in the same way. Since the LAAs use LLMs for determining actions, a temperature setting of the LLMs may provide some randomness to the outputs, resulting in different actions being performed at each attempt. The controller may determine an attempt was successful or not successful by collecting feedback from the environment. In some embodiments, an LLM is given the task instruction, the actions, and/or the observations from the environment with a prompt asking the LLM to determine whether the task was successfully performed.
[0032]:
Observations 104 from this action and subsequent actions may not result in a correct answer to the question. For a next attempt, agent selection 208 may select a “reflective” LAA, and communicate an agents message 210 to the reflective LAA with the task instruction 116, the set of actions 114 from the failed attempt and the corresponding observations 104
- and upon determining that the subsequent SIR is not satisfactory, supplying the SIR as an iterative action to the RLM that is configured to:
[0077]:
The system may continue to iterate and select additional actions following the same process of receiving observations from the environment, selecting an LAA, inputting a prompt to the selected LAA, receiving an action recommendation from the LAA, and performing the action, etc. The system may determine (e.g., by the controller) whether the actions successfully completed the task instruction. The system may perform another attempt by restarting the environment and trying a new set of actions determined in the same way. Since the LAAs use LLMs for determining actions, a temperature setting of the LLMs may provide some randomness to the outputs, resulting in different actions being performed at each attempt. The controller may determine an attempt was successful or not successful by collecting feedback from the environment. In some embodiments, an LLM is given the task instruction, the actions, and/or the observations from the environment with a prompt asking the LLM to determine whether the task was successfully performed. In some embodiments, a human user indicates whether the task was successful.
[0076]:
At step 506, the system causes the action to be performed on the environment thereby causing the first state of the environment to change to a second state. In some embodiments, the system determines a second action based on a determination that the action was not successful. For example, after a failed attempt, the system may start a second attempt, and the prompts used during the second attempt may indicate to the selected LAA what action was tried previously that failed.
- recompute the reward in response to the subsequent iterative action;
[0050]:
the backpropagation from the last layer 343 to the input layer 341 may be conducted for a number of training samples in a number of iterative training epochs. In this way, parameters of the neural network may be gradually updated in a direction to result in a lesser or minimized loss, indicating the neural network has been trained to generate a predicted output value closer to the target output value with improved prediction accuracy
[0019]:
When a controller is given a task prompt describing a target task, the controller determines which LAA from a pool of available LAAs is best suited for predicting a next action at every iteration.
[0080]:
For a metric for comparing the different models, the experiments mainly use the reward score in each environment to evaluate the performances of LAAs.
[0080]:
In HotPotQA environment, the reward is defined as the F1 score grading between agent answer and ground-truth answer.
[0074]:
In some embodiments, the output includes a reflection on the performance of one or more past actions performed on the environment. For example, a reflective LAA may be given a prompt to reflect on previous actions or attempts, and recommend how the approach may be changed. Subsequent prompts to the LAAs may be updated on subsequent attempts based on the reflection in order to improve the performance.
- and recompute the predicted temperature state based on the most recent recomputed reward;
[0050]:
the neural network has been trained to generate a predicted output value closer to the target output value with improved prediction accuracy. Training may continue until a stopping criterion is met, such as reaching a maximum number of epochs or achieving satisfactory performance on the validation data.
[0077]:
the system may determine (e.g., by the controller) whether the actions successfully completed the task instruction. The system may perform another attempt by restarting the environment and trying a new set of actions determined in the same way. Since the LAAs use LLMs for determining actions, a temperature setting of the LLMs may provide some randomness to the outputs, resulting in different actions being performed at each attempt. [BRI: another attempt is the recomputing]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI, Gardner and Liu.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
Gardner teaches reinforcement learning, baseline and reward in baseline.
Liu teaches recomputed reward.
One of ordinary skill would have motivation to combine MALKI, PRYZANT, Gardner and Liu that provides improved language model autonomous agents (Liu [0021])
In regard to claim 16:
MALKI and PRYZANT do not explicitly disclose:
- wherein the baseline prompt, the iterative prompt, and the subsequent iterative prompt each comprise a search vector in an embedding space of the MLM.
However, Gardner discloses:
- wherein the baseline prompt, the iterative prompt, and the subsequent iterative prompt each comprise a search vector in an embedding space of the MLM.
[0048]:
The architecture may include transformer blocks with attention mechanisms to model context and identify relevant patterns across the entire input sequence. The attention weights and feedforward layers may transform the input embeddings into higher level contextual representations used to make the word predictions.
[0045]:
a search application may enable keyword searches of the content items 128.
[0074]:
Pre-trained semantic search engines can help identify contextual text passages for a given topic.
[0840]:
Raw input documents are cleaned, parsed, segmented, tokenized etc. to normalize them for downstream summarization steps. [BRI: the input embedding are the initial dense vector representation of tokens word. Perhaps known to a POSTA, the search vector is not a separate space; it is a contextualized representation in the same embedding space as the input embeddings]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI, PRYZANT and Gardner.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
Garner teaches reinforcement learning for reward and prediction based on reward.
One of ordinary skill would have motivation to combine MALKI, PRYZANT and Gardner to provide substantial technical improvement for engineering prompts with configurable abstraction parameters to control summarization by LLMs (Gardner [0029]) and over existing approaches to text summarization, such as in the areas of abstraction control, iterative refinement, and/or information retention (Gardner [0038]).
In regard to claim 17:
MALKI and PRYZANT do not explicitly disclose:
- comprise a probability distribution around one of the search vectors in the embedding space of the MLM.
However, Gardner discloses:
- comprise a probability distribution around one of the search vectors in the embedding space of the MLM.
[0058]:
Extractive summarization models can identify and extract the most salient sentences or phrases from the original text to produce abridged summaries. These models may be trained on text-summary pairs to learn to rank sentences based on importance. These models may use encoder-decoder architectures, converting sentences to vector representations and scoring for relevance. The system can be configured to use extractive summarization models to, for example, concatenate extracted snippets into summaries meeting length constraints.
[0156]:
the decoder model generates an optimized prompt sequence conditioned on the encoder outputs. [ BRI: encoder-decoder models can be trained on text–summary pairs to learn to rank sentences by importance, and they can also be used to convert sentences into vector representations for relevance scoring. They are also well-suited for predicting next-word distributions, which can be interpreted as probability distributions in the embedding space of a masked language model (MLM). The decoder’s output is a probability distribution over the vocabulary for each predicted token]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI, PRYZANT and Gardner.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
Garner teaches reinforcement learning for reward and prediction based on reward.
One of ordinary skill would have motivation to combine MALKI, PRYZANT and Gardner to provide substantial technical improvement for engineering prompts with configurable abstraction parameters to control summarization by LLMs (Gardner [0029]) and over existing approaches to text summarization, such as in the areas of abstraction control, iterative refinement, and/or information retention (Gardner [0038]).
In regard to claim 19:
MALKI discloses:
- further comprising: setting the MLM to an iterative temperature state corresponding to a temperature state of the SBR;
[0023]:
FIG. 13 is a formula representation of an incremental language model temperature parameter adjustment function;
[0031]:
a system is configured to submit a sequence of backward prompts at different settings of a language model parameter, and the language model parameter is one of: temperature, top-k, or top-p.
[0132]:
In some embodiments, the primary answer belongs to a first dataset 494, and at least a portion of the method 1000 is repeated with a second dataset 494, [BRI: in SBR modeling, the primary answer from the first dataset can be followed by iterative application of the same method to a second dataset to represent evolving temperature states. This approach leverages iterative dataset refinement and repeated sampling to improve accuracy and capture the dynamic nature of SBR processes
- issuing an iterative prompt to the MLM at the iterative temperature state;
[0162] :
In some embodiments which utilize an InterrogateLLM approach, this backward process is repeated multiple times, e.g., K times for each model in BLLM with variable 316 temperature values, [BR: this is an iterative, temperature-tuned prompting strategy that mimics backward chaining, allowing the model to explore, revise, and refine its reasoning path before finalizing the output]
- receiving a first response to the iterative prompt from the MLM at the iterative temperature state;
[0157]:
The FLLM is queried with a concatenation of the query Q on top of the prompt Pforward, which retrieves a generated answer 408 denoted by A*, signifying the response to the query Q. In other words, the prompt Pforward and the query Q are fed into the LLM as follows:
A*=FLLM([Pforward;Q]) (2)
[0159]:
Some embodiments perform or leverage a backward process for reconstructing the original query Q from the generated answer A*. An example embodiment creates a new prompt 404 by reversing the given prompt Pforward. The reversed prompt rearranges 406 the order within each of the query-answer pairs 426 to form answer-query pairs 419. This portion of a reversed prompt denoted as Pbackward can be expressed as follows:
Pbackward=a1,q1, . . . ,an,qn (3)
[0162]:
In some embodiments which utilize an InterrogateLLM approach, this backward process is repeated multiple times, e.g., K times for each model in BLLM with variable 316 temperature values,
- setting the MLM to the predicted temperature state;
[0023]:
FIG. 13 is a formula representation of an incremental language model temperature parameter adjustment function;
[0031]:
a system is configured to submit a sequence of backward prompts at different settings of a language model parameter, and the language model parameter is one of: temperature, top-k, or top-p. [BRI: Prompting LLMs to internally assess their own predictions (e.g., via calibration, internal consistency, or self-consistency selection) can improve accuracy, but temperature adjustment is essential to ensure that confidence estimates are reliable. Low temperature supports confident self-assessment, while moderate temperature can help explore diverse reasoning paths before final selection.
[0149]:
One veracity evaluation approach involves a self-evaluation technique, where models are trained to predict their knowledge of the answer to any given free-form question. This approach entails prompting the language model to internally assess the accuracy of its previous predictions, including estimating the likelihood that its generated response or answer is correct. This approach requires labeled data for model training, making it a supervised task.
- issuing the iterative prompt to the MLM at the predicted temperature state;
[0150];
prompt that includes a stochastic sample and a sentence from the generated text and predicts whether the sentence is supported by the stochastic sample. Internal testing of the InterrogateLLM utilized a SelfCheckGPT benchmark, but unless expressly stated otherwise herein for a particular embodiment, the embodiments taught herein do not rely on employing an additional prompt that includes a stochastic sample and a sentence from the generated text and on predicting whether the sentence is supported by the stochastic sample, for hallucination detection.
[0154]:
Some examples herein assume a source domain of textual queries and a target domain of textual answers, but as noted elsewhere the teachings herein also apply to other data types 425. Assume also that a few-shot prompt, a corresponding query Q and a LLM 132 denoted by FLLM 412, are provided. The terms “query” and “question” are used interchangeably herein with respect to model 132 prompts and outputs, unless indicated otherwise by reference numerals. Although this portion of the discussion assumes a provided few-shot prompt 402, tools and techniques taught herein are adaptable to many zero-shot tasks where the creation of few-shot prompts is feasible. The prompt is fed into the LLM to generate an answer to a query in the prompt.
[0176]:
The internal evaluation compared InterrogateLLM prototypes to baselines. One baseline was BERT-cosine, which employed a pre-trained BERT model to embed both the query and the generated answer, calculated the cosine similarity between them, and predicted “hallucination” if the similarity fell below a threshold BERTT. The threshold was determined by using a process described above, with BERT embeddings. Another baseline was SBERT-cosine, which employed an SBERT backbone, similar to BERT-cosine. Another baseline was ADA-cosine which was similar to BERT-cosine but employed the OpenAI model ada002. Another baseline was SelfCheckGPT with Prompt, which employed the same FLLM in each task. SelfCheckGPT generated N additional stochastic LLM response samples, denoted as S1, S2, . . . , Sn, using the same query. Then it scored the consistency between the generated response and the stochastic samples, by querying the same LLM to determine whether the i-th sentence in A* is supported by sample Si. The final inconsistency score was computed by averaging all the sentence scores.
- receiving a second response to the iterative prompt from the MLM at the predicted temperature state;
[0157]:
The FLLM is queried with a concatenation of the query Q on top of the prompt Pforward, which retrieves a generated answer 408 denoted by A*, signifying the response to the query Q. In other words, the prompt Pforward and the query Q are fed into the LLM as follows:
A*=FLLM([Pforward;Q]) (2)
[0159]:
Some embodiments perform or leverage a backward process for reconstructing the original query Q from the generated answer A*. An example embodiment creates a new prompt 404 by reversing the given prompt Pforward. The reversed prompt rearranges 406 the order within each of the query-answer pairs 426 to form answer-query pairs 419. This portion of a reversed prompt denoted as Pbackward can be expressed as follows:
Pbackward=a1,q1, . . . ,an,qn (3)
[0162]:
In some embodiments which utilize an InterrogateLLM approach, this backward process is repeated multiple times, e.g., K times for each model in BLLM with variable 316 temperature values,
[0149]:
One veracity evaluation approach involves a self-evaluation technique, where models are trained to predict their knowledge of the answer to any given free-form question. This approach entails prompting the language model to internally assess the accuracy of its previous predictions, including estimating the likelihood that its generated response or answer is correct. This approach requires labeled data for model training, making it a supervised task. [BRI: Prompting an LLM to internally assess its own predictions, adjusting temperature to balance exploration and certainty, and using iterative prompting to refine outputs is a powerful combination. It leverages self-evaluation for accuracy, temperature for confidence control, and iteration for continuous improvement, leading to more reliable and robust responses]
MALKI does not explicitly disclose:
- selecting a selected iterative response ("SIR") from the first and second responses to the iterative prompt; and determining if the SIR is satisfactory.
However, PRYZANT discloses:
- selecting a selected iterative response ("SIR") from the first and second responses to the iterative prompt; and determining if the SIR is satisfactory.
[0018]:
Other work applies discrete manipulations to prompts via Reinforcement Learning or LLM-based feedback. Such algorithms may also require low-level access to the LLM, may produce incomprehensible outputs, and/or may rely on directionless Monte Carlo search over the semantic space of prompts.
[0063] :
The LM input optimization technology framework, as described herein, was compared against the following baselines, which focused on nonparametric algorithms that are directly comparable to the LM input optimization technology: (a) Monte-Carlo (“MC”); (b) Reinforcement Learning (“RL”); and/or (c) AutoGPT. MC is an automatic prompt engineering algorithm that proposes an iterative but directionless
[0063]:
AutoGPT is an open-source AI agent, which relies on an agent-controlled feedback loop to improve its responses.
[0061];
he Azure® OpenAI™ LLM API service with a temperature of 0.0 during few-shot classification and 1.0 in all other contexts. For the nonparametric algorithms with broad applicability, default values and the same parameters were used throughout instead of conducting any hyperparameter search for the baseline or proposed algorithms.
[0112]:
The set of operations further includes receiving, from output of the LLM in response to the first feedback prompt, the one or more first textual gradients; providing, as input to the LLM, a first editing prompt requesting a first set of optimized prompts, based on the initial prompt and the one or more first textual gradients; and receiving, from output of the LLM, the first set of optimized prompts. The set of operations also includes selecting one or more first optimized prompts from at least the first set of optimized prompts based at least in part on evaluation of prompt performance using a secondary LLM that is finetuned based on a curated dataset for a specific subject area. The set of operations further includes instructing the LLM to perform task focused on the subject area by inputting the selected one or more first optimized prompts into the secondary LLM; and receiving, from the secondary LLM, results to the instructed task.
[0022]:
In some cases, computing system 105a may further include an automated prompt optimizer 125a that uses one or more LLMs 130a-130k (collectively, “LLMs 130”; in this case, first through Kth LLMs 130a-130k) to perform automated prompt optimization using textual gradients. Herein, an LLM, which is a type of language model (“LM”), may be a deep learning algorithm that can recognize, summarize, translate, predict, and/or generate text and/or other content based on knowledge gained from massive datasets.
[0035]:
a selection process is used to decide which candidate prompts are worth carrying forward to the next iteration. This loop allows for incremental improvements and exploration over multiple prompt candidates.
[0002]:
A selection algorithm is used to select one or more optimized prompts from the set of optimized prompts and/or the set of paraphrased prompts, and the process may be iteratively repeated with the selected one or more optimized prompts replacing the current prompt.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI and PRYZANT.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
One of ordinary skill would have motivation to combine MALKI and PRYZANT that can provide 31 % increase in initial prompt performance (PRYZANT [0019]).
In regard to claim 20:
MALKI, PRYZANT and Gardner do not explicitly disclose:
wherein the program instructions executable by the processor further cause the computing device to:
- upon determining that the SIR is satisfactory, setting the MLM to a temperature state corresponding to a temperature state of the SIR;
- and upon determining that the SIR is not satisfactory, supplying the SIR as an iterative action to the RLM that is configured to:
- recompute the reward in response to the iterative action;
- and recompute the predicted temperature state based on the recomputed reward;
- upon determining that the subsequent SIR is satisfactory, setting the MLM to a temperature state corresponding to a temperature state of the SIR;
- and upon determining that the subsequent SIR is not satisfactory, supplying the SIR as an iterative action to the RLM that is configured to:
- recompute the reward in response to the subsequent iterative action;
- and recompute the predicted temperature state based on the most recent recomputed reward;
However, Liu discloses:
wherein the program instructions executable by the processor further cause the computing device to:[ 0008]; [0027];
- upon determining that the SIR is satisfactory, setting the MLM to a temperature state corresponding to a temperature state of the SIR;
[Abstract]:
Embodiments described herein provide a method of predicting an action by a plurality of language model augmented agents (LAAs)
[0080]:
For a metric for comparing the different models, the experiments mainly use the reward score in each environment to evaluate the performances of LAAs
[0077]:
The system may continue to iterate and select additional actions following the same process of receiving observations from the environment, selecting an LAA, inputting a prompt to the selected LAA, receiving an action recommendation from the LAA, and performing the action, etc. The system may determine (e.g., by the controller) whether the actions successfully completed the task instruction. The system may perform another attempt by restarting the environment and trying a new set of actions determined in the same way. Since the LAAs use LLMs for determining actions, a temperature setting of the LLMs may provide some randomness to the outputs, resulting in different actions being performed at each attempt. The controller may determine an attempt was successful or not successful by collecting feedback from the environment. In some embodiments, an LLM is given the task instruction, the actions, and/or the observations from the environment with a prompt asking the LLM to determine whether the task was successfully performed.
- and upon determining that the SIR is not satisfactory, supplying the SIR as an iterative action to the RLM that is configured to:
[0077]:
The system may continue to iterate and select additional actions following the same process of receiving observations from the environment, selecting an LAA, inputting a prompt to the selected LAA, receiving an action recommendation from the LAA, and performing the action, etc. The system may determine (e.g., by the controller) whether the actions successfully completed the task instruction. The system may perform another attempt by restarting the environment and trying a new set of actions determined in the same way. Since the LAAs use LLMs for determining actions, a temperature setting of the LLMs may provide some randomness to the outputs, resulting in different actions being performed at each attempt. The controller may determine an attempt was successful or not successful by collecting feedback from the environment. In some embodiments, an LLM is given the task instruction, the actions, and/or the observations from the environment with a prompt asking the LLM to determine whether the task was successfully performed. In some embodiments, a human user indicates whether the task was successful.
[0076]:
At step 506, the system causes the action to be performed on the environment thereby causing the first state of the environment to change to a second state. In some embodiments, the system determines a second action based on a determination that the action was not successful. For example, after a failed attempt, the system may start a second attempt, and the prompts used during the second attempt may indicate to the selected LAA what action was tried previously that failed.
- recompute the reward in response to the iterative action;
[0050]:
the backpropagation from the last layer 343 to the input layer 341 may be conducted for a number of training samples in a number of iterative training epochs. In this way, parameters of the neural network may be gradually updated in a direction to result in a lesser or minimized loss, indicating the neural network has been trained to generate a predicted output value closer to the target output value with improved prediction accuracy
[0019]:
When a controller is given a task prompt describing a target task, the controller determines which LAA from a pool of available LAAs is best suited for predicting a next action at every iteration.
[0080]:
For a metric for comparing the different models, the experiments mainly use the reward score in each environment to evaluate the performances of LAAs.
[0080]:
In HotPotQA environment, the reward is defined as the F1 score grading between agent answer and ground-truth answer.
[0074]:
In some embodiments, the output includes a reflection on the performance of one or more past actions performed on the environment. For example, a reflective LAA may be given a prompt to reflect on previous actions or attempts, and recommend how the approach may be changed. Subsequent prompts to the LAAs may be updated on subsequent attempts based on the reflection in order to improve the performance.
- and recompute the predicted temperature state based on the recomputed reward;
[0050]:
the neural network has been trained to generate a predicted output value closer to the target output value with improved prediction accuracy. Training may continue until a stopping criterion is met, such as reaching a maximum number of epochs or achieving satisfactory performance on the validation data.
[0077]:
the system may determine (e.g., by the controller) whether the actions successfully completed the task instruction. The system may perform another attempt by restarting the environment and trying a new set of actions determined in the same way. Since the LAAs use LLMs for determining actions, a temperature setting of the LLMs may provide some randomness to the outputs, resulting in different actions being performed at each attempt. [BRI: another attempt is the recomputing]
- upon determining that the subsequent SIR is satisfactory, setting the MLM to a temperature state corresponding to a temperature state of the SIR;
[Abstract]:
Embodiments described herein provide a method of predicting an action by a plurality of language model augmented agents (LAAs)
[0080]:
For a metric for comparing the different models, the experiments mainly use the reward score in each environment to evaluate the performances of LAAs
[0077]:
The system may continue to iterate and select additional actions following the same process of receiving observations from the environment, selecting an LAA, inputting a prompt to the selected LAA, receiving an action recommendation from the LAA, and performing the action, etc. The system may determine (e.g., by the controller) whether the actions successfully completed the task instruction. The system may perform another attempt by restarting the environment and trying a new set of actions determined in the same way. Since the LAAs use LLMs for determining actions, a temperature setting of the LLMs may provide some randomness to the outputs, resulting in different actions being performed at each attempt. The controller may determine an attempt was successful or not successful by collecting feedback from the environment. In some embodiments, an LLM is given the task instruction, the actions, and/or the observations from the environment with a prompt asking the LLM to determine whether the task was successfully performed.
[0032]:
Observations 104 from this action and subsequent actions may not result in a correct answer to the question. For a next attempt, agent selection 208 may select a “reflective” LAA, and communicate an agents message 210 to the reflective LAA with the task instruction 116, the set of actions 114 from the failed attempt and the corresponding observations 104
- and upon determining that the subsequent SIR is not satisfactory, supplying the SIR as an iterative action to the RLM that is configured to:
[0077]:
The system may continue to iterate and select additional actions following the same process of receiving observations from the environment, selecting an LAA, inputting a prompt to the selected LAA, receiving an action recommendation from the LAA, and performing the action, etc. The system may determine (e.g., by the controller) whether the actions successfully completed the task instruction. The system may perform another attempt by restarting the environment and trying a new set of actions determined in the same way. Since the LAAs use LLMs for determining actions, a temperature setting of the LLMs may provide some randomness to the outputs, resulting in different actions being performed at each attempt. The controller may determine an attempt was successful or not successful by collecting feedback from the environment. In some embodiments, an LLM is given the task instruction, the actions, and/or the observations from the environment with a prompt asking the LLM to determine whether the task was successfully performed. In some embodiments, a human user indicates whether the task was successful.
[0076]:
At step 506, the system causes the action to be performed on the environment thereby causing the first state of the environment to change to a second state. In some embodiments, the system determines a second action based on a determination that the action was not successful. For example, after a failed attempt, the system may start a second attempt, and the prompts used during the second attempt may indicate to the selected LAA what action was tried previously that failed.
- recompute the reward in response to the subsequent iterative action;
[0050]:
the backpropagation from the last layer 343 to the input layer 341 may be conducted for a number of training samples in a number of iterative training epochs. In this way, parameters of the neural network may be gradually updated in a direction to result in a lesser or minimized loss, indicating the neural network has been trained to generate a predicted output value closer to the target output value with improved prediction accuracy
[0019]:
When a controller is given a task prompt describing a target task, the controller determines which LAA from a pool of available LAAs is best suited for predicting a next action at every iteration.
[0080]:
For a metric for comparing the different models, the experiments mainly use the reward score in each environment to evaluate the performances of LAAs.
[0080]:
In HotPotQA environment, the reward is defined as the F1 score grading between agent answer and ground-truth answer.
[0074]:
In some embodiments, the output includes a reflection on the performance of one or more past actions performed on the environment. For example, a reflective LAA may be given a prompt to reflect on previous actions or attempts, and recommend how the approach may be changed. Subsequent prompts to the LAAs may be updated on subsequent attempts based on the reflection in order to improve the performance.
- and recompute the predicted temperature state based on the most recent recomputed reward;
[0050]:
the neural network has been trained to generate a predicted output value closer to the target output value with improved prediction accuracy. Training may continue until a stopping criterion is met, such as reaching a maximum number of epochs or achieving satisfactory performance on the validation data.
[0077]:
the system may determine (e.g., by the controller) whether the actions successfully completed the task instruction. The system may perform another attempt by restarting the environment and trying a new set of actions determined in the same way. Since the LAAs use LLMs for determining actions, a temperature setting of the LLMs may provide some randomness to the outputs, resulting in different actions being performed at each attempt. [BRI: another attempt is the recomputing]
[0030]:
In some embodiments, agent selection 208 may select an agent based on which actions are available for the current state of environment 102. For example, the most recent observation 104 may indicate that environment 102 after the most recent action 114 is in a state that has the option of clicking one of four different buttons, with no other actions available.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine MALKI, PRYZANT, Gardner and Liu.
MALKI teaches LLM prompt session, and reinforcement learning.
PRYZANT teaches selecting baseline response ("SBR") from the first and second responses to the baseline prompt;
Gardner teaches reinforcement learning, baseline and reward in baseline.
Liu teaches recomputed reward.
One of ordinary skill would have motivation to combine MALKI, PRYZANT, Gardner and Liu that provides improved language model autonomous agents (Liu [0021])
Conclusion
Any inquiry concerning this communication or earlier communications from the
examiner should be directed to TIRUMALE KRISHNASWAMY RAMESH whose telephone number is (571)272-4605. The examiner can normally be reached by phone.
Examiner interviews are available via telephone, in-person, and video conferencing
using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at
http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached on phone (571-272-3768). The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be
obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit:
https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for
information about filing in DOCX format. For additional questions, contact the Electronic
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO
Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TIRUMALE K RAMESH/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights