DETAILED ACTION
This Office Action is in response to the correspondence filed by the applicant on 5/7/2024.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The Information Statements (IDS) filed on 9/26/2026, 7/18/2025, 10/13/2025, 2/18/2026, and 6/3/2026 have been accepted and considered in this office action and are in compliance with the provisions of 37 CFR 1.97.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 4-6, 8, 11, 14-16, and 18 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The independent claims 1, 11, and 20 recite receiving, “selecting, by one or more processors, one or more simple annotated question-answer pairs for an input data object comprising an input question and an input document from a reference dataset; selecting, by the one or more processors, one or more complex annotated question-answer pairs from the reference dataset; generating, by the one or more processors, a few-shot prompt based on the one or more simple annotated question-answer pairs and the one or more complex annotated question-answer pairs; and providing, by the one or more processors, the few-shot prompt to a large language model (LLM) to receive a predictive output for the input question.”
The limitations of the recited steps as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. That is, other than reciting, “memory and onr or more processors”, nothing in the claim element precludes the step from practically being performed in the mind. For example, a person can have a question to ask, select similar exemplar question-and-answer pairs and dissimilar exemplar question-and-answer pairs, and ask another person to answer the question in a manner similar to the similar pairs and in a manner dissimilar to the dissimilar pairs. The limitations, as drafted, are processes that, under its broadest reasonable interpretation, cover performance of the limitations in the mind.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
This judicial exception is not integrated into a practical application. In particular, the claims only recite additional elements – “memory and onr or more processors”. The additional elements in both steps is recited at a high-level of generality (i.e., as a generic processor performing a generic computer function of the recited steps) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using a processor to perform the recited steps amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claim is not patent eligible.
Regarding the dependent claims, claims 4 and 14 recite a mistake prone Q/A pair; claims 5 and 15 recite identifying a failure question scenario; claims 6 and 16 recite a context dissimilar Q/A pair; claims 8 and 18 recite an answer dissimilar Q/A pair.
Even though the disclosed invention is described in the specification as improving computer technology, the claim provides no meaningful limitations such that this improvement is realized. Therefore, the claim does not amount to significantly more than the abstract idea itself.
Accordingly, the limitations of the Claims, whether considered individually or as an ordered combination, are not sufficient to add significantly more to improve technological functionality. As such, claims 1, 4-6, 8, 11, 14-16, and 18 are rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter.
Allowable Subject Matter
Claims 5, 7, 9, 15, 17, and 19 are objected to as being dependent upon a rejected base claim, but would be allowable over prior art if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 6, 8, 10, 11-12, 16, 18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over YAO (Yao B, Chen G, Zou R, Lu Y, Li J, Zhang S, Sang Y, Liu S, Hendler J, Wang D. More samples or more prompts? exploring effective in-context sampling for llm few-shot prompt engineering. arXiv preprint arXiv:2311.09782v2. 2024 Apr 2.), and in further view of HACKMAN (US 2025/0124060 A1).
REGARDING CLAIM 1, YAO discloses a computer-implemented method comprising:
selecting, by one or more processors, one or more simple annotated [question-answer] pairs (YAO figure 1 –“sample and annotate n data”; pg.1 2nd Col – pg.2 1st Col –“As a prominent prompting strategy to exploit LLMs’ task-solving capabilities, especially for un-seen tasks, ICL inserts a few data examples as well as their corresponding annotations into the prompt input. The data examples, along with their annotations, serve as demonstrations2 for the targeting task. The demonstrations are expected to facilitate LLMs’ better understanding of the task narrative, the expected outputs, and potentially the underlying rationales needed for solving the task”; pg. 1 Abstract – “Extensive experiments with three open-source LLMs (FlanT5-XL, Mistral-7B, and Mixtral-8x7B) on four NLI datasets (e-SNLI, Multi-NLI, ANLI, and Contract-NLI) and one QA dataset (CommonsenseQA) illustrate that ICS can consistently enhance LLMs’ performance.”; pg. 5 Section 3.2 – “In this work, we utilize four NLI datasets of varying difficulties and fix three as the number of demonstrations per prompt input, consistent with the number of NLI categories. This setting also applies to the CQA task in our evaluation.”) for an input data object comprising an input question and an input document from a reference dataset (YAO pg. 3 2nd Col –“Given a natural language task instruction I and a datum to predict x ∈ D, LLMs can take the In-Context Learning (ICL) input format, denoted as:
{I + (xicl1 , yicl1 ) + ... + (xiclm , yiclm ) + x} (1)
where (xiclm, yiclm ) denotes an oracle-annotated in-context demonstration. We believe in-context demonstrations can provide LLMs with two types of knowledge: 1) explicit insights to interpret the task instruction I and expected outputs through(yicl1 , ..., yiclm ) and 2) implicit guidance for how to solve the task via demonstrations (xiclm → yiclm ).”; pg. 4 2st Col Similarity – “The similarity strategy shares the same procedure as the diversity strategy of calculating the averaged similarity score for each unlabeled data. Nevertheless, the similarity strategy aims to find examples that are of the highest averaged similarity to the whole unlabeled training data space so that the sampled data will most likely be similar to the actual testing data. The underlying concept of this strategy is analogous to a family of density-weighted sampling strategies that look for the ones that appear most in the unlabeled data space or are most similar to unlabeled data (Fujii et al., 1999; Xu et al., 2003; Haffari and Sarkar, 2009). We follow the same mathematical procedure 2 above to calculate and rank the unlabeled data by the averaged similarity score. Then, differing from the diversity strategy, we retrieve n highest-ranked examples from the ranked list.”; In other words, YAO teaches selecting demonstrations that is most likely similar to the actual testing data x, that is an input question in the CQA dataset. The demonstration forms annotated question-answer pairs, that is (xicl1 , yicl1 ) + ... + (xiclm , yiclm ), where each y is the answer to each question x.);
selecting, by the one or more processors, one or more complex annotated [question-answer] pairs from the reference dataset (YAO figure 1 –“sample and annotate n data”; pg. 4 2nd Col – “This strategy adheres to established cluster-based strategies (i.e., core-set) (Sener and Savarese, 2017; Yao et al., 2023a), aiming to identify examples representative of all unlabeled data while maximizing the diversity among these selected instances. The concept of ensuring data diversity derives from the established density-weighted sampling strategies (Settles and Craven, 2008; Shen et al., 2004). They assume the instances that can provide the most helpfulness should be the ones that are representative of the input space (He et al., 2023). In other words, the diversity among selected data should be maximized. Specifically, our strategy calculates the cosine similarity for each data xi, encoded with sentence transformer (Reimers and Gurevych, 2019), with the following formula, where embed represents sentence-transformer embedding: … Subsequently, we rank the data by similarity score and retrieve n examples with the same interval, ensuring the sampling diversity. for instance, to sample 4 demonstrations from 10 ranked unlabeled data, we choose the 1st, 4th, 7th, and 10th data.”);
generating, by the one or more processors, a few-shot prompt based on the one or more simple annotated question-answer pairs and the one or more complex annotated [question-answer] pairs (YAO figure 1 –“sample and annotate n data”; pg.1 2nd Col – pg.2 1st Col –“As a prominent prompting strategy to exploit LLMs’ task-solving capabilities, especially for un-seen tasks, ICL inserts a few data examples as well as their corresponding annotations into the prompt input. The data examples, along with their annotations, serve as demonstrations2 for the targeting task. The demonstrations are expected to facilitate LLMs’ better understanding of the task narrative, the expected outputs, and potentially the underlying rationales needed for solving the task”; pg. 3 2nd Col –“Given a natural language task instruction I and a datum to predict x ∈ D, LLMs can take the In-Context Learning (ICL) input format, denoted as:
{I + (xicl1 , yicl1 ) + ... + (xiclm , yiclm ) + x} (1)
where (xiclm, yiclm ) denotes an oracle-annotated in-context demonstration. We believe in-context demonstrations can provide LLMs with two types of knowledge: 1) explicit insights to interpret the task instruction I and expected outputs through (yicl1 , ..., yiclm ) and 2) implicit guidance for how to solve the task via demonstrations (xiclm → yiclm ).”; pg. 5 1st Col – “Hybrid Similar to the aforementioned line of ensemble strategies that incorporate different strategies altogether in Section 2, our hybrid strategy expects to benefit from both above-mentioned strategies, which aims to locate examples that are either representative of the sampling space or of the highest similarity to the whole space. Subsequently, this hybrid strategy comprises two steps: first, sample n/2 examples following the diversity strategy, then sample n/2 examples following the similarity strategy from the remaining list.”); and
providing, by the one or more processors, the few-shot prompt to a large language model (LLM) to receive a predictive output for the input question (YAO figure 1 –“determine and vote LLM’s most confident prediction”; pg.1 2nd Col – pg.2 1st Col –“As a prominent prompting strategy to exploit LLMs’ task-solving capabilities, especially for un-seen tasks, ICL inserts a few data examples as well as their corresponding annotations into the prompt input. The data examples, along with their annotations, serve as demonstrations2 for the targeting task. The demonstrations are expected to facilitate LLMs’ better understanding of the task narrative, the expected outputs, and potentially the underlying rationales needed for solving the task.”);.
YAO does not explicitly teach the [square-bracketed] limitation. YAO teaches using different databases for performing few-shot in-context learning wherein the databases include a QA dataset (Abst.) Thus, YAO suggests the request (x) is corresponding to a question and the output (y) corresponds to an answer. One of ordinary skill in the art would recognize that the (xicl1 , yicl1 ) + ... + (xiclm , yiclm ) of the in-context input prompt denote the question-answer pairs. Although YAO suggests the [question-answer] pairs, the Examiner provides HACKMAN for the clarity of the rejection.
HACKMAN explicitly teaches the [question-answer] limitation. HACKMAN discloses a method/system for in-context learning, wherein a prompt is generated to include examples comprising:
selecting, by one or more processors, one or more simple annotated [question-answer] pairs (HACKMAN Par 59 – “In some implementations, an embedding of user question 202 and embeddings of the question-and-answer pairs 308 (or embeddings of the question or answer of the question-and-answer pairs 308) may be used to select question-and-answer pairs. User question 202 and the question-and-answer pairs 308 may be represented as a vector in a vector space. The vector representations of the question-and-answer pairs 308 may then be processed to select one or more pairs that are closest (e.g., using a cosine similarity measure) to the vector representation of user question 202. In some implementations, pairs of questions and answers may be selected such that they relate to the same document as the question from the user.”) for an input data object comprising an input question and an input document from a reference dataset (HACKMAN Par 59 – “In some implementations, pairs of questions and answers may be selected such that they relate to the same document as the question from the user.”; Par 80 – “At step 640, a prompt for a language model may be created. The prompt may include a representation of the user question and a representation of the one or more question-and-answer pairs. Representation of the user question and the question-and-answer pair may include the text or modified text of the user question and the question-and-answer pairs. In some implementations, the representation of the question-and-answer pairs may include a representation as a dialogue history. The representations may include the pairs specified as previous interactions with a language model or as the context for conversations with a language model.”);
selecting, by the one or more processors, one or more complex annotated [question-answer] pairs from the reference dataset (HACKMAN Par 61 – “In some implementations, vector representations of the question-and-answer pairs 308 may then be processed to select one or more pairs that provide a sampling of different varieties of pairs. In some cases, selecting pairs that are closest may return variations (e.g., small variations in the wording of a question-and-answer pair). In some implementations, coverage of a greater variety of pairs may be selected by selecting pairs that are at a greater distance in the vector space.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method/system of YAO to include a question-answer pair, as taught by HACKMAN.
One of ordinary skill would have been motivated to include a question-answer pair, in order to provide a predicted answer for a user question by an LLM, so that the LLM can be utilized in a variety of scenarios according to a user’s needs.
REGARDING CLAIM 2, YAO in view of HACKMAN discloses the computer-implemented method of claim 1, wherein the reference dataset comprises a plurality of annotated question-answer pairs that comprises a plurality of reference questions, a plurality of reference answers (YAO Figure 1 – “Annotated Demonstration Candidates”; pg. 3 2nd Col –“Given a natural language task instruction I and a datum to predict x ∈ D, LLMs can take the In-Context Learning (ICL) input format, denoted as:
{I + (xicl1 , yicl1 ) + ... + (xiclm , yiclm ) + x} (1)
where (xiclm, yiclm ) denotes an oracle-annotated in-context demonstration. We believe in-context demonstrations can provide LLMs with two types of knowledge: 1) explicit insights to interpret the task instruction I and expected outputs through (yicl1 , ..., yiclm ) and 2) implicit guidance for how to solve the task via demonstrations (xiclm → yiclm ).”), and [a plurality of reference document contexts], and selecting the one or more simple annotated question-answer pairs (YAO figure 1 –“sample and annotate n data”; pg. 4 2st Col Similarity – “The similarity strategy shares the same procedure as the diversity strategy of calculating the averaged similarity score for each unlabeled data. Nevertheless, the similarity strategy aims to find examples that are of the highest averaged similarity to the whole unlabeled training data space so that the sampled data will most likely be similar to the actual testing data. The underlying concept of this strategy is analogous to a family of density-weighted sampling strategies that look for the ones that appear most in the unlabeled data space or are most similar to unlabeled data (Fujii et al., 1999; Xu et al., 2003; Haffari and Sarkar, 2009). We follow the same mathematical procedure 2 above to calculate and rank the unlabeled data by the averaged similarity score. Then, differing from the diversity strategy, we retrieve n highest-ranked examples from the ranked list.”) comprises:
generating, using a pretrained encoder-only language model, a plurality of reference embeddings for the plurality of annotated question-answer pairs based on the plurality of reference questions (YAO pg. 4 2nd col – “Specifically, our strategy calculates the cosine similarity for each data xi, encoded with sentence transformer (Reimers and Gurevych, 2019), with the following formula, where embed represents sentence-transformer embedding: s(x,D) = cos (embed (x), 1 / |D| ∑ j=1 … |D| embed (xj) ) (2)”; In other words, the questions of the QA pairs in the dataset D are encoded with the sentence transformer to obtain the embeddings of xj);
generating, using the pretrained encoder-only language model, an input embedding based on the input question (YAO pg. 4 2nd col – “Specifically, our strategy calculates the cosine similarity for each data xi, encoded with sentence transformer (Reimers and Gurevych, 2019), with the following formula, where embed represents sentence-transformer embedding: s(x,D) = cos (embed (x), 1 / |D| ∑ j=1 … |D| embed (xj) ) (2)”; In other words, the input question x is encoded with the sentence transformer to obtain the embedding of x.); and
selecting, using a nearest neighbor-based selection mechanism, the one or more simple annotated question-answer pairs based on an embedding similarity between the input embedding and the plurality of reference questions (YAO pg. 4 2st Col Similarity – “The similarity strategy shares the same procedure as the diversity strategy of calculating the averaged similarity score for each unlabeled data. Nevertheless, the similarity strategy aims to find examples that are of the highest averaged similarity to the whole unlabeled training data space so that the sampled data will most likely be similar to the actual testing data. The underlying concept of this strategy is analogous to a family of density-weighted sampling strategies that look for the ones that appear most in the unlabeled data space or are most similar to unlabeled data (Fujii et al., 1999; Xu et al., 2003; Haffari and Sarkar, 2009). We follow the same mathematical procedure 2 above to calculate and rank the unlabeled data by the averaged similarity score. Then, differing from the diversity strategy, we retrieve n highest-ranked examples from the ranked list.”).
YAO does not explicitly teach the [square-bracketed] limitation.
HACKMAN explicitly teaches the [question-answer] limitation. HACKMAN discloses a method/system for in-context learning, wherein a prompt is generated to include examples comprising: wherein the reference dataset comprises a plurality of annotated question-answer pairs that comprises a plurality of reference questions, a plurality of reference answers (HACKMAN Par 59 – “In some implementations, an embedding of user question 202 and embeddings of the question-and-answer pairs 308 (or embeddings of the question or answer of the question-and-answer pairs 308) may be used to select question-and-answer pairs. User question 202 and the question-and-answer pairs 308 may be represented as a vector in a vector space.”), and [a plurality of reference document contexts] (HACKMAN Par 59 – “In some implementations, pairs of questions and answers may be selected such that they relate to the same document as the question from the user.”; Par 80 – “At step 640, a prompt for a language model may be created. The prompt may include a representation of the user question and a representation of the one or more question-and-answer pairs. Representation of the user question and the question-and-answer pair may include the text or modified text of the user question and the question-and-answer pairs. In some implementations, the representation of the question-and-answer pairs may include a representation as a dialogue history. The representations may include the pairs specified as previous interactions with a language model or as the context for conversations with a language model.”; Par 81 – “In some applications, the user question may relate to one or more reference documents and the prompt may be created to include a representation of the reference document.”);, and selecting the one or more simple annotated question-answer pairs (HACKMAN Par 59 – “User question 202 and the question-and-answer pairs 308 may be represented as a vector in a vector space. The vector representations of the question-and-answer pairs 308 may then be processed to select one or more pairs that are closest (e.g., using a cosine similarity measure) to the vector representation of user question 202. In some implementations, pairs of questions and answers may be selected such that they relate to the same document as the question from the user.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method/system of YAO to include a context of a question-answer pair, as taught by HACKMAN.
One of ordinary skill would have been motivated to include a context of a question-answer pair, in order to select more useful example pairs for prompting an LLM so that a prediction generated by the LLM is more accurate.
REGARDING CLAIM 6, YAO in view of HACKMAN discloses the computer-implemented method of claim 1, wherein the one or more complex annotated question-answer pairs comprise a [context] dissimilar question-answer pair associated with a low question-[context] lexical overlap classification (YAO figure 1 –“sample and annotate n data”; pg. 4 2nd Col – “This strategy adheres to established cluster-based strategies (i.e., core-set) (Sener and Savarese, 2017; Yao et al., 2023a), aiming to identify examples representative of all unlabeled data while maximizing the diversity among these selected instances. The concept of ensuring data diversity derives from the established density-weighted sampling strategies (Settles and Craven, 2008; Shen et al., 2004). They assume the instances that can provide the most helpfulness should be the ones that are representative of the input space (He et al., 2023). In other words, the diversity among selected data should be maximized. Specifically, our strategy calculates the cosine similarity for each data xi, encoded with sentence transformer (Reimers and Gurevych, 2019), with the following formula, where embed represents sentence-transformer embedding: s(x,D) = cos (embed (x), 1 / |D| ∑ j=1 … |D| embed (xj) ) (2) Subsequently, we rank the data by similarity score and retrieve n examples with the same interval, ensuring the sampling diversity. for instance, to sample 4 demonstrations from 10 ranked unlabeled data, we choose the 1st, 4th, 7th, and 10th data.”).
YAO does not explicitly teach the [square-bracketed] limitation. In other words, YAO teaches including dissimilar question-answer pairs (e.g., 7th and/or 10th demonstration data out of 10 ranked demonstration data.) in a prompt for an LLM, wherein the dissimilar question-answer pairs are obtained by calculating similarity scores between an input question (x) and other questions (xj) in the dataset (D). However, YAO does not explicitly teach a [context] dissimilarity.
HACKMAN teaches the [square-bracketed] limitations. HACKMAN discloses a method/system for question answering with embedding vectors comprising the selected question-answer pairs (HACKMAN Par 56 – “In another example, the question-and-answer selector 310 may select from the question-and-answer pairs 308 based on a similarity measure.”) comprise a [context] dissimilar question-answer pair (HACKMAN Par 57 – “In some implementations, a similarity measure may be determined using semantic representations. Any appropriate semantic representation may be used, such as an embedding that represents text (e.g., a question or a question-and-answer sample) in a vector space in a manner that preserves information about the meaning of the text. For example, where two different text items represent similar information, the corresponding vector space embeddings may be close to each other in the vector space. For another example, where two different text items represent very different information, the corresponding vector space embeddings may be far from each other in the vector space.”; Par 61 – “In some implementations, vector representations of the question-and-answer pairs 308 may then be processed to select one or more pairs that provide a sampling of different varieties of pairs. In some cases, selecting pairs that are closest may return variations (e.g., small variations in the wording of a question-and-answer pair). In some implementations, coverage of a greater variety of pairs may be selected by selecting pairs that are at a greater distance in the vector space.”) associated with a low question-[context] lexical overlap classification (HACKMAN Par 59 – “In some implementations, pairs of questions and answers may be selected such that they relate to the same document as the question from the user.”; Par 80 – “At step 640, a prompt for a language model may be created. The prompt may include a representation of the user question and a representation of the one or more question-and-answer pairs. Representation of the user question and the question-and-answer pair may include the text or modified text of the user question and the question-and-answer pairs. In some implementations, the representation of the question-and-answer pairs may include a representation as a dialogue history. The representations may include the pairs specified as previous interactions with a language model or as the context for conversations with a language model.”; Par 81 – “In some applications, the user question may relate to one or more reference documents and the prompt may be created to include a representation of the reference document.”; Par 61 – “In some implementations, coverage of a greater variety of pairs may be selected by selecting pairs that are at a greater distance in the vector space.”; In other words, HACKMAN teaches the representation question-answer pairs and the input question representation including the contextual information. Thus, HACKMAN teaches selecting a context dissimilar question-answer pairs based on contextual similarity (e.g., pairs that are at a greater distance).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method/system of YAO to include a context dissimilar question-answer pair, as taught by HACKMAN.
One of ordinary skill would have been motivated to include a context dissimilar question-answer pair, in order to select more useful example pairs for prompting an LLM so that a prediction generated by the LLM is more accurate.
REGARDING CLAIM 8, YAO in view of HACKMAN discloses the computer-implemented method of claim 1, wherein the one or more complex annotated question-answer pairs comprise an [answer] dissimilar question-answer pair associated with a low question-[answer] overlap classification (YAO figure 1 –“sample and annotate n data”; pg. 4 2nd Col – “This strategy adheres to established cluster-based strategies (i.e., core-set) (Sener and Savarese, 2017; Yao et al., 2023a), aiming to identify examples representative of all unlabeled data while maximizing the diversity among these selected instances. The concept of ensuring data diversity derives from the established density-weighted sampling strategies (Settles and Craven, 2008; Shen et al., 2004). They assume the instances that can provide the most helpfulness should be the ones that are representative of the input space (He et al., 2023). In other words, the diversity among selected data should be maximized. Specifically, our strategy calculates the cosine similarity for each data xi, encoded with sentence transformer (Reimers and Gurevych, 2019), with the following formula, where embed represents sentence-transformer embedding: s(x,D) = cos (embed (x), 1 / |D| ∑ j=1 … |D| embed (xj) ) (2) Subsequently, we rank the data by similarity score and retrieve n examples with the same interval, ensuring the sampling diversity. for instance, to sample 4 demonstrations from 10 ranked unlabeled data, we choose the 1st, 4th, 7th, and 10th data.”).
YAO does not explicitly teach the [square-bracketed] limitation. In other words, YAO teaches including dissimilar question-answer pairs (e.g., 7th and/or 10th demonstration data out of 10 ranked demonstration data.) in a prompt for an LLM, wherein the dissimilar question-answer pairs are obtained by calculating similarity scores between an input question (x) and other questions (xj) in the dataset (D). However, YAO does not explicitly teach a [answer] dissimilarity.
HACKMAN teaches the [square-bracketed] limitations. HACKMAN discloses a method/system for question answering with embedding vectors comprising the selected question-answer pairs (HACKMAN Par 56 – “In another example, the question-and-answer selector 310 may select from the question-and-answer pairs 308 based on a similarity measure.”) comprise a [answer] dissimilar question-answer pair (HACKMAN Par 59 – “In some implementations, an embedding of user question 202 and embeddings of the question-and-answer pairs 308 (or embeddings of the question or answer of the question-and-answer pairs 308) may be used to select question-and-answer pairs.”; Par 61 – “In some implementations, vector representations of the question-and-answer pairs 308 may then be processed to select one or more pairs that provide a sampling of different varieties of pairs. In some cases, selecting pairs that are closest may return variations (e.g., small variations in the wording of a question-and-answer pair). In some implementations, coverage of a greater variety of pairs may be selected by selecting pairs that are at a greater distance in the vector space.”) associated with a low question-[answer] overlap classification (HACKMAN Par 60 – “In some implementations, the distance may be between the embedding of a user question and any of an embedding of the question-and-answer pair sample, an embedding of a question of the question-and-answer sample, or an embedding of the answer of the question-and-answer sample.”; Par 61 – “In some implementations, coverage of a greater variety of pairs may be selected by selecting pairs that are at a greater distance in the vector space.”; In other words, HACKMAN teaches the representation question-answer pairs and the input question representation including the contextual information. Thus, HACKMAN teaches selecting a context dissimilar question-answer pairs based on contextual similarity (e.g., pairs that are at a greater distance).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method/system of YAO to include an answer dissimilar question-answer pair, as taught by HACKMAN.
One of ordinary skill would have been motivated to include an answer dissimilar question-answer pair, in order to select more useful example pairs for prompting an LLM so that a prediction generated by the LLM is more accurate.
REGARDING CLAIM 10, YAO in view of HACKMAN discloses the computer-implemented method of claim 1, wherein generating the few-shot prompt based on the one or more simple annotated question-answer pairs and the one or more complex annotated question-answer pairs comprises:
generating the few-shot prompt from a prompt template associated with the input question (YAO figure 1 –“sample and annotate n data”; pg.1 2nd Col – pg.2 1st Col –“As a prominent prompting strategy to exploit LLMs’ task-solving capabilities, especially for un-seen tasks, ICL inserts a few data examples as well as their corresponding annotations into the prompt input. The data examples, along with their annotations, serve as demonstrations2 for the targeting task. The demonstrations are expected to facilitate LLMs’ better understanding of the task narrative, the expected outputs, and potentially the underlying rationales needed for solving the task”; pg. 3 2nd Col –“Given a natural language task instruction I and a datum to predict x ∈ D, LLMs can take the In-Context Learning (ICL) input format, denoted as:
{I + (xicl1 , yicl1 ) + ... + (xiclm , yiclm ) + x} (1)
where (xiclm, yiclm ) denotes an oracle-annotated in-context demonstration. We believe in-context demonstrations can provide LLMs with two types of knowledge: 1) explicit insights to interpret the task instruction I and expected outputs through (yicl1 , ..., yiclm ) and 2) implicit guidance for how to solve the task via demonstrations (xiclm → yiclm ).”);
modifying the few-shot prompt with the input document or a reference to the input document (YAO pg. 15 1ST Col – “Thus, we only evaluate the performance of those data. For CommonsenseQA, we design the prompt to be: Answer this commonsense question from the given choices.”; HACKMAN also teaches the limitation: Par 59 – “In some implementations, pairs of questions and answers may be selected such that they relate to the same document as the question from the user.”; Par 80 – “At step 640, a prompt for a language model may be created. The prompt may include a representation of the user question and a representation of the one or more question-and-answer pairs. Representation of the user question and the question-and-answer pair may include the text or modified text of the user question and the question-and-answer pairs. In some implementations, the representation of the question-and-answer pairs may include a representation as a dialogue history. The representations may include the pairs specified as previous interactions with a language model or as the context for conversations with a language model.”);
generating a plurality of in-context prompt examples by aggregating the one or more simple annotated question-answer pairs and the one or more complex annotated question-answer pairs (YAO pg. 5 1st Col – “Hybrid Similar to the aforementioned line of ensemble strategies that incorporate different strategies altogether in Section 2, our hybrid strategy expects to benefit from both above-mentioned strategies, which aims to locate examples that are either representative of the sampling space or of the highest similarity to the whole space. Subsequently, this hybrid strategy comprises two steps: first, sample n/2 examples following the diversity strategy, then sample n/2 examples following the similarity strategy from the remaining list.”); and
modifying the few-shot prompt with the plurality of in-context prompt examples (YAO pg. 3 2nd Col –“Given a natural language task instruction I and a datum to predict x ∈ D, LLMs can take the In-Context Learning (ICL) input format, denoted as:
{I + (xicl1 , yicl1 ) + ... + (xiclm , yiclm ) + x} (1)
where (xiclm, yiclm ) denotes an oracle-annotated in-context demonstration.).
REGARDING CLAIM 11, YAO in view of HACKMAN discloses a computing system comprising memory and one or more processors communicatively coupled to the memory, the one or more processors (YAO pg. 9 – “computing resources”; pg. 15 A Experiment Setup) configured to: performing the steps of claim 1; thus, it is rejected under the same rationale.
Claim 12 is similar to claim 2; thus, it is rejected under the same rationale.
Claim 16 is similar to claim 6; thus, it is rejected under the same rationale.
Claim 18 is similar to claim 8; thus, it is rejected under the same rationale.
REGARDING CLAIM 20, YAO in view of HACKMAN discloses one or more non-transitory computer-readable storage media including instructions that, when executed by one or more processors, cause the one or more processors to:perform the steps of claim 1; thus, it is rejected under the same rationale.
Claims 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over YAO in view of HACKMAN, and in further view of GAO (Gao T, Yao X, Chen D. Simcse: Simple contrastive learning of sentence embeddings. InProceedings of the 2021 conference on empirical methods in natural language processing 2021 Nov (pp. 6894-6910).).
REGARDING CLAIM 3, YAO in view of HACKMAN discloses the computer-implemented method of claim 2.
YAO in view of HACKMAN does not explicitly teach unsupervised training.
GAO discloses a method/system for contrastive learning for natural language processing, wherein the pretrained encoder-only language model is previously trained using an unsupervised training technique and the reference dataset (GAO pg. 6894 2nd Col – “Our unsupervised SimCSE simply predicts the input sentence itself with only dropout (Srivastava
et al., 2014) used as noise (Figure 1(a)). In other words, we pass the same sentence to the pre-trained encoder twice: by applying the standard dropout twice, we can obtain two different embeddings as “positive pairs”.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method/system of YAO in view of HACKMAN to include unsupervised training for obtaining embeddings, as taught by GAO.
One of ordinary skill would have been motivated to include unsupervised training for obtaining embeddings, in order to improve uniformity while avoiding degenerated alignment via dropout noise, thus improving the expressiveness of the representations.
Claim 13 is similar to claim 3; thus, it is rejected under the same rationale.
Claims 4 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over YAO in view of HACKMAN, and in further view of NGUYEN (Nguyen T, Wong E. In-context example selection with influences. arXiv preprint arXiv:2302.11042. 2023 Feb 21.).
REGARDING CLAIM 4, YAO in view of HACKMAN discloses the computer-implemented method of claim 1.
YAO in view of HACKMAN fails to teach a failure question scenario.
NGUYEN discloses a method/system for selecting positive and negative examples for prompting an LLM (NGUYEN pg. 3 Algorithm 1), wherein the one or more complex annotated question-answer pairs comprise a mistake prone annotated question-answer pair associated with a failure question scenario of the LLM (NGUYEN pg. 7 – “Negative selection. Similarly, our influence-based example selection methods can consistently identify low-performing examples. As shown in Table 2, results indicate that Influence (-) achieves the highest rank (2.90) among all other methods. Notably, in this context, One-shot slightly outperforms IC Datamodels (-), suggesting that selecting examples based on their individual validation performance can be effective. Compared to positive selection, the ability to pinpoint negative examples is equally meaningful: we can avoid these examples to achieve better ICL performance, or further study them to identify the factors that make them ineffective.”; pg. 8 Table 3: Negative examples identified by in-context influences on LLaMA-7B; Section 4.3 – “Additionally, we identify WIC example #3890 as a falsely annotated instance. Related to input-label mapping, Min et al. [Min+22b] has shown that label correctness is not important for good ICL performance.”; pg. 27 Table 11: Highly negative influence examples from the bottom 20 percentile bin for LLaMA-7B.;).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method/system of YAO to include a failure question scenario, as taught by NGUYEN.
One of ordinary skill would have been motivated to include a failure question scenario, in order to pinpoint negative examples to achieve better in-context learning performance.
Claim 14 is similar to claim 4; thus, it is rejected under the same rationale.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONATHAN C KIM whose telephone number is (571)272-3327. The examiner can normally be reached Monday to Friday 8:00 AM thru 4:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew C Flanders can be reached at 571-272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JONATHAN C KIM/Primary Examiner, Art Unit 2655
Prosecution Insights