Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-3, 8, 10-11, 15-16, and 19 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Lu et al. (NPL: Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering, published Oct. 2022, hereinafter “Lu”).
Regarding claim 1, Lu teaches a computer-implemented method for training a machine-learned sequence processing model, the method comprising:
obtaining, by a computing system comprising one or more processors, a plurality of training examples for training the machine-learned sequence processing model, wherein each training example of the plurality of training examples comprises an example query, an example response to the query, and an example trace of intermediate states from the example query to the example response (Lu, Fig. 1 description – “We construct the SCIENCEQA dataset where a data example consists of multimodal question answering information and the grounded lecture and explanation. We study if QA models can generate a reasonable explanation to reveal the chain-of-thought reasoning.”, Section 1 Paragraph 4 – “In this paper, we train language models to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop reasoning process to answer SCIENCEQA questions.”, Table 1, and Section B.1 Paragraph 7 – “Compute resources. We use two GeForce RTX 3090 GPU’s for fine-tuning baselines and UnifiedQA on the dataset.” – teaches obtaining, by a computing system comprising one or more processors (teaches compute resources), a plurality of training examples for training the machine-learned sequence processing model (trains language models to generate lectures and explanations as chain of thought), wherein each training example of the plurality of training examples comprises an example query, an example response to the query, and an example trace of intermediate states from the example query to the example response (data example consists of multimodal question, which is a query, answering, which is a response to the query, and grounded lecture and explanation, which is an example of the intermediate states or chain of thought from the query to the response)); and
for each respective training example of the plurality of training examples:
obtaining, by the computing system, a respective query associated with the respective training example (Lu, Section 4.2 Paragraph 1 – “The annotated lectures and explanations in SCIENCEQA serve as demonstrations of the chain of thought that mimics the multi-step reasoning steps of human beings. In this paper, we study if large language models can generate reasonable explanations as the chain of thought to reveal the thought process when answering SCIENCEQA questions.” and in Section 4.2 Paragraph 3 – “Different from standard prompting, we build GPT-3 via chain-of-thought (CoT) prompting, as shown in Figure 5. To be specific, for each test problem t, we map the prompt instruction I : {Ii}n,It into a textual format where {Ii}n refers to the instruction set of n-shot in-context examples from the training set, while It denotes the test instruction. Instead of the way where the explanation comes before the answer [54], we feed the instruction I into the encoder decoder model GPT-3 to generate the answer a followed by the lecture lect and explanation exp: M:{Ii}n,It → a,lect,exp.” – teaches obtaining, by the computing system, a respective query associated with the respective training example (applies language models, such as GPT-3, to SCIENCEQA questions, thus obtaining a respective query, or question, associated with the respective training example, such as the data examples of the SCIENCEQA dataset));
inputting, by the computing system, the respective query to the machine-learned sequence processing model (Lu, Section 4.2 Paragraph 1 – “The annotated lectures and explanations in SCIENCEQA serve as demonstrations of the chain of thought that mimics the multi-step reasoning steps of human beings. In this paper, we study if large language models can generate reasonable explanations as the chain of thought to reveal the thought process when answering SCIENCEQA questions.”, Section 4.2 Paragraph 3 – “Different from standard prompting, we build GPT-3 via chain-of-thought (CoT) prompting, as shown in Figure 5. To be specific, for each test problem t, we map the prompt instruction I : {Ii}n,It into a textual format where {Ii}n refers to the instruction set of n-shot in-context examples from the training set, while It denotes the test instruction. Instead of the way where the explanation comes before the answer [54], we feed the instruction I into the encoder decoder model GPT-3 to generate the answer a followed by the lecture lect and explanation exp: M:{Ii}n,It → a,lect,exp.”, and in Figs. 5 & 6 – teaches inputting, by the computing system, the respective query to the machine-learned sequence processing model (Fig. 5 shows inputting the respective query, or question Iiques, associated with the respective training example, example i, into the machine-learned sequence processing model, such as GPT-3));
obtaining, by the computing system and from the machine-learned sequence processing model:
a response to the respective query (Lu, Section 4.2 Paragraph 3 – “Different from standard prompting, we build GPT-3 via chain-of-thought (CoT) prompting, as shown in Figure 5. To be specific, for each test problem t, we map the prompt instruction I : {Ii}n,It into a textual format where {Ii}n refers to the instruction set of n-shot in-context examples from the training set, while It denotes the test instruction. Instead of the way where the explanation comes before the answer [54], we feed the instruction I into the encoder decoder model GPT-3 to generate the answer a followed by the lecture lect and explanation exp: M:{Ii}n,It → a,lect,exp.”, and in Figs. 5 & 6 – teaches obtaining, by the computing system and from the machine-learned sequence processing model, a response to the respective query (feeds the instruction I for example i into the encoder decoder model GPT-3 to generate the answer a. Fig. 5 shows receiving an answer, “Answer: The answer is answer : Iia”, in response to the respective query, or question Iiques)); and
a trace of intermediate states from the respective query to the response (Lu, Section 4.2 Paragraph 2 – “We make a format modification to develop UnifiedQA with the chain of thought (CoT), i.e., UnifiedQA is fine-tuned to generate a long sequence of text which consists of the answer followed by the lecture and explanation.”, and in Section 4.2 Paragraph 3 – “Different from standard prompting, we build GPT-3 via chain-of-thought (CoT) prompting, as shown in Figure 5. To be specific, for each test problem t, we map the prompt instruction I : {Ii}n,It into a textual format where {Ii}n refers to the instruction set of n-shot in-context examples from the training set, while It denotes the test instruction. Instead of the way where the explanation comes before the answer [54], we feed the instruction I into the encoder decoder model GPT-3 to generate the answer a followed by the lecture lect and explanation exp: M:{Ii}n,It → a,lect,exp.”, and in Figs. 5 & 6 – teaches obtaining, by the computing system and from the machine-learned sequence processing model, a trace of intermediate states from the respective query to the response (modifies UnifiedQA to generate answer followed by lecture and explanation, and as in Fig. 5 feeds the instruction I into the encoder decoder model GPT-3 to generate the answer a followed by the lecture lect and explanation exp, and in Fig. 5, “BECAUSE: lecture : Iilect explanation : Iiexp”));
evaluating, by the computing system, the response using a ground truth response associated with the respective training example (Lu, Section 5.1 Paragraph 1 – “The heuristics and VQA baselines treat our SCIENCEQA task as a multi-class classification problem with multiple options and are evaluated with the accuracy metrics. UnifiedQA and GPT-3 treat SCIENCEQA as a text generation problem. So the most similar option is selected as the final prediction to evaluate the question answering accuracy.”, Section 5.2 Paragraph 3 – “By further teaching the model to generate the answer along with lecture and explanation, the developed language model with chain-of-thought (UnifiedQABASE (CoT)) brings additional improvements of +3.21% (QCM→AE) and +3.99% (QCM→ALE). These results show that generating the chain of thought along with the answer benefits the reasoning ability of language models.”, Section 5.2 Paragraph 4 – “However, if we prompt GPT-3 with chain-of-thought prompting (QCM→ALE), we obtain the state-of-the-art result so far (75.17%).”, and in Table 4 – teaches evaluating, by the computing system, the response using a ground truth response associated with the respective training example (evaluates question answering accuracy to evaluate the response, thus using a ground truth response associated with the respective training example to determine accuracy));
evaluating, by the computing system, the trace using a ground truth trace associated with the respective training example, wherein the ground truth trace was obtained from annotations that were input by a human user after being presented with the query and the ground truth response, wherein the annotations comprise a description of step-by-step reasoning between the respective query and the ground truth response (Lu, Section 4.2 Paragraph 1 – “The annotated lectures and explanations in SCIENCEQA serve as demonstrations of the chain of thought that mimics the multi-step reasoning steps of human beings.”, Section 5.1 Paragraph 1 – “The generated lectures and explanations are evaluated by automatic metrics [44, 28, 49] and human scores by annotators.”, Section 5.3 Paragraph 1 – “We can see that GPT-3 (CoT) predicts the correct answer and generates a reasonable lecture and explanation to mimic the human thought process. We further report automatic metrics (BLEU-1/4[44], ROUGE-L[44], and (sentence) Similarity [49] to evaluate the generated lectures and explanations, as shown in Table 4... However, it’s commonly agreed that automatic evaluation of generated texts only provides a partial view and has to be complemented by a human study. By asking annotators to rate the relevance, correctness, and completeness of generated explanations, we find that the explanations generated by GPT-3 (CoT) conform best to human judgment.”, and in Table 4 – teaches evaluating, by the computing system, the trace using a ground truth trace associated with the respective training example (generated lectures and explanations are evaluated by automatic metrics, such as sentence Similarity, and scored by annotators), wherein the ground truth trace was obtained from annotations that were input by a human user after being presented with the query and ground truth response (annotated lectures and explanations serve as demonstrations of chain of thought that mimics reasoning of human beings, human annotators rate relevance, correctness, and completeness of trace), wherein the annotations comprise a description of step-by-step reasoning between the respective query and the ground truth response (annotated lectures and explanations are demonstrations of chain of thought that mimics human reasoning)); and
updating, by the computing system, one or more parameters of the machine-learned sequence processing model based on the evaluation of the response and based on the evaluation of the trace (Lu, Section 5.1 Paragraph 2 – “The VQA baselines are trained for a maximum number of 50 epochs with a learning rate of 5e−5. We fine-tune the UnifiedQA for 50k iterations and evaluate every 1k iteration. The training process is stopped following the early stopping strategy with a patience period of three evaluations. For GPT-3, we use the text-davinci-002 engine, which is the most capable model version suggested in the official documentation.” – teaches updating, by the computing system, one or more parameters of the machine-learned sequence processing model based on the evaluation of the response and based on the evaluation of the trace (UnifiedQA is fine-tuned for 50k iterations and evaluated, using evaluations as in Section 5.1 Paragraph 1, at every 1k iteration, thus updating one or more parameters of the machine-learned sequence processing model based on evaluations of both the response and the trace).
Claims 15 and 19 incorporate substantively all the limitations of claim 1 in a system and non-transitory computer-readable media, and are rejected on similar grounds as above. Lu teaches the processors and non-transitory computer-readable media in Section B.1 Paragraph 7 – “Compute resources. We use two GeForce RTX 3090 GPU’s for fine-tuning baselines and UnifiedQA on the dataset.”
Claim 19 incorporates all the limitations of claim 1, and recites further limitations regarding:
instructions that are executable by the one or more processors to cause the computing system to perform operations, the operations comprising:
inputting a runtime query to the machine-learned sequence processing model (Lu, Section B.1 Paragraph 9 – “GPT-3: Following default settings, we choose temperature, frequency penalty and presence penalty as 0.0, and top probability as 1.0. All experiments for GPT-3 are run via the online API. Experiments in Figure 7 are repeated four times with in-context examples listed in Table 9. Experiments in Table 3, 5, 6, and 7 are conducted using examples with the trial ID of 1.” and in Fig. 15 – teaches inputting a runtime (all experiments for GPT-3 are run online) query (Fig. 15 shows the questions input at runtime) to the machine-learned sequence processing model (GPT-3)); and
receiving a runtime response from the machine-learned sequence processing model, wherein the runtime response comprises a runtime trace of intermediate states from the runtime query to the runtime response (Lu, Section B.1 Paragraph 9 – “GPT-3: Following default settings, we choose temperature, frequency penalty and presence penalty as 0.0, and top probability as 1.0. All experiments for GPT-3 are run via the online API. Experiments in Figure 7 are repeated four times with in-context examples listed in Table 9. Experiments in Table 3, 5, 6, and 7 are conducted using examples with the trial ID of 1.” and in Fig. 15 – teaches receiving a runtime response from the machine-learned sequence processing model, wherein the runtime response comprises a runtime trace of intermediate states from the runtime query to the runtime response (Fig. 15 shows the runtime answer to the runtime question comprising runtime lectures and/or explanations (as in Fig. 15, the lectures/explanations appears after “BECAUSE:”) from the runtime question to the runtime answer)).
Regarding claim 2, Lu teaches the method of claim 1, wherein
the plurality of training examples comprises examples from multiple different task categories (Lu, Fig. 4 and Section 3.1 Paragraph 4 – “Each SCIENCEQA question belongs to one of the three subjects: natural science, language science, and social science. With each subject, questions are categorized first by the topic (Biology, Physics, Chemistry, etc.), then by the category (Plants, Cells, Animals, etc.), and finally by the specific skill (Classify fruits and vegetables as plant parts, Identify countries of Africa, etc.). SCIENCEQA has a total of 26 topics, 127 categories, and 379 skills. The tree map in Figure 4 visualizes the different subjects, topics, and categories and shows that SCIENCEQA questions are very diverse, spanning a wide range of domains.” – teaches wherein the plurality of training examples comprises examples from multiple different task categories (Fig. 4 shows wherein each training example belongs to different subjects, categorized by topic, category, and skill)).
Claim 16 is similar to claim 2, hence similarly rejected.
Regarding claim 3, Lu teaches the method of claim 2, wherein
the task categories comprise at least one or more of: question generation; explanation generation; or question and answer generation (Lu, Section 3 Paragraph 1 – “The explanation reveals a specific reason for the answer. To effectively answer the questions, a model often needs to be able to understand the multimodal content in the input and extract external knowledge, similar to how humans do. More importantly, the goal of SCIENCEQA is to aid development of a reliable model that is capable of generating a coherent chain of thought when arriving at the correct answer to reveal the multi-step reasoning process.” – teaches wherein the task categories comprise at least one or more of question generation, explanation generation, or question and answer generation (model needs to understand multimodal content to effectively answer questions and generate a coherent chain of thought, thus teaching that the tasks comprise explanation generation and/or question and answer generation)).
Regarding claim 8, Lu teaches the method of claim 1, comprising:
training, by the computing system, the machine-learned sequence processing model using other training examples without ground truth traces (Lu, Table 1 and Section 3.1 Paragraph 1 – “We randomly split the dataset into training, validation, and test splits with a ratio of 60:20:20… 83.9% of the questions are annotated with a lecture, while 91.3% of the questions feature an explanation. The cross-combination of these information sources diversifies the problem scenario: sometimes the model is given a lot of information from multiple sources, while at other times, the only source of information is the question itself.” – teaches training, by the computing system, the machine-learned sequence processing model using other training examples without ground truth traces (dataset used to train model contains questions without any lecture and/or explanation, thus teaching using other training examples without ground truth traces)).
Regarding claim 10, Lu teaches the method of claim 1, wherein
the respective query comprises an exemplar query, an exemplar trace, and an exemplar response (Lu, Section 5.4 Paragraph 2 – “We study the effect of prompt types and visualize the comparison in Figure 7 (a). It shows that prompting the GPT-3 model with both lectures and explanations (QCM→ALE) results in the highest accuracy on average and the smallest variance.” and in Fig. 7 – teaches wherein the respective query comprises an exemplar query, an exemplar trace, and an exemplar response (Fig. 7 shows accuracy v.s. different prompts with 4-shot examples, and prompting GPT-3 with both lectures and explanations QCM-ALE results in highest accuracy, thus the respective query comprises an exemplar query, an exemplar trace, and an exemplar response as used in the 4-shot prompting)).
Regarding claim 11, Lu teaches the method of claim 1, wherein
the respective query does not comprise an exemplar trace (Lu, Section 5.4 Paragraph 2 – “We study the effect of prompt types and visualize the comparison in Figure 7 (a). It shows that prompting the GPT-3 model with both lectures and explanations (QCM→ALE) results in the highest accuracy on average and the smallest variance.” and in Fig. 7 – teaches wherein the respective query does not comprise an exemplar trace (Fig. 7 shows accuracy v.s. different prompts with 4-shot examples, one of the examples QCM-A includes an exemplar query and exemplar response without an exemplar lecture and/or explanation, or trace, thus the respective query does not comprise an exemplar trace as used in the 4-shot prompting)).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 4-6, 17, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lu in view of Zhou et al. (NPL from IDS: Large Language Models are Human-Level Prompt Engineers, published Nov 2022, hereinafter “Zhou”).
Regarding claim 4, Lu teaches the method of claim 2, wherein the respective training example is associated with a particular task determined by
selecting a task category (Lu, Section 4.1 Paragraph 2 – “These in-context examples serve as an instruction for the language model to adjust to the specific task in SCIENCEQA.” and in Section 4.2 Paragraph 3 – “To be specific, for each test problem t, we map the prompt instruction I : {Ii}n,It into a textual format where {Ii}n refers to the instruction set of n-shot in-context examples from the training set, while It denotes the test instruction.” – teaches selecting a task category (in-context examples serve as an instruction for model to adjust to specific task, {Ii}n refers to instruction set of n-shot in-context examples from training set, thus selecting a task category for an instruction to adjust the model for a specific task));
selecting an instruction template associated with the task category, wherein the instruction template is configured to induce the machine-learned sequence processing model to generate traces when generating responses to input queries (Lu, Section 4.2 Paragraph 3 – “To be specific, for each test problem t, we map the prompt instruction I : {Ii}n,It into a textual format where {Ii}n refers to the instruction set of n-shot in-context examples from the training set, while It denotes the test instruction. Instead of the way where the explanation comes before the answer [54], we feed the instruction I into the encoder decoder model GPT-3 to generate the answer a followed by the lecture lect and explanation exp: M:{Ii}n,It → a,lect,exp” and in Fig. 5 - teaches selecting an instruction template associated with the task category (selects instruction {Ii}n which refers to instruction set of n-shot in-context examples from the training set, thus selecting an instruction template associated with the task of the example from the training set), wherein the instruction template is configured to induce the machine-learned sequence processing model to generate traces when generating responses to input queries (feeds instruction I into encoder decoder model to generate the answer a followed by lecture lect and explanation exp, thus the instruction template Ii for the respective example/task i is configured to induce the model to generate lectures and explanations, or traces, when generating responses, or answers a, to the input queries, or questions)); and
populating the instruction template using data from the dataset to obtain the respective query of the respective training example (Lu, Section 4.2 Paragraph 3 – “To be specific, for each test problem t, we map the prompt instruction I : {Ii}n,It into a textual format where {Ii}n refers to the instruction set of n-shot in-context examples from the training set, while It denotes the test instruction.” and in Fig. 5 – teaches populating the instruction template using data from the dataset to obtain the respective query of the respective training example (instruction {Ii}n which refers to instruction set of n-shot in-context examples from the training set, and as in Fig. 5 the instruction uses data from the dataset to obtain respective query Iiques of respective training example I, thus populating the instruction template using data from the dataset to obtain the respective query of the respective training example)).
Lu fails to explicitly teach selecting a dataset.
However, analogous to the field of the claimed invention, Zhou teaches:
selecting a dataset (Zhou, Section 3 Paragraph 1 – “We consider a task specified by a dataset Dtrain = {(Q,A)} of input/output demonstrations sampled from population X, and a prompted model M.” – teaches selecting a dataset (considers a task specified by a selected dataset Dtrain, example dataset may be TruthfulQA as in Zhou at Section 4.2));
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the dataset selection of Zhou to the chain of thought reasoning, sequence processing models, and instructions of Lu in order to select a dataset for tasks. Doing so would leverage the generalist capabilities of LLMs, and would generate and evaluate a set of instruction candidates based on demonstrations (Zhou, Introduction).
Claim 17 is similar to claim 4, hence similarly rejected.
Claim 20 is similar to claims 2 and 4, hence similarly rejected.
Regarding claim 5, the combination of Lu and Zhou teaches the method of claim 4, wherein
the instruction template is selected from a plurality of instruction templates (Lu, Section 4.2 Paragraph 3 – “To be specific, for each test problem t, we map the prompt instruction I : {Ii}n,It into a textual format where {Ii}n refers to the instruction set of n-shot in-context examples from the training set, while It denotes the test instruction.” and in Fig. 5 – teaches wherein the instruction template is selected from a plurality of instruction templates ({Ii}n refers to the instruction set of n-shot in-context examples from the training set, thus the instruction template Ii is selected from a plurality of instruction templates in instruction set {Ii}n)).
Regarding claim 6, the combination of Lu and Zhou teaches the method of claim 5, wherein
the plurality of instruction templates comprises at least ten instruction templates (Lu, Section 4.2 Paragraph 3 – “To be specific, for each test problem t, we map the prompt instruction I : {Ii}n,It into a textual format where {Ii}n refers to the instruction set of n-shot in-context examples from the training set, while It denotes the test instruction.” and in Fig. 5 – teaches wherein the instruction template is selected from a plurality of instruction templates ({Ii}n refers to the instruction set of n-shot in-context examples from the training set, thus the instruction template Ii is selected from a plurality of instruction templates in instruction set {Ii}n, where n set to 10 provides the instruction set of at least ten instruction templates)).
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lu and Zhou as applied to claims 1, 15, and 19 above, and further in view of Suzgun et al. (NPL: Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models, published Dec. 2022, hereinafter “Suzgun”).
Regarding claim 7, the combination of Lu and Zhou teaches the method of claim 4.
The combination of Lu and Zhou fails to explicitly teach wherein populating the instruction template comprises: populating the instruction template with one or more exemplar delimiters selected randomly from a plurality of exemplar delimiters.
However, analogous to the field of the claimed invention, Suzgun teaches: wherein populating the instruction template comprises:
populating the instruction template with one or more exemplar delimiters selected randomly from a plurality of exemplar delimiters (Suzgun, Section 4.1 Paragraph 1 – “We experimented with ten different text boundary markers (delimiter pairs), which may be divided into two categories: those whose opening and closing markers are identical (known as indistinguishable delimiters), and those whose markers are different (known as complementary delimiters). Specifically, we considered two indistinguishable pairs (viz., quotes and dashes) and eight complementary pairs: (1) curly brackets {·}, (2) square brackets [·], (3) angle brackets ⟨·⟩, (4) parentheses (·), (5) quotes " · ", (6) dashes– ·–, (7) triple angle brackets ⟨⟨⟨·⟩⟩⟩, (8) bracket quotes ⟩ " · ", (9) asterisk quotes * " · ", and (10) double curly bracket {{·}}.”, Section 4.2 Paragraph 1 – “We considered four manually-written template formats t ∈ T for our discrete prompts: … Note that [d1] and [d2] denote the opening and closing elements of the chosen delimiter-pair, respectively.”, and Fig. 2 – teaches populating the instruction template (template formats) with one or more exemplar delimiters ([d1] and [d2] denote the selected delimiter pair) selected randomly from a plurality of exemplar delimiters (delimiters of the constructed prompts, the prompts utilized in Fig. 2, are randomly selected from the plurality of indistinguishable delimiters and complementary delimiters, which are the exemplar delimiters, to produce several candidates)).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the random selection of exemplar delimiters of Suzgun to the prompts, chain of thought reasoning, and models of Lu and Zhou in order to select delimiters for populating instruction templates. Doing so would provide a large impact on performance as delimiter choice affects prompt design (Suzgun, Abstract).
Claim(s) 9 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lu in view of Bai et al. (NPL: Constitutional AI: Harmlessness from AI Feedback, published Dec. 2022, hereinafter “Bai” ).
Regarding claim 9, Lu teaches the method of claim 8.
Lu fails to explicitly teach wherein the plurality of training examples are less than ten percent of a sum of a quantity of the plurality of training examples and a quantity of the other training examples without ground truth traces.
However, analogous to the field of the claimed invention, Bai teaches: wherein
the plurality of training examples are less than ten percent of a sum of a quantity of the plurality of training examples and a quantity of the other training examples without ground truth traces (Bai, Section 3.2 Paragraph 1 – “For red teaming prompts (i.e. partial conversations), we collected 42,496 human-written prompts as discussed and shared in [Ganguli et al., 2022], and generated a further 140,335 prompts by few-shot prompting a pre trained model, giving a total of 182,831.” and in Section 4.2 Paragraph 1 – “For the purpose of doing controlled tests, all the RL runs in this paper use the same set of training prompts, which consists of all the HF and model-generated prompts used for SL-CAI (Section 3.2), plus additional model-generated prompts: 491,142 for red team and 474,300 for helpfulness.” – teaches wherein the plurality of training examples (human-written prompts) are less than ten percent of a sum of a quantity of the plurality of training examples and a quantity of the other training examples without ground truth traces (collected 42,496 human-written prompts and 140,335 prompts by few-shot prompting a pre-trained model, along with additional model-generated prompts: 491,142 for red team and 474,300 for helpfulness. The human-written prompts are less than ten percent of a sum of the quantity of human written prompts and model generated prompts without ground truth traces)).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the split of training examples and other training examples without ground truth traces of Bai to the prompts, chain of thought reasoning, and models of Lu. Doing so enable the control of AI behavior more precisely and with far less human labels (Bai, Abstract), and thus develop techniques that do not require humans to supervise all aspects of AI behavior (Bai, Introduction).
Claim 18 is similar to claims 8 and 9, hence similarly rejected.
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lu in view of Taylor et al. (NPL: Galactica: A Large Language Model for Science, published Nov. 2022, hereinafter “Taylor”).
Regarding claim 12, Lu teaches the method of claim 1.
Lu fails to explicitly teach wherein the response and the trace are generated in a single forward pass of the machine-learned sequence processing model.
However, analogous to the field of the claimed invention, Taylor teaches: wherein
the response and the trace are generated in a single forward pass of the machine-learned sequence processing model (Taylor, Section 3.1 Paragraph 1 – “There are two limitations with chain-of-thought. First, it relies on prompt discovery to find a prompt that elicits robust step-by-step reasoning; i.e. minimizes mistakes from doing too much in a single forward pass.” – teaches wherein the response and the trace are generated in a single forward pass of the machine-learned sequence processing model (finds a prompt that elicits chain-of-thought that minimizes mistakes generated from a single forward pass, thus the response and the trace are generated in a single-forward pass of the machine-learned sequence processing model)).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the generation in a single forward pass of Taylor to the queries, responses, and traces of Lu. Doing so would improve the capability of predictions smoothly with scale (Taylor, Introduction) and provide chain-of-thought reasoning in a single forward pass (Taylor, Section 3.1).
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lu in view of Ouyang et al. (NPL from IDS: Training language models to follow instructions with human feedback, published March 2022, hereinafter “Ouyang”).
Regarding claim 13, Lu teaches the method of claim 1, wherein
the query comprises an instruction (Lu, Section 4.2 Paragraph 3 – “To be specific, for each test problem t, we map the prompt instruction I : {Ii}n,It into a textual format where {Ii}n refers to the instruction set of n-shot in-context examples from the training set, while It denotes the test instruction.” and in Fig. 5 – teaches wherein the query (question) comprises an instruction (Fig. 5 shows the question comprising an instruction Iiques))
Lu fails to explicitly teach wherein the one or more parameters are updated to increase a likelihood that the machine-learned sequence processing model generates an output that follows the instruction.
However, analogous to the field of the claimed invention, Ouyang teaches:
wherein the one or more parameters are updated to increase a likelihood that the machine-learned sequence processing model generates an output that follows the instruction (Ouyang, Section 1 Paragraph 3 – “We focus on fine-tuning approaches to aligning language models. Specifically, we use reinforcement learning from human feedback (RLHF; Christiano et al., 2017; Stiennon et al., 2020) to fine-tune GPT-3 to follow a broad class of written instructions (see Figure 2).”, Section 3.1 Paragraph 2-4 – “Step 1: Collect demonstration data, and train a supervised policy. Our labelers provide demonstrations of the desired behavior on the input prompt distribution (see Section 3.2 for details on this distribution). We then fine-tune a pretrained GPT-3 model on this data using supervised learning. Step 2: Collect comparison data, and train a reward model. We collect a dataset of comparisons between model outputs, where labelers indicate which output they prefer for a given input. We then train a reward model to predict the human-preferred output. Step 3: Optimize a policy against the reward model using PPO. We use the output of the RM as a scalar reward. We fine-tune the supervised policy to optimize this reward using the PPO algorithm (Schulman et al., 2017).”, and in Fig. 2 – teaches wherein the one or more parameters are updated (fine-tuning) to increase a likelihood that the machine-learned sequence processing model generates an output that follows the instruction (utilizes reinforcement learning from human feedback to fine-tune GPT-3 to follow a broad class of written instructions, thus one or more parameters are updated to increase a likelihood that the machine-learned sequence processing model generates an output that follows the instruction)).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the updates to increase a likelihood that the model follows instructions of Ouyang to the instructions, updates, and model of Lu. Doing so would avert unintended behaviors of deployed language models and align language models by training them to act in accordance with user intentions (Ouyang, Introduction).
Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lu in view of Reynolds et al. (NPL: Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm, published Feb. 2021, hereinafter “Reynolds”).
Regarding claim 14, Lu teaches the method of claim 1.
Lu fails to explicitly teach wherein the trace comprises a chain of intermediate responses to intermediate queries.
However, analogous to the field of the claimed invention, Reynolds teaches: wherein
the trace comprises a chain of intermediate responses to intermediate queries (Reynolds, Section 4.7 Paragraph 2 – “We instead propose harnessing the language model itself via metaprompts, seeds encapsulating a more general intention that will unfold into a specific prompt when combined with additional information, such as the task question.”, Section 4.7 Paragraph 3 – “A metaprompt may be something as short as a phrase such as “This problem asks us to”, a seemingly innocuous fragment which, by prompting for a statement of the problem’s intention, sets the stage for a serial explanation of a procedure to solve the problem. Alternatively, a metaprompt may take the form of a fill-in-the-blank template which constrains the response along a predetermined procedure, but allows the model to fill in the details specific to the problem.”, and in Fig. 3 – teaches wherein the trace comprises a chain of intermediate response to intermediate queries (a metaprompt is a fragment of a prompt that sets the stage for a serial explanation of a procedure to solve the problem, and Fig. 3 shows the chain of intermediate responses to intermediate queries, the generic serializing metaprompt is broken down into intermediate queries with a chain of intermediate responses “f(f(3)) = f(3*3) = 3*3*3” and “f(3) = 3*3 = 9, so (f(f(3)) = 27”)).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the trace comprising a chain of intermediate responses to intermediate queries of Reynolds to the queries and model of Lu. Doing so would provide novel prompts which force a language model to break a problem into components before producing a verdict (Reynolds, Introduction).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Wang et al. (NPL: Iteratively Prompt Pre-trained Language Models for Chain of Thought, published Dec. 2022) a method for iterative prompting which progressively elicits relevant knowledge from pre-trained language models for multi-step inference with chain of thought reasoning. Teaches providing intermediate traces for intermediate queries. Teaches optimizing the parameters of the model based on evaluations.
Yao et al. (NPL: REACT: Synergizing Reasoning and Acting in Language Models, published Oct. 2022) teaches methods for exploring the use of large language models to generate both reasoning traces and task-specific actions in an interleaved manner. Teaches updating contexts based on reasoning traces to support future reasoning or acting. Teaches wherein humans may control or correct agent behavior to further align the model with human feedback.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LOUIS C NYE whose telephone number is 571-272-0636. The examiner can normally be reached Monday - Friday 9:00AM - 5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MATT ELL can be reached at 571-270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LOUIS CHRISTOPHER NYE/Examiner, Art Unit 2141
/MATTHEW ELL/Supervisory Patent Examiner, Art Unit 2141
Prosecution Insights