DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The IDS dated 1/28/2026 has been considered and placed in the application file.
Response to Amendment and Arguments.
Applicant’s arguments, see page 2, filed 4/13/2026, with respect to claims 1,3-6, 8-15, 17-21,23 rejection have been fully considered and are not persuasive. Applicant argues that “The claimed invention provides a technical solution to a technical problem inherent in the field of natural language processing, by allowing for distinguishing between consensus outputs and stochastic outliers (hallucinations). Large language models (LLMs) implemented as autoregressive neural networks are inherently probabilistic and stochastic. A significant technical challenge in this field is the tendency of such models to "hallucinate," i.e., to generate text that is grammatically correct but factually incorrect or logically inconsistent. Standard LLM implementations often provide no mechanism to quantify the uncertainty of a response, leaving the user with an unreliable output. The Specification describes how the claimed invention distinguishes between consensus outputs and hallucinations:” The Examiner respectfully disagrees gathering answers then seeing which one is most common then choosing based on which one was mentioned the most is not an improvement of technology; this would be an improvement of an abstract idea. This covers performance in the human mind. A person analyzing a survey to see what’s the most common answer that people have. Furthermore, the claims do not reflect an improvement of hallucinations of an LLM. Hence being a mental process. The claims are not patent eligible.
Applicant further argues that "training one or more of the pluralities of language models using a training set generated based on the intermediate answers." Claim 22 recites: "generating the training set by: for each of one or more of the intermediate answers, determining whether the intermediate answer meets a criterion; and for each of the one or more intermediate answers, in response to determining that the intermediate answer does not meet the criteria, including the intermediate answer in the training set." The Examiner respectfully disagrees gathering answers then training a model with those answers without a criteria is not an improvement of technology. Examiner suggests to look at claim 22 where a criteria is mentioned (emphasis added). Hence being a mental process. The claims are not patent eligible.
Therefore, the 101 rejection of claims 1,3-6, 8-15, 17-21,23 is maintained.
Applicant’s arguments with respect to claim(s) 1,3-6, 8-15, 17-23 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. The newly modified claim limitation “providing the input to a plurality of language models, wherein each language model is configured to generate at least an intermediate answer to the prompt from the input, and wherein each language model comprises an auto-regressive neural network trained on a language modeling task;
generating an answer to the prompt, wherein the answer comprises natural language text, by performing a probabilistic inference over the distribution, the answer comprising natural language text comprising: generating the answer based on each cluster of intermediate answers in the distribution.” necessitates the new ground of rejection.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1,3-6, 8-15, 17-21,23 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claims 1, 6, 15, and 20, Further claim 1 recites A method comprising:
obtaining a prompt comprising natural language text;
obtaining a set of documents comprising natural language text;
generating an input comprising at least the set of documents and the prompt;
providing the input to a plurality of language models, wherein each language model is configured to generate at least an intermediate answer to the prompt from the input;
generating a distribution from the intermediate answers; and
generating an answer to the prompt by performing a probabilistic inference over the distribution, the answer comprising natural language text.
Further claim 15 states A system comprising:
one or more computers; and
one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform respective operations comprising:
Further claim 20 states One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform respective operations comprising:
The limitation of “obtaining …”, “obtaining…”, “generating…”, “providing…”, “generating…”, and “generating…” , as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. For example, a person receiving a query and documents from a user. The person would than use them both to answer the query, but he would answer the query in a plurality of ways. Further grouping the answers that are alike. Further selecting a final answer among them.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claims recite an abstract idea.
This judicial exception is not integrated into a practical application. In particular, the claim only recites additional elements that are computer components “computer” (specification states “The system 600 is intended to include various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers, including vehicles installed on base units or pod units of modular vehicles.”), “storage” (specification states “The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.”), and “language models” (specification states “In some implementations, the embedding server can use an embedding engine that can be finetuned on training data for a particular domain, such as the legal domain. In some examples, the embedding engine can be an encoder neural network or a large language model such as Gemini, Gemma, or PaLM.”) recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using the computer components amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claims are not patent eligible.
Claims 3, 8, and 17 additionally claim 3 recites the method of claim 1, wherein the plurality of language models comprises instances of a same language model. In particular, the claim only recites additional elements that are computer components “language models” (specification states “In some implementations, the embedding server can use an embedding engine that can be finetuned on training data for a particular domain, such as the legal domain. In some examples, the embedding engine can be an encoder neural network or a large language model such as Gemini, Gemma, or PaLM.”) recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
Claims 4 and 18 additionally claim 4 recites the method of claim 1, wherein the plurality of language models comprises different language models. In particular, the claim only recites additional elements that are computer components “language models” (specification states “In some implementations, the embedding server can use an embedding engine that can be finetuned on training data for a particular domain, such as the legal domain. In some examples, the embedding engine can be an encoder neural network or a large language model such as Gemini, Gemma, or PaLM.”) recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
Claims 5 and 19 additionally claim 5 recites The method of claim 1, wherein the method further comprises: generating a modified input comprising at least the answer and the set of documents; providing the modified input to a plurality of language models, wherein each language model is configured to generate at least a secondary intermediate answer to the prompt from the modified input; generating a second distribution from the secondary intermediate answers; and generating a response to the modified input by performing a probabilistic inference over the second distribution, the response comprising natural language text. However, these limitations encompass a person using the query the document and the answer to further generate more answers. Further grouping the second set of answers and finally choosing one. Thus, the claim is directed towards a mental process. In particular, the claim only recites additional elements that are computer components “language models” (specification states “In some implementations, the embedding server can use an embedding engine that can be finetuned on training data for a particular domain, such as the legal domain. In some examples, the embedding engine can be an encoder neural network or a large language model such as Gemini, Gemma, or PaLM.”) recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
Claims 9 additionally recites the method of claim 6, wherein the input to each language model comprises a different prompt. However, these limitations encompass a person asked a plurality of questions. Thus, the claim is directed towards a mental process. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claims 10 additionally recites the method of claim 6, wherein obtaining a set of documents comprising natural language text further comprises obtaining a subset of the set of documents, wherein each document in the subset comprises text that is relevant to the prompt. However, these limitations encompass a person receiving a document to answer the question he receives. Further the document is related to the question and the person separated the document into portions. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claims 11 additionally recites the method of claim 6, wherein the method further comprises: receiving a request for an alternative to the answer; and generating a second answer comprising natural language text to the prompt. However, these limitations encompass a answering the query and being asked to provide another answer in natural text. Thus, the claim is directed towards a mental process. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claims 12 additionally recite the method of claim 6, wherein the method further comprises: receiving a request for an explanation for the answer; and generating an explanation comprising natural language text for the answer. However, these limitations encompass a person answering a query and being requested to provide an explanation on the answer in natural language text. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claim 13 additionally recites The method of claim 6, wherein the method further comprises: obtaining a second prompt comprising a deterministic answer comprising natural language text to the prompt; generating a modified input comprising at least the second prompt and the set of documents; providing the modified input to a plurality of language models, wherein each language model is configured to generate at least an intermediate answer to the prompt from the modified input; for each language model: generating a distribution of a plurality of intermediate answers by providing the modified input to the language model multiple times; and generating an answer comprising natural language text to the prompt by performing a probabilistic inference over each distribution. However, these limitations encompass a person receiving a second prompt and the answer to the first prompt , further including the documents to answer the second prompt. In a plurality of ways. Further grouping the together and choosing a final answer. Thus, the claim is directed towards a mental process. In particular, the claim only recites additional elements that are computer components “language models” (specification states “In some implementations, the embedding server can use an embedding engine that can be finetuned on training data for a particular domain, such as the legal domain. In some examples, the embedding engine can be an encoder neural network or a large language model such as Gemini, Gemma, or PaLM.”) recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
Claim 14 additionally recites The method of claim 6, wherein the method further comprises: generating a second prompt that comprises different text with a same meaning as the text of the prompt; for each language model: generating a first distribution of a plurality of first intermediate answers by providing the input to the language model multiple times; generating a second distribution of a plurality of second intermediate answers by providing an input comprising at least the set of documents and the second prompt to the language model; and generating the answer by performing a probabilistic inference over each distribution. However, these limitations encompass a person receiving a query. Further paraphrasing the query. Then answering the query in multiple of ways. Then grouping the first set of answers and choosing one. Further answering the query to generate a second set of answering further grouping them and choosing one. Thus, the claim is directed towards a mental process. In particular, the claim only recites additional elements that is “user interface” where its pre-solution by receiving something from the user. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
Claims 21 and 23 additionally claim 21 recite The method of claim 1, further comprising: training one or more of the plurality of language models using a training set generated based on the intermediate answers. However, these limitations encompass a person gathering answers and learning from them. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claims 20 is drawn to a “signal” per se as recited in the preamble and as such is non-statutory subject matter. In the Published Specification it states “The memory 620 stores information within the system 600. In one implementation, the memory 620 is a computer-readable medium. In one implementation, the memory 620 is a volatile memory unit. In another implementation, the memory 620 is a non-volatile memory unit.”, the term “computer-readable medium " is not defined as to what the scope of the term is meant to encompass. Hence, one of ordinary skilled in the art can interpret such term to include transitory signals and non-transitory signals. It does not appear that a claim reciting a signal encoded with functional descriptive material falls within any of the categories of patentable subject matter set forth in § 101. First, a claimed signal is clearly not a "process" under § 101 because it is not a series of steps. The other three § 101 classes of machine, compositions of matter and manufactures "relate to structural entities and can be grouped as 'product' claims in order to contrast them with process claims." 1 D. Chisum, Patents § 1.02 (1994).
The Applicant' s Specification presents a broad definition as to what the “computer-readable medium " covers and is being made to include transitory and non-transitory signals. The Applicant' s Published Specification states “The memory 620 stores information within the system 600. In one implementation, the memory 620 is a computer-readable medium. In one implementation, the memory 620 is a volatile memory unit. In another implementation, the memory 620 is a non-volatile memory unit” ,refers to the “storage medium”. Hence, it appears that the claims appear to be drawn towards transitory signals, which is not subject matter eligible. In order to overcome the present rejection, the Applicant is advised to amend the claims by using the following terminology: "non-transitory machine readable storage medium." Such example terminology has been also found in the Official Gazette 1351 OG 212.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1,3, 4, 6,8,10,12,15,17,18, 20, 21, and 23 are rejected under 35 U.S.C. 103 as obvious over Wang, Xuezhi, et al. "Self-consistency improves chain of thought reasoning in language models." arXiv preprint arXiv:2203.11171 (2022), in view of US Patent Publication US 20240095460 A1, (Xu; Peng.) in view of Chen, Lingjiao, Matei Zaharia, and James Zou. “Frugalgpt: How to use large language models while reducing cost and improving performance.” arXiv preprint arXiv:2305.05176 (2023).
Claim 1, 6, 15, and 20
Regarding Claim 1 and 6, Wang teach 1. A method comprising: obtaining a prompt comprising natural language text;
(Fig 1 shows obtaining a prompt)
generating a distribution from the intermediate answers by clustering the intermediate answers based on a similarity of each intermediate answer to each other intermediate answer; and
(Fig 1 shows marginalizing to aggregate a final answer
Page 1 last paragraph "…we first sample from the language model’s decoder to generate a diverse set of reasoning paths; each reasoning path might lead to a different final answer,…"
page 3 paragraph 1 "Finally, we aggregate the answers by marginalizing out the sampled reasoning paths and choosing the answer that is the most consistent among the generated answers.")
generating an answer to the prompt, wherein the answer comprises natural language text, by performing a probabilistic inference over the distribution, the answer comprising natural language text comprising:
(Fig 1 shows the final answer in natural language text. Where the final answer comes from marginalizing.
Page 3 paragraph 2 "After sampling multiple (ri ai) from the model’s decoder, self-consistency applies a marginalization over ri by taking a majority vote over ai, i.e., argmaxa m i=1 1(ai = a), or as we defined as the most “consistent” answer among the final answer set.")
generating the answer based on each cluster of intermediate answers in the distribution.
(Fig 1 shows the final answer in natural language text. Where the final answer comes from marginalizing.1
Page 1 last paragraph "…we first sample from the language model’s decoder to generate a diverse set of reasoning paths; each reasoning path might lead to a different final answer,…"
page 3 paragraph 1 "Finally, we aggregate the answers by marginalizing out the sampled reasoning paths and choosing the answer that is the most consistent among the generated answers."
Page 3 paragraph 2 "After sampling multiple (ri ai) from the model’s decoder, self-consistency applies a marginalization over ri by taking a majority vote over ai, i.e., argmaxa m i=1 1(ai = a), or as we defined as the most “consistent” answer among the final answer set.")
Wang do not explicitly teach all of obtaining a set of documents comprising natural language text;
generating an input comprising at least the set of documents and the prompt;
providing the input to a plurality of language models, wherein each language model is configured to generate at least an intermediate answer to the prompt from the input;, and wherein each language model comprises an auto-regressive neural network trained on a language modeling task;
However, Xu teach
obtaining a set of documents comprising natural language text;
(Paragraph 3 "Embodiments of the present disclosure relate to dialogue systems for automotive systems and applications. Systems and methods are disclosed that generate and/or receive audio data (and/or text data corresponding to the audio data) representing speech from a user, where the speech may include a question associated with a vehicle or other machine (e.g., autonomous or semi-autonomous vehicle, construction equipment, landscaping equipment, warehouse vehicles, aircraft, water-based vehicles, etc.). The systems and methods may then use one or more techniques to retrieve information associated with a context of the speech. For a first example, the systems and methods may use a retrieval system(s) to retrieve one or more question/answer pairs associated with the speech, such as from a database(s). For a second example, the systems and methods may use the retrieval system(s) to retrieve contextual information related to the speech, such as contextual information from a (fixed or live) text-based knowledge base—such as a manual, a vehicle manual, a machine manual, a document, etc.—that is stored in the database(s). In either of these examples, the systems and methods of the present disclosure may input data representing the speech, data representing the information associated with the context, and/or other data into a language model(s) (e.g., a large language model(s)). The language model(s) may then process the data and, based on the processing, output data associated with the speech. For example, if the speech includes the question associated with the vehicle, the language model(s) may output information (e.g., an answer) associated with the question. The systems and methods may then provide the information to the user.")
generating an input comprising at least the set of documents and the prompt;
(Fig 6 shows the prompt entailing contextual information (documents)
Paragraph 29 "The system(s) may then use the text data representing the transcript, data representing the question/answer pair(s), data representing the portion(s) of the contextual information, and/or additional data to generate a prompt associated with the speech. The system(s) may then input, into a language model(s) (e.g., a large language model(s)), prompt data representing the prompt. As described herein, the language model(s) may include any type of language model(s), such as a large language model (LLM), generative language model(s) (e.g., a Generative Pretrained Transformer (GPT), etc.), a representation language model(s) (e.g., a Bidirectional Encoder Representations from Transformers (BERT), etc.), and/or any other type of language model. The language model(s) may then process the prompt data and, based on the processing, output data associated with the speech. For example, if the speech represents a question associated with the vehicle, then the output data may represent information (e.g., an answer) associated with the question. The system(s) may then provide the output to the user, such as by outputting audio associated with the output using one or more speakers.")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Xu to provide a “obtaining a set of documents comprising natural language text; generating an input comprising at least the set of documents and the prompt;” Doing so would Output an answer based on the information, as recognized by Xu. (Paragraph 3).
Wang in view of Xu do not explicitly teach all of
providing the input to a plurality of language models, wherein each language model is configured to generate at least an intermediate answer to the prompt from the input;, and wherein each language model comprises an auto-regressive neural network trained on a language modeling task;
However, Chen teaches
providing the input to a plurality of language models, wherein each language model is configured to generate at least an intermediate answer to the prompt from the input;, and wherein each language model comprises an auto-regressive neural network trained on a language modeling task;
(See table one for a list of auto regressive models. Note: all providers API are autoregressive models.
page 3 section 2 scope and problems statement: LLM marketplace. " We consider answering queries via the LLM market, which comprises K different LLM APIs, denoted by {fi(·)}K i=1. Each fi(·) : P → A is a function that, given a prompt p from the prompt space P, generates an answer from the answer distribution A. Note that to use LLM APIs, one has to convert each query q to some corresponding prompt first. LLM APIs are associated with their own cost, typically consisting of three components: a portion proportional to the length of the prompt, a portion proportional to the length of the generated answer, and (sometimes) a fixed cost per query. Formally, given a prompt p, the cost of using the ith LLM API is denoted by ci(p) ˜ci,2 fi(p) + ˜ci,1 p + ˜ci,0, where ˜ci,j,j = 0,1,2 are constants")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang in view of Xu to incorporate the teachings of Chen to provide a “providing the input to a plurality of language models, wherein each language model is configured to generate at least an intermediate answer to the prompt from the input;” Doing so would Have better performance than a single LLM, as recognized by Chen. (page 8 LLM diversity).
Regarding Claim 15, Wang further teaches
15. A system comprising:
one or more computers; and
(section 1 paragraph 2 "In this paper, we introduce a novel decoding strategy called self-consistency to replace the greedy decoding strategy used in chain-of-thought prompting (Wei et al., 2022), that further improves language models’ reasoning performance by a significant margin."
It is implicant to have a computer to run a language model)
one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform respective operations comprising:
(section 1 paragraph 2 "In this paper, we introduce a novel decoding strategy called self-consistency to replace the greedy decoding strategy used in chain-of-thought prompting (Wei et al., 2022), that further improves language models’ reasoning performance by a significant margin."
It is implicant to have memory to run a language model)
Claim 15 contains limitations similar to those found in claims 1,6,20 and therefore are not patent eligible for the same reasons.
Regarding Claim 20, Wang teach
One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform respective operations comprising:
(section 1 paragraph 2 "In this paper, we introduce a novel decoding strategy called self-consistency to replace the greedy decoding strategy used in chain-of-thought prompting (Wei et al., 2022), that further improves language models’ reasoning performance by a significant margin."
It is implicant to have memory to run a language model)
Claim 20 contains limitations similar to those found in claims 1,6,15 and therefore are not patent eligible for the same reasons.
Claim 3, 8 ,and 17
Regarding Claim 3 ,8, and 17, Wang in view of Xu in view of Chen, furthermore, Chen teaches the method of claim 1, wherein the plurality of language models comprise instances of a same language model.
(Page 1 section 1 introduction "… Given the heterogeneous cost and quality, how to effectively and efficiently leverage the full set of LLM options is a key challenge for pracitioners. If the tasks are relatively simple, then aggregating multiple responses from GPT-J [WK21] (whose size is 30x smaller than GPT-3) offers performance similar to GPT-3 [ANC+22], leading to financial and environmental savings. However, the performance of GPT-J can be much worse on difficult tasks [TLI+23]…. ")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang in view of Xu to incorporate the teachings of Chen to provide a “The method of claim 1, wherein the plurality of language models comprises instances of a same language model.” Doing so would Provide performance like gpt3 and reduce cost and environmental impact, as recognized by Chen. (page 1 section 1 introduction).
Claim 4, 18
Regarding Claim 4 ,and 18, Wang in view of Xu in view of Chen, furthermore, Chen teaches the method of claim 1, wherein the plurality of language models comprise different language models.
(See table one for a list of auto regressive models.
page 3 section 2 scope and problems statement: LLM marketplace. " We consider answering queries via the LLM market, which comprises K different LLM APIs, denoted by {fi(·)}K i=1. Each fi(·) : P → A is a function that, given a prompt p from the prompt space P, generates an answer from the answer distribution A. Note that to use LLM APIs, one has to convert each query q to some corresponding prompt first. LLM APIs are associated with their own cost, typically consisting of three components: a portion proportional to the length of the prompt, a portion proportional to the length of the generated answer, and (sometimes) a fixed cost per query. Formally, given a prompt p, the cost of using the ith LLM API is denoted by ci(p) ˜ci,2 fi(p) + ˜ci,1 p + ˜ci,0, where ˜ci,j,j = 0,1,2 are constants")
See claim one for rationale.
Claim 10
Regarding Claim 10, Wang in view of Xu in view of Chen, furthermore, Xu teaches the method of claim 6, wherein obtaining a set of documents comprising natural language text further comprises obtaining a subset of the set of documents, wherein each document in the subset comprises text that is relevant to the prompt.
(Paragraph 3 "Embodiments of the present disclosure relate to dialogue systems for automotive systems and applications. Systems and methods are disclosed that generate and/or receive audio data (and/or text data corresponding to the audio data) representing speech from a user, where the speech may include a question associated with a vehicle or other machine (e.g., autonomous or semi-autonomous vehicle, construction equipment, landscaping equipment, warehouse vehicles, aircraft, water-based vehicles, etc.). The systems and methods may then use one or more techniques to retrieve information associated with a context of the speech. For a first example, the systems and methods may use a retrieval system(s) to retrieve one or more question/answer pairs associated with the speech, such as from a database(s). For a second example, the systems and methods may use the retrieval system(s) to retrieve contextual information related to the speech, such as contextual information from a (fixed or live) text-based knowledge base—such as a manual, a vehicle manual, a machine manual, a document, etc.—that is stored in the database(s). In either of these examples, the systems and methods of the present disclosure may input data representing the speech, data representing the information associated with the context, and/or other data into a language model(s) (e.g., a large language model(s)). The language model(s) may then process the data and, based on the processing, output data associated with the speech. For example, if the speech includes the question associated with the vehicle, the language model(s) may output information (e.g., an answer) associated with the question. The systems and methods may then provide the information to the user."
Paragraph 27 "The retrieval system(s) may then retrieve at least a portion of the information that is associated with the text data. In some examples, to retrieve the portion of the information, and similar to the question/answer pairs above, the information stored within the database(s) may be associated with embeddings. For instance, and as described in more detail herein, a first portion of the information may be associated with a first embedding, a second portion of the information may be associated with a second embedding, a third portion of the information may be associated with a third embedding, and/or so forth. As such, the retrieval system(s) may use the generated embedding and the embeddings associated with the portions of the information to determine scores for the portions of the information. The retrieval system(s) may then retrieve a threshold amount of the information that is associated with the highest score(s). In some examples, the threshold amount of the information may include a threshold number of portions such as, but not limited to, one portion of the information, two portions of the information, five portions of the information, and/or any other number of portions of the information. Additionally, or alternatively, in some examples, the threshold amount of the information may include a threshold number of words such as, but not limited to, one word of the information, ten words of the information, one hundred words of the information, two hundred words of the information, and/or any other number of words of the information"
Paragraph 54 "The retrieval component 114 may then use the embedding 404 associated with the text data 204 and the embeddings 508 to select one or more portions 506 of the information 504. In some examples, the retrieval component 114 selects a threshold number of the portions 506. The threshold number may include, but is not limited to, one portion 506, two portions 506, five portions 506, ten portions 506, and/or any other number of portions 506. In some examples, the retrieval component 114 selects portions 506 until reaching a threshold number of characters and/or words. For instance, the threshold number of words may include, but is not limited to, ten words, fifty words, one hundred words, two hundred words, and/or any other number of words.")
See claim one for rationale.
Claim 12
Regarding Claim 12, Wang in view of Xu in view of Chen, furthermore, Wang teaches
12. The method of claim 6, wherein the method further comprises:
receiving a request for an explanation for the answer; and
(Section introduction "where a language model is prompted to generate a series of short sentences that mimic the reasoning process a person might employ in solving a task. For example, given the question “If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?”, instead of directly responding with “5”, a language model would be prompted to respond with the entire chain-of-thought: “There are 3 cars in the parking lot already. 2 more arrive. Now there are 3 + 2 =5cars. The answer is 5.”.")
generating an explanation comprising natural language text for the answer.
(Section introduction "where a language model is prompted to generate a series of short sentences that mimic the reasoning process a person might employ in solving a task. For example, given the question “If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?”, instead of directly responding with “5”, a language model would be prompted to respond with the entire chain-of-thought: “There are 3 cars in the parking lot already. 2 more arrive. Now there are 3 + 2 =5cars. The answer is 5.”.")
Claim 21, 23
Regarding Claim 21 and 23, Wang in view of Xu in view of Chen, furthermore, Chen teaches
21. (New) The method of claim 1, further comprising:
training one or more of the plurality of language models using a training set generated based on the intermediate answers.
(See figure 2d where it shows the finetuning
page 5 section strategy 2: LLM approximation "… Another example of LLM approximation is model fine-tuning. As shown in Figure 2(d), this process consists of three steps: first, collect a powerful but expensive LLM API’s responses to a few queries; second, use the responses to fine-tune a smaller and more affordable AI model; and finally, employ the fine-tuned model for new queries. In addition to cost savings, the fine-tuned model often does not require lengthy prompts, thus providing latency improvements as a byproduct.")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang in view of Xu to incorporate the teachings of Chen to provide a “The method of claim 1, wherein the plurality of language models comprises instances of a same language model.” Doing so would Have cost saving and latency improvements, as recognized by Chen. (page 5 section strategy 2: LLM approximation).
Claims 5,13,19 is rejected under 35 U.S.C. 103 as obvious over Wang, Xuezhi, et al. "Self-consistency improves chain of thought reasoning in language models." arXiv preprint arXiv:2203.11171 (2022), in view of US Patent Publication US 20240095460 A1, (Xu; Peng.) in view of Chen, Lingjiao, Matei Zaharia, and James Zou. “Frugalgpt: How to use large language models while reducing cost and improving performance.” arXiv preprint arXiv:2305.05176 (2023). in further view of Gao, Luyu, et al. “RARR: Researching and Revising What Language Models Say, Using Language Models.” ArXiv.org, 2022, arxiv.org/abs/2210.08726v2. Accessed 7 Jan. 2026.
Claim 5 and 19
Regarding Claim 5 and 19, Wang teach
generating a second distribution from the secondary intermediate answers; and
(Fig 1 shows marginalizing to aggregate a final answer
Page 1 last paragraph "…we first sample from the language model’s decoder to generate a diverse set of reasoning paths; each reasoning path might lead to a different final answer,…"
page 3 paragraph 1 "Finally, we aggregate the answers by marginalizing out the sampled reasoning paths and choosing the answer that is the most consistent among the generated answers.")
generating a response to the modified input by performing a probabilistic inference over the second distribution, the response comprising natural language text.
(Fig 1 shows the final answer in natural language text. Where the final answer comes from marginalizing.
Page 3 paragraph 2 "After sampling multiple (ri ai) from the model’s decoder, self-consistency applies a marginalization over ri by taking a majority vote over ai, i.e., argmaxa m i=1 1(ai = a), or as we defined as the most “consistent” answer among the final answer set.")
Wang in view of Xu do not explicitly teach all of plurality of language models, wherein each language model is configured to generate at least a secondary intermediate answer to the prompt from the modified input;
However, Chen teaches plurality of language models, wherein each language model is configured to generate at least a secondary intermediate answer to the prompt from the modified input;
(Paragraph 6 "Another example aspect of the present disclosure is directed to a computer-implemented method. The method can include obtaining, by a computing system including one or more processors, conversation data. The conversation data can be descriptive of a conversation history. The method can include processing, by the computing system, the conversation data with a language encoding model to generate a language representation. The language representation can be descriptive of semantics associated with the conversation history. The method can include processing, by the computing system, the language representation with a plurality of machine-learned language models to generate a plurality of candidate outputs. The plurality of machine-learned language models may have been trained based on learned sentiment distributions associated with a latent space. The method can include processing, by the computing system, the language representation and the plurality of candidate outputs with a dialogue management model to determine a dialogue response." )
See claim one for rationale.
Wang in view of Xu in view of Chen do not explicitly teach all of
generating a modified input comprising at least the answer and the set of documents;
providing the modified input to a
However, Gao teach
generating a modified input comprising at least the answer and the set of documents;
(Fig1 shows the answer and document being an input.
section 3 approach first paragraph "… Next, the revision stage takes the original text x and the retrieval results (q1 e11) and produces a revised text y(e.g.,“ Millie In between premiered on 1October2014onCBBC.”)."
Section 3.2 revision paragraph 2 "The revision stage initializes y = x and iterates through each pair (q e) = (qi eij) in the retrieval result. For each pair, we perform two steps:…")
providing the modified input to a
(Fig1 shows the answer and document being an input.
section 3 approach first paragraph "… Next, the revision stage takes the original text x and the retrieval results (q1 e11) and produces a revised text y(e.g.,“ Millie In between premiered on 1October2014onCBBC.”)."
Section 3.2 revision paragraph 2 "The revision stage initializes y = x and iterates through each pair (q e) = (qi eij) in the retrieval result. For each pair, we perform two steps:…")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang in view of Xu in view of Chen to incorporate the teachings of Gao to provide a “generating a modified input comprising at least the answer and the set of documents; providing the modified input to a ” Doing so would increase the total percentage of attributed sentences, as recognized by Gao. (Section 3.2 revision paragraph 1).
Claim 13
Regarding Claim 13, Wang in view of Xu in view of Chen, in further view of Gao, furthermore, Wang teaches the method of claim 6, wherein the method further comprises: for each language model:
generating a distribution of a plurality of intermediate answers by providing the modified input to the language model multiple times; and
(Fig 1 shows marginalizing to aggregate a final answer
Page 1 last paragraph "…we first sample from the language model’s decoder to generate a diverse set of reasoning paths; each reasoning path might lead to a different final answer,…"
page 3 paragraph 1 "Finally, we aggregate the answers by marginalizing out the sampled reasoning paths and choosing the answer that is the most consistent among the generated answers.")
generating an answer comprising natural language text to the prompt by performing a probabilistic inference over each distribution.
(Fig 1 shows the final answer in natural language text. Where the final answer comes from marginalizing.
Page 3 paragraph 2 "After sampling multiple (ri ai) from the model’s decoder, self-consistency applies a marginalization over ri by taking a majority vote over ai, i.e., argmaxa m i=1 1(ai = a), or as we defined as the most “consistent” answer among the final answer set.")
Wang in view of Xu in view of Chen, in further view of Gao, furthermore, Chen teaches plurality of language models, wherein each language model is configured to generate at least an intermediate answer to the prompt from the modified input;
(Paragraph 6 "Another example aspect of the present disclosure is directed to a computer-implemented method. The method can include obtaining, by a computing system including one or more processors, conversation data. The conversation data can be descriptive of a conversation history. The method can include processing, by the computing system, the conversation data with a language encoding model to generate a language representation. The language representation can be descriptive of semantics associated with the conversation history. The method can include processing, by the computing system, the language representation with a plurality of machine-learned language models to generate a plurality of candidate outputs. The plurality of machine-learned language models may have been trained based on learned sentiment distributions associated with a latent space. The method can include processing, by the computing system, the language representation and the plurality of candidate outputs with a dialogue management model to determine a dialogue response." )
See claim 1 for rationale.
Wang in view of Xu in view of Chen, in further view of Gao, furthermore, Gao teaches
13. The method of claim 6, wherein the method further comprises:
obtaining a second prompt comprising a deterministic answer comprising natural language text to the prompt;
(Fig1 shows the answer and document being an input.
section 3 approach first paragraph "… Next, the revision stage takes the original text x and the retrieval results (q1 e11) and produces a revised text y(e.g.,“ Millie In between premiered on 1October2014onCBBC.”)."
Section 3.2 revision paragraph 2 "The revision stage initializes y = x and iterates through each pair (q e) = (qi eij) in the retrieval result. For each pair, we perform two steps:…")
generating a modified input comprising at least the second prompt and the set of documents;
(Fig1 shows the answer and document being an input.
section 3 approach first paragraph "… Next, the revision stage takes the original text x and the retrieval results (q1 e11) and produces a revised text y(e.g.,“ Millie In between premiered on 1October2014onCBBC.”)."
Section 3.2 revision paragraph 2 "The revision stage initializes y = x and iterates through each pair (q e) = (qi eij) in the retrieval result. For each pair, we perform two steps:…")
providing the modified input to a
(Fig1 shows the answer and document being an input.
section 3 approach first paragraph "… Next, the revision stage takes the original text x and the retrieval results (q1 e11) and produces a revised text y(e.g.,“ Millie In between premiered on 1October2014onCBBC.”)."
Section 3.2 revision paragraph 2 "The revision stage initializes y = x and iterates through each pair (q e) = (qi eij) in the retrieval result. For each pair, we perform two steps:…")
See claim five for rationale.
Claims 9 are rejected under 35 U.S.C. 103 as obvious over Wang, Xuezhi, et al. "Self-consistency improves chain of thought reasoning in language models." arXiv preprint arXiv:2203.11171 (2022), in view of US Patent Publication US 20240095460 A1, (Xu; Peng.) in view of Chen, Lingjiao, Matei Zaharia, and James Zou. “Frugalgpt: How to use large language models while reducing cost and improving performance.” arXiv preprint arXiv:2305.05176 (2023). in view of US Patent US 12001462 B1, (Madisetti; Vijay.)
Claim 9
Regarding Claim 9, Wang in view of Xu in view of Chen, do not explicitly teach
9. The method of claim 6, wherein the input to each language model comprises a different prompt.
However, Madisetti teaches the method of claim 6, wherein the input to each language model comprises a different prompt.
(Col 8 lines 34-42 "Referring now to FIG. 13 is an illustration of combining h-LLMs in parallel, is described in more detail. User 1200 enters a prompt in user interface 1202. The prompt 1204 is sent to an AI Input Broker 1206 which generates multiple derived prompts by adding more contextual information. The derived prompts are sent to multiple h-LLMs 1208 which process the prompt in parallel generating multiple results. The AI Output Broker 1210 processes the results and sends the processed results 1212 to the user 1200.")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang in view of Xu in view of Chen to incorporate the teachings of Madisetti to provide a “The method of claim 6, wherein the input to each language model comprises a different prompt.” Doing so would Cause the best result to be sent to the user, as recognized by Madisetti. (col8 lines 4-6).
Claims 11 are rejected under 35 U.S.C. 103 as obvious over Wang, Xuezhi, et al. "Self-consistency improves chain of thought reasoning in language models." arXiv preprint arXiv:2203.11171 (2022), in view of US Patent Publication US 20240095460 A1, (Xu; Peng.) in view of Chen, Lingjiao, Matei Zaharia, and James Zou. “Frugalgpt: How to use large language models while reducing cost and improving performance.” arXiv preprint arXiv:2305.05176 (2023)., in further view of US Patent US 11301466 B2, (Tomita; Yu.)
Claim 11
Regarding Claim 11, Wang in view of Xu in view of Chen, do not explicitly teach the method of claim 6, wherein the method further comprises: receiving a request for an alternative to the answer; and generating a second answer comprising natural language text to the prompt.
However, Tomita teaches 11. The method of claim 6, wherein the method further comprises: receiving a request for an alternative to the answer; and
(Col 21 lines 10-21 " In a case where an answer displayed as the first response displayed on the user terminal 3 corresponds to the intended question, the user 1 selects the displayed FAQ question sentence, and thus an FAQ response sentence is displayed on the user terminal 3. Therefore, the question of the user 1 is solved (step S1103). Meanwhile, when the user 1 determines that the answer displayed as the displayed first response does not correspond to the intended question sentence and selects “Not applicable”, a telegraphic message indicating “Not applicable” is transmitted to the server apparatus 100 (step S1104).")
generating a second answer comprising natural language text to the prompt.
(Col 10 lines 52-64 "Further, in a case where the user 1 desires another answer different from the answer applied alone, the second response is made with a plurality of answers determined in order from the answer rank “2” by using the high-accuracy threshold S2. The accuracy “0.65” in the answer rank “2” is equal to or less than the single application threshold S1, and thus the accuracy “0.55” in the answer rank “3” is added. The total accuracy is “1.20”, which is equal to or larger than the high-accuracy threshold S2. Further, the total number of answers is two, which is equal to or less than the maximum number of answers P1. Therefore, in the second response, two answers in the answer ranks “2” and “3” are displayed on the user terminal 3 as optional sentences. ")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang in view of Xu in view of Chen to incorporate the teachings of Tomita to provide a “The method of claim 6, wherein the method further comprises: receiving a request for an alternative to the answer; and generating a second answer comprising natural language text to the prompt.” Doing so would Make the answer variable according to the accuracy instead of fixed, as recognized by Tomita. (col 4 lines 11-26).
Claims 14 are rejected under 35 U.S.C. 103 as obvious over Wang, Xuezhi, et al. "Self-consistency improves chain of thought reasoning in language models." arXiv preprint arXiv:2203.11171 (2022), in view of US Patent Publication US 20240095460 A1, (Xu; Peng.) in view of Chen, Lingjiao, Matei Zaharia, and James Zou. “Frugalgpt: How to use large language models while reducing cost and improving performance.” arXiv preprint arXiv:2305.05176 (2023). in view of Gao, Luyu, et al. “RARR: Researching and Revising What Language Models Say, Using Language Models.” ArXiv.org, 2022, arxiv.org/abs/2210.08726v2. Accessed 7 Jan. 2026 in further view of Zhengbao Jiang, Frank F. Xu, Jun Araki, Graham Neubig; How Can We Know What Language Models Know?. Transactions of the Association for Computational Linguistics 2020; 8 423–438. doi: https://doi.org/10.1162/tacl_a_00324
Claim 14
Regarding Claim 14, Wang teach
for each language model:
generating a first distribution of a plurality of first intermediate answers by providing the input to the language model multiple times;
(Fig 1 shows marginalizing to aggregate a final answer
Page 1 last paragraph "…we first sample from the language model’s decoder to generate a diverse set of reasoning paths; each reasoning path might lead to a different final answer,…"
page 3 paragraph 1 "Finally, we aggregate the answers by marginalizing out the sampled reasoning paths and choosing the answer that is the most consistent among the generated answers.")
generating a second distribution of a plurality of second intermediate answers and
(Fig 1 shows marginalizing to aggregate a final answer
Page 1 last paragraph "…we first sample from the language model’s decoder to generate a diverse set of reasoning paths; each reasoning path might lead to a different final answer,…"
page 3 paragraph 1 "Finally, we aggregate the answers by marginalizing out the sampled reasoning paths and choosing the answer that is the most consistent among the generated answers.")
generating the answer by performing a probabilistic inference over each distribution.
(Fig 1 shows the final answer in natural language text. Where the final answer comes from marginalizing.
Page 3 paragraph 2 "After sampling multiple (ri ai) from the model’s decoder, self-consistency applies a marginalization over ri by taking a majority vote over ai, i.e., argmaxa m i=1 1(ai = a), or as we defined as the most “consistent” answer among the final answer set.")
Wang in view of Xu in view of Chen do not explicitly teach all of
generating a second prompt that comprises different text with a same meaning as the text of the prompt;
by providing an input comprising at least the set of documents and the second prompt to the language model;
However, Gao teach by providing an input comprising at least the set of documents and the second prompt to the language model;
(Fig1 shows the answer and document being an input.
section 3 approach first paragraph "… Next, the revision stage takes the original text x and the retrieval results (q1 e11) and produces a revised text y(e.g.,“ Millie In between premiered on 1October2014onCBBC.”)."
Section 3.2 revision paragraph 2 "The revision stage initializes y = x and iterates through each pair (q e) = (qi eij) in the retrieval result. For each pair, we perform two steps:…")
See claim five for rationale.
Wang in view of Xu in view of Chen in further view of Gao do not explicitly teach all of generating a second prompt that comprises different text with a same meaning as the text of the prompt;
However, Zhengbao teach
14. The method of claim 6, wherein the method further comprises:
generating a second prompt that comprises different text with a same meaning as the text of the prompt;
(Section 3.2 paraphrasing-based generation "Our second method for generating prompts is more targeted—it aims to improve lexical diversity while remaining relatively faithful to the original prompt. Specifically, we do so by performing paraphrasing over the original prompt into other semantically similar or identical expressions. For example, if our original prompt is ‘‘x shares a border with y’’, it may be paraphrased into ‘‘x has a common border with y’’ and ‘‘x adjoins y’’. This is conceptually similar to query expansion techniques used in information retrieval that reformulate a given query to improve retrieval performance (Carpineto and Romano, 2012).")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang in view of Xu in view of Chen in further view of Gao to incorporate the teachings of Zhengbao to provide a “generating a second prompt that comprises different text with a same meaning as the text of the prompt;” Doing so would Improve lexical diversity , as recognized by Zhengbao. (section 3.2 paraphrasing-based generation).
Claims 22 are rejected under 35 U.S.C. 103 as obvious over Wang, Xuezhi, et al. "Self-consistency improves chain of thought reasoning in language models." arXiv preprint arXiv:2203.11171 (2022), in view of US Patent Publication US 20240095460 A1, (Xu; Peng.) in view of Chen, Lingjiao, Matei Zaharia, and James Zou. “Frugalgpt: How to use large language models while reducing cost and improving performance.” arXiv preprint arXiv:2305.05176 (2023) in further view of Cobbe, Karl, et al. "Training verifiers to solve math word problems." arXiv preprint arXiv:2110.14168 (2021).
Claim 22
Regarding Claim 22, Wang in view of Xu in further view of Chen do not explicitly teach all of 22. (New) The method of claim 21, further comprising generating the training set by: for each of one or more of the intermediate answers, determining whether the intermediate answer meets a criteria; and for each of the one or more intermediate answers, in response to determining that the intermediate answer does not meet the criteria, including the intermediate answer in the training set.
However, Cobbe teaches
22. (New) The method of claim 21, further comprising generating the training set by:
for each of one or more of the intermediate answers, determining whether the intermediate answer meets a criteria; and
(page 2 introduction "…We propose training verifiers to evaluate the correctness of model generated solutions, similar to concurrent work by Shen et al. (2021a). At test time, we sample a fixed number of candidate solutions and select the solution ranked highest by the verifier. Verifiers benefit both from their inherent optionality and from verification being a simpler task than generation in general."
Page 5 section 4 methods "We investigate two methods to solve problems in GSM8K: finetuning and verification. Finetuning, our baseline method, uses the same language modeling objective as the generative pretraining in GPT-3 (Brown et al., 2020). At test time, we judge performance by autoregressively sampling a single low temperature solution and checking whether the final answer is correct. In contrast, verification consists of sampling multiple high temperature solutions, assigning each solution a score, and outputting the highest ranked solution. Verifiers are trained to judge the correctness of solutions, with the training signal determined solely by whether or not the solution reached the correct final answer."
Page 8 section 4.2 verification "2. Sample 100 completions from the generator for each training problem and label each solution as correct or incorrect.")
for each of the one or more intermediate answers, in response to determining that the intermediate answer does not meet the criteria, including the intermediate answer in the training set.
(Page 8 section 4.2 verification "As shown in Figure 4, we train the verifier as follows: 1. Finetune a model (the “generator”) for 2 epochs on the training set. 2. Sample 100 completions from the generator for each training problem and label each solution as correct or incorrect. 3. Train a verifier for a single epoch on this dataset.")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang in view of Xu to incorporate the teachings of Chen to provide a “22. (New) The method of claim 21, further comprising generating the training set by: for each of one or more of the intermediate answers, determining whether the intermediate answer meets a criteria; and for each of the one or more intermediate answers, in response to determining that the intermediate answer does not meet the criteria, including the intermediate answer in the training set.” Doing so would Prevent catastrophic mistakes, as recognized by Chen. (page 1 section 1 introduction).
Reference Cited
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Prystawski, Ben, Michael Li, and Noah Goodman. "Why think step by step? reasoning emerges from the locality of experience." Advances in Neural Information Processing Systems 36 (2023): 70926-70947. to Prystawski et al. discloses generating intermediate answers and evaluating them.
Chen, Angelica, et al. "Two failures of self-consistency in the multi-step reasoning of LLMs." arXiv preprint arXiv:2305.14279 (2023). to Chen et al. discloses evaluating answers.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALI M HASSAN whose telephone number is (571)272-5331. The examiner can normally be reached Monday - Friday 8:00am - 4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Paras Shah can be reached at (571)270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALI M HASSAN/Examiner, Art Unit 2653
/Paras D Shah/Supervisory Patent Examiner, Art Unit 2653
07/02/2026
Prosecution Insights