Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under, including the fee set forth in 37 CFR1.17(e), was filed in this application after final rejection. Since this application is eligiblefor continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e)has been timely paid, the finality of the previous Office action has been withdrawnpursuant to 37 CFR 1.114. Applicant's submission filed on 05/19/2026 has been entered.
Status of the Claims
Claims 1-22 are pending.
Response to Applicant’s Arguments
In response to “For example, on page 8 of the Office Action, the Office argued that Liu teaches the elements of "wherein the response scores include a first response score and the first response score reflects an accuracy of the first response in relation to an expected output format" and cited paragraphs 35 and 37-38 of Liu. The Office further argued that the cited passages teach the element since the passage describes (emphasis added) "a prompt provides a desired output format 204 such as instructing LLM 120 to effectively identify answerable passages and an answering format example 222, a reasoning and output format 224, and a demonstrative example 226 that initially identifies the relevant information and then summarize the relevant information like chain of thought and generate the final answer, i.e., select the most accurate final answer/response according to the rating/ranking of the equation in paragraphs 32-33."” And “The cited paragraphs and elements of Liu relied on by the office are not supported by Liu's provisional application. Liu's provisional application does not include Figs. 2A-2C nor descriptions of "output format 204", "answering format example 222", or "reasoning and output format 224." Liu's provisional fails to include any mention of "format" at all. Likewise, Liu's provisional application provides no examples of the equation referenced in paragraphs 32-33. Thus, the Office provided improper citation elements in the later published application, which are not found in the provisional application”.
Exemplary claim 1 recites “creating a prompt for a language model, the prompt comprising:…an expected output format, wherein the expected output format requests a quotation of the first reference document or a citation to the first reference document”.
Under the broadest reasonable interpretation, the prompt’s expected output format requests the language model to either output (1) a quotation of the first reference document or (2) a citation to the first reference document.
Liu discloses “In FIG. 2B, the prompt includes an answer format example 212, a reasoning and output format 214, and a demonstrative example 216 that shows the reasoning is to prune irrelevant passages and using the few remaining relevant ones for answer generation. Specifically, this prompt may instruct LLM 120 to effectively identify answerable passages through a process of selective elimination. As a result, the demonstration 216 involves differentiating irrelevant passages from the ones that can provide an answer, and subsequently generating the final response based on the few relevant passages” (¶37).
In support of the above disclosure, Appendix I discloses “We initially focus on two CoT strategies incorporating in-context learning, demonstrating a pruning strategy and a selection strategy on retrieved passages to guide the LLM’s answer generation” (lines 082-086).
Specifically, for “Prune Prompt”: “This necessitates the LLM to accurately identify answerable passages through a process of elimination. Consequently, the demonstration involves discerning irrelevant passages to the question at hand, pinpointing the appropriate passage that can provide an answer, and consequently delivering the final response” (lines 141-147) where relevant documents or text passages are fetched to aid the answer generation process (lines 033-035).
Therefore, Appendix I fully supported Liu regarding the prompt comprising a prune prompt to instruct or guide the LLM’s answer generation to “discerning irrelevant passages to the question at hand, pinpointing the appropriate passage that can provide an answer”.
By guiding or instructing the LLM to pinpoint the appropriate passage that can provide the answer, the prompt’s pruning guidance or instruction is an expected output format because the pruning guidance or instruction requested the LLM to provide “(1) a quotation of a first reference document or (2) a citation to the first reference document” by pinpointing the appropriate passage that can provide the answer.
In response to “Likewise, Liu's provisional application provides no examples of the equation referenced in paragraphs 32-33. Thus, the Office provided improper citation elements in the later published application, which are not found in the provisional application” and “Assuming arguendo that Liu provides support in the provisional, Liu still fails to teach at least that "the first response score reflects an accuracy of the first response in relation to the expected output format," as recited in claim 1. The Office action cited paragraphs 32-33 of Liu as allegedly teaching the above-cited elements. However, Liu's cited majority-voting disclosure concerns selecting a final answer from an answer pool generated from multiple retrieved source documents. Liu explains that each retrieved source document is independently fed with the input question to the LLM to generate candidate answers, and those generated answers form an answer pool (see paragraph 32). Liu then states that a majority voting mechanism may be applied to the answer pool to determine the final answer, for example, by a human evaluator or by another LLM selecting the "best answer" (see paragraph 33). Nothing in Liu describes computing a score for a response that measures whether that response accurately complies with an expected output format as claimed”.
Liu discloses “…a retriever model may select top-K relevant passages in response to an input question. An LLM may then generate a respective answer using each selected passage, respectively, to form an answer pool. A language model may then rate and/or rank the answers in the pool to generate the final response to the input question” (¶21) and “…after the retriever model 110 retrieves top-k source documents 112a-n, each source document 112a-n is independently fed, together with the input question 102, to the LLM 120, to each generate a candidate answer. The generated answers form an answer pool 122” (¶32).
In support of the above disclosure, Appendix I discloses at Fig.1:
PNG
media_image1.png
486
626
media_image1.png
Greyscale
Further, Liu discloses (¶33): “In one embodiment, a majority voting mechanism may be then applied to this answer pool 122 to determine the final answer 125”:
PNG
media_image2.png
28
396
media_image2.png
Greyscale
In support of the above disclosure, Appendix I discloses in “First Concatenation Second Separate”: “initially, we employ the concatenation method to obtain the predicted answer from LLMs. If the LLM determines that the input passages are unable to provide a response to the question, we then proceed to the second round where we utilize a separate approach to predict the answer pool. Finally, we employ a majority vote system to select the final answer.” (lines 165-172).
Therefore, even if Appendix I does not provide support for the majority vote equation
PNG
media_image2.png
28
396
media_image2.png
Greyscale
, Appendix I fully supports Liu’s disclosure to use a majority vote mechanism to rate and/or rank the answers in the answer pool to generate the final answer to the input question.
Furthermore, the answers in the answer pool being rated or ranked were the results of pruning prompt instructing the LLM to guide the LLM’s answer generation (i.e., expected output format) pinpointing the appropriate passage that can provide an answer (Liu, ¶37 “Specifically, this prompt may instruct LLM 120 to effectively identify answerable passages through a process of selective elimination. As a result, the demonstration 216 involves differentiating irrelevant passages from the ones that can provide an answer, and subsequently generating the final response based on the few relevant passages”; Appendix I, lines 141-147, “Prune Prompt”: “This necessitates the LLM to accurately identify answerable passages through a process of elimination. Consequently, the demonstration involves discerning irrelevant passages to the question at hand, pinpointing the appropriate passage that can provide an answer, and consequently delivering the final response”).
Therefore, when rating / ranking the answers in the pool that are appropriate passages pinpointed by the LLM by a majority vote mechanism to generate the final response to the input question (Liu, ¶21 and ¶33), the respective answers’ rating / ranking measures how effectively the pruning prompt instructed or guided (i.e., requested) LLM 120 to identify answerable passages through a process of selective elimination (Liu, ¶37; Appendix I, lines 084-086 “demonstrating a pruning strategy…to guide the LLM’s answer generation”) or how accurately the Prune Prompt necessitates the LLM to identify answerable passages through a process of elimination (Appendix I, lines 141-143).
Say it another way, because the prompt pruning instruction or guidance to the LLM to selectively eliminate irrelevant passages while pinpointing the appropriate passage (i.e., prompt pruning is the expected output format) to provide answers forming the answering pool, the rating / ranking of the answers in the pool rates or ranks or otherwise scores how effectively or accurately the prompt pruning instruction / guidance (i.e., expected output format) guided the LLM’s answer generation to generate the correct answer to the input question.
Claim Rejections - 35 USC § 103
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 103 that form the basis for the rejections under this section made in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 3-21 are rejected under 35 USC 103(a) as being unpatentable over Xu et al. (US 2024/0095460 A1) in view of Liu et al. (US 2024/0428044 A1 as supported by Appendix I 63/510074).
Regarding Claims 1, 11, and 18, Xu discloses a system (¶32 and Fig. 1, dialogue systems), comprising:
at least one server computer (¶179 and Fig. 9D, server 978) comprising at least one processor (¶179, CPU 980) and at least one memory (¶186, memory 1004), the at least one server computer configured to:
receive a natural language question (¶¶33-34, processing audio data 104 to generate text data 106 (per ¶35, audio data 202 / text data 204 and transcript 206) representing a task / question being requested by the user);
determine that the natural language question relates to a first reference document (¶¶47-48, a retrieval component 114 retrieves contextual data 116 associated with text data 106 from information database 112 (which stores manual, document, webpages per ¶47) based on contextual information being related to the same topic, component, feature as the question represented by text data 106);
compute an embedding vector for the natural language question, wherein the embedding vector represents the natural language question in a vector space (¶41, generate embeddings 204 based on text data 204 representing transcript 206);
select one or more question-and-answer pairs from a set of available question-and-answer pairs using the embedding vector (¶39, retrieve one or more question / answer pairs that are associated with the text data 106 from information database 112 (per ¶36, database 112 stores a number of question / answer pairs) based on a question / answer pair is associated with the text data 106 as being related to the same topic, component, feature; ¶42, retrieval component 108 uses embedding 404 associated with text data 204 and embeddings 402 associated with question / answer pairs to retrieve a threshold number of question / answer pairs);
create a prompt for a language model (Fig. 1, prompt component 118 and language model 124), the prompt comprising:
a representation of the natural language question, a representation of the one or more question-and-answer pairs, and an expected output format, wherein the expected output format requests a quotation of the first reference document or a citation to the first reference document (¶58, prompt component 118 uses text data 106, question / answer data 110, and contextual data 116 to generate prompt data 122 by ordering text from text data 106, question / answer data 110, and contextual data 116 using a given order; in view of ¶49 and ¶51, Fig. 5, contextual information comprising portions 506 where each portion 506 is a quotation or citation of OEM manual 502; ¶¶61-62, prompt data 122 comprising contextual data 116 prompts the language model to output contextual data 120; i.e., the prompt data 122 prompts the language model to output contextual data 120 comprising at least one quotation or citation of OEM manual 502 corresponding to audio data 104 including a question about a component of the vehicle);
submit the prompt to the language model (¶61, input the prompt data into a language model);
receive a plurality of responses from the language model, the plurality of responses including a first response (¶71, the language model 124 processes prompt data 122 to generate output data 126 associated with the question).
Xu does not teach compute response scores for the plurality of responses.
Liu teaches a question answering framework that generates a plurality of responses / answers to an input question based on retrieving a plurality of supporting documents in parallel and selects one or more relevant answers as final response (¶21; supported by Appendix I, Fig. 1);
the framework creates a prompt for a language model, the prompt comprising an expected output format requesting a quotation of a first reference document or a citation to the first reference document (¶37, the prompt shows the reasoning is to prune irrelevant passages and using the few remaining relevant ones for answer generation; see Appendix I, lines 141-143, “Prune Prompt” necessitates the LLM to accurately identify answerable passages through a process of elimination: discerning irrelevant passages to the question at hand, pinpointing the appropriate passage that can provide an answer (i.e., quotation or citation to a reference document), and consequently delivering the final response; the prune prompt guides or instructs the LLM to have an expected output format comprising a quotation or citation to the first reference document);
submitting the prompt to the language model to receive a plurality of response including a first response (¶37, prompt instructs LLM 120 to effectively identify answerable passages through a process of selective elimination; Appendix I, lines 141-147, “Prune Prompt” necessitates the LLM to accurately identify answerable passages through a process of elimination);
compute response scores for the plurality of responses (¶21, a language model may then rate or rank the answers in the pool to generate the final response to the input question; ¶33, apply majority voting mechanism to answer pool 122 to determine the final answer 125; Appendix I, lines 165-172, “First Concatenation Second Separate”: “we utilize a separate approach to predict the answer pool. Finally, we employ a majority vote system to select the final answer”), wherein the response scores include a first response score and the first response score reflects an accuracy of the first response in relation to the expected output format (¶21 and ¶33, when forming an answer pool of answers comprising the relevant passages and then rating / ranking the answers (i.e., answer / response scoring) in the pool to generate the final answer by majority voting mechanism, the rating / ranking measures how effectively (¶37, “this prompt may instruct LLM 120 to effectively identify answerable passages”) or accurately (Appendix I, line 136 “Prune Prompt”, lines 141-143 “this necessitates the LLM to accurately identify answerable passages”) the prune prompt guided or instructed the LLM to generate the few relevant passages / pinpointed appropriate passages as the expected output format of quotation / citation to the first reference document);
select the first response using the response scores (¶¶32-33, select a most accurate final answer using a majority voting mechanism; see Appendix I, lines 170-172, “Finally, we employ a majority vote system to select the final answer”); and
determine an answer to the natural language question using the first response (¶39, select the final answer; Appendix I, lines 170-172).
It would’ve been obvious to one ordinarily skilled in the art before the effective filing date of the invention to compute response scores reflecting accuracy of respective response to an expected output format in order to generate the most accurate final answer (Liu, ¶32).
Further regarding claim 18, Xu discloses one or more non-transitory, computer-readable media comprising computer-executable instructions that, when executed, cause at least one processor to perform actions of claims 1 and 11 (¶190, memory 1004 storing computer readable instructions for CPU 1006 of ¶188).
Regarding Claim 3, Liu discloses wherein computing the first response score comprises determining an inclusion of a quotation or a citation from the first reference document (Liu, ¶37, instruct LLM 120 to effectively identify answerable passages through a process of selective elimination; in view of ¶21, forming the answer pool using the passages and then rate and / or rank the answers in the pool means the rating and ranking answerable passages (per Appendix I, lines 145-146, pinpointed appropriate passage that can provide an answer)).
Regarding Claim 4, Liu discloses wherein computing the first response score comprises verifying content of the quotation in the first reference document (Liu, ¶39, LLM 120 determines that input passages 112a-n are unable to provide an answer to question 102, produce an answer pool by independently generating an answer for each passage 112a-n and apply an LLM prediction to select the final answer from answer pool; see Appendix I, lines 167-172 and Fig. 1).
Regarding Claim 5, Xu discloses wherein selecting the first response comprises creating a second prompt for a second language model, wherein the second prompt includes the representation of the natural language question and the first response (¶62, if user continues to ask questions associated with the vehicle, the prompt component 118 uses the contextual data 120 to continue generating the prompt data 122 for the questions, where the contextual data 120 represents a context associated with outputs to previous questions; alternatively, ¶64, process 100 to use a question represented by text data 106 to prompt the language model 124 to generate output data for comparison with answer from testing question / answer pair to determine system accuracy).
Regarding Claim 6, Xu discloses wherein the second prompt asks the second language model to determine a validity of the first response to the natural language question (¶¶64-65, perform process 100 using question represented by text data 106 to generate output data 126 associated with the question and compare the answer from a testing question / answer pair to the answer represented by output data 126 to determine if the dialogue system is accurate).
Regarding Claims 7 and 19, Xu discloses wherein: the prompt comprises at least a portion of a reference document or a link to the reference document (¶58, the prompt data 122 includes contextual data 116; ¶¶47-48, contextual data 116 associated with text data 106 corresponds to manual, document, and webpage in information database 112); and
selecting the first response comprises creating a second prompt for a second language model (¶¶63-64, when determine whether the dialogue system is accurate, using a question represented by text data 106 to prompt language model(s) 124 (i.e., more than one LM or second LM; compare to LM 126 in Fig. 1B of Liu and Appendix I, Fig. 1, multiple LLMs in parallel in the Separate Approach) to generate output data 126 associated with the question), wherein the second prompt includes the representation of the natural language question and the first response (¶64, the question is from a testing question / answer pair so as to compare the text answer with answer represented by output data 126; i.e., the prompt to language model (s) 124 or a second LM 124 comprises the question represented by text data 106 and a corresponding test answer / response, and the response represented by output data 126 to perform a validation process).
Regarding Claim 8, Xu discloses wherein: wherein the second prompt asks the second language model to verify that the first response is consistent with the reference document (¶64, compare the answer from the testing question / answer pair to the answer represented by the output data 126; see also Liu, Fig. 1B, ¶33, LLM 120 generates respective answers including the first answer / first response as prompt to LLM 126 to select the most accurate answer as the final answer; Appendix I, lines 175-187, Separation as the filter then Concatenation (Fig. 1): using LLMs in the Separation Approach of Fig. 1 to curate a pool of potential answers as the first round and then using the LLM in the Concatenation Approach to choose the final answer).
Regarding Claim 9, Xu discloses determining pair scores for the set of available question-and-answer pairs (¶43, determine scores for the question / answer pairs using embedding 404 and embeddings 402); and
selecting the one or more question-and-answer pairs using the pair scores (¶46, determine a first question / answer pair with the highest score that is most relevant to the text data 106).
Regarding Claim 10, Xu discloses wherein determining the pair scores comprises at least one of:
determining a similarity of a question-and-answer pair to the natural language question (¶¶41-42, use embedding 404 associated with text data and embeddings 402 associated with question / answer pairs to identify the question / answer pairs that are the most similar to the transcript 206 represented by the text data 204; ¶43, determine scores 406(1)-(6) for the question / answer pairs to identify the question / answer pairs that are most similar to the transcript 206);
determining a number of hallucinations generated by the language model when a question-and-answer pair was used in a previous prompt;
determining a number of citations generated by the language model when a question-and-answer pair was used in a previous prompt; or
determining a number of times a question-and-answer pair was used in a previous prompt.
Regarding Claim 12, Xu discloses wherein the at least one server computer is further configured to create the prompt for the language model by including the representation of the natural language question and the representation of the one or more question-and-answer pairs (¶58, prompt component 118 uses text data 106, question / answer data 110, and contextual data 116 to generate prompt data 122) as a dialogue history with the language model (¶62, prompt data 122 includes contextual data representing a context associated with outputs to previous questions).
Regarding Claim 15, Xu discloses wherein the at least one server computer is further configured to present, at a user interface, the answer to the natural language question (¶71 and ¶75, vehicle interface to provide the information by displaying content associated with output data 126).
Regarding Claim 16, Xu discloses wherein computing the response scores comprises obtaining a plurality of response embedding vectors for the plurality of responses (¶49 and ¶53, Fig. 5, extracting contextual information by transforming words from contextual portions 506 to dense vectors / embeddings 508; ¶62, LM 124 outputs contextual data 120 (i.e., embedding vectors) as part of output data 126; compare Liu, Fig. 1B, output data 126 comprises embeddings 508(1)-(6) to constitute the answer pool 122).
Regarding Claim 17, Xu discloses wherein obtaining the plurality of response embedding vectors comprises querying a third-party service (¶216, extract contextual information from private or public cloud storages).
Regarding Claim 20, Xu discloses wherein at least one of the one or more question-and-answer pairs relates to a second reference document (¶47, information may include text from multiple OEM manuals associated with multiple models of vehicles).
Regarding Claim 21, Xu discloses wherein determining the answer comprises removing a quotation or a citation from the first response (¶48, retrieving a threshold amount of contextual information for output as a portion of output data 126 means removing the remaining amount of contextual information from the manuals as possible response for output data 126).
Claims 2 and 22 are rejected under 35 USC 103(a) as being unpatentable over Xu et al. (US 2024/0095460 A1) in view of Liu et al. (US 2024/0428044 A1) as applied to claims 1 and 18, in further view of Bolcer et al. (US 2025/0086211 A1).
Regarding Claims 2 and 22, Xu does not disclose wherein computing the first response score comprises determining a number or severity of hallucinations in the first response.
Bolcer discloses a system receiving user query comprising a question (¶27) to prompt a LLM to generate an answer / response (¶29 and ¶45). The system computes response scores comprising determining a number or severity of hallucinations in a response (¶51, leverage an LLM to generate responses based on user’s original query; ¶53, verify the accuracy of system generated answers by evaluating their similarity to a cluster of semantically related documents; ¶54, vectorized each record’s clusters to generate a respective answer and compare to the generated answer to determine a respective degree of similarity or hallucination score) and causing the language model to regenerate a response when the response scores are below a threshold value (Fig. 4 and claim 5, determine if hallucination is acceptable as compared to a threshold; e.g., ¶78, determine whether or not to rewrite prompt at 403A (or at 403B per ¶80) into something that has a lower hallucination prediction).
It would’ve been obvious to one ordinarily skilled in the art before the effective filing date of the invention to at least determine a severity of hallucinations in the first response to compute at least the first response score and causing the language model to regenerate a response when the response scores are below a threshold value in order to determine when the language model will fail and became hallucinatory according to similarity based validation (Bolcer, ¶28).
Allowable Subject Matter
Claims 13-14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims:
Regarding Claims 13-14, Xu discloses wherein the at least one server computer is further configured to:
create and submit a second prompt to a second language model (¶64, process 100 to use a question represented by text data 106 to prompt the language models 124 (i.e., at least a second language model 124) to generate output data for comparison with answer from testing question / answer pair to determine system accuracy; compare Liu, Fig. 1B, this is parallel to LLM 126 / second LM to select the most accurate answer or Appendix I, lines 175-187, where LLM in the “separate approach” creates answering pools of answers (rated and ranked by majority vote mechanism) as prompts to the LLM in the “concatenation approach” to choose the final answer).
However, Xu and Liu combination does not disclose or render obvious receive a second plurality of responses from the second language model;
compute second response scores for the second plurality of responses; and
select the first response from the plurality of responses and the second plurality of responses.
Specifically, the configuration of Liu at Fig. 1B applies major vote mechanism to rate and rank the answers in the pool and then prompts the LLM 126 to generate the final answer without second response scores.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to examiner Richard Z. Zhu whose telephone number is 571-270-1587 or examiner’s supervisor Hai Phan whose telephone number is 571-272-6338. Examiner Richard Zhu can normally be reached on M-Th, 0730:1700.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RICHARD Z ZHU/Primary Examiner, Art Unit 2654 06/06/2026
Prosecution Insights