CTNF 18/947,752 CTNF 101717 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claim Rejections - 35 USC § 112 07-30-02 AIA The following is a quotation of 35 U.S.C. 112(b): (b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention. The following is a quotation of 35 U.S.C. 112 (pre-AIA), second paragraph: The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention. 07-34-01 Claim 6, 12 and 18 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Regarding claim 6, it displays multiple undefined variables have been newly introduced in the limitation without a proper description, complicating the understanding of the scope of the claim. More guidance on the meaning of each variable in the claim is needed. Appropriate correction is required. Regarding claim 12, arguments analogous to claim 6 are applicable. Regarding claim 18, arguments analogous to claim 6 are applicable. Claim Rejections - 35 USC § 101 07-04-01 AIA 07-04 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claim 6, 12 and 18 rejected under 35 U.S.C. 101 as being directed to a patent-ineligible subject matter. The dependent claim 6 include all the aspects of claim 1. Limitation of claim 1 include: accessing LLM responses that include a query, the answers to the query, citations and evidence passages and determine a grounding quality score, then tuning the LLM to satisfy a grounding quality score. Those steps, with the exception of tuning the LLM to satisfy a grounding quality score, constitute an abstract idea directed to a mental process that can be executed by a human mentally or using pen and paper, which is a judicial exception to patent eligibility. A human can read a query, the answers to the query, citations and evidence passages and determine a grounding quality score. In the case of claim 1 tuning the LLM to satisfy a grounding quality score could be considered a practical application. In claim 6 the tuning aspect is further narrowed down to a mathematical formula that maximize certain elements, which transform the tuning aspect into a mathematical concept. Rendering a mathematical concept applied to a mental process, which under MPEP 2106 is considered an abstract idea. The claim recites additional elements such as a generic LLM, which is general purpose software being used as a tool to implement the mental process and mathematical calculation. Thus, the additional element doesn’t integrate the mathematical concept or mental process into a practical application and the sole purpose of the system is not significantly more than performing the mathematical concept and the mental steps listed. The dependent claim 12 include all the aspects of claim 7. Limitation of claim 7 include: accessing LLM responses that include a query, the answers to the query, citations and evidence passages and determine a grounding quality score, then tuning the LLM to satisfy a grounding quality score. Those steps, with the exception of tuning the LLM to satisfy a grounding quality score, constitute an abstract idea directed to a mental process that can be executed by a human mentally or using pen and paper, which is a judicial exception to patent eligibility. A human can read a query, the answers to the query, citations and evidence passages and determine a grounding quality score. In the case of claim 7 tuning the LLM to satisfy a grounding quality score could be considered a practical application. In claim 12 the tuning aspect is further narrowed down to a mathematical formula that maximize certain elements, which transform the tuning aspect into a mathematical concept. Rendering a mathematical concept applied to a mental process, which under MPEP 2106 is considered an abstract idea. The claim recites additional elements such as an LLM, one or more computers and one or more storage devices, which are generic programs and generic computer components being used as tool to implement the abstract idea. They don’t integrate the mathematical concept or mental process into a practical application and the sole purpose of the system is not significantly more than performing the mathematical concept and the mental steps listed. The dependent claim 18 include all the aspects of claim 13. Limitation of claim 13 include: accessing LLM responses that include a query, the answers to the query, citations and evidence passages and determine a grounding quality score, then tuning the LLM to satisfy a grounding quality score. Those steps, with the exception of tuning the LLM to satisfy a grounding quality score, constitute an abstract idea directed to a mental process that can be executed by a human mentally or using pen and paper, which is a judicial exception to patent eligibility. A human can read a query, the answers to the query, citations and evidence passages and determine a grounding quality score. In the case of claim 13 tuning the LLM to satisfy a grounding quality score could be considered a practical application. In claim 18 the tuning aspect is further narrowed down to a mathematical formula that maximize certain elements, which transform the tuning aspect into a mathematical concept. Rendering a mathematical concept applied to a mental process, which under MPEP 2106 is considered an abstract idea. The claim recites additional elements such as an LLM, a computer storage medium and one or more computers, which are generic programs and generic computer components being used as tool to implement the abstract idea. They don’t integrate the mathematical concept or mental process into a practical application and the sole purpose of the system is not significantly more than performing the mathematical concept and the mental steps listed. Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-23-aia AIA The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or nonobviousness. 07-20-02-aia AIA This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention. 07-21-aia AIA Claim s 1, 5, 7, 11, 13 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Bolcer; Gregory Alan et al. (US 20250086211 A1) hereinafter BOLCER in view of Dingliwal; Saket et al. (US 20250005298 A1) hereinafter DINGLIWAL . Regarding claim 1, BOLCER teaches: A computer-implemented method, comprising: accessing responses from a large language model, each response comprising data including: a query; “The query 101 is a prompt that comprises the first input given to a language model. The query 101 may comprise questions, tasks, instructions or a series of answers and responses either by the user or by another system. A prompt template may contain instructions to guide the language model, a set of few-shot examples to help the language model generate a better response and/or specific questions directed at the language model.” (BOLCER [0029]). an answer to the query, the answer comprising one or more statements; “The query 101 is a prompt that comprises the first input given to a language model. The query 101 may comprise questions, tasks, instructions or a series of answers and responses either by the user or by another system. A prompt template may contain instructions to guide the language model, a set of few-shot examples to help the language model generate a better response and/or specific questions directed at the language model.” (BOLCER [0029]). “The grounded Q&A 303 leverages an LLM to generate responses based on the user's original query 101 and documents 107 retrieved by the system. This multi-step process involves the evaluation of documents 107, extraction of salient phrases and the construction of a coherent response. ” (BOLCER [0051]). determining a grounding quality of the answer based on the evidence passages and the statements using an attribution evaluation model, wherein the grounding quality is a quantification of attribution of the statements in the answer to a document corpus; “The truthfulness validator 109 comprises a vector store 301, a grounded Q&A 303 and a similarity validator 305 for cross-verifying the reliability and accuracy of system-generated answers 111. Through confirmation of answer consistency with the content of semantically similar documents 107, the similarity validator 305 enhances the overall quality and trustworthiness of the system's outputs 111, proposing a confidence score (or a hallucination score) to the final answer 111. ” (BOLCER [0049]). BOLCER does not teach, but DINGLIWAL teaches: and citations linking each statement to an evidence passage in a corpus; “FIG. 5 illustrates an example input record which may be used for instruction fine-tuning of a pre-trained LLM to obtain an annotated summarization LLM that provides evidence for its summaries of conversations, according to at least some embodiments. In the depicted embodiment, a set of annotated summary examples 502, at least some of which include instruction prompts may be used for instruction fine-tuning 510 of a pre-trained LLM 512. A given instruction prompt 503 included in one of the examples may provide guidance in natural language to the pre-trained LLM, such as “summarize the conversation with numeric citations indicating the evidence for each sentence in the summary”.” (DINGLIWAL [0050]). tuning the large language model to obtain an adapted large language model that satisfies a grounding constraint based on a grounding quality score. “FIG. 5 illustrates an example input record which may be used for instruction fine-tuning of a pre-trained LLM to obtain an annotated summarization LLM that provides evidence for its summaries of conversations, according to at least some embodiments. ” (DINGLIWAL [0006]). “Several different evidence mapping LLMs may be fine-tuned in some embodiments, e.g., using respective data sets or respective model architectures and algorithms. These alternative LLMs, each of which may be considered a candidate for production use, may be evaluated relative to one another automatically using other machine learning models such as question-generated models, question-answering models, and textual entailment models in various embodiments, without requiring tedious and error-prone evaluation by human teams. These other models, referred to as evaluation models, may themselves comprise LLMs in some implementations. The evaluation models may be provided examples of the transcripts and corresponding annotated summaries generated by the different candidate evidence mapping models, and generate respective quality scores for each candidate evidence mapping model based on analysis of the examples. ... ” (DINGLIWAL [0021]). It would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to include in the teachings of BOLCER the capability to provide evidence citations in the LLM output and tune the LLM based on the grounding quality score. The benefit and motivation of such modification is discussed by DINGLIWAL in the following portion: “The present disclosure relates to methods and apparatus for reducing the probability of hallucinations in the output produced by large language models (LLMs), such as LLMs that are utilized to generate summaries of doctor-patient conversations, by requiring the LLMs to provide evidence for various subsets of their output, and by evaluating the evidence-providing LLMs in an automated data-driven manner to select the LLMs that are least likely to generate hallucinations. ... ” (DINGLIWAL [0019]). Regarding claim 5, the rejection of claim 1 is incorporated, furthermore BOLCER teaches: BOLCER does not teach, but DINGLIWAL teaches: The computer-implemented method of claim 1, wherein determining a grounding quality of the answer based on the evidence passages and the statements using an attribution evaluation model comprises determining a sum of attribution evaluation model outputs, each output based on an evidence passage and statement provided as input to the attribution evaluation model. “Several different evidence mapping LLMs may be fine-tuned in some embodiments, e.g., using respective data sets or respective model architectures and algorithms. These alternative LLMs, each of which may be considered a candidate for production use, may be evaluated relative to one another automatically using other machine learning models such as question-generated models, question-answering models, and textual entailment models in various embodiments, without requiring tedious and error-prone evaluation by human teams. These other models, referred to as evaluation models, may themselves comprise LLMs in some implementations. The evaluation models may be provided examples of the transcripts and corresponding annotated summaries generated by the different candidate evidence mapping models, and generate respective quality scores for each candidate evidence mapping model based on analysis of the examples. ... ” (DINGLIWAL [0021]). “In at least some embodiments, an aggregation of quality scores (such as a weighted mean quality score) from different evaluation methodologies such as the TE-based methodology and the QGA methodology may be used to generate an overall quality score for each EMM which is evaluated. ” (DINGLIWAL [0033]). It would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to include in the teachings of BOLCER the capability to provide evidence citations in the LLM output and generate a sum of the scores obtained. The benefit and motivation of such modification is discussed by DINGLIWAL in the following portion: “… The aggregated score may, for example, be the mean of the scores generated for the different answer pairs in one implementation; in other implementations, statistics other than the mean may be used. The aggregated summary scores may be used as quality metrics to rank the EMMs relative to each other, with higher aggregated scores indicating higher quality and hence higher rank (element 1619) in the depicted embodiment. …”(DINGLIWAL [0096]). Regarding claim 7, BOLCER teaches: A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the operations of: accessing responses from a large language model, each response comprising data including: a query; “The query 101 is a prompt that comprises the first input given to a language model. The query 101 may comprise questions, tasks, instructions or a series of answers and responses either by the user or by another system. A prompt template may contain instructions to guide the language model, a set of few-shot examples to help the language model generate a better response and/or specific questions directed at the language model.” (BOLCER [0029]). “This disclosure describes a system and method, using one or more processors, for grounding large language models using real-time content feeds and reference data. The system is able to receive/generate queries, prompt an AI model and evaluate the answer given by the AI model. A hallucination score is generated according to the evaluation of the answer. According to the hallucination score, the AI model may be iteratively accessed to improve the truthfulness of the answer.” (BOLCER [ABSTRACT]). an answer to the query, the answer comprising one or more statements; “The query 101 is a prompt that comprises the first input given to a language model. The query 101 may comprise questions, tasks, instructions or a series of answers and responses either by the user or by another system. A prompt template may contain instructions to guide the language model, a set of few-shot examples to help the language model generate a better response and/or specific questions directed at the language model.” (BOLCER [0029]). “The grounded Q&A 303 leverages an LLM to generate responses based on the user's original query 101 and documents 107 retrieved by the system. This multi-step process involves the evaluation of documents 107, extraction of salient phrases and the construction of a coherent response. ” (BOLCER [0051]). determining a grounding quality of the answer based on the evidence passages and the statements using an attribution evaluation model, wherein the grounding quality is a quantification of attribution of the statements in the answer to a document corpus; “The truthfulness validator 109 comprises a vector store 301, a grounded Q&A 303 and a similarity validator 305 for cross-verifying the reliability and accuracy of system-generated answers 111. Through confirmation of answer consistency with the content of semantically similar documents 107, the similarity validator 305 enhances the overall quality and trustworthiness of the system's outputs 111, proposing a confidence score (or a hallucination score) to the final answer 111. ” (BOLCER [0049]). BOLCER does not teach, but DINGLIWAL teaches: and citations linking each statement to an evidence passage in a corpus; “FIG. 5 illustrates an example input record which may be used for instruction fine-tuning of a pre-trained LLM to obtain an annotated summarization LLM that provides evidence for its summaries of conversations, according to at least some embodiments. In the depicted embodiment, a set of annotated summary examples 502, at least some of which include instruction prompts may be used for instruction fine-tuning 510 of a pre-trained LLM 512. A given instruction prompt 503 included in one of the examples may provide guidance in natural language to the pre-trained LLM, such as “summarize the conversation with numeric citations indicating the evidence for each sentence in the summary”.” (DINGLIWAL [0050]). tuning the large language model to obtain an adapted large language model that satisfies a grounding constraint based on a grounding quality score. “FIG. 5 illustrates an example input record which may be used for instruction fine-tuning of a pre-trained LLM to obtain an annotated summarization LLM that provides evidence for its summaries of conversations, according to at least some embodiments. ” (DINGLIWAL [0006]). “Several different evidence mapping LLMs may be fine-tuned in some embodiments, e.g., using respective data sets or respective model architectures and algorithms. These alternative LLMs, each of which may be considered a candidate for production use, may be evaluated relative to one another automatically using other machine learning models such as question-generated models, question-answering models, and textual entailment models in various embodiments, without requiring tedious and error-prone evaluation by human teams. These other models, referred to as evaluation models, may themselves comprise LLMs in some implementations. The evaluation models may be provided examples of the transcripts and corresponding annotated summaries generated by the different candidate evidence mapping models, and generate respective quality scores for each candidate evidence mapping model based on analysis of the examples. ... ” (DINGLIWAL [0021]). It would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to include in the teachings of BOLCER the capability to provide evidence citations in the LLM output and tune the LLM based on the grounding quality score. The benefit and motivation of such modification is discussed by DINGLIWAL in the following portion: “The present disclosure relates to methods and apparatus for reducing the probability of hallucinations in the output produced by large language models (LLMs), such as LLMs that are utilized to generate summaries of doctor-patient conversations, by requiring the LLMs to provide evidence for various subsets of their output, and by evaluating the evidence-providing LLMs in an automated data-driven manner to select the LLMs that are least likely to generate hallucinations. ... ” (DINGLIWAL [0019]). Regarding claim 11, the rejection of claim 7 is incorporated and arguments analogous to claim 5 are applicable. Regarding claim 13, BOLCER teaches: A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: accessing responses from a large language model, each response comprising data including: a query; “The query 101 is a prompt that comprises the first input given to a language model. The query 101 may comprise questions, tasks, instructions or a series of answers and responses either by the user or by another system. A prompt template may contain instructions to guide the language model, a set of few-shot examples to help the language model generate a better response and/or specific questions directed at the language model.” (BOLCER [0029]). “This disclosure describes a system and method, using one or more processors, for grounding large language models using real-time content feeds and reference data. The system is able to receive/generate queries, prompt an AI model and evaluate the answer given by the AI model. A hallucination score is generated according to the evaluation of the answer. According to the hallucination score, the AI model may be iteratively accessed to improve the truthfulness of the answer.” (BOLCER [ABSTRACT]). an answer to the query, the answer comprising one or more statements; “The query 101 is a prompt that comprises the first input given to a language model. The query 101 may comprise questions, tasks, instructions or a series of answers and responses either by the user or by another system. A prompt template may contain instructions to guide the language model, a set of few-shot examples to help the language model generate a better response and/or specific questions directed at the language model.” (BOLCER [0029]). “The grounded Q&A 303 leverages an LLM to generate responses based on the user's original query 101 and documents 107 retrieved by the system. This multi-step process involves the evaluation of documents 107, extraction of salient phrases and the construction of a coherent response. ” (BOLCER [0051]). determining a grounding quality of the answer based on the evidence passages and the statements using an attribution evaluation model, wherein the grounding quality is a quantification of attribution of the statements in the answer to a document corpus; “The truthfulness validator 109 comprises a vector store 301, a grounded Q&A 303 and a similarity validator 305 for cross-verifying the reliability and accuracy of system-generated answers 111. Through confirmation of answer consistency with the content of semantically similar documents 107, the similarity validator 305 enhances the overall quality and trustworthiness of the system's outputs 111, proposing a confidence score (or a hallucination score) to the final answer 111. ” (BOLCER [0049]). BOLCER does not teach, but DINGLIWAL teaches: and citations linking each statement to an evidence passage in a corpus; “FIG. 5 illustrates an example input record which may be used for instruction fine-tuning of a pre-trained LLM to obtain an annotated summarization LLM that provides evidence for its summaries of conversations, according to at least some embodiments. In the depicted embodiment, a set of annotated summary examples 502, at least some of which include instruction prompts may be used for instruction fine-tuning 510 of a pre-trained LLM 512. A given instruction prompt 503 included in one of the examples may provide guidance in natural language to the pre-trained LLM, such as “summarize the conversation with numeric citations indicating the evidence for each sentence in the summary”.” (DINGLIWAL [0050]). tuning the large language model to obtain an adapted large language model that satisfies a grounding constraint based on a grounding quality score. “FIG. 5 illustrates an example input record which may be used for instruction fine-tuning of a pre-trained LLM to obtain an annotated summarization LLM that provides evidence for its summaries of conversations, according to at least some embodiments. ” (DINGLIWAL [0006]). “Several different evidence mapping LLMs may be fine-tuned in some embodiments, e.g., using respective data sets or respective model architectures and algorithms. These alternative LLMs, each of which may be considered a candidate for production use, may be evaluated relative to one another automatically using other machine learning models such as question-generated models, question-answering models, and textual entailment models in various embodiments, without requiring tedious and error-prone evaluation by human teams. These other models, referred to as evaluation models, may themselves comprise LLMs in some implementations. The evaluation models may be provided examples of the transcripts and corresponding annotated summaries generated by the different candidate evidence mapping models, and generate respective quality scores for each candidate evidence mapping model based on analysis of the examples. ... ” (DINGLIWAL [0021]). It would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to include in the teachings of BOLCER the capability to provide evidence citations in the LLM output and tune the LLM based on the grounding quality score. The benefit and motivation of such modification is discussed by DINGLIWAL in the following portion: “The present disclosure relates to methods and apparatus for reducing the probability of hallucinations in the output produced by large language models (LLMs), such as LLMs that are utilized to generate summaries of doctor-patient conversations, by requiring the LLMs to provide evidence for various subsets of their output, and by evaluating the evidence-providing LLMs in an automated data-driven manner to select the LLMs that are least likely to generate hallucinations. ... ” (DINGLIWAL [0019]). Regarding claim 17, the rejection of claim 13 is incorporated and arguments analogous to claim 5 are applicable . 07-21-aia AIA Claim s 2, 4, 8, 10, 14 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Bolcer; Gregory Alan et al. (US 20250086211 A1) hereinafter BOLCER in view of Dingliwal; Saket et al. (US 20250005298 A1) hereinafter DINGLIWAL in further view of RUSSELL; James Simon et al. (US 20240296295 A1) hereinafter RUSSELL . Regarding claim 2, the rejection of claim 1 is incorporated, furthermore BOLCER teaches: The computer-implemented method of claim 1, further comprising: iteratively processing a query by the adapted large language model, comprising: initially processing the query by the adapted large language model to obtain an answer for the query and citations for passages in the answer; “FIG. 1 illustrates an example system for grounding LLMs using real-time content feeds and reference data, in accordance with various example implementations of this disclosure. The system receives an NL query 101 and generates an answer with sources 111. A first subsystem 103 performs the process of converting NL queries into structured data queries to generate a call to a record service 105 that returns fetched records 107. A second subsystem 109 receives the query 101 and the fetched records 107 and performs a similarity-based truthfulness validation to generate the answer with sources 111.” (BOLCER [0027]). for each subsequent iteration: determining which passages in the answer are not supported by a citation; “With the answer the NLP 405 gave, we can generate a new answer based on using our reference data and vector data analysis across all billions of data points. In this case, for instance, we rewrite to include a list of “high confidence”, “possible”, and “not likely” or we want to label or just not included things with low scores based on sorting the hallucination scores. ” (BOLCER [0084]). “This disclosure describes a system and method, using one or more processors, for grounding large language models using real-time content feeds and reference data. The system is able to receive/generate queries, prompt an AI model and evaluate the answer given by the AI model. A hallucination score is generated according to the evaluation of the answer. According to the hallucination score, the AI model may be iteratively accessed to improve the truthfulness of the answer.” (BOLCER [ABSTRACT]). adding the passages that are supported by a citation to a list of relevant passages; “With the answer the NLP 405 gave, we can generate a new answer based on using our reference data and vector data analysis across all billions of data points. In this case, for instance, we rewrite to include a list of “high confidence”, “possible”, and “not likely” or we want to label or just not included things with low scores based on sorting the hallucination scores. ” (BOLCER [0084]). “The grounded Q&A 303 leverages an LLM to generate responses based on the user's original query 101 and documents 107 retrieved by the system. This multi-step process involves the evaluation of documents 107, extraction of salient phrases and the construction of a coherent response. ” (BOLCER [0051]). BOLCER in view of DINGLIWAL does not teach, but RUSSELL teaches: if there are one or more passages in the answer that are not supported by a citation, then, for each passage not supported, processing the passage not supported to generate a subsequent answer and citations for passages in the subsequent answer; “For each identified asserted quote, the verification postprocessor 206 forms a quote query for the corresponding extracted text. The quote query is configured to perform a string-matching search against the content of the source document 222. The quote query may be transmitted to the productivity application 110 as part of data communications 220.” (RUSSELL [0050]). “ ... In some examples, the non-verified portions of the output may be removed before the output is displayed, and/or the prompt may be revised and/or resubmitted to the LLM to generate a second output from the LLM. By implementing such improvements in prompt generation and output verification, the overall likelihood of hallucinations is reduced, and when such hallucinations do occur, their negative effect of potentially conveying inaccuracies is reduced or removed entirely. ” (RUSSELL [0014]). otherwise processing the query by the adapted large language model to obtain a subsequent answer for the query based on the passages and citations for passages in the subsequent answer. “For examples where all the quotes are unverified (e.g., none of the quotes could be verified), the LLM output may be discarded and the LLM prompt (or a revised LLM prompt) may be provided to the LLM 108 to be processed again to generate a second output from the LLM. The second output may then be analyzed for verification again. Alternatively or additionally, the responsive results that are provided for display may be an error message that indicates a verified summary and/or answer cannot be generated. In other examples, the original LLM output is retained, and an unverified indicator is incorporated into the responsive result to indicate that no quotes could be verified as being attributable to the content of the source document 222. ” (RUSSELL [0056]). “The LLM prompt may be revised by selecting an alternative prompt template, adjusting the emphasis on the verbatim quote suggestion, and/or adding additional context. At operation 424, the revised prompt is provided to the LLM. The LLM then processes the revised prompt to generate a revised LLM output. The method 400 may then flow back to operation 408 where the method repeats with the revised LLM output. ” (RUSSELL [0077]). It would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to include in the teachings of BOLCER in view of DINGLIWAL the capability to generate alternative passages when the passages are not supported. The benefit and motivation of such modification is discussed by RUSSELL in the following portion: “The present technology provides, among other things, programmatic solutions to reducing the likelihood that a large language model (LLM) will return hallucinated content and/or verifying that the responses produced by the LLM are actually supported by the document for which the request was generated. For instance, an LLM prompt for requesting the summarization of a source document and/or providing a question about the source document is generated to encourage the reduction of hallucinated content from the LLM. In some cases where specific instructions are provided in the prompt for the LLM to produce only verbatim quotes, the LLM may still hallucinate elements of its responses. With the present technology, the quotes and/or statements in the outputs from the LLM may be programmatically verified and checked against the source document. ” (RUSSELL [0002]). Regarding claim 4, the rejection of claim 2 is incorporated, furthermore BOLCER teaches: by a retriever model to generate supplemental passages to be processed by the adapted large language model. “This disclosure describes a system and method for handling NL queries, retrieving relevant information from a database and validating generated answers. The system and method leverage recursive calls to LLMs with specialized prompts and employs vector embeddings to improve the accuracy of information retrieval tasks. ” (BOLCER [0026]). BOLCER in view of DINGLIWAL does not teach, but RUSSELL teaches: The computer-implemented method of claim 2, wherein processing the passage not supported to generate a subsequent answer and citations for passages in the subsequent answer comprises processing the passage not supported “... In some examples, the non-verified portions of the output may be removed before the output is displayed, and/or the prompt may be revised and/or resubmitted to the LLM to generate a second output from the LLM. By implementing such improvements in prompt generation and output verification, the overall likelihood of hallucinations is reduced, and when such hallucinations do occur, their negative effect of potentially conveying inaccuracies is reduced or removed entirely. ” (RUSSELL [0014]). It would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to include in the teachings of BOLCER in view of DINGLIWAL the capability to generate alternative passages when the passages are not supported. The benefit and motivation of such modification is discussed by RUSSELL in the following portion: “The present technology provides, among other things, programmatic solutions to reducing the likelihood that a large language model (LLM) will return hallucinated content and/or verifying that the responses produced by the LLM are actually supported by the document for which the request was generated. For instance, an LLM prompt for requesting the summarization of a source document and/or providing a question about the source document is generated to encourage the reduction of hallucinated content from the LLM. In some cases where specific instructions are provided in the prompt for the LLM to produce only verbatim quotes, the LLM may still hallucinate elements of its responses. With the present technology, the quotes and/or statements in the outputs from the LLM may be programmatically verified and checked against the source document. ” (RUSSELL [0002]). Regarding claim 8, the rejection of claim 7 is incorporated and arguments analogous to claim 2 are applicable. Regarding claim 10, the rejection of claim 8 is incorporated and arguments analogous to claim 4 are applicable. Regarding claim 14, the rejection of claim 13 is incorporated and arguments analogous to claim 2 are applicable. Regarding claim 16, the rejection of claim 14 is incorporated and arguments analogous to claim 4 are applicable . 07-21-aia AIA Claim s 3, 9 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over BOLCER in view of DINGLIWAL in view of RUSSELL in further view of Chakhvadze; Artur et al. (US 20250045025 A1) hereinafter CHAKHVADZE . Regarding claim 3, the rejection of claim 2 is incorporated, furthermore BOLCER teaches: The computer-implemented method of claim 2, wherein iteratively processing the query by the adapted large language model to obtain an answer for the query and citations for passages in the answer comprises iteratively processing the query by the adapted large language model “This disclosure describes a system and method, using one or more processors, for grounding large language models using real-time content feeds and reference data. The system is able to receive/generate queries, prompt an AI model and evaluate the answer given by the AI model. A hallucination score is generated according to the evaluation of the answer. According to the hallucination score, the AI model may be iteratively accessed to improve the truthfulness of the answer.” (BOLCER [ABSTRACT]). “Trusted or trustworthy AI is that all best practices have been employed in the selection of the training data, use of technologies, and include accommodations for accuracy, explainability, privacy and reliability. A user trusts the system to follow a reasonable process to give the best answers with available information and controls. These types of systems are efficient at adhering to strategic goals for how and why the AI system was design and also are designed to efficiently use limited time, resources or information. ” (BOLCER [0072]). BOLCER in view of DINGLIWAL in view of RUSSELL does not teach, but CHAKHVADZE teaches: until an inference budget is exhausted. At operation 604, the processor determines that the collection of data units corresponds to a number of tokens that exceeds an upper limit of tokens (e.g., token budget) that a large language model can process. Token budget can refer to a number of tokens that a large language model can process in a single API call.” (CHAKHVADZE [0080]). It would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to include in the teachings of BOLCER in view of DINGLIWAL in further view of RUSSELL the capability to iterate until an inference budget is exhausted, in this case is token based. The benefit and motivation of such modification is discussed by CHAKHVADZE in the following portion: “… This solution requires a user to provide a collection of data units (e.g., paragraphs of a document or a set of documents) and exhaustively search for a set of subsequent data units that most closely meet the token budget of a large language model. User only needs to pass the collection into a function provided by the software library, and write a normal “for” loop over the chunk iterator described herein, returned by the function. ” (CHAKHVADZE [0029]). Regarding claim 9, the rejection of claim 8 is incorporated and arguments analogous to claim 3 are applicable. Regarding claim 15, the rejection of claim 14 is incorporated and arguments analogous to claim 3 are applicable. Allowable Subject Matter 07-43-02 Claims 6, 12, and 18 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) and the rejection(s) under 35 U.S.C. 101 set forth in this Office action and to include all of the limitations of the base claim and any intervening claims. Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to HECTOR J. CRESPO FEBLES whose telephone number is (571)272-4512. The examiner can normally be reached Mon - Fri 7:30 - 5:00. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached at (571) 272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /H.J.C./ Examiner, Art Unit 2657 /DANIEL C WASHBURN/ Supervisory Patent Examiner, Art Unit 2657 Application/Control Number: 18/947,752 Page 2 Art Unit: 2657 Application/Control Number: 18/947,752 Page 3 Art Unit: 2657 Application/Control Number: 18/947,752 Page 4 Art Unit: 2657 Application/Control Number: 18/947,752 Page 5 Art Unit: 2657 Application/Control Number: 18/947,752 Page 6 Art Unit: 2657 Application/Control Number: 18/947,752 Page 7 Art Unit: 2657 Application/Control Number: 18/947,752 Page 8 Art Unit: 2657 Application/Control Number: 18/947,752 Page 9 Art Unit: 2657 Application/Control Number: 18/947,752 Page 10 Art Unit: 2657 Application/Control Number: 18/947,752 Page 11 Art Unit: 2657 Application/Control Number: 18/947,752 Page 12 Art Unit: 2657 Application/Control Number: 18/947,752 Page 13 Art Unit: 2657 Application/Control Number: 18/947,752 Page 14 Art Unit: 2657 Application/Control Number: 18/947,752 Page 15 Art Unit: 2657 Application/Control Number: 18/947,752 Page 16 Art Unit: 2657 Application/Control Number: 18/947,752 Page 17 Art Unit: 2657 Application/Control Number: 18/947,752 Page 18 Art Unit: 2657 Application/Control Number: 18/947,752 Page 19 Art Unit: 2657 Application/Control Number: 18/947,752 Page 20 Art Unit: 2657 Application/Control Number: 18/947,752 Page 21 Art Unit: 2657 Application/Control Number: 18/947,752 Page 22 Art Unit: 2657 Application/Control Number: 18/947,752 Page 23 Art Unit: 2657 Application/Control Number: 18/947,752 Page 24 Art Unit: 2657
Prosecution Insights