DETAILED ACTION
Claims 1 – 16 and 18 – 21 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant’s submission filed on 19 march 2026 has been entered.
Response to Amendment
With regard to the Final Office Action from 19 November 2025, the Applicant has filed a response on 19 March 2026.
Claim 17 has been cancelled.
New claim — 21 has been added.
Response to Arguments
With regard to the 35 U.S.C. 103 rejection given to independent claims, the Applicant indicates (Remarks: page 8 par 3) that the applied references fail to teach of ‘executing a string-matching query against the content of the source document to determine that the text string from the first quote is present in the content of the source document,’ and ‘based on the text string of the first quote being present in the content of the source document and the text string of the second quote not being present in the content of the source document, generating responsive results including a portion of the LLM output not including the text string of the second quote and a verification indicator that the first quote is verified.’ Considering the Examiner’s use of the Menick, Jacob, et al., the Examiner acquiesces and will address the claims by their current presentation in the following section. This also applies to the use of Menick et al. for claim 9 (Remarks: page 9). The Applicant indicates here that the Menick, Jacob, et al. reference teaches of reduction in hallucinations but does not teach of the claimed verification features. The Examiner indicates here that the Menick, Jacob, et al. reference is actually directed to performing verification, but would differ in the sense that it does not present a verification indication as the claimed invention requires, rather, it simply goes through its process to ensure that quotes being cited are being verified. The Menick, Jacob, et al. reference is able to check for string matching between the obtained quote and the source, as provided on Page 35 F.1. ‘Wrong quote’ which this reference indicates, gives a bad syntax penalty. This establishes teaching for the string-matching. As indicated earlier, this Menick, Jacob, et al. would not be applied to teach of the claimed verification indicator, even though the reference does provide verification.
The Applicant further argues against the use of the Gao, Luyu, et al. reference (Remarks: page 10 par 2–3) regarding the teaching of ‘based on the first text string not being present in the content of the source document, providing a second LLM prompt as input to the LLM, the second LLM prompt comprising the interrogation request, the content of the source document, and second instructions for inclusion of verbatim quotes from the source document.’ The Applicant rightly provides that this Gao, Luyu, et al. reference directly modifies the text output to be a revised output which can be further edited, but contrary to the claimed invention, it does not submit another prompt to cause a new output to be generated. The Gao, Luyu, et al. reference does provide teaching for editing a quote when an agreement is not met with the source, as provided in Figure 2. The editing of the quote, then leads to obtaining a new quote, that is then checked to ensure it is in agreement. This is equivalent to checking that the first quote is not present, and then checking the edited quote, as a second quote, to then find out that this edited (second) quote is present in the source. For the sake of moving the application forward though, the Examiner will also address this limitation by its current presentation in the appropriate art rejection section.
The Applicant argues against the Examiner’s 35 U.S.C. 101 rejection of the claims being directed to a judicial exception without significantly more. The Applicant presents again (Remarks: pages 10 – 11) that the present technology is directed to an improvement by reducing hallucinations in LLM-based results, and this need not be explicitly stated in the claims. For example, claim 1 has now been amended to indicate that the invention is directed to reducing hallucinations in LLM outputs. This statement is recited in the preamble of the claim and does not particularly add to the limitations of the claim. Also, stating its presence for the reduction of hallucinations is a mental process by itself, given that a human could go through several document checks after an initial citation of documents in order to ensure that the indicated documents are properly addressed and cited to, without introducing false citations. The further steps of the claims are still mental recitations by themselves. The presence of the LLM is simply provided as a tool being used to enact the process. The generation being performed by the LLM based on the current limitations can be performed in the human mind. The Applicant would need to provide a limitation regarding the LLM that would remove its presence from the realm of being a mental process. This could be through the particular training scheme of the LLM that’s being applied. The Examiner maintains the 101 rejection in this case.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1 – 16 and 18 – 21are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception without significantly more.
Independent claim 1 recites the limitations of reducing hallucinations in outputs from a large language model by receiving an interrogation request about a source document with content, generating an LLM prompt that includes the interrogation request, the content of the source document and instructions for including verbatim quotes from the source document, putting the LLM prompt as an input to an LLM, receiving an LLM output that includes asserted first and second quotes from the source document, extracting a text string from the asserted first and second quotes that are present in the LLM output, performing string-matching query against the content of the source document to determine that the text string of the first quote is present in the content of the source document and that the text string of the second quote is not present in the content of the source document, in the event that the text string of the first quote is present and the text string of the second quote is not present, generating responsive results that include the portion of the LLM output indicating the first quote and absent the portion of the LLM output indicating the second quote, and a verification indicator that indicates that the LLM output indicating the first quote is verified, and finally causing the responsive results to be displayed.
Nothing mentioned in the claims precludes it from being performed in the human mind. The entire process involves data collection through receiving a request, providing an LLM prompt and extracting first and second text strings, data generation for generating a prompt as well as generating responsive results and a verification indicator, and data presentation for displaying responsive results. The claim can be performed by: a human receiving an interrogation request about a source document in textual form, the human generates and writes out a prompt that includes the interrogation request, the content of the source document and instructions to be followed for including direct quotes from the source document, the human provides the prompt to a second human and receives from the second human, an output that includes two asserted quotes from the source document, the human extracts text strings from the asserted quotes, matches the text strings against the content of the source document to determine that the first text string is in the content of the source document and that the second text string is not in the content of the source document, and based on the string matching, the human generates responsive results that include the output provided in the first text string but not that of the second text string, as well as a verification label to show that the output is verified, and finally displaying the responsive results. The claim hereby recites a mental process.
Independent claim 9 recites the limitations of performing attribution verification for outputs from a large language model by receiving an interrogation request about a source document with content, generating a first LLM prompt that includes the interrogation request, the content of the source document and instructions for including verbatim quotes from the source document, putting the LLM prompt as an input to an LLM, receiving an LLM output that includes a first asserted quote and a second asserted quote from the source document, extracting a first text string from the first asserted quote and a second text string from the second asserted quote as presented in the LLM output, executing string-matching query against the content of the source document to determine that the first text string is not present in the content of the source document and that the second string is present in the content of the source document, based on the string-matching query, generating responsive results that include the LLM output that includes the second asserted quote but without the first asserted quote and a verification indicator that indicates that the LLM output is verified, and finally causing the responsive results to be displayed.
Nothing mentioned in the claim precludes it from being performed in the human mind. The entire process involves data collection through receiving a request, providing an LLM prompt and extracting first and second text string, data generation for generating a first LLM prompt as well as generating responsive results and a verification indicator, and data presentation for displaying responsive results. The claim can be performed by: a human receiving an interrogation request about a source document in textual form, the human generates and writes out a first prompt that includes the interrogation request, the content of the source document and instructions to be followed for including direct quotes from the source document, the human provides the prompt to a second human and receives from the second human, an output that includes a first asserted quote and a second asserted quote from the source document, the human extracts a first text string from the first asserted quote and a second text string from the second asserted quote, matches strings against the content of the source document to determine that the first text string is not in the content of the source document and that the second text string is in the content of the source document, and based on the string matching, the human generates responsive results that include the output provided by the second quote but not including that of the first quote as well as a verification code to show that the output is verified, and finally displaying the responsive results. The claim hereby recites a mental process.
Independent claim 16 recites the limitations of performing attribution verification for outputs from a large language model by receiving an interrogation request about a source document with content, generating a first LLM prompt that includes the interrogation request, the content of the source document and instructions for including verbatim quotes from the source document, putting the LLM prompt as an input to an LLM, receiving a first LLM output that includes a first asserted quote from the source document, extracting a first text string from the first asserted quote as presented in the first LLM output, executing string-matching query against the content of the source document to determine that the first text string is not present in the content of the source document, based on the text string not being present in the content, providing a second LLM prompt which comprises the interrogation request, the content of the source document and second instructions for including verbatim quotes from the source document into the LLM, receiving a second LLM output that includes a second asserted quote from the source document, extracting a second text string from the second asserted quote as presented in the second LLM output, executing string-matching query against the content of the source document to determine that the second text string is present in the content of the source document, generating responsive results from the second LLM output to include the second asserted quote, and displaying these response results.
Nothing mentioned in the claim precludes it from being performed in the human mind. The entire process involves data collection through receiving a request, providing LLM prompts and extracting text strings, data generation for generating first and second LLM prompts. The claim can be performed by: a human receiving an interrogation request about a source document in textual form, the human generates and writes out a first prompt that includes the interrogation request, the content of the source document and instructions to be followed for including direct quotes from the source document, the human provides the prompt to a second human and receives from the second human, an output that includes a first asserted quote from the source document, the human extracts a first text string from the first asserted quote, matches a string against the content of the source document to determine that the first text string is not in the content of the source document, and based on the string matching showing that the text string is not present in the content of the source document, the human provides a second prompt to the second human, such that the second prompt comprises the interrogation request, the content of the source document and second instructions for directly including quotes from the source document, receives from the second human again, a second output that includes a second asserted quote from the source document, the human extracts a second text string from the second asserted quote, matches a string against the content of the source document to determine that the second text string is in the content of the source document, and based on the string matching showing that the text string is present in the content of the source document, the human generates responsive results from the second output that includes the second asserted quote, which then gets displayed as the responsive results. The claim hereby recites a mental process.
This judicial exception is not integrated into a practical application as the claims simply teach of analysing data and presenting data. While the claims mention a processor, and a memory, these are recited in generic terms.
The invention is not tied to any particular defining structure and simply provides instructions to apply the judicial exception. The technique can be performed by a generic computer which would be presented as a tool to implement the abstract idea (classifiable as automation of the mental process steps). The Specification in [0015] provides a generic computer as one of several computer devices, suitable to read upon the limitations of these claims. This is recited at a high level of generality that it amounts to no more than mere instructions to apply the exception using a generic computer. The claims also present the use of a large language model and the Specification in [0019] provides an LLM as a generative machine learning model trained to under and generate sequences of tokens, however recited as a generic machine learning model recited in the claims without specificity on the details of the large language model being applied.
The claim does not provide any detail on the training of the large language model, nor precisely what large language model gets applied to providing an LLM output. The presence of an LLM serves the purpose of providing nothing more than mere instructions to implement an abstract idea on a generic computer. An LLM model is used to generally apply the abstract idea without limiting how the trained LLM functions. The LLM is presented at a high level of generality that it amounts to using a computer with a generic machine learning model to apply the abstract idea. The claims just state providing an LLM prompt to an LLM and receiving from the LLM, an LLM output, but does not provide any additional detail. The claims therefore do not include additional elements that would be sufficient to amount to significantly more than the judicial exception because the invention is not tied to a practical application.
The claims provide techniques that amount to no more than mere instructions that apply the judicial exception which can be performed by a generic device. Merely mentioning the processor and memory amount to no more than general-purpose hardware used as tools to implement the abstract idea and does not provide any particular application other than applying it for the purpose of implementing a judicial exception. Mere instructions to apply an exception using a generic device cannot provide an inventive concept. Claims 1, 9 and 16 are not eligible.
Claim 2 provides teaching for the LLM output further including a source identifier for the asserted first quote. A human may provide an identification of the source as a citation for an asserted quote. This does not integrate any practical application nor does it provide any additional element sufficient to amount to more than the mentioned judicial exception.
Claim 3 provides teaching for generating the responsive results to include incorporating the source identifier as a link to a position of the asserted first quote in the source document. A human may include a source identifier as a written link to the page number/line numbers that the asserted quote appears in the source document. This does not integrate any practical application nor does it provide any additional element sufficient to amount to more than the mentioned judicial exception.
Claim 4 provides teaching for receiving the source identifier, and that in response to receiving the selection of the source identifier, causing the display of the source document position to show the asserted first quote. A human may receive the selection of the source identifier and write out or highlight the position of the asserted quote as available on the source document. This does not integrate any practical application nor does it provide any additional element sufficient to amount to more than the mentioned judicial exception.
Claim 5 provides teaching for the interrogation request being one of a summarisation request or a question about the source document that be answered from the source document. A human may provide a query requesting a summary or regarding a question about the source document, such that the answer would be available in the source document. This does not integrate any practical application nor does it provide any additional element sufficient to amount to more than the mentioned judicial exception.
Claim 6 provides that the interrogation request is based on a user input. A human may wite out an interrogation request. This does not integrate any practical application nor does it provide any additional element sufficient to amount to more than the mentioned judicial exception.
Claim 7 provides that the interrogation request is automatically generated upon the source document being accessed. A human may have instructions to write an interrogation request right after receiving a source document. This does not integrate any practical application nor does it provide any additional element sufficient to amount to more than the mentioned judicial exception.
Claim 8 provides teaching for storing the responsive results with the source document. A human may save the responsive results in the same location the source document is saved. This does not integrate any practical application nor does it provide any additional element sufficient to amount to more than the mentioned judicial exception.
Claim 10 provides teaching for generating the responsive results by including the removal of the first asserted quote. A human may generate and present responsive results by removing the first asserted quote. This does not integrate any practical application nor does it provide any additional element sufficient to amount to more than the mentioned judicial exception.
Claim 11 provides that generating the responsive results includes marking the second asserted quote as verified with the verification indicator. A human may mark the second asserted quote as verified with a verification indicator marking. This does not integrate any practical application nor does it provide any additional element sufficient to amount to more than the mentioned judicial exception.
Claim 12 provides teaching for the LLM output further including a first source identifier for the first asserted quote and a second source identifier for the second asserted quote. A human may provide an identification of the source as a citation for an asserted quote. This does not integrate any practical application nor does it provide any additional element sufficient to amount to more than the mentioned judicial exception.
Claim 13 provides that the interrogation request is automatically generated upon the source document being accessed. A human may have instructions to write an interrogation request right after receiving a source document. This does not integrate any practical application nor does it provide any additional element sufficient to amount to more than the mentioned judicial exception.
Claim 14 provides teaching for the interrogation request being a summarisation request. A human may provide a query requesting a summary. This does not integrate any practical application nor does it provide any additional element sufficient to amount to more than the mentioned judicial exception.
Claim 15 provides teaching for executing the string-matching query by preprocessing the extracted text string and the document content by performing at least one of white space removal, punctuation removal and letter case changing. A human may manually perform any one or more of these preprocessing tasks. This does not integrate any practical application nor does it provide any additional element sufficient to amount to more than the mentioned judicial exception.
Claim 18 provides teaching for generating first responsive results to include an LLM output with the asserted quote and an unverified indicator, and then causing a display of the first responsive results. A human may write out generated responsive results that include an output from another human, the result including a received output and an indication that the result is currently unverified, and then presenting this responsive result for display on a piece of paper. This does not integrate any practical application nor does it provide any additional element sufficient to amount to more than the mentioned judicial exception.
Claim 19 provides that the second LLM prompt is the same as the first LLM prompt. A human may present the same prompt twice or write it out to be presented at two different locations. This does not integrate any practical application nor does it provide any additional element sufficient to amount to more than the mentioned judicial exception.
Claim 20 provides revising the first LLM prompt to form a second LLM prompt by adding additional emphasis on producing a verbatim quote in the second instructions. A human may revise first prompt that is to be sent out by adding additional emphasis on producing a verbatim quote. This does not integrate any practical application nor does it provide any additional element sufficient to amount to more than the mentioned judicial exception.
Claim 21 provides that the responsive results include a selectable source indicator that causes a source document to be displayed and navigated to a portion of the source document where the second quote is present. A human may present a response with clear information regarding the source of the response and how to navigate to that source. This does not integrate any practical application nor does it provide any additional element sufficient to amount to more than the mentioned judicial exception.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 3, 4, 5, 6, 9, 11, 12, 14 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over TUNSTALL-PEDOE et al. (WO 2023/161630 A1: hereafter — Tunstall-Pedoe) in view of Menick, Jacob, et al. (“Teaching language models to support answers with verified quotes.” arXiv preprint arXiv:2203.11147 (2022): hereafter — Menick) further in view of Gao, Luyu, et al. (“Attributed text generation via post-hoc research and revision.” arXiv preprint arXiv:2210.08726 (2022): hereafter — Gao).
For claim 1, Tunstall-Pedoe discloses a system for reducing hallucinations in outputs from a large language model (LLM) (Tunstall-Pedoe: page 8 par 2 — avoiding hallucinations; Page 6 Par 1 — interacting with an LLM), comprising:
at least one processor (Tunstall-Pedoe: Page 10 line 16 — a processor); and
memory storing instructions that, when executed by the at least one processor (Tunstall-Pedoe: Page 10 line 16 — memory as a part of a computer sytem), cause the system to perform operations comprising:
receiving an interrogation request [[about a source document having content]] (Tunstall-Pedoe: Page 10 lines 13–14 — receiving a natural language query (as the interrogation request));
generating an LLM prompt [[that includes the interrogation request, the content of the source document, and instructions for inclusion of verbatim quotes from the source document]] (Tunstall-Pedoe: Page 15 Par 2 — LLM generating continuation data (as the claimed LLM prompt));
[[executing a string-matching query against the content of the source document to]] determine that the text string from the first quote is present in the content of the source document and the text string from the second quote is not present in the content of the source document (Tunstall-Pedoe: Page 8 par 2–3 — checking sections of the continuation generated by the LLM against the database and retrieving sources where the continuation match text contained within the database; Page 7 par 2 — validating natural language for factual accuracy (indicating a check for text strings of quotes that are either present or not present in the content of the source document); Page 133 par 3 — avoiding factual inaccuracies (indicating a check to ensure that the second text string is not present in the source content, as this would be considered a factual inaccuracy));
based on the text string of the first quote being present in the content of the source document and the text string of the second quote note being present in the content of the source document, generating responsive results including a portion of the LLM output not including the text of the string of the second quote [[and a verification indicator indicating that the first quote is verified]] (Tunstall-Pedoe: Page 8 par 2–3 — checking sections of the continuation generated by the LLM against the database and retrieving sources where the continuation match text contained within the database; Page 7 par 2 — validating natural language for factual accuracy (indicating a check for text strings of quotes that are either present or not present in the content of the source document); Page 15 Par 2 — ensuring a fact-checked output is generated and presented to the user; Page 133 Par 2 — removing assertions that are factual inaccuracies from display to a user);
causing the responsive results to be displayed (Tunstall-Pedoe: Page 19 par 3 — outputting the responsive answer on a display).
The reference of Tunstall-Pedoe provides teaching for the above but fails to provide the further teachings of this claim, for which the reference of Menick is now introduced to teach as:
receiving an interrogation request about a source document having content (Menick: Figure 1 — a user provides a query; Page 2 Par 6 — an input query is given and the system retrieves relevant document to address it (the documents being known to have content));
generating an LLM prompt that includes the interrogation request, the content of the source document, and instructions for inclusion of verbatim quotes from the source document (Menick: Page 5 Par 2 — combining a question (interrogation request) with retrieved documents into a prompt; Page 37 H. — forming prompts using a desired syntax; Page 39 J. — prompting by using Inline Evidence Syntax in order to enforce verbatim quotes (thereby teaching of instructions for inclusion of verbatim quotes); Page 2 Par 4 — a special syntax for a language model to use when providing answers in order to constrain the outputs of the model to be exact quotes from the retrieved documents);
providing the LLM prompt as input into an LLM (Menick: Page 2 Par 6 — given an input query, the system retrieves relevant documents … and presents the language model a large context drawn from multiple documents (teaching of inputting the prompt into a language model); Page 2 Par 4 — the use of a large language model);
receiving, from the LLM, an LLM output including a plurality of asserted quotes from the source document, including a first quote and a second quote (Menick: Section 1. Figure 1 — examples showing the generation of LLM outputs that include asserted quotes from the source document (the Wikipedia page); page Section 2.10 From the user’s perspective — WebGPT links claims to quotes, allowing the model to link multiple supported claims into an answer (teaching of obtaining several quotes to indicate the presence of a first quote and a second quote));
extracting a text string from the asserted quote in the LLM output (Menick: Page 2 par 4 — generating answers that can be verbatim quotes of supporting evidence as a single string evidence (indicating the extraction of a text string from the section used as the asserted quote to answer a question as obtained by the LLM); Page 3 section 2.1. — inline evidence syntax to show a verification indicator which extracts as part of the answer, a text string from the obtained source document as a precise quote from the document);
executing a string-matching query against the content of the source document to determine that the text string from the first quote is present in the content of the source document and the text string from the second quote is not present in the content of the source document (Menick: Page 2 par 4 — generating answers that can be verbatim quotes of supporting evidence as a single string evidence (indicating the extraction of a text string from the section used as the asserted quote to answer a question and that it is present in the source document); Page 24 Appendix — matching snippet position inside scrapped document; Page 11 par 1 — finding the closest matching sequence of n+2 sentences regarding the number of sentences in the answer; Page 35 F.1 — ‘Wrong quote: if the quote is not matching verbatim the full source’ (as a clear string matching check between the quote and content of the source document));
based on the text string of the first quote being present in the content of the source document and the text string of the second quote not being present in the content of the source document, generating responsive results including a portion of the LLM output not including the text of the string of the second quote [[and a verification indicator indicating that the first quote is verified]] (Menick: Page 2 par 4 — generating answers that can be verbatim quotes of supporting evidence as a single string evidence (indicating the extraction of a text string from the section used as the asserted quote to answer a question and that it is present in the source document); Figure 1 — an inclusion of source in the response serves as a verification indicator; Page 35 F.1. — checking for bad syntax penalty (indicating an attempt at string matching of a quote verbatim, to the source so as to apply it, and penalise it when it instead does not match verbatim); Page 3 par 4 — ‘… pointing the user to an easily verified quote as we do in GopherCite …’; Page 8, first bullet of 2.10. — ‘GopherCite provides exact and succinct quotes supporting the claim …’ all these to show that an output is verified).
The reference of Tunstall-Pedoe provides teaching for a system able to receive a query and generate an LLM prompt to be presented to an LLM engine while checking that certain quotes not contained in the source content should be removed from being published. This differs from the claimed invention in that the claimed invention further provides teaching for providing the LLM prompt to an LLM in order to generate an LLM output that extracts and matches text strings from an asserted quote. This isn’t new to the art as it is seen to be taught by the reference of Menick above.
Hence, before the effective filing date of the claimed invention, one of ordinary skill in the art would have found it obvious to improve upon the teaching of Tunstall-Pedoe which checks that certain quotes are not contained in source content and are to be removed from being published, by incorporating the known teaching of Menick which provides generating LLM outputs that are matched strings as an asserted quote to the content source, to thereby come up with the claimed invention. The combination of both prior art elements would have provided the predictable result of ensuring that the provided user input query results are those that properly match the source documents from which the answers are being obtained from, further ensuring reduction in hallucination. See KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 415-421, 82 USPQ2d 1385, 1395-97 (2007).
The combination of Tunstall-Pedoe in view of Menick provides teaching for generating verified results for display to a user. This combination however fails to explicitly teach of a verification indicator indicating that the quote is verified.
The reference of Gao is now introduced to teach this as:
based on the text string of the first quote being present in the content of the source document and the text string of the second quote not being present in the content of the source document, generating responsive results including a portion of the LLM output not including the text of the string of the second quote and a verification indicator indicating that the first quote is verified (Gao: Page 6 Col 1 Section 4 — outputting a label to indicate whether a statement is supported or refuted (the label stands as a verification indicator)).
Hence, before the effective filing date of the claimed invention, one of ordinary skill in the art would have found it obvious to improve upon the teaching of the combination of Tunstall-Pedoe in view of Menick which provides query results as verified quotes, by incorporating the teaching of Gao which provides an indication label as the verification indicator, thereby coming up with the claimed invention. The combination of both prior art elements would have provided the predictable result of a visual marker/label for the person reading the query responses to be assured that the responses are indeed trustworthy. See KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 415-421, 82 USPQ2d 1385, 1395-97 (2007).
For claim 2, claim 1 is incorporated and the combination of Tunstall-Pedoe in view of Menick further in view of Gao discloses the system, wherein the LLM output further includes a source identifier for the asserted first quote (Menick: Figure 1 — indicating a source identifier as Wikipedia).
For claim 3, claim 2 is incorporated and the combination of Tunstall-Pedoe in view of Menick further in view of Gao discloses the system, wherein generating the responsive results includes incorporating the source identifier as a link to a position of the asserted first quote in the source document (Menick: page 12 par 3 — a model that provides output as a single answer that includes a quote and URL as well as indicating the page (a link to a position of the asserted quote)).
For claim 4, claim 3 is incorporated and the combination of Tunstall-Pedoe in view of Menick further in view of Gao discloses the system, wherein the operations further comprise:
receiving a selection of the source identifier (Menick: page 10 3.2 — ‘The “supporting evidence” is formed by choosing a random sentence from the dataset-provided Wikipedia document’ showing the selection of a source identifier); and
in response to receiving the selection, causing a display of the source document positioned to show the asserted first quote (Menick: Figure 1 — an inclusion of source in the response serves as a verification indicator and asserted quote with the response).
For claim 5, claim 1 is incorporated and the combination of Tunstall-Pedoe in view of Menick further in view of Gao discloses the system, wherein the interrogation request is one of a summarization request or a question about the source document that can be answered from the source document (Tunstall-Pedoe: Page 20 Par 2 — a request to generate a summary).
For claim 6, claim 1 is incorporated and the combination of Tunstall-Pedoe in view of Menick further in view of Gao discloses the system, wherein the interrogation request is based on a user input (Menick: Figure 1 — a user provides a query; Page 2 Par 6 — an input query is given and the system retrieves relevant document to address it (the documents being known to have content)).
For claim 9, Tunstall-Pedoe discloses a computer-implemented method for performing attribution verification for outputs from a large language model (LLM) (Tunstall-Pedoe: page 8 par 2 — avoiding hallucinations; Page 6 Par 1 — interacting with an LLM), comprising:
receiving an interrogation request [[about a source document having content]] (Tunstall-Pedoe: Page 10 lines 13–14 — receiving a natural language query (as the interrogation request));
generating a first LLM prompt [[that includes the interrogation request, the content of the source document, and instructions for inclusion of verbatim quotes from the source document]] (Tunstall-Pedoe: Page 15 Par 2 — LLM generating continuation data (as the claimed LLM prompt));
[[executing a string-matching query against the content of the source document to]] determine that the first text string is not present in the content of the source document and that the second text string is present in the content of the source document (Tunstall-Pedoe: Page 8 par 2–3 — checking sections of the continuation generated by the LLM against the database and retrieving sources where the continuation match text contained within the database; Page 7 par 2 — validating natural language for factual accuracy (indicating a check for text strings of quotes that are either present or not present in the content of the source document); Page 133 par 3 — avoiding factual inaccuracies (indicating a check to ensure that the first text string is not present in the source content, as this would be considered a factual inaccuracy));
based on the string-matching query, generating responsive results including the LLM output, including the second quote, [[and a verification indicator,]] wherein the responsive results do not include the first asserted quote (Tunstall-Pedoe: Page 8 par 2–3 — checking sections of the continuation generated by the LLM against the database and retrieving sources where the continuation match text contained within the database; Page 7 par 2 — validating natural language for factual accuracy (indicating a check for text strings of quotes that are either present or not present in the content of the source document); Page 15 Par 2 — ensuring a fact-checked output is generated and presented to the user; Page 133 Par 2 — removing assertions that are factual inaccuracies from display to a user (not including the first asserted quote)); and
causing the responsive results to be displayed (Tunstall-Pedoe: Page 19 par 3 — outputting the responsive answer on a display).
The reference of Tunstall-Pedoe provides teaching for the above but fails to provide the further teachings of this claim, for which the reference of Menick is now introduced to teach as:
receiving an interrogation request about a source document having content (Menick: Figure 1 — a user provides a query; Page 2 Par 6 — an input query is given and the system retrieves relevant document to address it (the documents being known to have content));
generating a first LLM prompt that includes the interrogation request, the content of the source document, and instructions for inclusion of verbatim quotes from the source document (Menick: Page 5 Par 2 — combining a question (interrogation request) with retrieved documents into a prompt; Page 37 H. — forming prompts using a desired syntax; Page 39 J. — prompting by using Inline Evidence Syntax in order to enforce verbatim quotes (thereby teaching of instructions for inclusion of verbatim quotes); Page 2 Par 4 — a special syntax for a language model to use when providing answers in order to constrain the outputs of the model to be exact quotes from the retrieved documents)
providing the LLM prompt as input into an LLM (Menick: Page 2 Par 6 — given an input query, the system retrieves relevant documents … and presents the language model a large context drawn from multiple documents (teaching of inputting the prompt into a language model); Page 2 Par 4 — the use of a large language model);
receiving, from the LLM, an LLM output including a first asserted quote and a second asserted quote from the source document (Menick: Page 35 Par 1 (considering the bullet points) — showing that several asserted quotes can be made to provide a response to a prompt (teaching the presence of first and second asserted quotes));
extracting a first text string, from the first asserted quote, and a second text string from the second asserted quote in the LLM output (Menick: Page 2 Par 4 — collecting a single string of verbatim quotes of supporting evidence as inline evidence (teaching of extracting a string from the asserted quote as obtained by the LLM); Page 3 2.1. — the generated answers have supporting evidence inline in a single string, an inline evidence syntax to show a verification indicator which extracts as part of the answer, a text string from the obtained source document as a precise quote from the document);
executing a string-matching query against the content of the source document to determine that the first text string is not present in the content of the source document and that the second text string is present in the content of the source document (Menick: Page 2 par 4 — generating answers that can be verbatim quotes of supporting evidence as a single string evidence (indicating the extraction of a text string from the section used as the asserted quote to answer a question and that it is present in the source document); Page 24 Appendix — matching snippet position inside scrapped document; Page 11 par 1 — finding the closest matching sequence of n+2 sentences regarding the number of sentences in the answer); Page 35 F.1. (considering the bullet points) — penalising a response when there’s a wrong quote in that the quote used is not matching verbatim the full source);
based on the string-matching query, generating responsive results including the LLM output, including the second quote, [[and a verification indicator,]] wherein the responsive results do not include the first asserted quote (Menick: Page 2 par 4 — generating answers that can be verbatim quotes of supporting evidence as a single string evidence (indicating the extraction of a text string from the section used as the asserted quote to answer a question and that it is present in the source document); Figure 1 — an inclusion of source in the response serves as a verification indicator; Page 3 par 4 — ‘… pointing the user to an easily verified quote as we do in GopherCite …’; Page 8, first bullet of 2.10. — ‘GopherCite provides exact and succinct quotes supporting the claim …’ all these to show that an output is verified).
The same motivation for combination applied to claim 1 above for introducing the Menick reference is applicable here still.
The combination of Tunstall-Pedoe in view of Menick provides teaching for generating verified results for display to a user. This combination however fails to explicitly teach of a verification indicator indicating that the quote is verified.
The reference of Gao is now introduced to teach this as:
based on the string-matching query, generating responsive results including the LLM output, including the second quote, and a verification indicator, wherein the responsive results do not include the first asserted quote (Gao: Page 6 Col 1 Section 4 — outputting a label to indicate whether a statement is supported or refuted (the label stands as a verification indicator))
The same motivation for combination applied to claim 1 above for introducing the Gao reference is applicable here still.
For claim 11, claim 9 is incorporated and the combination of Tunstall-Pedoe in view of Menick further in view of Gao discloses the method, wherein generating the responsive results includes marking the second asserted quote as verified with the verification indicator (Menick: Page 2 par 4 — generating answers that can be verbatim quotes of supporting evidence as a single string evidence (indicating the extraction of a text string from the section used as the asserted quote to answer a question and that it is present in the source document); Figure 1 — an inclusion of source of the response in the response area and that serves as a verification indicator).
For claim 12, claim 9 is incorporated and the combination of Tunstall-Pedoe in view of Menick further in view of Gao discloses the method, wherein the LLM output further includes a first source identifier for the first asserted quote and a second source identifier for the second asserted quote (Menick: Page 1 Figure 1 — showing two responses with their respective source identifiers).
For claim 14, claim 9 is incorporated and the combination of Tunstall-Pedoe in view of Menick further in view of Gao discloses the method, wherein the interrogation request is a summarization request (Tunstall-Pedoe: Page 20 Par 2 — a request to generate a summary).
For claim 15, claim 9 is incorporated and the combination of Tunstall-Pedoe in view of Menick further in view of Gao discloses the method, wherein executing the string-matching query further comprises preprocessing the extracted text string and the document content to perform at least one of removing white space, removing punctuation, or changing letter case (Menick: Page 39 K. — preprocessing documents to remove whitespace).
Claims 7 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Tunstall-Pedoe (WO 2023/161630 A1) in view of Menick (“Teaching language models to support answers with verified quotes.”) further in view of Gao (“Attributed text generation via post-hoc research and revision.”) as applied to claims 1 and 9, and further in view of SHEN et al. (US 2010/0057687 A1: hereafter — Shen).
For claim 7, claim 1 is incorporated but the combination of Tunstall-Pedoe in view of Menick further in view of Gao fails to teach the limitation of this claim, for which Shen is now introduced to teach as the system, wherein the interrogation request is automatically generated upon the source document being accessed (Shen: [0019] — generating future queries based on language models that represent previous queries; FIG. 2 — data collection (accessing source documents) to then lead to the generation of future queries; [0031] — the presence of a language model leading to the generation of future queries).
The combination of Tunstall-Pedoe in view of Menick further in view of Gao provides teaching for receiving an interrogation request, but differs from the claimed invention in that the claimed invention further provides teaching for the automatic generation of an interrogation request upon accessing source document. This isn’t new to the art as the reference of Shen is seen to teach above.
Hence, before the effective filing date of the claimed invention, one of ordinary skill in the art would have found it obvious to incorporate the teaching of Shen which automatically generates prompts for a language model in the present of content containing prior prompts, with the teaching of the combination of Tunstall-Pedoe in view of Menick further in view of Gao which receives an interrogation request for the purpose of delivering it to a large language model, to thereby come up with the claimed invention. The combination of both prior art elements would have provided the predictable result of generating several queries suitable to train the large language model, making the system more robust to be able to address possible future queries. See KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 415-421, 82 USPQ2d 1385, 1395-97 (2007).
For claim 13, claim 9 is incorporated but the combination of Tunstall-Pedoe in view of Menick further in view of Gao fails to disclose the limitation of this claim, for which the reference of Shen is now introduced to teach as:
the method, wherein the interrogation request is automatically generated upon the source document being accessed (Shen: [0019] — generating future queries based on language models that represent previous queries; FIG. 2 — data collection (accessing source documents) to then lead to the generation of future queries; [0031] — the presence of a language model leading to the generation of future queries).
The same motivation combination which introduced the reference of Shen in claim 7 above is applicable here still.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Tunstall-Pedoe (WO 2023/161630 A1) in view of Menick (“Teaching language models to support answers with verified quotes.”) further in view of Gao (“Attributed text generation via post-hoc research and revision.”) as applied to claim 1, and further in view of OSMON et al. (US 2021/0097096 A1: hereafter — Osmon).
For claim 8, claim 1 is incorporated but the combination of Tunstall-Pedoe in view of Menick further in view of Gao fails to disclose the limitation of this claim, for which the reference of Osmon is now introduced to teach as the system, wherein the operations further comprise storing the responsive results with the source document (Osmon: [0025] — having a response database that store responses (responsive results)).
The combination of Tunstall-Pedoe in view of Menick further in view of Gao provides teaching for storing source documents, especially that used for training (Menick: Page 32 D.2. the presence of stored documents applied to training) but differs from the claimed invention in that the claimed invention further provides teaching storing the responsive results. This isn’t new to the art as the reference of Osmon is seen to teach above.
Hence, before the effective filing date of the claimed invention, one of ordinary skill in the art would have found it obvious to incorporate the teaching of Osmon which stores responsive results, with the teaching of the combination of Tunstall-Pedoe in view of Menick further in view of Gao which stores source documents, to thereby come up with the claimed invention. The combination of both prior art elements would have provided the predictable result of not having to consume resources on generating new responsive results when the system encounters a query that requires an already-available response. See KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 415-421, 82 USPQ2d 1385, 1395-97 (2007).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Tunstall-Pedoe PEDOE (WO 2023/161630 A1) in view of Menick (“Teaching language models to support answers with verified quotes.”) further in view of Gao (“Attributed text generation via post-hoc research and revision.”) as applied to claim 9, and further in view of SABAPATHY et al. (US 20223/0419051 A1: hereafter — Sabapathy).
For claim 10, claim 9 is incorporated but the combination of Tunstall-Pedoe in view of Menick further in view of Gao fails to explicitly disclose the limitation of this claim, for which the reference of Sabapathy is now introduced to teach as the method, wherein generating the responsive results includes removing the first asserted quote (Sabapathy: [0106] — removing hallucinated terms from a paraphrased contextual summary (removing asserted quotes that were hallucinated, meaning they weren’t found in the source content)).
The combination of Tunstall-Pedoe in view of Menick further in view of Gao provides teaching for having a first asserted quote when providing a response to a prompt. It differs from the claimed invention in that the claimed invention further provides teaching for removing the first asserted quote. This isn’t new to the art as the reference of Sabapathy is seen to teach above.
Hence, before the effective filing date of the claimed invention, one of ordinary skill in the art would have found it obvious to incorporate the teaching of Sabapathy which removes hallucinated terms as the first asserted quote, with the teaching of the combination of Tunstall-Pedoe in view of Menick further in view of Gao which provides teaching for the presence of a first asserted quote, to thereby come up with the claimed invention. The combination of both prior art elements would have provided the predictable result of ensuring that made-up responses which might be incorrect are not presented as verified answers to a query. See KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 415-421, 82 USPQ2d 1385, 1395-97 (2007).
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Tunstall-Pedoe (WO 2023/161630 A1) in view of Menick (“Teaching language models to support answers with verified quotes.”).
For claim 16, Tunstall-Pedoe discloses a computer-implemented method for performing attribution verification for outputs from a large language model (LLM) (Tunstall-Pedoe: page 8 par 2 — avoiding hallucinations; Page 6 Par 1 — interacting with an LLM), comprising:
receiving an interrogation request [[about a source document having content]] (Tunstall-Pedoe: Page 10 lines 13–14 — receiving a natural language query (as the interrogation request));
generating a first LLM prompt [[that includes the interrogation request, the content of the source document, and first instructions for inclusion of verbatim quotes from the source document]] (Tunstall-Pedoe: Page 15 Par 2 — LLM generating continuation data (as the claimed LLM prompt));
[[executing a string-matching query against the content of the source document to]] determine that the first text string is not present in the content of the source document (Tunstall-Pedoe: Page 8 par 2–3 — checking sections of the continuation generated by the LLM against the database and retrieving sources where the continuation match text contained within the database; Page 7 par 2 — validating natural language for factual accuracy (indicating a check for text strings of quotes that are either present or not present in the content of the source document); Page 133 par 3 — avoiding factual inaccuracies (indicating a check to ensure that the first text string is not present in the source content, as this would be considered a factual inaccuracy));
based on the first text string not being present in the content of the source document, providing a second LLM prompt as input to the LLM, [[the second LLM prompt comprising the interrogation request, the content of the source document, and second instructions for inclusion of verbatim quotes from the source document]] (Tunstall-Pedoe: Page 133 par 2 — when a factual inaccuracy is encountered, an alternative continuation is generated by the LLM (when the first text string which is a factual inaccuracy is not present in the source content, a new second prompt is generated));
[[executing a string-matching query against the content of the source document]] to determine that the second string is present in the content of the source document (Tunstall-Pedoe: Page 8 par 2–3 — checking sections of the continuation generated by the LLM against the database and retrieving sources where the continuation match text contained within the database; Page 7 par 2 — validating natural language for factual accuracy (indicating a check for text strings of quotes that are either present or not present in the content of the source document));
generating responsive results from the second LLM output, wherein the responsive results include at least the second asserted quote (Tunstall-Pedoe: Page 8 par 2–3 — checking sections of the continuation generated by the LLM against the database and retrieving sources where the continuation match text contained within the database; Page 7 par 2 — validating natural language for factual accuracy (indicating a check for text strings of quotes that are either present or not present in the content of the source document); Page 15 Par 2 — ensuring a fact-checked output is generated and presented to the user); and
causing a display of the responsive results (Tunstall-Pedoe: Page 19 par 3 — outputting the responsive answer on a display).
The reference of Tunstall-Pedoe provides teaching for the above but fails to provide the further teachings of this claim, for which the reference of Menick is now introduced to teach as:
providing the first LLM prompt as input into an LLM (Menick: Page 2 Par 6 — given an input query, the system retrieves relevant documents … and presents the language model a large context drawn from multiple documents (teaching of inputting the prompt into a language model); Page 2 Par 4 — the use of a large language model);
receiving, from the LLM, a first LLM output including a first asserted quote from the source document (Menick: Section 1. Figure 1 — examples showing the generation of LLM outputs that include asserted quotes from the source document (the Wikipedia page));
extracting a first text string from the asserted quote in the first LLM output (Menick: Page 2 par 4 — generating answers that can be verbatim quotes of supporting evidence as a single string evidence (indicating the extraction of a text string from the section used as the asserted quote to answer a question as obtained by the LLM); Page 3 section 2.1. — inline evidence syntax to show a verification indicator which extracts as part of the answer, a text string from the obtained source document as a precise quote from the document);
executing a string-matching query against the content of the source document to determine that the first text string is not present in the content of the source document (Menick: Page 2 par 4 — generating answers that can be verbatim quotes of supporting evidence as a single string evidence (indicating the extraction of a text string from the section used as the asserted quote to answer a question and that it is present in the source document); Page 24 Appendix — matching snippet position inside scrapped document; Page 11 par 1 — finding the closest matching sequence of n+2 sentences regarding the number of sentences in the answer; Page 35 F.1 — ‘Wrong quote: if the quote is not matching verbatim the full source’ (as a clear string matching check between the quote and content of the source document));
based on the first text string not being present in the content of the source document, providing a second LLM prompt as input to the LLM, the second LLM prompt comprising the interrogation request, the content of the source document, and second instructions for inclusion of verbatim quotes from the source document (Menick: Page 5 Par 2 — combining a question (interrogation request) with retrieved documents into a prompt; Page 37 H. — forming prompts using a desired syntax; Page 39 J. — prompting by using Inline Evidence Syntax in order to enforce verbatim quotes (thereby teaching of instructions for inclusion of verbatim quotes); Page 2 Par 4 — a special syntax for a language model to use when providing answers in order to constrain the outputs of the model to be exact quotes from the retrieved documents);
receiving a second output from the LLM including a second asserted quote from the source document (Menick: Section 1. Figure 1 — examples showing the generation of LLM outputs that include asserted quotes from the source document (the Wikipedia page));
extracting a second text string from the asserted quote in the second LLM output (Menick: Page 2 par 4 — generating answers that can be verbatim quotes of supporting evidence as a single string evidence (indicating the extraction of a text string from the section used as the asserted quote to answer a question as obtained by the LLM); Page 3 section 2.1. — inline evidence syntax to show a verification indicator which extracts as part of the answer, a text string from the obtained source document as a precise quote from the document);
executing a string-matching query against the content of the source document to determine that the second string is present in the content of the source document (Menick: Page 2 par 4 — generating answers that can be verbatim quotes of supporting evidence as a single string evidence (indicating the extraction of a text string from the section used as the asserted quote to answer a question and that it is present in the source document); Page 24 Appendix — matching snippet position inside scrapped document; Page 11 par 1 — finding the closest matching sequence of n+2 sentences regarding the number of sentences in the answer; Page 35 F.1 — ‘Wrong quote: if the quote is not matching verbatim the full source’ (as a clear string matching check between the quote and content of the source document)).
The same motivation for combination applied to claim 1 above for introducing the Menick reference is applicable here still.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Tunstall-Pedoe (WO 2023/161630 A1) in view of Menick (“Teaching language models to support answers with verified quotes.”) as applied to claim 16, further in view of Sharevski, Filipo, et al. (“Misinformation warning labels: Twitter's soft moderation effects on COVID-19 vaccine belief echoes.” arXiv preprint arXiv:2104.00779 (2021): hereafter — Sharevski).
For claim 18, claim 16 is incorporated and the combination of Tunstall-Pedoe in view of Menick discloses the method, wherein the responsive results are first responsive results (Menick: Page 2 par 4 — generating answers that can be verbatim quotes of supporting evidence as a single string evidence), and the method further comprises:
generating first responsive results including the first LLM output with the asserted quote [[and an unverified indicator]] (Menick: Page 2 par 4 — generating answers that can be verbatim quotes of supporting evidence as a single string evidence (indicating the extraction of a text string from the section used as the asserted quote to answer a question and that it is present in the source document); Figure 1 — an inclusion of source in the response serves as a verification indicator; Page 35 F.1. — checking for bad syntax penalty (indicating an attempt at string matching of a quote verbatim, to the source so as to apply it, and penalise it when it instead does not match verbatim)); and
causing a display of the first responsive results (Tunstall-Pedoe: Page 19 par 3 — outputting the responsive answer on a display).
The combination of Tunstall-Pedoe in view of Menick fails to disclose the further limitation of this claim regarding the presence of an unverified indicator. The reference of Sharevski is now introduced to teach this instead:
generating first responsive results including the first LLM output with the asserted quote and an unverified indicator (Sharevski: 5.5 — Twitter adding labels to tweets indicating unverified claims (providing an unverified indicator)).
The combination of Tunstall-Pedoe in view of Menick provides teaching for generating responsive results to an input query. It differs from the claimed invention in that the claimed invention further provides teaching for generating an unverified indicator. This isn’t new to the art as the reference of Sharevski is seen to teach above.
Hence, before the effective filing date of the claimed invention, one of ordinary skill in the art would have found it obvious to incorporate the teaching of Sharevski which provides an unverified label to messages, with the teaching of the combination of Tunstall-Pedoe in view of Menick which generates responsive results, to thereby come up with the claimed invention. The combination of both prior art elements would have provided the predictable result of informing a reader that the provided information may not be valid so that the reader can be better informed on how to make use of the presented message. See KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 415-421, 82 USPQ2d 1385, 1395-97 (2007).
Claims 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Tunstall-Pedoe (WO 2023/161630 A1) in view of Menick (“Teaching language models to support answers with verified quotes.”) as applied to claim 16, further in view of Gao (“Attributed text generation via post-hoc research and revision.”).
For claim 19, claim 16 is incorporated but the combination of Tunstall-Pedoe in view of Menick fails to disclose the limitation of this claim, for which the reference of Gao is now introduced to instead teach as the method, wherein the second LLM prompt is the same as the first LLM prompt (Gao: 3.2 — checking if there’s a disagreement between the evidence (this is taken as the content source) and the passage y (passage y is the output), when there is a disagreement, y gets edited, with a revision stage being the initialisation of y = x (Figure 1 shows x as the input, showing the second prompt as the first prompt).
The combination of Tunstall-Pedoe in view of Menick provides teaching for the generation of LLM prompts, both first and second, but differs from the claimed invention in that the claimed invention further provides that the first LLM prompt is the second LLM prompt. This isn’t new to the art as the reference of Gao goes to show above.
Hence, before the effective filing date of the claimed invention, one of ordinary skill in the art would have found it obvious to incorporate the known teaching of Gao which provides that the first and second LLM prompt are the same, into improving upon the teaching of the combination of Tunstall-Pedoe in view of Menick which generates the LLM prompts, to thereby come up with the claimed invention. The combination of both prior art elements would have provided a confirmation and verification of the generated responses being obtained through different routes but the same prompt. See KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 415-421, 82 USPQ2d 1385, 1395-97 (2007).
For claim 20, claim 16 is incorporated and as applied to claim 19 above, the combination of Tunstall-Pedoe in view of Menick further in view of Gao discloses the method, further comprising revising the first LLM prompt to form the second LLM prompt (Gao: 3.2 — checking if there’s a disagreement between the evidence (this is taken as the content source) and the passage y (passage y is the output), when there is a disagreement, y gets edited, with a revision stage being the initialisation of y = x (Figure 1 shows x as the input, showing the second prompt as the first prompt),
wherein revising the first LLM prompt includes adding additional emphasis on producing a verbatim quote in the second instructions (Menick: Page 2 Par 4 — ensuring that answers are generated to produce verbatim quotes with a generative approach).
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Tunstall-Pedoe (WO 2023/161630 A1) in view of Menick (“Teaching language models to support answers with verified quotes.”) as applied to claim 16, further in view of Gao (“Attributed text generation via post-hoc research and revision.”) as applied to claim 1, and further in view of Gray et al. (US 2024/0220735 A1: hereafter — Gray1).
For claim 21, claim 1 is incorporated but the combination of Tunstall-Pedoe in view of Menick further in view of Gao fails to disclose the limitation of this claim, for which the reference of Gray is now introduced to teach as:
the system, wherein the responsive results further include a selectable source indicator, where selection of the source indicator causes a source document to be displayed and navigated to a portion of the source document where the second quote is presented (Gray: [0131] — the natural language summary includes linkified portions, source identifiers, that can be selected to cause navigation to corresponding search result documents).
The combination of Tunstall-Pedoe in view of Menick further in view of Gao provides teaching for providing responsible results, but differs from the claimed invention in that the claimed invention further provides teaching for the responsive results including a selectable source indicator that causes a source document to be displayed and navigated to a portion of it. This isn’t new to the art as the reference of Gray is seen to teach above.
Hence, before the effective filing date of the claimed invention, one of ordinary skill in the art would have found it obvious to incorporate the known teaching of Gray which presents natural language responses that includes links, source identifiers which are presented and can be navigated to, into improving upon the teaching of the combination of Tunstall-Pedoe in view of Menick further in view of Gao which provides responsible results, to thereby come up with the claimed invention. The combination of both prior art elements would have provided the predictable result of granting the user the opportunity to be able to personally verify the provided responses. See KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 415-421, 82 USPQ2d 1385, 1395-97 (2007).
Conclusion
The prior art made of record and not relied upon is considered pertinent to Applicant’s disclosure.
See PTO-892.
Any inquiry concerning this communication or earlier communications from the Examiner should be directed to OLUWADAMILOLA M. OGUNBIYI whose telephone number is (571)272-4708. The Examiner can normally be reached Monday – Thursday (8:00 AM – 5:30 PM Eastern Standard Time).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, Applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the Examiner by telephone are unsuccessful, the Examiner’s Supervisor, PARAS D. SHAH can be reached at (571) 270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/OLUWADAMILOLA M OGUNBIYI/Examiner, Art Unit 2653
/Paras D Shah/Supervisory Patent Examiner, Art Unit 2653
05/15/2026
1 This reference was filed on 09 August 2023, but has a provisional application with the application number 63/436,416, having an earlier filing date of 30 December 2022. The information relied upon in this rejection has been verified to be contained [00112] of the Specification of this provisional application.
Prosecution Insights