Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on September 03, 2025 is being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 13 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 13, the claim states ”maximizing a coverage over the one or more claims”. The phrase is unclear because the claim does not specify how “coverage” is measured or what standard determines when coverage has been maximized. For example, it is unclear whether coverage concerns the number of claims addressed, the degree of evidentiary support for each claim, or another measurement.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 25 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because the recited “one or more computer readable media” are not limited to non-transitory storage media. Under the broadest reasonable interpretation, the limitation encompasses transitory propagating signals, which do not fall within any of the four statutory categories.
Claim [1-25] rejected under 35 U.S.C. 101 because the claimed invention is directed to
mental process without significantly more.
Regarding claim 1, A computer-implemented method for improved attribution of model- generated content, the method comprising: obtaining, by a computing system comprising one or more computing devices, a first textual content generated by a first machine-learned language model; retrieving, by the computing system, one or more evidence sources relating to the first textual content generated by the first machine-learned language model; generating, by the computing system, a second textual content as an output of a second machine-learned language model, based on a comparison between the first textual content and the one or more evidence sources; and associating, by the computing system, one or more portions of the one or more evidence sources as attributions for the first or second textual content.
[A person could perform the underlying process by reading a draft, locating reference materials, comparing the draft with those materials, revising unsupported documents, and attaching citations. ].
As described above, these limitations can be carried out as a series of mental steps.
This judicial exception is not integrated into a practical application because the only additional elements recited are the computing system comprising one or more computing devices, the first machine language model and the second machine learning model and these additional elements are nothing more than using a generic computer as a tool to implement the mental process.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the computing system, the first machine language model and the second machine learning model, and these additional elements are nothing more than instructions to apply the mental process using general-purpose hardware.
Regarding claim 2, the method of claim 1, wherein retrieving, by the computing system, the one or more evidence sources comprises: generating, by the computing system, one or more queries based on the first textual content; and retrieving, by the computing system, the one or more evidence sources based on the one or more queries.
[A person could read the first textual content, formulate search questions based on its subject matter, and use those questions to locate relevant reference materials. These steps constitute collecting and organizing information through a mental process.]
As described above, these limitations can be carried out as a series of mental steps.
This judicial exception is not integrated into a practical application because the only additional element recited is the computing system and this additional element is nothing more than using a generic computer as a tool to implement the mental process.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the computing system is the only additional element and this additional element is nothing more than instructions to apply the mental process using general-purpose hardware.
Regarding claim 3, The method of claim 2, wherein retrieving, by the computing system, the one or more evidence sources based on the one or more queries comprises a web search.
[A person could search an organized collection of information for materials relevant to a query.]
As described above, these limitations can be carried out as a series of mental steps. This judicial exception is not integrated into a practical application because the only additional elements recited is the computing system and this additional element is nothing more than using a generic computer as a tool to implement the mental process.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the computing system is the only additional element and this additional element is nothing more than instructions to apply the mental process using general-purpose hardware.
Regarding claim 4, The method of claim 1, wherein generating, by the computing system, one or more queries based on the first textual content comprises prompting a third machine-learned language model.
[A person could read textual content, identify its subject matter or factual assertions, and formulate corresponding search questions.].
As described above, these limitations can be carried out as a series of mental steps.
This judicial exception is not integrated into a practical application because the only additional elements recited are the computing system and a third machine-learned language model and these additional elements are nothing more than using a generic computer as a tool to implement the mental process.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the computing system and a third machine-learned language model are the only additional elements and these additional elements are nothing more than instructions to apply the mental process using general-purpose hardware.
Regarding claim 5, The method of claim 1, wherein generating, by the computing system, one or more queries based on the first textual content comprises: identifying one or more claims associated with the first textual content; and generating a query associated with at least one of the one or more claims.
[A person could read textual content, identify its factual assertions and formulate a search question relating to one of those assertions.]
As described above, these limitations can be carried out as a series of mental steps. This judicial exception is not integrated into a practical application because the only additional elements recited is the computing system and this additional element is nothing more than using a generic computer as a tool to implement the mental process.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the computing system is the only additional element and this additional element is nothing more than instructions to apply the mental process using general-purpose hardware.
Regarding claim 6, The method of claim 1 any preceding claim, comprising:
performing at least one of:
prompting a third machine-learned language model to identify one or more claims associated with the first textual content; or prompting a third machine-learned language model to generate a query associated with at least one of the one or more claims.
[A person could review textual content to identify its factual assertions or formulate a search question concerning an identified assertion].
As described above, these limitations can be carried out as a series of mental steps.
This judicial exception is not integrated into a practical application because the only additional elements recited is the third machine learned language model and this additional element is nothing more than using a generic computer as a tool to implement the mental process.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the third machine learned language model is the only additional element and this additional element is nothing more than instructions to apply the mental process using general-purpose hardware.
Regarding claim 7, The method of claim 6, wherein prompting a third machine-learned language model comprises few-shot prompting.
[A person could provide several examples showing factual assertions should be identified or how corresponding search queries should be generated and then request another result based on those examples. This involves teaching by example, observation, and judgement.].
As described above, these limitations can be carried out as a series of mental steps.
This judicial exception is not integrated into a practical application because the only additional elements recited are a computer system and a third language model and these additional elements are nothing more than generic computer implementing the mental process using general-purpose software and general-purpose hardware. Few shot promoting merely implements the abstract information analysis process using a generic machine learned language model.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the only additional elements recited are a computer system and a third language model and these additional elements are nothing more than instructions to apply the mental process using general-purpose software and hardware.
Regarding claim 8, The method of claim 6, wherein prompting a third machine-learned language model comprises chain-of-thought prompting.
[A person could perform the task by considering the content, reasoning through immediate steps, and formulating an appropriate query. ]
As described above, these limitations can be carried out as a series of mental steps.
This judicial exception is not integrated into a practical application because the only additional element recited is a third language model and this additional element is nothing more than generic computer implementing the mental process using general-purpose software and general-purpose hardware. Chain of thought promoting uses a generally recited language model to automate human reasoning and query formulation and therefore does not add significantly more to the abstract idea.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the only additional element recited is a third language model and this additional element is nothing more than instructions to apply the mental process using general-purpose software and hardware.
Regarding claim 9, The method of claim 1, further comprising: extracting, by the computing system, one or more evidence snippets from the one or more evidence sources retrieved by the computing system; and comparing, by the computing system, the one or more evidence snippets extracted by the computing system to the first textual content.
[A person could review reference documents, select relevant passages, and compare passages with statements in the original text to determine whether they correspond].
As described above, these limitations can be carried out as a series of mental steps.
This judicial exception is not integrated into a practical application because the only additional elements recited is a computer system and this additional element is nothing more than generic computer implementing the mental process using general-purpose software and general-purpose hardware.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the only additional elements recited is a computer system and this additional element is nothing more than instructions to apply the mental process using general-purpose software and hardware.
Regarding claim 10, The method of claim 9, wherein extracting, by the computing system, one or more evidence snippets from the one or more evidence sources retrieved by the computing system comprises: determining one or more candidate evidence snippets; generating, for each candidate evidence snippet, a respective score based on a query that was used to retrieve the one or more evidence sources; and selecting a subset of candidate evidence snippets based on the respective scores.
[A person could identify potentially relevant passages, evaluate and rank each passage according to relevance to a search question, and select passages having the best rankings.]
As described above, these limitations can be carried out as a series of mental steps.
This judicial exception is not integrated into a practical application because the only additional element recited is a computer system and this additional element is nothing more than generic computer implementing the mental process using general-purpose software and general-purpose hardware.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the only additional element recited is a computer system and this additional element is nothing more than instructions to apply the mental process using general-purpose software and hardware.
Regarding claim 11, The method of claim 10, wherein the respective scores are relevance scores generated by a machine-learned query-document relevance model.
[A person could compare a search query with each candidate passage, judge how relevant each passage is to the query, and assign a corresponding relevance score.]
As described above, these limitations can be carried out as a series of mental steps. This judicial exception is not integrated into a practical application because the only additional element recited is a machine-learned query-document relevance model and this additional element is nothing more than generic computer implementing the mental process using general-purpose software and general-purpose hardware.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the only additional element recited is a machine-learned query-document relevance model and this additional element is nothing more than instructions to apply the mental process using general-purpose software and hardware.
Regarding claim 12, The method of claim 10, wherein: the one or more evidence sources are textual documents; and determining one or more candidate evidence snippets comprises running a sliding window across a respective textual document.
[A person could review a textual document one portion at a time, including successive or overlapping portions, and treat each reviewed portion as a candidate evidence snippet.].
As described above, these limitations can be carried out as a series of mental steps. This judicial exception is not integrated into a practical application because the only additional element recited is a computing system which is automating examination of the document through a sliding window and this additional element is nothing more than generic computer implementing the mental process using general-purpose software and general-purpose hardware.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the only additional elements recited is a computing system which is automating examination of the document through a sliding window and this additional element is nothing more than instructions to apply the mental process using general-purpose software and hardware.
Regarding claim 13, The method of claim 10, wherein selecting a subset of candidate evidence snippets based on the respective score comprises: maximizing a coverage over the one or more claims.
[A person could review the candidate passages and select the group that collectively addresses the greatest number of claims in the first textual content. ]
As described above, these limitations can be carried out as a series of mental steps.
No additional elements are recited. Therefore, the claim does not describe a practical application or significantly more than the mental process.
Regarding claim 14, The method of claim 9, wherein comparing, by the computing system, the one or more evidence snippets extracted by the computing system to the first textual content comprises: determining one or more first levels of agreement, with respect to one or more claims associated with the first textual content, between the one or more evidence snippets and the first textual content.
[A person could identify factual claims in a written passage, compare each claim with relevant evidence passages, and judge the degree to which the evidence supports, contradicts, or does not address the claim].
As described above, these limitations can be carried out as a series of mental steps.
This judicial exception is not integrated into a practical application because the only additional elements recited is a computer system and this additional element is nothing more than generic computer implementing the mental process using general-purpose software and general-purpose hardware.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the only additional elements recited is a computer system and this additional element is nothing more than instructions to apply the mental process using general-purpose software and hardware.
Regarding claim 15, The method of claim 14, wherein determining one or more first levels of agreement, with respect to one or more claims associated with the first textual content, between the one or more evidence snippets and the first textual content comprises: evaluating the first textual content and the evidence snippet with a machine- learned agreement model.
[A person could read the first textual content and an evidence passage, compare their respective statements, and determine whether they agree].
As described above, these limitations can be carried out as a series of mental steps.
This judicial exception is not integrated into a practical application because the only additional element recited is machine- learned agreement model and this additional element is nothing more than generic computer implementing the mental process using general-purpose software and general-purpose hardware.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the only additional element recited is a machine- learned agreement model and this additional element is nothing more than instructions to apply the mental process using general-purpose software and hardware.
Regarding claim 16, the method of claim 15, wherein evaluating the first textual content and the evidence snippet with a machine-learned agreement model comprises prompting the machine-learned agreement model with chain-of-thought prompting.
[A person could perform this by reviewing the text and evidence, considering intermediate reasoning steps, and reaching an agreement. ]
As described above, these limitations can be carried out as a series of mental steps.
This judicial exception is not integrated into a practical application because the only additional element recited is a using a chain of thought prompting with a generally recited agreement model and this additional element is nothing more than generic computer implementing the mental process using general-purpose software and general-purpose hardware.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the only additional element recited is a chain of thought prompting with a generally recited agreement model and this additional element is nothing more than instructions to apply the mental process using general-purpose software and hardware.
Regarding claim 17 , The method of claim 15, wherein evaluating the first textual content and the evidence snippet with a machine-learned agreement model comprises prompting the machine-learned agreement model with few-shot prompting.
[A person could perform this evaluation after reviewing several examples showing how evidence is classified as supporting, contradicting, or not addressing a statement].
As described above, these limitations can be carried out as a series of mental steps.
This judicial exception is not integrated into a practical application because the only additional element recited is a using a few shot prompting with a generally recited agreement model and this additional element is nothing more than generic computer implementing the mental process using general-purpose software and general-purpose hardware.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the only additional element recited is a few shot prompting with a generally recited agreement model and this additional element is nothing more than instructions to apply the mental process using general-purpose software and hardware.
Regarding claim 18, The method of claim 14, wherein generating, by the computing system, a second textual content as an output of a second machine-learned language model comprises: identifying a respective claim of the one or more claims associated with the first textual content, wherein the first level of agreement with respect to the respective claim indicates a lack of complete agreement; and generating a second textual content, such that a second level of agreement, with respect to the respective claim and between the second textual content and the one or more evidence snippets, indicates a greater degree of agreement than the first level of agreement with respect to the respective claim.
[A person could compare a statement with supporting evidence, identify that the statement is not fully supported, revise the statement based on the evidence, and determine that the revised statement agrees more closely with the evidence.]
As described above, these limitations can be carried out as a series of mental steps.
This judicial exception is not integrated into a practical application because the only additional element recited is a second machine learning model and this additional element is nothing more than generic computer implementing the mental process using general-purpose software and general-purpose hardware.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the only additional element recited is second machine learning model and this additional element is nothing more than instructions to apply the mental process using general-purpose software and hardware.
Regarding claim 19, The method of claim 1, wherein generating, by the computing system, a second textual content as an output of a second machine-learned language model comprises prompting the second machine-learned language model using chain-of-thought prompting..
[A person could compare a statement with the evidence, reason through the inconsistences, and revise the statement accordingly.]
As described above, these limitations can be carried out as a series of mental steps.
This judicial exception is not integrated into a practical application because the only additional element recited is a chain of thought prompting with a second machine-learned language model and this additional element is nothing more than generic computer implementing the mental process using general-purpose software and general-purpose hardware.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the only additional element recited is chain of thought prompting with a second machine-learned language model and this additional element is nothing more than instructions to apply the mental process using general-purpose software and hardware.
Regarding claim 20, The method of claim 1, wherein generating, by the computing system, a second textual content as an output of a second machine-learned language model comprises prompting the second machine-learned language model using few-shot prompting.
[A person could review several examples demonstrating how unsupported statements were revised and then revise another statement consistently with those examples.]
As described above, these limitations can be carried out as a series of mental steps.
This judicial exception is not integrated into a practical application because the only additional element recited is a few shot prompting with a second machine-learned language model and this additional element is nothing more than generic computer implementing the mental process using general-purpose software and general-purpose hardware.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the only additional element recited is few shot prompting with a second machine-learned language model and this additional element is nothing more than instructions to apply the mental process using general-purpose software and hardware.
Regarding claim 21, The method of claim 1, further comprising: determining an edit distance based on one or more differences between the first textual content and the second textual content; and based on the edit distance, determining whether to output the second textual content..
[A person could compare an original passage with a revised passage, identify or count their textual difference, and determine whether the revision differs too substantially from the original to be used].
As described above, these limitations can be carried out as a series of mental steps.
This judicial exception is not integrated into a practical application because the only additional element recited is computing system and this additional element is nothing more than generic computer implementing the mental process using general-purpose software and general-purpose hardware.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the only additional element recited is a computing system and this additional element is nothing more than instructions to apply the mental process using general-purpose software and hardware.
Regarding claim 22, The method of claim 1, further comprising: determining an agreement score between the second textual content and one or more evidence snippets extracted from the one or more evidence sources; determining a preservation score indicative of a similarity between the first textual content and the second textual content; and based on a combination of the agreement score and the preservation score, determining whether to output the second textual content.
[A person could evaluate how well revised text agrees with supporting evidence, separately evaluate how closely the revision preserves the original text, balance the two evaluations, and decide whether to use the revision].
As described above, these limitations can be carried out as a series of mental steps.
This judicial exception is not integrated into a practical application because the only additional element recited is computing system and this additional element is nothing more than generic computer implementing the mental process using general-purpose software and general-purpose hardware.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the only additional element recited is a computing system and this additional element is nothing more than instructions to apply the mental process using general-purpose software and hardware.
Regarding claim 23, The method of claim 1, wherein: the one or more evidence sources are textual documents.
[A person could obtain and review written documents to locate evidence concerning statements in the first textual content.].
As described above, these limitations can be carried out as a series of mental steps.
No additional elements are recited. Therefore, the claim does not describe a practical application or significantly more than the mental process.
Regarding claim 24, A computing system, comprising: one or more processors; and one or more non-transitory computer-readable media storing instructions that are executable by the one or more processors to cause the computing system to perform one or more operations, the operations comprising: obtaining, by a computing system comprising one or more computing devices, a first textual content generated by a first machine-learned language model; retrieving, by the computing system, one or more evidence sources relating to the first textual content generated by the first machine-learned language model; generating, by the computing system, a second textual content as an output of a second machine-learned language model, based on a comparison between the first textual content and the one or more evidence sources; and associating, by the computing system, one or more portions of the one or more evidence sources as attributions for the first or second textual content.
[A person could perform the underlying process by reading a draft, locating reference materials, comparing the draft with those materials, revising unsupported documents, and attaching citations. ].
As described above, these limitations can be carried out as a series of mental steps.
This judicial exception is not integrated into a practical application because the only additional elements recited are the processors, one or more non-transitory computer-readable media storing instructions, the first machine language model and the second machine learning model and these additional elements are nothing more than using a generic computer as a tool to implement the mental process.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, processors, one or more non-transitory computer-readable media storing instructions, the first machine language model and the second machine learning model, and these additional elements are nothing more than instructions to apply the mental process using general-purpose hardware.
Regarding claim 25, one or more computer-readable media storing instructions that are executable by a computing system to cause the computing system to perform one or more operations, the operations comprising: obtaining, by a computing system comprising one or more computing devices, a first textual content generated by a first machine-learned language model; retrieving, by the computing system, one or more evidence sources relating to the first textual content generated by the first machine-learned language model; generating, by the computing system, a second textual content as an output of a second machine-learned language model, based on a comparison between the first textual content and the one or more evidence sources; and associating, by the computing system, one or more portions of the one or more evidence sources as attributions for the first or second textual content.
[A person could perform the underlying process by reading a draft, locating reference materials, comparing the draft with those materials, revising unsupported documents, and attaching citations. ].
As described above, these limitations can be carried out as a series of mental steps.
This judicial exception is not integrated into a practical application because the only additional elements recited are computer-readable media storing instructions that are executable by a computing system, the first machine language model and the second machine learning model and these additional elements are nothing more than using a generic computer as a tool to implement the mental process.
This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, computer-readable media storing instructions that are executable by a computing system, the first machine language model and the second machine learning model, and these additional elements are nothing more than instructions to apply the mental process using general-purpose hardware.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim [ 1, 2, 3, 5, 9, 14, 15, 18, 23, 24, 25 ] are rejected under 35 U.S.C. 103 as being unpatentable over Galitsky (US 12,175,187 B2, hereinafter Galitsky) in view of Abraham (US 20220237382 A1, hereinafter Abraham).
Regarding claim 1,
Galitsky teaches
A computer-implemented method for improved attribution of model- generated content, the method comprising: obtaining, by a computing system comprising one or more computing devices, a first textual content generated by a first machine-learned language model;
[Column 7, lines 63-64 "Deep Learning generated content 118 can be generated by deep learning generation module 120" where raw text/DL generated content 118 is mapped to first textual content and deep learning generation module 120 is mapped to first machine learned language model];
[Column 8, lines 57- 60 "Prior to performing the operations of method 200, raw text may be obtained (e.g., from DL generated content 118 of FIG. 1, from DLG module 120 of FIG. 1, or any suitable source).
retrieving, by the computing system, one or more evidence sources relating to the first textual content generated by the first machine-learned language model;
[Column 9, lines 10- 16 "To find relevant sentences from a knowledge base (e.g., the web, a corpus of text, etc.) with which the raw text sentence can be corrected, the correction module may develop one or more search queries
from the raw text sentence using a discourse tree generated from the raw text sentence based at least in part on Rhetorical Structure Theory."];
[Column 17, lines 51-58 "Returning to FIG. 2, at 203, the one or more search queries generated at 202 using one or more of the techniques discussed above, may be executed to obtain a list of candidate
true sentences. For example, the one or more queries may be provided (e.g., via a search API) to one or more knowledge bases. An example of a knowledge base may include Wikipedia®, Bing®, Google®, and the like."];
[Column 17, lines 61-65 "A set of search results may be obtained based on executing the search queries against the knowledge base(s). Each sentence (or some number of sentences) of one
or more search results may serve as a candidate true sentence with which the raw text sentence can be corrected." where knowledge base search results maps to evidence sources and retrieved using queries generated from the raw text sentence maps to relating to the first textual content];
generating, by the computing system, a second textual content as an
[Column 8, lines 47-52 "Computing device 102 may include correction module 122. In some embodiments, correction module 122 may be configured to identify and implement one or more corrections to raw text provided as input ( e.g., raw text generated by the DLG module 120, raw text retrieved from DL generated content 118, etc.). Operations"];
[Column 18, lines 11-12 "Correction module 122 can implement method 600 on computing device 102 of FIG. 1."];
[Column 18, lines 14-22 "It will be appreciated that method 600 can repeat additional times in the event that a candidate true sentence (e.g., a sentence from one of the search results obtained at 203 of FIG. 3) is not deemed to be sufficiently appropriate or accurate. Further, method 600 can iterate multiple times to determine whether a candidate true sentence aligns with a raw
text sentence sufficiently to provide replacement fragments for the raw text sentence."];[Column 17, lines 66-67 , Column 18, lines 1-5 "At 204, the raw text sentence and each candidate true sentence can be generalized. That is the syntactic and semantic representations of the raw text sentence may be
mapped to syntactic and semantic representations of each candidate sentence to determine if a given candidate sentence may be used to correct/replace an entity of the raw text sentence."];
[Column 23, lines 34-36 "For example, a syntactically matching and semantically closest true sentence candidate may be selected."];
[Column 23, lines 33-34 "At 207, a candidate true statement may be selected (e.g., based on the semantic/syntactic scores assigned at 204)."];
[Column 23, lines 40-42 "At 208, entity and/or phrase substitutions may be made using entities and/or phrases of the selected candidate true sentence"];
[Column 24, lines 39-46 "At 212, output may be generated…..the output may include the corrected raw sentence, and/or an indication that the correction was successful and/or an indication of the particular
corrections made."];
Galitsky does not teach
output of a second machine-learned language model
associating, by the computing system, one or more portions of the one or more evidence sources as attributions for the first or second textual content.
However, Abraham does teach
output of a second machine-learned language model
[0124 "In some implementations, the claim verification module is trained with a third set of ground truth to link detected claims to one or more pieces of evidence within related references and to determine whether the one or more pieces of evidence support or refute the one or more claims" where claim verification module compares whether evidence supports or refutes the claim];
[0107 " A system may utilize a language model to facilitate text generation or modification to provide the recommendation 808 based on evidence from a set of relevant references (e.g., as determined by the evidence extraction module 112 and the claim verification module 114)." where language model used to generate or modify the textual recommendations maps to second machine learning model and modified textual recommendations map to the second textual content];
[0107 "In some implementations, as shown in FIG. 8, the user input 806 triggers the presentation of a recommendation 808 for modifying the claim associated with the refute indicator 804. In particular, the recommendation 808 comprises modifying the text “47% of lung cancer cases” to instead state “87% of lung cancer cases”"];
associating, by the computing system, one or more portions of the one or more evidence sources as attributions for the first or second textual content
[0035 "Furthermore, at least some implementations of the present disclosure link claims or procedures detected in a body of text to relevant content from the corpus of references and selectively display the relevant content in association with the linked claims,.." ];
[0127 "Act 1114 may additionally or alternatively entail presenting a representation of the set of references in the user interface in association with the one or more claims, presenting a representation of one or more pieces of evidence of the set of references in the user interface in association with the one or more claims, and/or presenting a recommendation for modifying the one or more claims based on the set of references." where one or more pieces of evidence from the set of references maps to one or more portions of the one or more evidence sources, the displayed evidence /reference identifies the evidentiary basis for the associated portion of text is mapped to attribution];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Galitsky with Abraham such that correction module 122 uses a language model to generate corrected text based on a comparison between the original model generated text and retrieved evidence and associates the relevant evidence with the original or corrected text. This would improve the accuracy of model generated content by correcting unsupported statements while enabling users to identify and evaluate the evidence supporting the content.
Regarding claim 2, the rejection of claim 1 is incorporated.
Galitsky does teach
The method of claim 1, wherein retrieving, by the computing system, the one or more evidence sources comprises:
generating, by the computing system, one or more queries based on the first textual content;
[Column 9, lines 6-7 "At 202, a number of queries may be formed from a given raw text sentence"];
[Column 9, lines 10-16 "To find relevant sentences from a knowledge base (e.g., the web, a corpus of text, etc.) with which the raw text sentence can be corrected, the correction module may develop one or more search queries
from the raw text sentence using a discourse tree generated from the raw text sentence based at least in part on Rhetorical Structure Theory." where raw text sentence generated from the queries is the first textual content];
and retrieving, by the computing system, the one or more evidence sources based on the one or more queries.
[Column 17, lines 55-65 "For example, the one or more queries may be provided (e.g., via a search API) to one or more knowledge bases. An example of a knowledge base may
include Wikipedia®, Bing®, Google®, and the like. A knowledge base may additionally, or alternatively, include a corpus of documents (e.g., text) accessible to the correction
module 122. A set of search results may be obtained based on executing the search queries against the knowledge base(s). Each sentence (or some number of sentences) of one
or more search results may serve as a candidate true sentence with which the raw text sentence can be corrected" where candidate true statements contained in the retrieved search results maps to evidence sources and executing the search queries against the knowledge base maps to retrieving based on the queries];
Regarding claim 3, the rejection of claim 2 is incorporated.
Galitsky does teach
The method of claim 2, wherein retrieving, by the computing system,
the one or more evidence sources based on the one or more queries comprises a web search.
[Column 17, lines 55-58 "For example, the one or more queries may be provided (e.g., via a search API) to one or more knowledge bases. An example of a knowledge base may include Wikipedia®, Bing®, Google®, and the like."];
Regarding claim 5, the rejection of claim 1 is incorporated.
Galitsky does not teach
The method of claim 1, wherein generating, by the computing system, one or more queries based on the first textual content comprises:
identifying one or more claims associated with the first textual content; and
generating a query associated with at least one of the one or more claims.
However, Abraham teaches
based on the first textual content comprises:
identifying one or more claims associated with the first textual content; and
[0009 "The system is also configured, in some instances, to input the textual content into the claim detection module to detect one or more claims within the selection of textual content and,"];
generating a query associated with at least one of the one or more claims.
[0010 "… to automatically search the one or more reference repositories for one or more related references that have content that is determined by the evidence extraction module to be related to the detected one or more claims"].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Galitsky with Abraham to identify claims within the model generated text before generating queries associated with the claims because Galitsky already identifies potentially untrue portions of generated text and generates queries from those portions. This would have predictably provided more targeted queries and improved retrieval of evidence relevant to verifying and correcting the identified claims.
Regarding claim 9, the rejection of claim 1 is incorporated.
Galitsky does teach
The method of claim 1, further comprising:
extracting, by the computing system, one or more evidence snippets from the one or more evidence sources retrieved by the computing system; and
[Column 17, lines 61-65 "A set of search results may be obtained based on executing the search queries against the knowledge base(s). Each sentence (or some number of sentences) of one
or more search results may serve as a candidate true sentence with which the raw text sentence can be corrected." where knowledge base search results maps to evidence sources and each sentence of one or more search results maps to evidence snippets extracted from the sources];
comparing, by the computing system, the one or more evidence snippets extracted by the computing system to the first textual content.
[Column 17, lines 66-67 , Column 18, lines 1-5 "At 204, the raw text sentence and each candidate true sentence can be generalized. That is the syntactic and semantic representations of the raw text sentence may be mapped to syntactic and semantic representations of each candidate sentence to determine if a given candidate sentence may be used to correct/replace an entity of the raw text sentence." where raw sentence generated are mapped to first textual content and candidate sentence extracted from search results are mapped to extracted evidence snippets. ];
[Column 23, lines 34-36 "For example, a syntactically matching and semantically closest true sentence candidate may be selected." where syntactically matching and semantically closet true sentence is mapped to comparing];
[Column 23, lines 40-42 "At 208, entity and/or phrase substitutions may be made using entities and/or phrases of the selected candidate true sentence"];
Regarding claim 14, the rejection of claim 9 is incorporated.
Galitsky does not teach
The method of claim 9, wherein comparing, by the computing system, the one or more evidence snippets extracted by the computing system to the first textual content comprises: determining one or more first levels of agreement, with respect to one or more claims associated with the first textual content, between the one or more evidence snippets and the first textual content.
However, Abraham teaches
The method of claim 9, wherein comparing, by the computing system, the one or more evidence snippets extracted by the computing system to the first textual content comprises:
[0080 "For example, in some implementations, a claim verification module 114 is configured to analyze context of a claim and context of evidence to determine similarity or dissimilarity therebetween (e.g., whether the content of the claim and the content of the evidence are associated with similar or dissimilar conclusions, or whether the content of the evidence substantiates the content of the claim)."];
determining one or more first levels of agreement, with respect to one or more claims associated with the first textual content, between the one or more evidence snippets and the first textual content.
[0086 "One will appreciate, in view of the present disclosure, that a claim verification module 114 may determine multiple sets of references or pieces of evidence (e.g., a set of supporting references or pieces of evidence and a set of refuting references or pieces of evidence) and that a set of references or pieces of evidence may comprise evidence that supports a claim as well as evidence that refutes a claim."];
[0087 "In some implementations, the claim verification module 114 associates a classification confidence score with its determination as to whether evidence supports or refutes a claim. The classification confidence score may numerically represent an amount of confidence in a label assigned to a piece of evidence vis-à-vis a claim (e.g., a label of whether the piece of evidence supports the claim or refutes the claim). In some instances, the classification confidence score is based on an error or loss metric or is based on a similarity or dissimilarity of context and/or conclusion between the claim and the evidence (e.g., represented as a percentage)." where first level of agreement between an evidence snippet and a claim in the first textual content maps to the classification confidence score. ];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Galitsky with Abraham to determine respective levels of agreement between retrieved evidence and claims in the generated text. This would provide a quantifiable indication of whether and to what extent the evidence supports or refutes each claim.
Regarding claim 15, the rejection of claim 14 is incorporated.
Galitsky does not teach
The method of claim 14, wherein determining one or more first levels of agreement, with
respect to one or more claims associated with the first textual content, between the one or more evidence snippets and the first textual content comprises: evaluating the first textual content and the evidence snippet with a machine- learned agreement model.
However, Abraham teaches
The method of claim 14, wherein determining one or more first levels of agreement,
with respect to one or more claims associated with the first textual content, between the one or more evidence snippets and the first textual content comprises: evaluating the first textual content and the evidence snippet with a machine- learned agreement model.
[0081 "In some implementations, the claim verification module 114 is initially or further trained using tagged training data under a full supervision or weak supervision training approach."]
[0080 "For example, in some implementations, a claim verification module 114 is configured to analyze context of a claim and context of evidence to determine similarity or dissimilarity therebetween (e.g., whether the content of the claim and the content of the evidence are associated with similar or dissimilar conclusions, or whether the content of the evidence substantiates the content of the claim)."]
[0087 "In some implementations, the claim verification module 114 associates a classification confidence score with its determination as to whether evidence supports or refutes a claim" where trained claim verification module maps to the machine learned agreement module, analyzing the claim context and evidence context maps to evaluating first text and evidence snipped, and classification confidence score maps to agreement level];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Galitsky with Abraham to automatically evaluate the similarity or dissimilarity between each claim and its evidence and produce more reliable agreement determinations.
Regarding claim 18, the rejection of claim 14 is incorporated.
Galitsky does not teach
The method of claim 14, wherein generating, by the computing system, a second textual content as an output of a second machine-learned language model comprises: identifying a respective claim of the one or more claims associated with the first textual content, wherein the first level of agreement with respect to the respective claim indicates a lack of complete agreement; and generating a second textual content, such that a second level of agreement, with respect to the respective claim and between the second textual content and the one or more evidence snippets, indicates a greater degree of agreement than the first level of agreement with respect to the respective claim.
However, Abraham teaches
The method of claim 14, wherein generating, by the computing system, a second textual content as an output of a second machine-learned language model comprises:
identifying a respective claim of the one or more claims associated with the first textual content, wherein the first level of agreement with respect to the respective claim indicates a lack of complete agreement; and
[0106 "FIG. 8 illustrates text 802 that is different from the text 202 hitherto described. In particular, the text 802 states that tobacco smoking accounts for about 47% of lung cancer cases (in contrast with the text 202, which states that tobacco smoking accounts for about 87% of lung cancer cases). FIG. 8 illustrates a refute indicator 804 associated with the claim that tobacco smoking accounts for about 47% of lung cancer cases, indicating that the claim verification module 114 has determined that the claim is strongly refuted by evidence (e.g., refuted with high confidence)." where first textual content is mapped to 47% claim, first level agreement is mapped to strongly refuted which implies lack of agreement,]
generating a second textual content, such that a second level of agreement, with respect to the respective claim and between the second textual content and the one or more evidence snippets,
indicates a greater degree of agreement than the first level of agreement with respect to the respective claim.
[0107 "FIG. 8 also shows user input 806 directed to the refute indicator 804 (or directed to the claim that underlies the refute indicator 804). In some implementations, as shown in FIG. 8, the user input 806 triggers the presentation of a recommendation 808 for modifying the claim associated with the refute indicator 804. In particular, the recommendation 808 comprises modifying the text “47% of lung cancer cases” to instead state “87% of lung cancer cases”. A system may utilize a language model to facilitate text generation or modification to provide the recommendation 808 based on evidence from a set of relevant references (e.g., as determined by the evidence extraction module 112 and the claim verification module 114)" where 87% claim generated through modification maps to second textual content,…];
[0091 "The example depicted in FIG. 4 shows that the claim that tobacco smoking accounts for about 87% of lung cancer cases is strongly supported by evidence represented within the corpus of references 304 (e.g., as determined using the evidence extraction module 112 and the claim verification module 114)." where strongly supported is mapped to second agreement level].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Galitsky with Abraham by identifying a claim that lacks agreement with retrieved evidence and generating revised text having a greater agreement with that evidence. This would automatically correct unsupported claims and improve the factual accuracy and reliability of generated text.
Regarding claim 23, the rejection of claim 1 is incorporated.
Galitsky teaches
The method of claim 1, wherein: the one or more evidence sources are textual documents.
[Column 5, lines 51-55 "The structure and content flow of the raw text may be utilized, while factoids may be collected from true texts mined from a knowledge base (e.g., the web, a corpus of text/documents, etc.) to correspond to the raw text."];
[Column 5, lines 64-67 "One or more queries are formed from these untrue phrases and search available knowledge bases (e.g., the Internet, a corpus of documents, etc.) for sentences that may be used to replace" where evidence sources map to knowledge base/corpus searched for candidate true sentences];
[Column 17, lines 58-61 "A knowledge base may additionally, or alternatively, include a corpus of documents (e.g., text) accessible to the correction module 122."]
Regarding claim 24,
Galitsky teaches
A computing system, comprising: [[0]]one or more processors; and one or more non-transitory computer-readable media storing instructions that are executable by the one or more processors to cause the computing system to perform one or more operations, the operations comprising:
[Column 2, lines 57-60 “The exemplary method(s) discussed herein can be implemented
on systems and/or devices including one or more processors and/or stored as instructions on a non-transitory computer-readable medium.”];
obtaining, by a computing system comprising one or more computing devices, a first textual content generated by a first machine-learned language model;
[Column 7, lines 63-64 "Deep Learning generated content 118 can be generated by deep learning generation module 120" where raw text/DL generated content 118 is mapped to first textual content and deep learning generation module 120 is mapped to first machine learned language model];
[Column 8, lines 57- 60 "Prior to performing the operations of method 200, raw text may be obtained (e.g., from DL generated content 118 of FIG. 1, from DLG module 120 of FIG. 1, or any suitable source).
retrieving, by the computing system, one or more evidence sources relating to the first textual content generated by the first machine-learned language model;
[Column 9, lines 10- 16 "To find relevant sentences from a knowledge base (e.g., the web, a corpus of text, etc.) with which the raw text sentence can be corrected, the correction module may develop one or more search queries
from the raw text sentence using a discourse tree generated from the raw text sentence based at least in part on Rhetorical Structure Theory."];
[Column 17, lines 51-58 "Returning to FIG. 2, at 203, the one or more search queries generated at 202 using one or more of the techniques discussed above, may be executed to obtain a list of candidate
true sentences. For example, the one or more queries may be provided (e.g., via a search API) to one or more knowledge bases. An example of a knowledge base may include Wikipedia®, Bing®, Google®, and the like."];
[Column 17, lines 61-65 "A set of search results may be obtained based on executing the search queries against the knowledge base(s). Each sentence (or some number of sentences) of one
or more search results may serve as a candidate true sentence with which the raw text sentence can be corrected." where knowledge base search results maps to evidence sources and retrieved using queries generated from the raw text sentence maps to relating to the first textual content];
generating, by the computing system, a second textual content as an
[Column 8, lines 47-52 "Computing device 102 may include correction module 122. In some embodiments, correction module 122 may be configured to identify and implement one or more corrections to raw text provided as input ( e.g., raw text generated by the DLG module 120, raw text retrieved from DL generated content 118, etc.). Operations"];
[Column 18, lines 11-12 "Correction module 122 can implement method 600 on computing device 102 of FIG. 1."];
[Column 18, lines 14-22 "It will be appreciated that method 600 can repeat additional times in the event that a candidate true sentence (e.g., a sentence from one of the search results obtained at 203 of FIG. 3) is not deemed to be sufficiently appropriate or accurate. Further, method 600 can iterate multiple times to determine whether a candidate true sentence aligns with a raw
text sentence sufficiently to provide replacement fragments for the raw text sentence."];[Column 17, lines 66-67 , Column 18, lines 1-5 "At 204, the raw text sentence and each candidate true sentence can be generalized. That is the syntactic and semantic representations of the raw text sentence may be
mapped to syntactic and semantic representations of each candidate sentence to determine if a given candidate sentence may be used to correct/replace an entity of the raw text sentence."];
[Column 23, lines 34-36 "For example, a syntactically matching and semantically closest true sentence candidate may be selected."];
[Column 23, lines 33-34 "At 207, a candidate true statement may be selected (e.g., based on the semantic/syntactic scores assigned at 204)."];
[Column 23, lines 40-42 "At 208, entity and/or phrase substitutions may be made using entities and/or phrases of the selected candidate true sentence"];
[Column 24, lines 39-46 "At 212, output may be generated…..the output may include the corrected raw sentence, and/or an indication that the correction was successful and/or an indication of the particular
corrections made."];
Galitsky does not teach
output of a second machine-learned language model
associating, by the computing system, one or more portions of the one or more evidence sources as attributions for the first or second textual content.
However, Abraham does teach
output of a second machine-learned language model
[0124 "In some implementations, the claim verification module is trained with a third set of ground truth to link detected claims to one or more pieces of evidence within related references and to determine whether the one or more pieces of evidence support or refute the one or more claims" where claim verification module compares whether evidence supports or refutes the claim];
[0107 " A system may utilize a language model to facilitate text generation or modification to provide the recommendation 808 based on evidence from a set of relevant references (e.g., as determined by the evidence extraction module 112 and the claim verification module 114)." where language model used to generate or modify the textual recommendations maps to second machine learning model and modified textual recommendations map to the second textual content];
[0107 "In some implementations, as shown in FIG. 8, the user input 806 triggers the presentation of a recommendation 808 for modifying the claim associated with the refute indicator 804. In particular, the recommendation 808 comprises modifying the text “47% of lung cancer cases” to instead state “87% of lung cancer cases”"];
associating, by the computing system, one or more portions of the one or more evidence sources as attributions for the first or second textual content
[0035 "Furthermore, at least some implementations of the present disclosure link claims or procedures detected in a body of text to relevant content from the corpus of references and selectively display the relevant content in association with the linked claims,.." ];
[0127 "Act 1114 may additionally or alternatively entail presenting a representation of the set of references in the user interface in association with the one or more claims, presenting a representation of one or more pieces of evidence of the set of references in the user interface in association with the one or more claims, and/or presenting a recommendation for modifying the one or more claims based on the set of references." where one or more pieces of evidence from the set of references maps to one or more portions of the one or more evidence sources, the displayed evidence /reference identifies the evidentiary basis for the associated portion of text is mapped to attribution];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Galitsky with Abraham such that correction module 122 uses a language model to generate corrected text based on a comparison between the original model generated text and retrieved evidence and associates the relevant evidence with the original or corrected text. This would improve the accuracy of model generated content by correcting unsupported statements while enabling users to identify and evaluate the evidence supporting the content.
Regarding claim 25,
Galitsky teaches
One or more computer-readable media storing instructions that are executable by a computing system to cause the computing system to perform one or more operations, the operations comprising: [Column 37, lines 64-67 “The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program
modules, and other data for computing subsystem 1300.”];
obtaining, by a computing system comprising one or more computing devices, a first textual content generated by a first machine-learned language model;
[Column 7, lines 63-64 "Deep Learning generated content 118 can be generated by deep learning generation module 120" where raw text/DL generated content 118 is mapped to first textual content and deep learning generation module 120 is mapped to first machine learned language model];
[Column 8, lines 57- 60 "Prior to performing the operations of method 200, raw text may be obtained (e.g., from DL generated content 118 of FIG. 1, from DLG module 120 of FIG. 1, or any suitable source).
retrieving, by the computing system, one or more evidence sources relating to the first textual content generated by the first machine-learned language model;
[Column 9, lines 10- 16 "To find relevant sentences from a knowledge base (e.g., the web, a corpus of text, etc.) with which the raw text sentence can be corrected, the correction module may develop one or more search queries
from the raw text sentence using a discourse tree generated from the raw text sentence based at least in part on Rhetorical Structure Theory."];
[Column 17, lines 51-58 "Returning to FIG. 2, at 203, the one or more search queries generated at 202 using one or more of the techniques discussed above, may be executed to obtain a list of candidate
true sentences. For example, the one or more queries may be provided (e.g., via a search API) to one or more knowledge bases. An example of a knowledge base may include Wikipedia®, Bing®, Google®, and the like."];
[Column 17, lines 61-65 "A set of search results may be obtained based on executing the search queries against the knowledge base(s). Each sentence (or some number of sentences) of one
or more search results may serve as a candidate true sentence with which the raw text sentence can be corrected." where knowledge base search results maps to evidence sources and retrieved using queries generated from the raw text sentence maps to relating to the first textual content];
generating, by the computing system, a second textual content as an
[Column 8, lines 47-52 "Computing device 102 may include correction module 122. In some embodiments, correction module 122 may be configured to identify and implement one or more corrections to raw text provided as input ( e.g., raw text generated by the DLG module 120, raw text retrieved from DL generated content 118, etc.). Operations"];
[Column 18, lines 11-12 "Correction module 122 can implement method 600 on computing device 102 of FIG. 1."];
[Column 18, lines 14-22 "It will be appreciated that method 600 can repeat additional times in the event that a candidate true sentence (e.g., a sentence from one of the search results obtained at 203 of FIG. 3) is not deemed to be sufficiently appropriate or accurate. Further, method 600 can iterate multiple times to determine whether a candidate true sentence aligns with a raw
text sentence sufficiently to provide replacement fragments for the raw text sentence."];[Column 17, lines 66-67 , Column 18, lines 1-5 "At 204, the raw text sentence and each candidate true sentence can be generalized. That is the syntactic and semantic representations of the raw text sentence may be
mapped to syntactic and semantic representations of each candidate sentence to determine if a given candidate sentence may be used to correct/replace an entity of the raw text sentence."];
[Column 23, lines 34-36 "For example, a syntactically matching and semantically closest true sentence candidate may be selected."];
[Column 23, lines 33-34 "At 207, a candidate true statement may be selected (e.g., based on the semantic/syntactic scores assigned at 204)."];
[Column 23, lines 40-42 "At 208, entity and/or phrase substitutions may be made using entities and/or phrases of the selected candidate true sentence"];
[Column 24, lines 39-46 "At 212, output may be generated…..the output may include the corrected raw sentence, and/or an indication that the correction was successful and/or an indication of the particular
corrections made."];
Galitsky does not teach
output of a second machine-learned language model
associating, by the computing system, one or more portions of the one or more evidence sources as attributions for the first or second textual content.
However, Abraham does teach
output of a second machine-learned language model
[0124 "In some implementations, the claim verification module is trained with a third set of ground truth to link detected claims to one or more pieces of evidence within related references and to determine whether the one or more pieces of evidence support or refute the one or more claims" where claim verification module compares whether evidence supports or refutes the claim];
[0107 " A system may utilize a language model to facilitate text generation or modification to provide the recommendation 808 based on evidence from a set of relevant references (e.g., as determined by the evidence extraction module 112 and the claim verification module 114)." where language model used to generate or modify the textual recommendations maps to second machine learning model and modified textual recommendations map to the second textual content];
[0107 "In some implementations, as shown in FIG. 8, the user input 806 triggers the presentation of a recommendation 808 for modifying the claim associated with the refute indicator 804. In particular, the recommendation 808 comprises modifying the text “47% of lung cancer cases” to instead state “87% of lung cancer cases”"];
associating, by the computing system, one or more portions of the one or more evidence sources as attributions for the first or second textual content
[0035 "Furthermore, at least some implementations of the present disclosure link claims or procedures detected in a body of text to relevant content from the corpus of references and selectively display the relevant content in association with the linked claims,.." ];
[0127 "Act 1114 may additionally or alternatively entail presenting a representation of the set of references in the user interface in association with the one or more claims, presenting a representation of one or more pieces of evidence of the set of references in the user interface in association with the one or more claims, and/or presenting a recommendation for modifying the one or more claims based on the set of references." where one or more pieces of evidence from the set of references maps to one or more portions of the one or more evidence sources, the displayed evidence /reference identifies the evidentiary basis for the associated portion of text is mapped to attribution];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Galitsky with Abraham such that correction module 122 uses a language model to generate corrected text based on a comparison between the original model generated text and retrieved evidence and associates the relevant evidence with the original or corrected text. This would improve the accuracy of model generated content by correcting unsupported statements while enabling users to identify and evaluate the evidence supporting the content.
Claim [4, 6] are rejected under 35 U.S.C. 103 as being unpatentable over Galitsky (US 12,175,187 B2, hereinafter Galitsky) in view of Abraham (US 20220237382 A1
, hereinafter Abraham) and in further view of Nakano (Nakano, Reiichiro, et al. "Webgpt: Browser-assisted question-answering with human feedback." arXiv preprint arXiv:2112.09332 (2021), hereinafter Nakano).
Regarding claim 4, the rejection of claim 1 is incorporated.
Galitsky in view of Abraham do not teach
The method of claim l, wherein generating, by the computing system, one or more queries
based on the first textual content comprises prompting a third machine-learned language model.
However, Nakano teaches
[Abstract "We fine-tune GPT-3 to answer long-form questions using a text-based web browsing environment, which allows the model to search and navigate the web" where the fine tune GPT-3 model maps to the third machine learned language model];
[Page 2, Figure 1 shows receiving a textual question (prompting the model), the model then generates the query ]
PNG
media_image1.png
456
924
media_image1.png
Greyscale
[Page 3 Table 1:
PNG
media_image2.png
341
859
media_image2.png
Greyscale
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Galitsky in view Abraham with Nakano to employ Nakano 's language model as a third machine learned language model for generating search queries from the original model generated text, because Galitsky already generates queries from such text to retrieve corrected evidence, and Nakano teaches using a language model to generate and submit web-search queries. This would have predictably improved query formulation and retrieval of relevant evidence for correcting the text.
Regarding claim 6, the rejection of claim 1 is incorporated.
Galitsky in view of Abraham do teach
associated with at least one of the one or more claims.
Abraham teaches associated with at least one of the one or more claims
[0009 "The system is also configured, in some instances, to input the textual content into the claim detection module to detect one or more claims within the selection of textual content and,.."];
[0108 "For example, claim verification 904 may include detecting claims within the document 902 using a claim detection module 110, automatically searching within a corpus of references for references and/or evidence related to the detected claims using an evidence extraction module 112, and determining whether the related references and/or evidence support or refute the detected claims."];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Galitsky with Abraham to retrieve evidence specifically relevant to each claim, thereby improving the accuracy and reliability of Galitsky’s evidence based evaluation and correction.
Galitsky in view of Abraham do not teach
prompting a third machine-learned language model to generate a query
However, Nakano teaches prompting a third machine-learned language model to generate a query
[Page 2, "We create a text-based web-browsing environment that a fine-tuned language model can interact with."];
PNG
media_image3.png
147
1020
media_image3.png
Greyscale
[Page 2, Figure 1 shows receiving a textual question (prompting the model) concerning a factual subject then generates the query ]
PNG
media_image4.png
172
1102
media_image4.png
Greyscale
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Galitsky in view Abraham with Nakano to generate a search query associated with a claim identified in the model generated text, because the combined system already identifies claims and generates queries to retrieve corrective evidence, while Nakano teaches using a language model to generate and submit web search queries. This would have predictably improved query formulation and retrieval of evidence relevant to verifying and correcting the claims.
Claim [7] is rejected under 35 U.S.C. 103 as being unpatentable over Galitsky (US 12,175,187 B2, hereinafter Galitsky) in view of Abraham (US 20220237382 A1
, hereinafter Abraham) and in further view of Nakano (Nakano, Reiichiro, et al. "Webgpt: Browser-assisted question-answering with human feedback." arXiv preprint arXiv:2112.09332 (2021), hereinafter Nakano) and in further view of Brown (Brown, Tom, et al. "Language models are few-shot learners." Advances in neural information processing systems 33 (2020): 1877-1901, hereinafter, Brown).
Regarding claim 7, the rejection of claim 6 is incorporated.
Galitsky in view of Abraham in view of Nakano teach
The method of claim 6, wherein prompting a third machine-learned language model
Nakano teaches prompting a third machine-learned language model
[Abstract "We fine-tune GPT-3 to answer long-form questions using a text-based web browsing environment, which allows the model to search and navigate the web" where the fine tune GPT-3 model maps to the third machine learned language model];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Galitsky in view Abraham with Nakano to automate the generation of target search queries and retrieve more relevant evidence for Galitsky and Abraham’s evidence based evaluation and correction process.
However, Galitsky in view of Abraham in view of Nakano do not teach comprises
few-shot prompting.
But Brown teaches few-shot prompting
[Abstract " Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks."].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Galitsky in view Abraham in view of Nakano with Brown by applying Brown's few shot prompting technique to Nakano 's language model used to generate search queries, because Brown teaches that providing a few textual demonstrations enables a language model to perform a specified task without additional fine tuning. This modification would have predictably improved the language model's ability to generate relevant search queries for retrieving evidence concerning identified claims.
Claim [8] is rejected under 35 U.S.C. 103 as being unpatentable over Galitsky (US 12,175,187 B2, hereinafter Galitsky) in view of Abraham (US 20220237382 A1
, hereinafter Abraham) and in further view of Nakano (Nakano, Reiichiro, et al. "Webgpt: Browser-assisted question-answering with human feedback." arXiv preprint arXiv:2112.09332 (2021), hereinafter Nakano) and in further view of Wei (Wei, Jason, et al. "Chain-of-thought prompting elicits reasoning in large language models." Advances in neural information processing systems 35 (2022): 24824-24837., hereinafter Wei).
Regarding claim 8, the rejection of claim 6 is incorporated.
Galitsky in view of Abraham in view of Nakano do teach
The method of claim 6, wherein prompting a third machine-learned language model
Nakano teaches prompting a third machine-learned language model
[Abstract "We fine-tune GPT-3 to answer long-form questions using a text-based web browsing environment, which allows the model to search and navigate the web" where the fine tune GPT-3 model maps to the third machine learned language model];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Galitsky in view Abraham with Nakano prompting of a third machine learned language model to generate queries based on the first textual content. This would automate targeted query generation and retrieve more relevant evidence for the evidence based evaluation and correction process.
However, Galitsky in view of Abraham in view of Nakano do not teach comprises
chain-of-thought prompting.
But Wei teaches chain-of-thought prompting
[Abstract " In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain-of thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting." where large language model is the third machine learned language model];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Galitsky in view Abraham in view of Nakano with Wei by applying Wei's chain of thought prompting technique to Nakano's language model, because Wei teaches that providing intermediate reasoning steps as prompt improves a language model's performance on reasoning task. This modification would improve the model's ability to analyze identified claims and generate relevant evidence search queries.
Claim [10, 11, 13] are rejected under 35 U.S.C. 103 as being unpatentable over Galitsky (US 12,175,187 B2, hereinafter Galitsky) in view of Abraham (US 20220237382 A1
, hereinafter Abraham) and in further view of Baker (US 9,940,367 Bl, hereinafter Baker).
Regarding claim 10, the rejection of claim 9 is incorporated.
Galitsky teaches
The method of claim 9, wherein extracting, by the computing system, one or more evidence
snippets from the one or more evidence sources retrieved by the computing system comprises: determining one or more candidate evidence snippets;
[Column 17, lines 61-65 "A set of search results may be obtained based on executing the search queries against the knowledge base(s). Each sentence (or some number of sentences) of one or more search results may serve as a candidate true sentence with which the raw text sentence can be corrected" where search results obtained from knowledge base maps to retrieved evidence sources; individual sentences or groups of sentences from the search results map to evidence snippets];
[Column 18, lines 19-22 "Further, method 600 can iterate multiple times to determine whether a candidate true sentence aligns with a raw text sentence sufficiently to provide replacement fragments for the raw text sentence." where candidate true sentences identified from the search results for possible use in correcting raw text maps to candidate evidence snippets. ];
and selecting a subset of candidate evidence snippets based on the respective scores.
[Column 23, lines 7-12 "But if there is more than one candidate true sentence, then the semantic and syntactic alignment scores for the different candidate true sentences can be compared and a candidate true sentence (e.g., the 10 highest scored candidate true sentence) can be selected to correct aspects of the raw text sentence."];
Galitsky in view of Abraham do not teach
generating, for each candidate evidence snippet, a respective score based on a query that was used to retrieve the one or more evidence sources
However, Baker teaches
generating, for each candidate evidence snippet, a respective score based on a query that was used to retrieve the one or more evidence sources
[Column 12, lines 4-7 "The process 800 determines, for each candidate answer passage, a query term match score (806). The query term match score is a score that is a measure of similarity of the query terms to the candidate answer passage."].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Galitsky in view Abraham with Baker by generating query based relevance scores for the candidate evidence snippets and selecting a subset based on those scores. This would prioritize the evidence most relevant to the generated queries and improve the accuracy and efficiency of claim verification.
Regarding claim 11, the rejection of claim 10 is incorporated.
Galitsky in view of Abraham does not teach
The method of claim 10, wherein the respective scores are relevance scores generated by a machine-learned query-document relevance model.
However, Baker teaches the method of claim 10, wherein the respective scores are relevance scores generated by a machine-learned query-document relevance model.
[Column 12 " lines 4-11 "The process 800 determines, for each candidate answer passage, a query term match score (806). The query term match score is a score that is a measure of similarity of the query terms to the candidate answer passage. In some implementations, the query dependent scorer 142 determines a score that is proportional to a number of instances of matches of query terms to terms of the candidate answer passage."];
[Column 11, lines 31-39 "The answer passage scorer 124 receives candidate answer passages 132 from the answer passage generator 122 and scores each passage by combining scoring signals that
predict how likely the passage is to answer the question. In some implementations, the answer passage scorer 124 includes a query dependent scorer 142 and a query independent scorer 144 that respectively generate a query dependent score and a query independent score" where respective query/answer scores map to the respective scores, scores predicting how likely each candidate passage is to answer the query is mapped to relevance scores, the query dependent scorer maps to the query document relevance model and candidate answer passage extracted maps to document];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Galitsky in view Abraham with Baker’s query dependent relevance model to generate relevance scores between each query and candidate evidence passage. It would provide an objective measure of passage relevance and improve identification of evidence responsive to the generated query.
Regarding claim 13, the rejection of claim 10 is incorporated.
Galitsky in view of Abraham do teach
The method of claim 10, wherein selecting a subset of candidate evidence snippets based on the
Abraham teaches
[0013 "In some instances, the system is configured to identify a document comprising a body of text and detect one or more claims within the body of text using a claim detection module having been trained to detect claims within bodies of text….The system is also configured, in some instances, in response to detecting the one or more claims within the body of text, to automatically search within a corpus of references for one or more related references using an evidence extraction module having been trained to identify related references from the corpus of references based on content of detected claims."];
[0016 "Each set of related references of the plurality of sets of references is determined by the evidence extraction module to be relevant to a respective claim of the plurality of claims."];
[0017 "In some instances, the system is also configured, for each respective claim of the plurality of claims, to automatically determine whether a respective set of related references of the plurality of sets of references supports or refutes the respective claim using a claim verification module having been trained to determine whether the one or more pieces of evidence support or refute the one or more claims."]
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Galitsky with Abraham by evaluating the generated text on a claim by claim basis and retrieving evidence relevant to the identified claims. This would provide more complete evidentiary support for the generated text and improve accuracy of the correction process.
However, Galitsky in view of Abraham do not teach based on respective scores to maximize
that coverage.
However, Baker teaches based on respective scores to maximize that coverage.
[Column 12, lines 4-7 "The process 800 determines, for each candidate answer passage, a query term match score (806). The query term match score is a score that is a measure of similarity of the
query terms to the candidate answer pass"];
[Column 12, lines 49-51 "The process 800, for each candidate answer passage, determines an answer score based on the query dependent score and query independent score (814)." ];
[Column 14, lines 28-37"As described above, the answer term match score is a measure of similarity of the answer terms to the candidate answer passage. However, the answer seeking queries do not describe what the user is looking for, as the answer is unknown to the user. Thus, the query dependent scorer 142 first finds a set of likely answer terms and compares the set of likely answer terms to the candidate answer passage to generate the answer term match score. In some implementations, the set of likely answer terms is derived from the top N ranked resources returned for the query." where the scoring passages would maximize coverage by favoring collection of passage that provide relevant evidence];
[Column 11, lines 44-49 A score combiner 144 combines the scores for each candidate answer passage 132 to generate a corresponding
answer score 134. The answer score is then used to select a candidate answer passage to provide with the search results, or, alternately, may be adjusted according to additional
scoring processes."];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Galitsky in view Abraham with Baker to select the most relevant available evidence for each claim, reduce redundant evidence directed to same claim, and maximize the number of claims addressed by the selected evidence snippets.
Claim [12] are rejected under 35 U.S.C. 103 as being unpatentable over Galitsky (US 12,175,187 B2, hereinafter Galitsky) in view of Abraham (US 20220237382 A1
, hereinafter Abraham) and in further view of Baker (US 9,940,367 Bl, hereinafter Baker) and in further view of Zeng (US 7,289,985 B2, hereinafter, Zeng).
Regarding claim 12, the rejection of claim 10 is incorporated.
Galitsky in view of Abraham in view of Baker teaches
The method of claim 10, wherein: the one or more evidence sources are textual
documents; and determining one or more candidate evidence snippets
Baker teaches
[Column 5, lines 64-65 "The candidate answer passages are generated from the resources that are identified as being responsive to the query"];
[Column 7, lines 61-64 "The process 600, for each resource, identifies passage units in the resource (604). A passage unit can be a complete sentence, a portion of a sentence, a header, or content of
structured data, such as a list entry or a cell value"];
[Column 8, lines 21-23 "The process 600, for each resource, generates, from passage units that satisfy the set of passage unit selection criterion, a set of candidate answer passages 132 ( 608)."
where retrieved responsive resources are evidence sources and the set of candidate answer passages are the candidate evidence snippets];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Galitsky in view Abraham with Baker determining candidate evidence snippets from the retrieved sources. This would help to identify portions of the retrieved sources suitable for comparison with the generated text, thereby improving efficiency and accuracy of the evaluation process.
However, Galitsky in view of Abraham in view of Baker do not teach running a sliding window
across a respective textual document.
But Zeng teaches running a sliding window across a respective textual document.
[Column 8, lines 7-11 "To generate a snippet description, search provider 132 locates one or more blocks from a retrieved KB article 106 determined to be relevant to search query 122 for the snippet
description,….Search provider 132 identifies the one or more blocks with a sliding window of configurable size that is applied to portions of the retrieved document."];
[Column 8, lines 18-21 "For each application of the sliding window to a portion of a retrieved KB article 106, search provider 132 measures the amount of query-related information carried by text delineated
by the sliding window." where the retrieved KB article is the respective textual document and the blocks identified by moving the sliding window across are the candidate evidence snippets];
It would have been prima facie obvious to one of ordinary skill in the art before the effective
filing date of the claimed invention to combine the Galitsky in view Abraham in view of Baker with Zeng
by applying Zeng's sliding window to the retrieved textual documents when determining candidate evidence snippets. This would systematically allow examination portions of retrieved document and identify passage containing information relevant to the query. This would thereby improve selection of relevant evidence snippets for scoring and attribution.
Claim [16, 19] are rejected under 35 U.S.C. 103 as being unpatentable over Galitsky (US 12,175,187 B2, hereinafter Galitsky) in view of Abraham (US 20220237382 A1
, hereinafter Abraham) and in further view Wei (Wei, Jason, et al. "Chain-of-thought prompting elicits reasoning in large language models." Advances in neural information processing systems 35 (2022): 24824-24837., hereinafter Wei).
Regarding claim 16, the rejection of claim 15 is incorporated.
Galitsky in view of Abraham teaches the method of claim 15, wherein evaluating the first textual
content and the evidence snippet with a machine-learned agreement model comprises:
prompting the machine-learned agreement model
Abraham teaches
[0080 "In some instances, the claim verification module 114 comprises or is based on an NLP model that is configured to analyze the parts of speech and/or other content of a body of text to detect particular language and/or context associated with claims or assertions."];
[0080 "For example, in some implementations, a claim verification module 114 is configured to analyze context of a claim and context of evidence to determine similarity or dissimilarity therebetween (e.g., whether the content of the claim and the content of the evidence are associated with similar or dissimilar conclusions, or whether the content of the evidence substantiates the content of the claim)."]
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Galitsky with Abraham by prompting Abraham’s machined learned agreement model to evaluate the first textual content against the retrieved evidence snippets. This would automate the determination of whether the generated text is supported or contradicted by evidence, thereby improving the accuracy of the correction process.
However, Galitsky in view of Abraham do not teach chain of thought prompting.
Wei teaches chain of thought prompting.
[Abstract " In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain-of thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting." where large language model is the third machine learned language model];
It would have been prima facie obvious to one of ordinary skill in the art before the effective
filing date of the claimed invention to combine the Galitsky in view Abraham with Wei to cause the model to use intermediate reasoning when comparing claims with evidence, thereby improving the reliability and explainability of its agreement determinations.
Regarding claim 19, the rejection of claim 1 is incorporated.
Galitsky in view of Abraham teaches the method of claim, wherein generating, by the computing system, a second textual content as an output of a second machine- learned language model comprises prompting the second machine-learned language model
Abraham teaches
[0107 " A system may utilize a language model to facilitate text generation or modification to provide the recommendation 808 based on evidence from a set of relevant references (e.g., as determined by the evidence extraction module 112 and the claim verification module 114)." where language model is mapped to the second language model];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Galitsky in view Abraham by prompting Abraham’s second machine learning model to generate the second textual content, thereby improving the accuracy of the revised text based on the retrieved evidence.
However, Galitsky in view of Abraham do not teach chain of thought prompting.
Wei teaches chain of thought prompting.
[Abstract " In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain-of thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting." where large language model is the third machine learned language model];
It would have been prima facie obvious to one of ordinary skill in the art before the effective
filing date of the claimed invention to combine the Galitsky in view Abraham with Wei’s chain of thought prompting to Abraham’s second language model when generating the revised textual content. This would enable the model to reason through the relationship between the original claim and retrieved evidence, thereby generating a more accurate and evidence consistent revision.
Claim [17, 20] are rejected under 35 U.S.C. 103 as being unpatentable over Galitsky (US 12,175,187 B2, hereinafter Galitsky) in view of Abraham (US 20220237382 A1
, hereinafter Abraham) and in further view Brown (Brown, Tom, et al. "Language models are few-shot learners." Advances in neural information processing systems 33 (2020): 1877-1901, hereinafter, Brown).
Regarding claim 17, the rejection of claim 15 is incorporated.
Galitsky in view of Abraham teaches
The method of claim 15, wherein evaluating the first textual content and the evidence snippet
with a machine-learned agreement model comprises prompting the machine-learned agreement model
Abraham teaches
[0080 "In some instances, the claim verification module 114 comprises or is based on an NLP model that is configured to analyze the parts of speech and/or other content of a body of text to detect particular language and/or context associated with claims or assertions."];
[0080 "For example, in some implementations, a claim verification module 114 is configured to analyze context of a claim and context of evidence to determine similarity or dissimilarity therebetween (e.g., whether the content of the claim and the content of the evidence are associated with similar or dissimilar conclusions, or whether the content of the evidence substantiates the content of the claim)."]
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Galitsky in view Abraham by using Abraham’s machine learned agreement model to evaluate the first textual content against the evidence snippets, thereby improving the accuracy of the agreement determination.
Galitsky in view of Abraham does not teach few shot prompting
However, Brown does teach few shot prompting
[Abstract " Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks."].
It would have been prima facie obvious to one of ordinary skill in the art before the effective
filing date of the claimed invention to combine the Galitsky in view Abraham with Brown’s by providing examples of claim and evidence comparisons, thereby enabling the model to perform more accurate agreement determinations without additional training.
Regarding claim 20, the rejection of claim 1 is incorporated.
Galitsky in view of Abraham teaches
The method of claim, wherein generating, by the computing system, a second textual content as an output of a second machine- learned language model comprises prompting the second machine-learned language model using
Abraham teaches
[0107 " A system may utilize a language model to facilitate text generation or modification to provide the recommendation 808 based on evidence from a set of relevant references (e.g., as determined by the evidence extraction module 112 and the claim verification module 114)." where language model is mapped to the second language model];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Galitsky in view Abraham by prompting Abraham’s second machine learned language model to generate the second textual content, thereby improving the accuracy of the revised text based on the retrieved evidence.
Galitsky in view of Abraham does not teach few shot prompting
However, Brown does teach few shot prompting
[Abstract " Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks."].
It would have been prima facie obvious to one of ordinary skill in the art before the effective
filing date of the claimed invention to combine the Galitsky in view Abraham with Brown’s few shot prompting to Abraham’s second language model when generating the revised textual content. This would provide examples of evidence based revisions, thereby improving the model’s ability to generate accurate revisions without additional training or fine tuning.
Claim [21, 22] are rejected under 35 U.S.C. 103 as being unpatentable over Galitsky (US 12,175,187 B2, hereinafter Galitsky) in view of Abraham (US 20220237382 A1
, hereinafter Abraham) and in further view Badger (US 8,782,556 B2, hereinafter, Badger).
Regarding claim 21, the rejection of claim 1 is incorporated.
Galitsky teaches the method of claim 1, further comprising:
determining an
[Column 22, lines 27-29 "The correction module 122 may compare the original CDT for the raw text sentence and the resultant CDT for the corrected text sentence"];
based on the
[Column 22, lines 39-52 "At 212, output may be generated. If the corrections/ substitutions performed were found to be acceptable (e.g., the corrected entities agree with similar entities in the same
sentence/other sentences, and the discourse structure of the raw text sentence was maintained), the output may include the corrected raw sentence, and/or an indication that the correction was successful and/or an indication of the particular corrections made. Conversely, if the corrections/ substitutions performed were found to be unacceptable ( e.g., the corrected entities did not agree with similar entities in the same sentence/other sentences, or the discourse structure of the raw text sentence was not sufficiently maintained), the output may include the raw text sentence and/or an indication that the correction was unsuccessful."];
Galitsky in view of Abraham does not teach edit distance
However, Badger teaches edit distance
[Column 20, lines 39-49 "At process block 1084, a weighted edit distance score for the input word(s) over one or more likely candidate words is determined. In some embodiments, the weighted edit distance score can be determined using a simple edit distance as described in FIG. 1 OC and accompanying text, above. In other embodiments, the weighted edit distance score can be determined
using a more complex edit distance technique, such as that described in FIG. lOD and accompanying text, above. In some examples, all eight penalty categories described above can be used, while other examples may use additional or fewer categories."].
It would have been prima facie obvious to one of ordinary skill in the art before the effective
filing date of the claimed invention to combine the Galitsky in view Abraham with Badger’s by calculating an edit distance between the original and revised text and using that distance to determine whether to output the revision. This would prevent excessive or unintended changes and ensure that the corrected text remains sufficiently similar to the original text.
Regarding claim 22, the rejection of claim 1 is incorporated.
Galitsky teaches the method of claim 1, further comprising:
[Column 24, lines 39-52 "At 212, output may be generated. If the corrections/ substitutions performed were found to be acceptable (e.g.,
the corrected entities agree with similar entities in the same sentence/other sentences, and the discourse structure of the raw text sentence was maintained), the output may include
the corrected raw sentence, and/or an indication that the correction was successful and/or an indication of the particular corrections made. Conversely, if the corrections/
substitutions performed were found to be unacceptable ( e.g., the corrected entities did not agree with similar entities in the same sentence/other sentences, or the discourse structure of
the raw text sentence was not sufficiently maintained), the output may include the raw text sentence and/or an indication that the correction was unsuccessful."]
Galitsky does not teach
determining an agreement score between the second textual content and one or more evidence snippets extracted from the one or more evidence sources;
determining a preservation score indicative of a similarity between the first textual content and the second textual content;
and based on a combination of the agreement score and the preservation score, determining whether to
However, Abraham teaches
determining an agreement score between the second textual content and one or more evidence snippets extracted from the one or more evidence sources;
[0087 "In some implementations, the claim verification module 114 associates a classification confidence score with its determination as to whether evidence supports or refutes a claim. The classification confidence score may numerically represent an amount of confidence in a label assigned to a piece of evidence vis-à-vis a claim (e.g., a label of whether the piece of evidence supports the claim or refutes the claim)." where classification confidence score maps to agreement score];
[0107 "FIG. 8 also shows user input 806 directed to the refute indicator 804 (or directed to the claim that underlies the refute indicator 804). In some implementations, as shown in FIG. 8, the user input 806 triggers the presentation of a recommendation 808 for modifying the claim associated with the refute indicator 804. In particular, the recommendation 808 comprises modifying the text “47% of lung cancer cases” to instead state “87% of lung cancer cases”. A system may utilize a language model to facilitate text generation or modification to provide the recommendation 808 based on evidence from a set of relevant references (e.g., as determined by the evidence extraction module 112 and the claim verification module 114)" where 87% claim generated through modification maps to second textual content and pieces of evidence obtained from relevant references maps to evidence snippets.];
It would have been prima facie obvious to one of ordinary skill in the art before the effective
filing date of the claimed invention to combine the Galitsky with Abraham’s agreement score between the second textual content and the retrieved evidence snippets to determine whether to output the second textual content. This would ensure that the second textual content is supported by the retrieved evidence before it is outputted.
Galitsky in view of Abraham do not teach
determining a preservation score indicative of a similarity between the first textual content and the second textual content;
and based on a combination of the
However, Badger teaches
determining a preservation score indicative of a similarity between the first textual content and the second textual content;
[Column 20, lines 51-53 "At process block 1090, the weighted word probabilities from process block 1080 and the weighted edit distances from process block 1084 are combined into
a combined score. In order to determine if auto-correction should be suppressed or performed, when given an input word ( e.g., a word received
with a touch screen keyboard) and a word suggested by the system, a formula can be applied that combines the negative logarithm of the word probability with a weighted edit distance
score:" where edit distance score is the preservation score indicative of the similarity];
[Column 21 lines 1-3 "Word2 is a candidate word (e.g., a possible suggestion candidate or auto-correction replacement word). C, is an auto-correction threshold (an adjustable constant)."];
and based on a combination of the
[Column 21, lines 21-29 "At process block 1094, the combined score (final result) can be used to generate auto-corrections, suppress auto-corrections,
determine suggestion candidates, or rank suggestion candidates. The final result (the right-hand side of the inequality above) can be compared against a flexible threshold
value (C,). Adjusting the comparison threshold value C, determines the frequency and aggressiveness of auto-correction or candidate generation." where combined score maps to agreement score and preservation score to determine whether to output the correction. ];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Galitsky in view of Abraham with Badger by additionally determining a preservation score indicating the similarity between the first and second textual content and combining the preservation score with Abraham’s agreement score when determining whether to output the second textual content. This would be done to ensure that the output is both supported by the evidence and sufficiently preserves the original content.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHEZA ABDUL AZIZ whose telephone number is (571)272-9610. The examiner can normally be reached Monday-Friday 7:30am-5pm Alternate Fridays off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached at (571) 272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHEZA ABDUL AZIZ/Examiner, Art Unit 2657
/DANIEL C WASHBURN/Supervisory Patent Examiner, Art Unit 2657
