DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
All objections/rejections not mentioned in this Office Action have been withdrawn by the Examiner.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 25 June 2026 has been entered.
Status of the Claims
Prior to entry of the amendment(s) and/or consideration of the argument(s), the status of the claims is as follows.
Claim(s) 1-13 and 15-20 is/are pending.
Claims 1, 3-14, and 16-18 are objected to because of informalities.
Claims 1-12 and 19 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement.
Claim(s) 1-11, 13, 15, and 18-20 are rejected under 35 U.S.C. §101 because the claimed invention is directed to an abstract idea without significantly more.
Claim(s) 13, 15, and 20 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Emrey (U.S. Pat. App. Pub. No. 2024/0386207, hereinafter Emrey).
Claims 1-3, 8-12, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey in view of Hurst (U.S. Pat. App. Pub. No. 2002/0016796, hereinafter Hurst), with further support from Non-Patent Literature to Jurafsky (Daniel Jurafsky and James H. Martin. 2020. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models, 3rd edition Draft. Published: 30 December 2020. <Retrieved on: 14 Nov 2025><URL: https://teaching.bb-ai.net/Student-Projects/Winograd-Challenge-Papers/Jurafsky-Martin-Speech-and-Language-Processing.pdf.>, hereinafter Jurafsky).
Claims 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey and Hurst as applied to claim 1 above, and further in view of Salloum (U.S. Pat. App. Pub. No. 2019/0065462, hereinafter Salloum), with further support from Jurafsky.
Claims 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey and Hurst as applied to claim 1 above, and further in view of Mirhaji (U.S. Pat. No. 8,429,179, hereinafter Mirhaji).
Claims 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey and Hurst as applied to claim 1 above, and further in view of Kelsey (U.S. Pat. App. Pub. No. 2018/0260472, hereinafter Kelsey).
Claims 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey and Hurst as applied to claim 1 above, and further in view of Rahman (U.S. Pat. App. Pub. No. 2024/0330661, hereinafter Rahman).
Claims 16-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey as applied to claim 13 above, and further in view of De Foor (U.S. Pat. App. Pub. No. 2025/0005301, hereinafter DeFoor).
Response to Amendments
Applicant’s amendment filed on 25 June 2026 has been entered.
In view of the amendment to the claim(s), the amendment of claim(s) 1 and 13 and the cancellation of claim(s) 2 have been acknowledged and entered.
After entry of the amendment, claim(s) 1, 3-13 and 15-20 remain pending.
In view of the amendment to claim(s) 1, the rejection of claim(s) 1-12 and 19 under 35 U.S.C. §112 as previously presented is withdrawn.
In view of the amendment of claim(s) 1 and 13 and the cancellation of claim(s) 2, the rejection of claims 2 under 35 U.S.C. §101 is withdrawn. The rejection(s) of claim(s) 1, 3-11, 13, 15, and 19-20 under 35 U.S.C. §101 is/are maintained as modified in response to amendment, for the reasons provided in the action below.
In view of the amendment to claim(s) 1 and 13 and the cancellation of claim(s) 2, the rejection of claims 1-13 and 15-20 under 35 U.S.C. §102 and §103 is withdrawn.
In light of the amended claims, new grounds for rejection under 35 U.S.C. §103 are provided in the action below.
Response to Arguments
Applicant’s arguments regarding the subject matter rejections under 35 U.S.C. §101, see pages 11-13 of the Response to Final Office Action dated 28 May 2026, which was received on 25 June 2026 (hereinafter Response and Office Action, respectively), have been fully considered.
Regarding the rejection of claims 1-11, 13, 15, and 19-20 under 35 U.S.C. §101, applicant asserts that claims 1 and 13, as amended, overcome the rejection under 101 for the following reasons. First, applicant maintains that the claims 1 and 13 “reflect an improvement in the functioning of a computer, or an improvement to other technology or technical field” and that the “specification describes the invention such that the improvement would be apparent to one of ordinary skill in the art.” These arguments remain not persuasive.
Applicant's arguments fail to comply with 37 CFR 1.111(b) because they amount to a general allegation that the claims define a patentable invention without distinctly and specifically pointing out the supposed errors in the examiner's action. Examiner’s explanations regarding the 35 USC 101 rejection, as presented at pages 6-12 of the Office Action, are incorporated herein without further recitation.
Applicant’s response appears to have chosen to ignore the explanations provided in the Office Action and restate their original previous position with further detail. It is noted that said further detail is directly addressed in the Office Action. As such, the continued restatement of addressed positions with no further analysis amounts to little more than attorney argument that the claims define a patentable invention. Said attorney argument is not responsive to the rejection provided and ignores the discussion and considerations presented in the Office Action. As the application of MPEP 2106.04(d)(1), Enfish, and Desjardins, have been specifically addressed in light of applicant’s claims and specification, in entirety, applicant is directed to the previously provided explanation for further guidance. As such, the arguments are not persuasive and the rejection is maintained as amended.
Applicant’s arguments regarding the prior art rejections under 35 U.S.C. §102/103, see pages 9-11 of the Response, have been fully considered.
With respect to the rejection(s) of claim(s) 1 under 35 U.S.C. 102(a)(2) as being obvious in light of Emrey in view of Hurst, with further evidence from Jurafsky, applicant asserts that Emrey and Hurst fails to teach or suggest all limitations of claim 1. Applicant’s arguments are persuasive. As such, the rejections of claim 1 under 35 U.S.C. §103 is withdrawn.
With respect to the rejection(s) of claim(s) 13 under 35 U.S.C. 102(a)(2) as being anticipated by Emrey with further evidence from Jurafsky, applicant asserts that Emrey fails to teach or suggest all limitations of claim 13 as amended. Applicant’s arguments are persuasive. As such, the rejections of claim 13 under 35 U.S.C. §102(a)(2) is withdrawn.
Applicant further argues that the rejection(s) of dependent claims 3-12 and 15-18 and independent claims 19 and 20 should be withdrawn for at least the same reasons as independent claims 1 and 13. Applicant’s arguments in light of the amended claims are persuasive. As such, the rejections of claims 2-12 and 15-20 under 35 U.S.C. §102 and 35 U.S.C. §103 are withdrawn.
However, upon further consideration, new ground(s) of rejection under 35 U.S.C. §103 are made in light of combinations of Emrey, Jurafsky, Salloum, Mirhaji, Kelsey, Rahman, De Foor, and newly cited reference Biegert (U.S. Pat. App. Pub. No. 2019/0026259, hereinafter Biegert).
Finally, as explained in the Office Action at the Examiner’s Note, claims 19 and 20 are interpreted as independent claims. However, both in the Patent Application Fee Determination Record (SB-06) filed on 24 June 2026 and in related argument in the Response, applicant maintains that claims 1 and 13 are independent claims and claims 19 and 20 are dependent claims. These actions are understood as traversal of the interpretation.
The Applicant has not provided any further statement and therefore, the Examiner directs the Applicant to the below rationale.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 3-13, and 15-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Regarding claims 1, 13, 19, and 20, the phrase “a language model” is not supported by the application as filed. Claims 1, 13, 19 and 20 recites the limitation “identifying text data” that was output or generated “by a language model of a question and answer computer system” at lines 7-8. However, “a language model” is not supported by the specification as filed.
The specification does not describe or even recite “a language model” generally. Though the specification does describe a “Large Language Model” (LLM) in the context of Question and Answer systems (Instant Application, [0016]), a LLM is a specific non-representative species of the genus of “language models”. The genus of “language models” is vastly broader and incorporates models with fundamentally different architectures, such as n-gram models, HMMs, and rule-based parsers, as well as non-transformer neural networks such as RNNs and LSTMs. A transformer-based LLM is not representative of the older statistical models, and, as such, the example provided is not representative of the genus such that a person of ordinary skill in the art would understand that the applicant possessed the broader concept of using any language model to achieve the invention. Therefore, claims 1, 13, 19, and 20 contain limitations which incorporate new matter and are rejected under 35 U.S.C. 112(a).
Regarding claims 2-13 and 15-18, claims 2-13 and 15-18 depend from claims 1 and 13, respectively, and incorporate all limitations therein. Therefore, claims 2-13 and 15-18 are rejected for at least the same reasons as claims 1 and 13.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. §101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 1, 3-11, 13, 15, and 18-20 are rejected under 35 U.S.C. §101 because the claimed invention is directed to an abstract idea without significantly more.
To determine subject matter eligibility for each of the recited claims above, we turn to the subject matter eligibility test, also referred to as the Alice/Mayo test, described in MPEP 2106. Regarding step 1 of the subject matter eligibility test, we first determine if the claims are directed to a statutory category. The independent claim(s) 1, and mutatis mutandis claim(s) 19, recites “identifying text data, wherein the text data was automatically generated by a language model of a question and answer computer system; identifying, within the text data, a list of items and introductory text that is associated with the list of items, wherein the list of items comprises multiple items introduced by the introductory text; for each item in the list of items: generating a claim that is based on the introductory text and said each item by prepending the introductory text to each item; and adding the claim to a set of claims that is associated with the text data; wherein the set of claims comprises multiple claims; for each claim in the set of claims: generating, by a machine learned model, a score that reflects a level of support of said each claim in a set of documents; adding the score to a set of scores for the set of claims; wherein the set of scores comprises multiple scores; and causing, to be presented on a screen of a computing device, data that is based on the set of scores; wherein the method is performed by one or more computing devices.”
The independent claim(s) 13, and mutatis mutandis claim(s) 20, recites “identifying text data that was output by a language model of a question and answer computer system based on a prompt and a plurality of documents; identifying a set of claims within the text data; generating a plurality of combinations of the plurality of documents, wherein each combination of the plurality of combinations is based on two or more documents in the plurality of documents, wherein the plurality of combinations comprises a first combination and a second combination, wherein the first combination and the second combination include the same document; for each claim in the set of claims: generating, by a machine learned model, for said each claim, a score that reflects a level of support of said each claim in a combination; wherein generating the score is performed for each combination of the plurality of combinations; wherein the score for said each claim reflects a level of support of said each claim in said each combination; adding the score to a set of scores for said each claim; and causing data to be presented on a screen of a computing device based on the set of scores; wherein the method is performed by one or more computing devices.” As the claims recites a process and an article of manufacture, the claim is directed to one of the statutory categories under step 1 of the subject matter eligibility test.
In Step 2A of the test, which is a Two Prong analysis, we then determine if the claim is directed to a judicial exception. For Step 2A, Prong One, we first ask if the claim recites an abstract idea, Law of Nature, or Natural Phenomena. Regarding claim(s) 1 and 19, the limitations of “identifying…”, “identifying…”, “generating…”, “adding…”, “generating…”, “adding…”, and “causing…” as drafted covers managing personal behavior or relationships or interactions between people, which is a method of organizing human activity. Respectively, regarding claim(s) 13 and 20, the limitations of “identifying…”, “identifying…”, “generating…”, “generating…”, “adding…”, and “causing…” as drafted covers managing personal behavior or relationships or interactions between people, which is a method of organizing human activity. Regarding claims 1, 13, and 19-20, as described with reference to a conversation between a first person and a second person, the first person receives text data from a second person (such as an unprompted statement of fact or in response to a question, and includes one or more asserted facts). The first person then, in attempting to verify the asserted facts, determines one or more specific details in the statement which relate to and define the asserted facts. For example, the text data “Abraham Lincoln was born in a log cabin in Illinois in February of 1809” results in a list of items based on a native speaker of a language understanding the statement made. In this example, the list of specific details includes Lincoln’s birthdate (“Abraham Lincoln was born… in February of 1809”) and his place of birth (“Abraham Lincoln was born in a log cabin in Illinois…”). Each of the asserted facts are interpreted by the first person as a factual claim from the second person (e.g., the first claim “Abraham Lincoln was born in February of 1809” and the second claim “Abraham Lincoln was born in a log cabin in Illinois.” As this step is generally performed as a mental activity by the first person and the second person, as participants in a conversation, the generation of a list and the determination of claims can be performed as multiple steps or as a single combined step, each of which can be written as a list (such as in the case of more complex statements). The first person can then review reliable documents to verify the first claim (e.g., According to Textbook 1, the first claim is completely accurate, as Abraham Lincoln was verifiably born on February 12, 1809) and the second claim (e.g., According to Primary Literature 1, the first claim is only partially accurate, as Abraham Lincoln was verifiably born in a log cabin, but the log cabin was in Kentucky, not Illinois) made by the second person. Further, it is well known to indicate the relative correctness of a factual assertion in a conversation using percentages (e.g., the first claim would be determined to be 100% accurate and the second claim might be considered to be 75% accurate, as Lincoln was born in a log cabin and later lived in Illinois, even though he was not born there). The first person can then respond to the second person based on the assessment (e.g., “The first part of that statement is right, but the second part is not accurate. He was born in a log cabin, but that was in Kentucky. He practiced law in Illinois, though.”), where the response can be provided in any number of ways based on types of conversation (e.g., email), convenience, or personal preference. Further, as indicated with respect to claim 13, data may also be grouped in a variety of ways, such as based on the number of documents in the group, content, or the like, as desired for digestibility of the information. The use of a threshold provides no further distinction over the above described human activity, beyond adding a clerical task of partitioning data based on relationship to the claims made. Therefore, the claims are directed to human activity, and, thus, directed to an abstract idea which is a judicial exception.
In Step 2A, Prong Two of the analysis, we next determine if the claim recites additional elements which integrate the judicial exception into a practical application. The judicial exception recited in claims 1, 13, and 19-20 is not integrated into a practical application. In particular, claim(s) 1, 13, and 19-20 recite additional elements of “identifying… a list of items and introductory text”, “for each item…generating a claim”, “generating a combination of the plurality of documents…”, “generating a score…”, “to be presented on a screen,” a “computing device,” “non-transitory storage media,” and “a first combination and a second combination, wherein the first combination and the second combination include the same document,” as per the independent claims. The “identifying… a list of items and introductory text” is insignificant pre-solution activity. It merely describes the generic receipt of text data, which is a necessary antecedent for any computer-based information processing. The inclusion of “output by a question and answer computer system based on a prompt and a plurality of documents” merely recites a specific environment upon which the pre-solution activity is performed and provides not relevant further limitation of the additional element. The “for each item… generating a claim” is a generic computer implementation of the human activity. The “generating a claim” is a generic description of a mental process which is naturally and instantaneously performed by each participant in a conversation. Though the generic description is intended for implementation in a computer, there are no provisions for actually generating a claim using the computer components or further explanations for how a generated claim might differ from standard human interpretation of language. The “generating a combination of the plurality of documents” the further human activity of verifying the source facts through corroboration, which is related to the dialogue verification. Applicant has merely indicated specific steps which are taken by a human to corroborate facts in the process of factual verification. The generation of a score is the heart of the abstract idea itself, the determination based on verifiable facts, whether asserted facts are accurate or not. The claim does not specify what the score is beyond the general relationship between “the score” and “reflect[ing] a level of support”, or how the score is determined such that the score amounts to more than a numerical reflection of a human decision on verifiability/accuracy. The use of a screen, recited as “to be presented on a screen” is a recitation of the typical operation of a computer screen, where the screen, as the visual portion of the described computing device, performs “predictab[le]” functions based on “vague, functional descriptions”, such that the screen “simply provides the environment in which the abstract idea… is carried out.” (In re TLI Communications LLC Patent Litigation, 823 F.3d 607, 613-14 (Fed. Cir. 2016) quoting Alice, 134 S.Ct. at 2359, quoting Mayo, 132 S.Ct. at 1294)). As such, the presentation on a screen does not meaningfully contribute to a practical application of the abstract idea. Regarding the computer components individually, these are general-purpose computer components with no provisions for the practical application of the abstract idea. Regarding “a first combination and a second combination, wherein the first combination and the second combination include the same document” the claim language refers to overlapping datasets which is merely a format for the underlying facts. The existence of the same document being in multiple sets is a well-known fact of data collection itself. Without further detail which might explain the utility or importance of this known fact, the mere restatement of the above known occurrence fails to meaningfully contribute to the practical application of the abstract idea. The “screen,” the “computing device,” and the “non-transitory storage media,” is/are not meaningfully integrated into the practical application of the abstract ideas recited in claim(s) 1, 13, and 19-20. Accordingly, the additional elements of the claims fail to integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
Regarding Step 2B of the analysis, we next determine if the claim recites additional elements which amount to substantially more than the judicial exception. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of using a “screen,” the “computing device,” and the “non-transitory storage media,” to perform the described dialogue receipt and verification amounts to no more than mere instructions to apply the exception using generic computer components. Mere instructions to apply an exception using a generic computing device or general purpose computer component cannot provide an inventive concept. (See Alice Corp. v. CLS Bank, 573 U.S. 208, 221, 223, 110 USPQ2d 1976, 1982-84 (2014) (quoting Mayo Collaborative Servs. V. Prometheus Labs., Inc., 566 U.S. 66, 72, 101 USPQ2d 1961, 1965). As well, the remaining claim limitations are well-known, routine, and conventional such as to not qualify as an inventive concept. Specifically, transmission of data over a network is well known, as is evidenced by OIP Techs, Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015). Further, “merely identifying a [graphic] user interface,” or more generally a screen, with relation to determinations which “can be performed in the human mind or using a pencil and paper” has been deemed insufficient to render otherwise abstract claims as non-abstract, where the court further indicated that the resulting device, system and/or method was “still missing” an “improved structure or function.” See Broadband iTV, Inc. v. Amazon.com, Inc., 113 F.4th 1359, 1367-68 (Fed. Cir. 2024). The court has consistently held that “[s]teps that do nothing more than spell out what it means to ‘apply it on a computer’ cannot confer patent-eligibility.” Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1371-72 (Fed.Cir. 2015)(citing Alice, 134 S.Ct. at 2359 (warning against a § 101 analysis that turns on the draftsman's art (citing Parker v. Flook, 437 U.S. 584, 593, 98 S.Ct. 2522, 57 L.Ed.2d 451 (1978))).
Further, the above discussion expressly considering the changes to 2106.04(d)(1) in light of the decision in Enfish and Ex Parte Desjardins are incorporated here by reference from the Office Action without further recitation.
Therefore, and in light of the preceding analysis, the claims do not amount to significantly more than the judicial exception. For these reasons, claims 1, 13, and 19-20 are not patent eligible.
With respect to claim(s) 3, the claims relate to the preprocessing of pronouns to their associated named entity. As performed by a person, these steps appear to refer to the mental process of understanding the meaning of a pronoun based on context. No additional limitation is present.
With respect to claim(s) 4, the claims relate to the preprocessing of a numbered list, including the removal of reference numerals. As performed by a person, these steps appear to refer to the mental process of recognizing important information in the conversation (e.g., If the second speaker presents a list with numerical organization, such as “Abraham Lincoln was born (1) in a log cabin (2) in Illinois (3) in February of 1809.”, it is understood that the numerical order doesn’t change the asserted facts and can be ignored for comprehension purposes). No additional limitation is present.
With respect to claim(s) 5, the claims relate to the preprocessing of a flattened list, which is a list which is implicitly incorporated into one or more sentences, based on the presence of punctuation or appropriate grammar elements. As performed by a person, these steps appear to refer to the mental process of recognizing listed items without an explicit list format based on language structure. No additional limitation is present.
With respect to claim(s) 6, the claims relate to the removal of sentences which have limited content value. As performed by a person, these steps appear to refer to the mental process of remembering information in a conversation based on the level of usefulness of the information. No additional limitation is present.
With respect to claim(s) 7, the claim is believed to relate to detection of a reference to a person in the second person point of view and the clarification of ambiguities related to the reference. As performed by a person, these steps appear to refer to the mental process of applying context from previous portions of dialogue to disambiguate a pronoun in the context of the question answer scheme. No additional limitation is present.
With respect to claim(s) 8, the claim relates to associating the scores with a label. As performed by a person, these steps appear to refer to the mental process of applying context from previous portions of dialogue to disambiguate a pronoun in the context of the question answer scheme. No additional limitation is present.
With respect to claim(s) 9, the claim relates to the determination of mathematical thresholds for determining reliability of a factual claim. As performed by a person, these steps appear to refer to the mathematical process of determining a threshold for the comparison score. No additional limitation is present.
With respect to claim(s) 10, the claim relates to the determination of average scores for the scores given to the factual claims. As performed by a person, these steps appear to refer to the mathematical process of calculating an average score for the related comparison scores. No additional limitation is present.
With respect to claim(s) 11, the claim relates to the determination of an associated label for a minimum value of the scores. As performed by a person, these steps appear to refer to the mental and clerical process of labeling claims based on their level of verifiability. No additional limitation is present.
With respect to claim(s) 15, the claim relates to limiting the organization of documents to being less than the total number of documents available. As performed by a person, these steps appear to refer to the mental process of limiting the documents to the most probative available documents. No additional limitation is present.
With respect to claim(s) 18, the claim relates to reviewing the document set based on the size of the set overall and moving forward with the analysis if the set is within a predefined threshold. As performed by a person, these steps appear to refer to the mental process of determining that a set of documents is too large and reducing the set prior to comparing with the factual claims. No additional limitation is present.
These claims further do not remedy the judicial exception being integrated into a practical application and further fail to include additional elements that are sufficient to amount to significantly more than the judicial exception. As such, for the same reasons as described above with reference to independent claim(s) 1, 13, and 19-20, dependent claim(s) 3-11, 15, and 18 are not patent eligible.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 8-12, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey in view of Biegert, with further support from Jurafsky.
Regarding claim 1, Emrey discloses A method comprising (Systems and methods described with reference to “detecting errors and hallucinations in generative model output data”; Emrey, ¶ [0046]): identifying text data, wherein the text data was automatically generated by a language model of a question and answer computer system (“One or more processors of fact extraction stage 206 may be configured to receive inputs such as generative model output data 202 and ground truth information 204,” each of which may be generated by a language model as part of the NLG system.; Emrey, ¶ [0029], [0043], [0049]); identifying, within the text data, a list of items and introductory text that is associated with the list of items… (Though not expressly recited in the context of a list of items, the “Fact extraction stage 206 may comprise one or more natural language processing (NLP) models configured to extract entities, relationships between entities, and constraints defining the semantic context of the generative model output data 202” and exemplary models which may be incorporated, can include “named entity recognition models, coreference resolution models, and/or syntactic/semantic parsing models,” where the syntactic/semantic parsing model performs semantic parsing to “to determine the meaning of an input (e.g., generative model output data 202), but with particular attention paid to the arrangement of words and the grammar in the input,” including lists, and performs syntactic parsing to analyze the grammar and structure of the sentence, where the outputs of all models in the fact extraction stage result in a “collection of outputs” which “may constitute all or part of data structure 208 representing generative model output data”. Together, these components generate the individual components of the text chunk, which represent the constituent factual assertions of the text chunk. It is further noted that the parsing of a list of items based on main clauses (syntactic) for a predicate argument structure (semantic) and subordinate clauses (syntactic) as corresponding arguments (semantic) is an implicit and native capability which is a fundamental function of the syntactic/semantic parsers, as known in the relevant art. See, for example, Jurafski at page 243, explaining the handing of conjunctions (lists) as a fundamental part of grammar and parsing, and page 390 which discusses semantic role labeling, as part of text normalization, which “is the task of assigning semantic role labels to the constituents of a sentence” where “Semantic roles are abstract models of the role an argument plays in the event described by the predicate.”; Emrey, ¶ [0053]); for each item in the list of items: generating a claim that is based on the introductory text and said each item… (“The one or more processors at fact extraction stage 206 may use the pre-processed data (e.g., generative model output data 202 and/or ground truth information 204) to identify factual information and/or assertations (e.g., NLG-generated claims)” which may be applied to “identify or extract entities from the output data that form part of one or more NLG-generated claims in the output data 202” where “A given data structure 210 representing fact data may characterize... a collection of facts (e.g., related facts)”; Emrey, ¶ [0055]); and adding the claim to a set of claims that is associated with the text data; wherein the set of claims comprises multiple claims (“The one or more processors of fact extraction stage 206 may be configured to not only detect and extract entities from inputs, but also optionally to categorize the extracted entities,” where the extracted entities from inputs are associated with the same input, and the extracted entities {set of entities} is pluralized, thus corresponding to multiple claims.; Emrey, ¶ [0056]); for each claim in the set of claims: generating, by a machine learned model, a score that reflects a level of support of said each claim in a set of documents (“The data structure 208 representing generative model output data” may be provided to “the fact comparison stage 212 configured to compare and assess whether the data structure 208 representing generative model output data comprises hallucinations and/or errors” where the fact comparison stage can “compare data structure 208 representing generative model output data against data structure 210 representing fact data (or vice versa)” and assign “a comparison score based on the comparison to data structure 210 representing fact data” where the comparison score can represents the level of similarity or dissimilarity from the ground truth data, and the “fact comparison stage 212 may comprise one or more natural language understanding (NLU) models” examples of which include “BERT, RoBERTa, GPT-n, T5, transformer, and their derivatives {by a machine learning model}”; Emrey, ¶ [0055], [0063], [0066]-[0069]); adding the score to a set of scores for the set of claims; wherein the set of scores comprises multiple scores (The system assigns “a comparison score based on the comparison” each of the data points of the “data structure 210 representing fact data,” thus generating a set of scores for each of the data points, and where in the context of multiple data points “individual data points may be assigned a score {comprises multiple scores}”; Emrey, ¶ [0068]-[0069]); and causing, to be presented on a screen of a computing device, data that is based on the set of scores (The system can “distinguish whether one or more data points of data structure 208 representing generative model output data is very similar, moderately similar, moderately dissimilar, or very dissimilar from the data structure 210 representing fact data” and “Based on the comparison” the system “may generate an output 214 indicative of identified hallucinations and/or errors” which “may be provided in a machine-readable and/or user-readable (e.g., user-friendly) format.”; Emrey, ¶ [0069], [0072]); wherein the method is performed by one or more computing devices (“Software 450, which can be stored in storage 440 and executed by processor 410, can include, for example, the programming that embodies the functionality of the present disclosure (e.g., as embodied in the systems, computers, servers, and/or devices as described above)” and “can also be stored and/or transported within any computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described above, that can fetch and execute instructions associated with the software from the instruction execution system, apparatus, or device”; Emrey, ¶ [0086]-[0087]). However, Emrey fails to expressly recite wherein the list of items comprises multiple items introduced by the introductory text, [and] generating a claim that is based on the introductory text and said each item by prepending the introductory text to each item.
Biegert teaches systems and methods for “expanding and/or contracting text associated with lists.” (Biegert, ¶ [0001]). Regarding claim 1, Biegert teaches identifying, within the text data, a list of items and introductory text that is associated with the list of items, wherein the list of items comprises multiple items introduced by the introductory text (“Operation 430 can further comprise identifying list items based on the location of conjunctions (also referred to as list item separators herein). For the purposes of the present disclosure, conjunctions can include, but are not limited to, word conjunctions (e.g., and, or) and punctuation conjunctions (e.g., commas).” including the detection of “each list item” and “Prefix modifiers” which are “a word or phrase appearing before (explicitly or impliedly) each list item {introductory text}” where explicitly or implicitly appearing before a list item is understood as introducing that list item in the context of the claim language.; Biegert, ¶ [0056]-[0057]); [and] for each item in the list of items: generating a claim that is based on the introductory text and said each item by prepending the introductory text to each item (“In operation 470, a set of list modification rules can be retrieved... based on... the number and type of list item separators, the number and type of parts of speech, [and] the number and type of prefix and suffix modifiers” and “The list modification rule set can contain information regarding list modifications such as, but not limited to, when and how to expand lists” which, as explained with reference to FIG. 7, can include “append[ing] a prefix modifier, or a suffix modifier, or both a prefix modifier and a suffix modifier adjacent to each list item,” thus generating a claim that is based on the prefix modifier {the introductory text} and each list item {said each item} individually by appending {which, as read in the context of the example, is prepending} the prefix modifier of the list {introductory text} to each list item {each item}; Biegert, ¶ [0061]-[0062], [0075]-[0077]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the error and hallucination detection systems of Emrey to incorporate the teachings of Biegert to include wherein the list of items comprises multiple items introduced by the introductory text, [and] generating a claim that is based on the introductory text and said each item by prepending the introductory text to each item. Emrey discloses a response verification system to prevent hallucinations, the system including a fact extraction stage with “syntactic/semantic parsing models.” However, Emrey is silent regarding the parsing of elliptical lists. Biegert discloses preprocessing techniques for “analyze content or other data to identify and manipulate one or more explicit or implied lists” such that the system can “clarify the meaning of the one or more explicit or implied lists to improve, for example, Q&A operations.” A person having ordinary skill in the art would be motivated to combine the hallucination prevention system of Emrey with the list expansion of Biegert, such that important information in lists can be precisely ingested, such as during a Q&A operation, resulting in improved responsiveness of output as disclosed by Biegert. (Biegert, ¶ [0002], [0031]).
Regarding claim 2, the rejection of claim 1 is incorporated. Emrey and Biegert disclose all of the elements of the current invention as stated above. Emrey further discloses wherein the text data was generated by a question and answer computer system (“Generative model output data 202 may comprise outputs produced by a generative model” including “question-answering models” where generative models includes LLMs; Emrey, ¶ [0029], [0050]).
Regarding claim 3, the rejection of claim 1 is incorporated. Emrey and Biegert disclose all of the elements of the current invention as stated above. Emrey further discloses further comprising: identifying, within the text data, a pronoun (“The coreference resolution models of fact extraction stage 206 may generate a data structure output that comprises an indication of various pronouns or other linguistic expressions in the generative model output data 202 related to a given extracted entity.”; Emrey, ¶ [0058]); identifying, within the text data, one or more nouns upon which the pronoun is based (The pronoun is identified with relation to named entity, as derived from “the output of one or more NER models that have identified entities” which, in the context of a pronoun, the recognized named entity is a noun.; Emrey, ¶ [0058]); and prior to dividing the text data into sentences, replacing the pronoun with the one or more nouns (The coreference resolution model replaces the pronoun with the noun in the generated “data structure output” which occurs prior to parsing the text data; Emrey, ¶ [0053], [0058]-[0059]).
Regarding claim 8, the rejection of claim 1 is incorporated. Emrey and Biegert disclose all of the elements of the current invention as stated above. Emrey further discloses further comprising: for each score of the set of scores: mapping said each score to a range of values from among a plurality of ranges of values (“system 100 may be configured to identify information including NLG-generated claims in generative model output data and compare these NLG-generated claims to ground truth information to determine whether the NLG-generated claims are grounded in trusted factual information” where “the one or more processors of fact comparison stage 212 may utilize one or more thresholds to distinguish whether one or more data points of data structure 208 representing generative model output data is very similar, moderately similar, moderately dissimilar, or very dissimilar from the data structure 210” As such, the scores for each claim are mapped based on the threshold to at least five sets of scores, which correspond to the above labels.”; Emrey, ¶ [0037], [0069]); identifying a label that is associated with the range of values (“Exemplary thresholds may be about 75% or greater similarity for data points which are very similar, about 50-75% similarity for data points which are moderately similar, about 25-50% similarity for data points which are moderately dissimilar, and about 0-25% similarity for data points which are very dissimilar.”; Emrey, ¶ [0069]); and assigning the label to the claim that corresponds to said each score, (“The overall comparison score may be measured against one or more thresholds to determine whether the data structure 208 representing generative model output data meets or exceeds a standard for similarity (or dissimilarity) relative to data structure 210 representing fact data.”; Emrey, ¶ [0069]) wherein the data that is based on the set of scores is also based on the label of each claim in the set of claims (The overall comparison score can be cumulative of the individual comparison scores, and “the one or more processors of fact comparison stage 212 may generate an output 214 indicative of identified hallucinations and/or errors” which “may be provided in a... user-readable (e.g., user-friendly) format,” where the overall comparison score, the individual comparison scores, and/or the various threshold determinations of similarity, are all understood as “indicative of identified hallucinations and/or errors”.; Emrey, ¶ [0068]-[0069], [0072]).
Regarding claim 9, the rejection of claim 1 is incorporated. Emrey and Biegert disclose all of the elements of the current invention as stated above. Emrey further discloses further comprising: based on the text data, identifying a plurality of claims that includes the set of claims (“Data points within data structure 208 representing generative model output data (e.g., representative of utterances, individual entities, etc.) may be assigned a comparison score based on the comparison to data structure 210 representing fact data.”; Emrey, ¶ [0068]); wherein each claim in the plurality of claims is associated with a different score of a plurality of scores that includes the set of scores (Each claim in the plurality of claims “Data points within data structure 208... individual entities... may be assigned a comparison score” where the comparison score is assigned individually.; Emrey, ¶ [0068]); identifying a minimum score from the plurality of scores (“a threshold of similarity vs. dissimilarity may be dichotomous, e.g., scores greater than or equal to about 50% may indicate similarity, whereas scores below about 50% may indicate dissimilarity.”; Emrey, ¶ [0068]); and assigning, to the text data, the minimum score as a grounding score (In a dichotomy, the threshold is the minimum score and the grounding score.; Emrey, ¶ [0068]).
Regarding claim 10, the rejection of claim 1 is incorporated. Emrey and Biegert disclose all of the elements of the current invention as stated above. Emrey further discloses further comprising: based on the text data, identifying a plurality of claims that includes the set of claims (“Data points within data structure 208 representing generative model output data (e.g., representative of utterances, individual entities, etc.) may be assigned a comparison score based on the comparison to data structure 210 representing fact data.”; Emrey, ¶ [0068]); wherein each claim in the plurality of claims is associated with a different score of a plurality of scores that includes the set of scores (Each claim in the plurality of claims “Data points within data structure 208... individual entities... may be assigned a comparison score” where the comparison score is assigned individually.; Emrey, ¶ [0068]); computing a mean score from the plurality of scores (“individual data points may be assigned a score that can be compiled (e.g., using a weighted or unweighted sum) with that of related data points to determine an overall comparison score for a given portion of data structure 208”; Emrey, ¶ [0068]); and assigning, to the text data, the mean score as a grounding score (“an overall comparison score for a given portion of data structure 208 representing generative model output data as compared to data structure 210 representing fact data.”; Emrey, ¶ [0068]).
Regarding claim 11, the rejection of claim 1 is incorporated. Emrey and Biegert disclose all of the elements of the current invention as stated above. Emrey further discloses further comprising: based on the text data, identifying a plurality of claims that includes the set of claims (“Data points within data structure 208 representing generative model output data (e.g., representative of utterances, individual entities, etc.) may be assigned a comparison score based on the comparison to data structure 210 representing fact data.”; Emrey, ¶ [0068]); wherein each claim in the plurality of claims is associated with a different score of a plurality of scores that includes the set of scores (Each claim in the plurality of claims “Data points within data structure 208... individual entities... may be assigned a comparison score” where the comparison score is assigned individually.; Emrey, ¶ [0068]); identifying a minimum score from the plurality of scores (“a threshold of similarity vs. dissimilarity may be dichotomous, e.g., scores greater than or equal to about 50% may indicate similarity, whereas scores below about 50% may indicate dissimilarity,” where In a dichotomy, the threshold is the minimum score and the grounding score.; Emrey, ¶ [0068]); mapping minimum score to a range of values from among a plurality of ranges of values (The minimum score is mapped to the range of values from 0-50%, which is from the two possible values in this example, and two constitutes a plurality.; Emrey, ¶ [0068]); identifying a label that is associated with the range of values (The label associated with the range of values is indicated “dissimilarity,” also referred to as ungrounded.; Emrey, ¶ [0068]); and assigning, to the text data, the label as a grounding label (“The overall comparison score may be measured against one or more thresholds to determine whether the data structure 208 representing generative model output data meets or exceeds a standard for similarity (or dissimilarity) relative to data structure 210 representing fact data.”, thus assigning both the value and the label to the text data.; Emrey, ¶ [0069]).
Regarding claim 12, the rejection of claim 1 is incorporated. Emrey and Biegert disclose all of the elements of the current invention as stated above. Emrey further discloses further comprising: based on the text data, identifying a plurality of claims that includes the set of claims (“Data points within data structure 208 representing generative model output data (e.g., representative of utterances, individual entities, etc.) may be assigned a comparison score based on the comparison to data structure 210 representing fact data.”; Emrey, ¶ [0068]); wherein each claim in the plurality of claims is associated with a different score of a plurality of scores that includes the set of scores (Each claim in the plurality of claims “Data points within data structure 208... individual entities... may be assigned a comparison score” where the comparison score is assigned individually.; Emrey, ¶ [0068]); for each score of the plurality of scores: mapping said each score to a range of values from among a plurality of ranges of values (“system 100 may be configured to identify information including NLG-generated claims in generative model output data and compare these NLG-generated claims to ground truth information to determine whether the NLG-generated claims are grounded in trusted factual information” where “the one or more processors of fact comparison stage 212 may utilize one or more thresholds to distinguish whether one or more data points of data structure 208 representing generative model output data is very similar, moderately similar, moderately dissimilar, or very dissimilar from the data structure 210” As such, the scores for each claim are mapped based on the threshold to at least five sets of scores, which correspond to the above labels.”; Emrey, ¶ [0037], [0069]); identifying a label that is associated with the range of values (“Exemplary thresholds may be about 75% or greater similarity for data points which are very similar, about 50-75% similarity for data points which are moderately similar, about 25-50% similarity for data points which are moderately dissimilar, and about 0-25% similarity for data points which are very dissimilar.”; Emrey, ¶ [0069]); assigning the label to the claim that corresponds to said each score (“The overall comparison score may be measured against one or more thresholds to determine whether the data structure 208 representing generative model output data meets or exceeds a standard for similarity (or dissimilarity) relative to data structure 210 representing fact data.”; Emrey, ¶ [0069]); including the label in a set of labels (For each of the thresholds and respective labels, “about 75% or greater similarity” for “very similar,” “about 50-75% similarity” for “moderately similar,” “about 25-50% similarity” for “moderately dissimilar,” and “about 0-25% similarity” for “very dissimilar,” these can be further classified in the described dichotomy, where “scores greater than or equal to about 50% may indicate similarity, whereas scores below about 50% may indicate dissimilarity,” thus each label is included in a set of labels; Emrey, ¶ [0069]); determining a number of labels, in the set of labels, that indicate that the claim that corresponds to the label is grounded (In the above labels sets, very similar and moderately similar labels for the claims, which is two labels, correspond to an indication of similarity which corresponds to grounded.; Emrey, ¶ [0069]); based on the number of labels, determining a ratio of the number of labels to a particular number of labels that are in the set of labels (Though not disclosed as a ratio of the number of labels in the set of labels that indicates that the claim that corresponds to the label is grounded. However, such a ratio implicitly exists in Emrey. It is further noted that the overall score, in one example, is described as a dichotomy. Further, individual claims, each of which have individual labels and can each correspond to 4 different labels each with an associated percentage of similarity required for said label. as such, there are a range of ratios for each of the 4 different labels, based on the number of individual claims, which will result in being above the dichotomous threshold for grounding or below it.; Emrey, ¶ [0068], [0069]); and assigning, to the text data, the ratio as a grounding score (Said ratios, as described above for the combination of individual claims and overall claim grounding, can be assigned as a grounding score, as each combination which produces an about 50% cumulative similarity results in grounding.; Emrey, ¶ [0068], [0069]).
Regarding claim 19, Emrey and Biegert disclose One or more non-transitory storage media storing instructions (“Software 450, which can be stored in storage 440 and executed by processor 410, can include, for example, the programming that embodies the functionality of the present disclosure (e.g., as embodied in the systems, computers, servers, and/or devices as described above)” and “can also be stored and/or transported within any computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described above, that can fetch and execute instructions associated with the software from the instruction execution system, apparatus, or device”; Emrey, ¶ [0086]-[0087]) which, when executed by one or more computing devices, cause performance of the method recited in Claim 1cause performance of the method recited in Claim 1 (See mapping of limitations presented with reference to claim 1; Emrey and Biegert, ¶ (See above)).
Claims 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey and Biegert as applied to claim 1 above, and further in view of Salloum.
Regarding claim 4, the rejection of claim 1 is incorporated. Emrey and Biegert disclose all of the elements of the current invention as stated above. Emrey further discloses wherein the method further comprises: prior to generating the score for each claim in the set of claims, removing… [numbers] (describes “the input data received by the system may be pre-processed,” which includes “segmentation, tokenization, stemming, lemmatization, parts-of-speech (POS) tagging, and/or other NLP pre-processing techniques” where text normalization is a well-known NLP pre-processing technique (example, Jurafsky, pg. 425 “remove numeric quantities”; pg. 570 “a first pass of text normalization to deal with numbers and abbreviations and other non-standard words”), and includes removing numbers, and other non-standard or non-semantically informative elements, from text, and where the pre-processing can occur prior to “the input data”, which can include the ground truth data, being subject to “further (e.g., substantive) processing of the data,” thus the preprocessing, and the text normalization, necessarily occurs prior to processing for generating the comparison score, as the system must receive the input data and the ground truth document data, before processing the same.; Emrey, ¶ [0054]-[0055]). However, Emrey fails to expressly recite wherein the list of items is a numbered list, further comprising...removing each number that precedes an item in the list of items.
Salloum teaches systems and methods for automatic formatting of documents. (Salloum, ¶ [0002]). Regarding claim 4, Salloum teaches wherein the list of items is a numbered list, wherein the method further comprises...removing each number that precedes an item in the list of items (Specifically discloses the preprocessing of a numbered list, where “Numbered lists have line-initial numbers replaced with dummy tokens. The first item in a numbered list has one token (‘NUM_LIST_1’), and all subsequent numbers have another (‘NUM_LIST_8’).”; Salloum, ¶ [0036]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the error and hallucination detection systems of Emrey, as modified by the list preprocessing of Biegert, to incorporate the teachings of Salloum to include wherein the list of items is a numbered list, further comprising...removing each number that precedes an item in the list of items. Salloum discloses uniformly subjecting documents for “text preprocessing” including the removal of numerals from numbered lists, “to better enable the translation system to reproduce punctuation and other formatting elements as well as to combat problems of sparsity for numerals”, which gives better consideration to informative numbers than prior art systems, thus retaining information which might otherwise be lost from standard text normalization, as recognized by Salloum. (Salloum, ¶ [0036]).
Claims 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey and Biegert as applied to claim 1 above, and further in view of Mirhaji.
Regarding claim 5, the rejection of claim 1 is incorporated. Emrey and Biegert disclose all of the elements of the current invention as stated above. Emrey further discloses wherein the list of items is within a sentence of the text data and is a flattened list (Though not described in the context of a list of items, the “Fact extraction stage 206 may comprise one or more natural language processing (NLP) models configured to extract entities, relationships between entities, and constraints defining the semantic context of the generative model output data 202” and exemplary models which may be incorporated, can include “named entity recognition models, coreference resolution models, and/or syntactic/semantic parsing models,” where the syntactic/semantic parsing model performs semantic parsing to understand the meaning of the individual components of a sentence, including the parsing of conjunctions (flattened lists), and performs syntactic parsing to analyze the grammar and structure of the sentence. Together, these components generate the individual components of the sentence, including the parsing of lists, which represent the constituent factual assertions. It is noted that the parsing of a list of items is an implicit and native capability which is a fundamental function of the syntactic/semantic parsers, as known in the relevant art.; Emrey, ¶ [0053]-[0054]). However, Emrey fails to expressly recite further comprising: determining that the list of items is a flattened list based on a number of commas or based on a number of phrases separated by commas in the sentence.
Mirhaji teaches systems and methods for “collection, integration and contextualization of information.” (Mirhaji, ¶ [0002]). Regarding claim 5, Mirhaji teaches further comprising: determining that the list of items is a flattened list based on a number of commas or based on a number of phrases separated by commas in the sentence (Discloses that a syntax ontology 135 is a conventional tool used by a parser 182 to parse text by establishing a basis for “identifying certain linguistic expressions.” This includes using “syntactic cues that may be reliably used for segmentation of a sentence”, such as “punctuations (for example, “.”, ″, “;”).” Further, Mirhaji provides specific example of this native function in FIG. 32, which shows the “output of a syntactic parser” breaking a single sentence “Large Blister on Toes and Abdomen,” which would be a flattened list per applicant’s specification, into its component facts.; Mirhaji, ¶ Col. 15, lines 5-14; Col. 20, lines 6-14, Col. 33, lines 16-18; FIG. 32).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the error and hallucination detection systems of Emrey, as modified by the list preprocessing of Biegert, to incorporate the teachings of Mirhaji to include further comprising: determining that the list of items is a flattened list based on a number of commas or based on a number of phrases separated by commas in the sentence. Mirhaji discloses systems and methods which “aid in the collection, representation and mining of data,” which “enable complete, reliable and fast collection and validation of information throughout various research projects, and among different participating locations,” with recognized “implications in multiple different contexts (decision support, research, quality of care, etc.),” including in a question answering capacity, as recognized by Mirhaji. (Mirhaji, Col. 1, lines 62-67, Col. 9, lines 3-10, Col. 13, lines 1-22).
Claims 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey and Biegert as applied to claim 1 above, and further in view of Kelsey (U.S. Pat. App. Pub. No. 2018/0260472, hereinafter Kelsey).
Regarding claim 6, the rejection of claim 1 is incorporated. Emrey and Biegert disclose all of the elements of the current invention as stated above. However, Emrey fails to expressly recite further comprising: identifying one or more filler sentences in the text data; removing the one or more filler sentences from consideration when generating a particular score for the text data.
Kelsey teaches systems and methods for “natural language processing.” (Kelsey, ¶ [0002]). Regarding claim 6, Kelsey teaches further comprising: identifying one or more filler sentences in the text data (“identification of paragraphs and/or discrete text chunks of related sentences having low content value”; Kelsey, ¶ [0057]); removing the one or more filler sentences from consideration when generating a particular score for the text data (In response to the “identification of paragraphs and/or discrete text chunks of related sentences having low content value” the “received text can be pre-processed and filtered”; Kelsey, ¶ [0056]-[0057]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the error and hallucination detection systems of Emrey, as modified by the list preprocessing of Biegert, to incorporate the teachings of Kelsey to include further comprising: identifying one or more filler sentences in the text data; removing the one or more filler sentences from consideration when generating a particular score for the text data. “The disclosed technologies adopt an approach of successive refinement in stages to simplify text, identify and extract semantic content, and obtain digested fragments of text” which includes “identification of paragraphs and/or discrete text chunks of related sentences having low content value,” where removal of sentences having a low content value provides the known benefit of “quality improvement can arise from process refinement” for “a succession of source documents,” which provides the known benefit of better quality source material, such as for question and answer systems, as recognized by Kelsey. (Kelsey, ¶ [0029], [0031]).
Claims 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey and Biegert as applied to claim 1 above, and further in view of Rahman.
Regarding claim 7, the rejection of claim 1 is incorporated. Emrey and Biegert disclose all of the elements of the current invention as stated above. Emrey further discloses further comprising: identifying, within the text data, a sentence that refers to a prompter in a second person manner (“The coreference resolution models of fact extraction stage 206 may generate a data structure output that comprises an indication of various pronouns or other linguistic expressions in the generative model output data 202 related to a given extracted entity,” where a pronoun is a reference to a person, such as you, s/he, etc., and where “an indication of various pronouns or other linguistic expressions in the generative model output data 202” is an identification of the sentence from the “generative model output data 202” which contains said indication; Emrey, ¶ [0058]); in response to identifying the sentence, identifying... [relevant information] associated with the sentence (“The coreference resolution models may receive a pre-processed input as described above with respect to the input of the named entity recognition (NER) models. The pre-processed input (e.g., generative model output data 202 and/or ground truth information 204)” where ground truth information can include the user input.; Emrey, ¶ [0034], [0058]); generating a new sentence that is based on the sentence and the [relevant information] (The coreference resolution model replaces the pronoun, as derived from the sentence, with the noun, as derived from the relevant information, in the generated “data structure output”; Emrey, ¶ [0053], [0058]-[0059]) generating a particular score for the text data that is based on the new sentence (“Data points within data structure 208 representing generative model output data (e.g., representative of utterances, individual entities, etc.) may be assigned a comparison score based on the comparison to data structure 210 representing fact data.”; Emrey, ¶ [0068]). However, Emrey fails to expressly recite in response to identifying the sentence, identifying a question that is associated with the sentence; [and] generating a new sentence that is based on the sentence and the question.
Rahman teaches systems and methods for “generating and correcting language model outputs.” (Rahman, ¶ [0002]). Regarding claim 7, Rahman teaches in response to identifying the sentence, identifying a question that is associated with the sentence (“responses generated by language model 204 can include inherent ambiguities. For example, pronouns such as “he”, “she”, “it” can refer to multiple different nouns, and an entity can be described in various ways across a text (e.g., “president of the United States”, “he”),” and “Request processing module 202 provides the response generated by language model 204 to NER model 208 in order to find named entities that can be used by coreference resolution module 210 to resolve ambiguous references in the response.”; Rahman, ¶ [0029]); generating a new sentence that is based on the sentence and the question (“Given the response and name entities as inputs, coreference resolution module 210 generates a resolved response.”; Rahman, ¶ [0029]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the error and hallucination detection systems of Emrey, as modified by the list preprocessing of Biegert, to incorporate the teachings of Rahman to include in response to identifying the sentence, identifying a question that is associated with the sentence; [and] generating a new sentence that is based on the sentence and the question. “The disclosed techniques” of Rahman “can identify a response as being incomplete when not all relevant portions of a context are included in the response, “ where the “incomplete response can be modified to include additional portions of the context that are relevant to a user request”, providing the known and obvious benefit of a complete and accurate response to a prompt, as recognized by Rahman. (Rahman, ¶ [0065]).
Claims 13, 15-18 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey in view of DeFoor.
Regarding claim 13, Emrey discloses A method comprising (Systems and methods described with reference to “detecting errors and hallucinations in generative model output data”; Emrey, ¶ [0046]): identifying text data that was output by a language model of a question and answer computer system (“One or more processors of fact extraction stage 206 may be configured to receive inputs such as generative model output data 202 and ground truth information 204” where the “outputs” are generated by question-answering models, including LLMs; Emrey, ¶ [0049], [0050]) based on a prompt and a plurality of documents (the output of the LLM is in response to a user input, and “ground truth information” which “may (or may not) be an input to the generative model” and “the ground truth information may include one or more (e.g., a collection of)…trusted or authoritative documents... factual information and/or information based on opinions...” and/or “‘desired’ truth representative of the user-curated set of documents”; Emrey, ¶ [0033]-[0034]); identifying a set of claims within the text data (“The one or more processors at fact extraction stage 206 may use the pre-processed data (e.g., generative model output data 202 and/or ground truth information 204) to identify factual information and/or assertations (e.g., NLG-generated claims)” which may be applied to “identify or extract entities from the output data that form part of one or more NLG-generated claims in the output data 202,” which may be either viewed as a single claim (e.g., one”) or a group (e.g., “or more”), which further, “may constitute all or part of data structure 208 representing generative model output data.”; Emrey, ¶ [0049], [0055], [0060]); generating a plurality of combinations of the plurality of documents, (Discloses “generat[ing]… data structure 210 representing fact data” which can be “a collection of documents (e.g. related documents). {a plurality of documents}” and “pairs of data structures 208 and 210” indicates a plurality of data structures 210 {a plurality of combinations}. It is noted that numerous portions of Emrey describe multiple data structures 210 (see also “one or more comparisons (e.g., as described below) using existing data structures 210” at [0062]); Emrey, ¶ [0049], [0060], [0062], [0066]) wherein each combination of the plurality of combinations is based on two or more documents in the plurality of documents, (the generation of data structure 210 can be based on “a collection of documents” {based on two or more documents} which can be taken from the broad set of all documents which qualify as “ground truth documents”; Emrey, ¶ [0060], [0062]) wherein the plurality of combinations comprises a first combination and a second combination… (Discloses a plurality of data structures 208 and 210, where a plurality of data structures includes at least a first data structure and a second data structure.; Emrey, ¶ [0049], [0060], [0062], [0066]); for each claim in the set of claims: generating, by a machine learned model, for said each claim, a score that reflects a level of support of said each claim in a combination (“The data structure 208 representing generative model output data” may be provided to “the fact comparison stage 212 configured to compare and assess whether the data structure 208 representing generative model output data comprises hallucinations and/or errors” where the fact comparison stage can “compare data structure 208 representing generative model output data against data structure 210 representing fact data (or vice versa)” and assign “a comparison score based on the comparison to data structure 210 representing fact data” where the comparison score can represents the level of similarity or dissimilarity from the ground truth data. Further “The system may be configured to compare data structure 208 representing generative model output data against …all data points in one or more of the data structures,” and “the data structure 208 representing generative model output data may be compared to the data structure 210 representing fact data” based on both “one data point from the data structure 208 representing generative model output data compared to more than one data point from the data structure 210 representing fact data” and “multiple data points from a first one of the data structures may each be compared to multiple data points from the second one of the data structures” which is understood to result in a score for each comparison, and each claim in data structure 208 being compared to each data point in each of the data structures 210.; Emrey, ¶ [0063], [0067]-[0069]); wherein generating the score is performed for each combination of the plurality of combinations (Specifically discloses “Detecting the degree of similarity between information in each of the data structures may comprise determining differences between data points (e.g., data values, nodes, etc.) in the data structures.” As such, the comparison is understood as being performed against all relevant data individually (i.e., “each of the data structures...”); Emrey, ¶ [0066]); wherein the score for said each claim reflects a level of support of said each claim in said each combination (As stated above the “the degree of similarity” is determined for “information in each of the data structures”, where the individual degree of similarity for the “pairs of data structures 208 and 210” as compared in a one to one, one to many and/or a many to many manner, reflects “whether generative model output data 202 is substantiated by ground truth information 204 {a level of support}.”; Emrey, ¶ [0066]-[0067]); adding the score to a set of scores for said each claim (The system assigns “a comparison score based on the comparison” each of the data points of the number of “data structure 210 representing fact data,” thus generating a set of scores for each of the data points.; Emrey, ¶ [0068]-[0069]); and causing data to be presented on a screen of a computing device based on the set of scores (The system can “distinguish whether one or more data points of data structure 208 representing generative model output data is very similar, moderately similar, moderately dissimilar, or very dissimilar from the data structure 210 representing fact data” and “Based on the comparison” the system “may generate an output 214 indicative of identified hallucinations and/or errors” which “may be provided in a machine-readable and/or user-readable (e.g., user-friendly) format.”; Emrey, ¶ [0069], [0072]); wherein the method is performed by one or more computing devices (“Software 450, which can be stored in storage 440 and executed by processor 410, can include, for example, the programming that embodies the functionality of the present disclosure (e.g., as embodied in the systems, computers, servers, and/or devices as described above)” and “can also be stored and/or transported within any computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described above, that can fetch and execute instructions associated with the software from the instruction execution system, apparatus, or device”; Emrey, ¶ [0086]-[0087]). However, Emrey fails to expressly recite wherein the first combination and the second combination include the same document.
DeFoor teaches systems and methods for “the evaluation of large volumes of documents” by large language models. (DeFoor, ¶ [0029]-[0030]). Regarding claim 13, DeFoor teaches wherein the plurality of combinations comprises a first combination and a second combination, wherein the first combination and the second combination include the same document (Discloses generating a plurality of text chunks {combinations}, where “text may be pre-divided into a number of different portions,” which necessarily includes a first text chunk {a first text portion} and a second text chunk {second text portion}, where exemplary “divisions of text into portions may include... lists of documents, documents, document sections, document pages, document paragraphs, and document sentences” which may be the result of a prior chunking process, and where, in various embodiments, the text chunks may include “text portions belonging to the same document, document section, paragraph, and/or sentence may be grouped together” or grouped separately. The same document being divided and grouped separately, results in the first combination and the second combination including the same document.; DeFoor, ¶ [0098], [0100], [0105]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the error and hallucination detection systems of Emrey to incorporate the teachings of DeFoor to include wherein the first combination and the second combination include the same document. Emrey discloses a sophisticated hallucination detection system which includes the partitioning of ground truth information into a plurality of data structures 210. However, Emrey fails to expressly recite exactly how these data structures 210 are generated from the overall ground truth information set. DeFoor discloses an automatic system for query evaluation including text sharding of a large ground truth dataset, for the generation of a plurality of selected text portions which are under a token limit threshold. A person of ordinary skill in the art would be motivated to combine the hallucination detection system of Emrey with the text sharding of DeFoor to assure that the prompt and all received ground truth information falls within the token limit of the model, providing the well-known benefit of assuring that the model is capable of receiving and processing the entire input, as understood in light of the disclosure of DeFoor. (DeFoor, ¶ [0095]).
Regarding claim 15, the rejection of claim 13 is incorporated. Emrey and DeFoor disclose all of the elements of the current invention as stated above. Emrey further discloses wherein each combination in the plurality of combinations comprises less than all of the plurality of documents (Though not expressly described as a determination that each combination comprises less than all of the documents which comprise the “ground truth information”, Discloses determining whether an “existing data structure 210” is “suitable for verification of generative model output data” and that such an existing data structure may have “insufficient information... to verify or refute a purported fact from the generative model output data.” As each of these data structures are generated from the ground truth information and may contain information which is not suitable for any specific “generative model output data”, the combinations of documents throughout all data structures 210” as generated for any specific data structure 208 necessarily contain less than the entirety of available ground truth information (i.e., only documents which are “suitable for verification” are included, and documents which are not suitable for verification are acknowledged to exist); Emrey, ¶ [0062]).
Regarding claim 16, the rejection of claim 13 is incorporated. Emrey and DeFoor discloses all of the elements of the current invention as stated above. However, Emrey fails to expressly recite wherein generating the plurality of combinations comprises: determining that total size of the plurality of documents is greater than a predefined threshold; creating a plurality of new combinations from the plurality of documents, wherein a first new combination in the plurality of new combinations is created by removing one of the plurality of documents from the plurality of documents, wherein a second new combination in the plurality of new combinations is created by removing another one of the plurality of documents from the plurality of documents; for each new combination in the plurality of new combinations: determining whether said each new combination exceeds the predefined threshold; if said each new combination exceeds the predefined threshold, then adding said each new combination to an OVER_THE_LIMIT set; if said each new combination does not exceed the predefined threshold, then adding said each new combination to the plurality of combinations.
DeFoor teaches systems and methods for “the evaluation of large volumes of documents” by large language models. (DeFoor, ¶ [0029]-[0030]). Regarding claim 16, DeFoor teaches wherein generating the plurality of combinations comprises: determining that total size of the plurality of documents is greater than a predefined threshold (As part of the text sharding method 500, “a text generation modeling system implementing a large language model may specify a size threshold in terms of a number of tokens (e.g., words).” and a the system can then make a “determination... as to whether the length of the selected text portion exceeds the maximum text chunk size,” where the selected text portion “may be the entirety of the text” and each of the text portions may refer to “documents” and/or “lists of documents”; DeFoor, ¶ [0097]-[0098], [0100]-[0101]); creating a plurality of new combinations from the plurality of documents, (“If it is determined that the length of the selected text portion {the plurality of documents} exceeds the maximum text chunk size, then at 510 one or more domain-specific text chunking constraints are identified” and “An updated text portion that does not exceed the maximum text chunk size is identified at 512” resulting in a plurality of “text portions selected at 506 and identified at 512” {a plurality of new combinations}”; DeFoor, ¶ [0102]-[0103]) wherein a first new combination in the plurality of new combinations is created by removing one of the plurality of documents from the plurality of documents, (The disclosed plurality of updated text portions includes at least a first updated text portion, where in the context of a dividing a larger selected text portion into the plurality of “smaller” updated text portions, from the perspective of the first updated text portion, the documents of all other updated text portions including the second updated text portion are “removed” {removing one of the plurality of documents} from the selected text portion to generate the first updated text portion.; DeFoor, ¶ [0095], [0103]) wherein a second new combination in the plurality of new combinations is created by removing another one of the plurality of documents from the plurality of documents (The disclosed plurality of updated text portions includes at least a first updated text portion, where in the context of a dividing a larger selected text portion into the plurality of “smaller” updated text portions, from the perspective of the first updated text portion, the documents of all other updated text portions including the first updated text portion are “removed” {removing another one of the plurality of documents} from the selected text portion to generate the second updated text portion.; DeFoor, ¶ [0095], [0103]); for each new combination in the plurality of new combinations: determining whether said each new combination exceeds the predefined threshold (Further “two or more of the text portions resulting from the division at 512” can be above or below “the maximum text chunk size” and, as such, “each of these may be assigned to a text chunk or chunks at operation 514” where the assignment itself may be a provisional assignment or it may be omitted, subject to FIG. 6, where “A determination is made at 516 as to whether to select an additional portion of the text” each updated text portion, where the plurality of updated text portions, as provisionally assigned to the respective text chunk, is then reviewed at 508 in the same manner as the selected text portion, including determining at 508 “whether the length of the selected text portion exceeds the maximum text chunk size. {whether each new combination exceeds the predefined threshold}”; DeFoor, ¶ [0101], [0106]-[0108]; FIG. 5); if said each new combination exceeds the predefined threshold, then adding said each new combination to an OVER_THE_LIMIT set (If “the length of” any of the plurality of updated text portions “exceeds the maximum text chunk size,” said updated text portion is subject to division, as described with reference to the selected text portion, which is understood as being part of an “OVER THE_LIMIT” set.; DeFoor, ¶ [0101], [0106]-[0108]; FIG. 5); if said each new combination does not exceed the predefined threshold, then adding said each new combination to the plurality of combinations (Once the selection process is complete, each of the “updated text portions” {new combinations} is added to a respective text chunk of the set of text chunks {the plurality of combinations}.; DeFoor, ¶ [0104], [0106], [0108]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the error and hallucination detection systems of Emrey to incorporate the teachings of DeFoor to include wherein generating the plurality of combinations comprises: determining that total size of the plurality of documents is greater than a predefined threshold; creating a plurality of new combinations from the plurality of documents, wherein a first new combination in the plurality of new combinations is created by removing one of the plurality of documents from the plurality of documents, wherein a second new combination in the plurality of new combinations is created by removing another one of the plurality of documents from the plurality of documents; for each new combination in the plurality of new combinations: determining whether said each new combination exceeds the predefined threshold; if said each new combination exceeds the predefined threshold, then adding said each new combination to an OVER_THE_LIMIT set; if said each new combination does not exceed the predefined threshold, then adding said each new combination to the plurality of combinations. Emrey discloses a sophisticated hallucination detection system which includes the partitioning of ground truth information into a plurality of data structures 210. However, Emrey fails to expressly recite exactly how these data structures 210 are generated from the overall ground truth information set. DeFoor discloses an automatic system for query evaluation including text sharding of a large ground truth dataset, for the generation of a plurality of selected text portions which are under a token limit threshold. A person of ordinary skill in the art would be motivated to combine the hallucination detection system of Emrey with the text sharding of DeFoor to assure that the prompt and all received ground truth information falls within the token limit of the model, providing the well-known benefit of assuring that the model is capable of receiving and processing the entire input, as understood in light of the disclosure of DeFoor. (DeFoor, ¶ [0095]).
Regarding claim 17, the rejection of claim 16 is incorporated. Emrey and DeFoor disclose all of the elements of the current invention as stated above. However, Emrey fail(s) to expressly recite further comprising, for each combination in the OVER_THE_LIMIT set: creating one or more new particular combinations from said each combination; for each new particular combination in the one or more new particular combinations: determining whether said each new particular combination exceeds the predefined threshold; if said each new particular combination exceeds the predefined threshold, then adding said each new particular combination to the OVER_THE_LIMIT set; and if said each new particular combination does not exceed the predefined threshold, then adding said each new particular combination to the plurality of combinations.
The relevance of DeFoor is described above with relation to claim 13. Regarding claim 17, DeFoor teaches further comprising, for each combination in the OVER_THE_LIMIT set (all updated text portion which exceed the size threshold, and thus are part of an “OVER THE_LIMIT” set, are reprocessed as shown in FIG. 5.; DeFoor, ¶ [0101], [0106]-[0108]; FIG. 5): creating one or more new particular combinations from said each combination (“If it is determined that the length of the... [updated text portion] {the plurality of documents} exceeds the maximum text chunk size, then at 510 one or more domain-specific text chunking constraints are identified” and “An updated text portion that does not exceed the maximum text chunk size is identified at 512” resulting in a plurality of “text portions selected at 506 and identified at 512” {one or more new particular combinations}”; DeFoor, ¶ [0102]-[0103]); for each new particular combination in the one or more new particular combinations: determining whether said each new particular combination exceeds the predefined threshold (The process described in FIG. 5 is iterative, as such the “two or more of the text portions resulting from the division at 512” can each be determined as above or below “the maximum text chunk size” and, as such, “each of these may be assigned to a text chunk or chunks at operation 514” where the assignment itself may be a provisional assignment or it may be omitted, subject to FIG. 6, where “A determination is made at 516 as to whether to select an additional portion of the text” each updated text portion, where the plurality of updated text portions, as provisionally assigned to the respective text chunk, is then reviewed at 508 in the same manner as the selected text portion, including determining at 508 “whether the length of the selected text portion exceeds the maximum text chunk size. {whether each new particular combination exceeds the predefined threshold}”; DeFoor, ¶ [0101], [0106]-[0108]; FIG. 5); if said each new particular combination exceeds the predefined threshold, then adding said each new particular combination to the OVER_THE_LIMIT set (If “the length of” any of the plurality of updated text portions “exceeds the maximum text chunk size,” said updated text portion is subject to division, as described with reference to the selected text portion, which is understood as being part of an “OVER THE_LIMIT” set.; DeFoor, ¶ [0101], [0106]-[0108]; FIG. 5); and if said each new particular combination does not exceed the predefined threshold, then adding said each new particular combination to the plurality of combinations (Once the selection process is complete, each of the “updated text portions” {new particular combinations} is added to a respective text chunk of the set of text chunks {the plurality of combinations}.; DeFoor, ¶ [0104], [0106], [0108]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the error and hallucination detection systems of Emrey to incorporate the teachings of DeFoor to include further comprising, for each combination in the OVER_THE_LIMIT set: creating one or more new particular combinations from said each combination; for each new particular combination in the one or more new particular combinations: determining whether said each new particular combination exceeds the predefined threshold; if said each new particular combination exceeds the predefined threshold, then adding said each new particular combination to the OVER_THE_LIMIT set; and if said each new particular combination does not exceed the predefined threshold, then adding said each new particular combination to the plurality of combinations. Emrey discloses a sophisticated hallucination detection system which includes the partitioning of ground truth information into a plurality of data structures 210. However, Emrey fails to expressly recite exactly how these data structures 210 are generated from the overall ground truth information set. DeFoor discloses an automatic system for query evaluation including text sharding of a large ground truth dataset, for the generation of a plurality of selected text portions which are under a token limit threshold. A person of ordinary skill in the art would be motivated to combine the hallucination detection system of Emrey with the text sharding of DeFoor to assure that the prompt and all received ground truth information falls within the token limit of the model, providing the well-known benefit of assuring that the model is capable of receiving and processing the entire input, as understood in light of the disclosure of DeFoor. (DeFoor, ¶ [0095]).
Regarding claim 18, the rejection of claim 13 is incorporated. Emrey and DeFoor discloses all of the elements of the current invention as stated above. Emrey further discloses further comprising: prior to generating the score, …[performing preprocessing of the documents] (“the input data received by the system may be pre-processed, or fact extraction stage 206 may comprise one or more pre-processing models to process the received input data prior to further (e.g., substantive) processing of the data.”; Emrey, ¶ [0054]); wherein generating the score is only performed in response to…[preprocessing of the documents] (The pre-processing can occur prior to “the input data”, which can include the ground truth data, being “further (e.g., substantive) processing of the data,” thus the preprocessing, and the text normalization, necessarily occurs prior to processing for generating the comparison score, as the system must receive the input data and the ground truth document data, before processing the same.; Emrey, ¶ [0054]-[0055]). However, Emrey fail(s) to expressly recite [wherein preprocessing of the documents includes]... determining whether a size of the combination is less than a predefined threshold; and wherein [further processing steps are]… only performed in response to determining that the size of the combination is less than the predefined threshold.
The relevance of DeFoor is described above with relation to claim 13. Regarding claim 18, DeFoor teaches [wherein preprocessing of the documents includes]... determining whether a size of the combination is less than a predefined threshold (As part of the text sharding method 500, which is a preprocessing step, “a text generation modeling system implementing a large language model may specify a size threshold in terms of a number of tokens (e.g., words).” and a the system can then make a “determination... as to whether the length of the selected text portion exceeds the maximum text chunk size,” where the selected text portion “may be the entirety of the text” and each of the text portions may refer to “documents” and/or “lists of documents” and, as a preprocessing step, this is prior to generating a score in the context of Emrey.; DeFoor, ¶ [0097]-[0098], [0100]-[0101]); and wherein [further processing steps are]… only performed in response to determining that the size of the combination is less than the predefined threshold (DeFoor performs the above process until all text chunks are comprised of updated text segments which are below the size threshold. As such, preprocessing is not complete until the size is less than a predetermined threshold. In the context of Emrey, since the score is not determined until after preprocessing is complete, the score would only be performed in response to determining that each “updated text portion... does not exceed the maximum text chunk size is identified at 512”; DeFoor, ¶ [0102]-[0103]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the error and hallucination detection systems of Emrey to incorporate the teachings of DeFoor to include [wherein preprocessing of the documents includes]... determining whether a size of the combination is less than a predefined threshold; and wherein [further processing steps are]… only performed in response to determining that the size of the combination is less than the predefined threshold. Emrey discloses a sophisticated hallucination detection system which includes the partitioning of ground truth information into a plurality of data structures 210. However, Emrey fails to expressly recite exactly how these data structures 210 are generated from the overall ground truth information set. DeFoor discloses an automatic system for query evaluation including text sharding of a large ground truth dataset, for the generation of a plurality of selected text portions which are under a token limit threshold. A person of ordinary skill in the art would be motivated to combine the hallucination detection system of Emrey with the text sharding of DeFoor to assure that the prompt and all received ground truth information falls within the token limit of the model, providing the well-known benefit of assuring that the model is capable of receiving and processing the entire input, as understood in light of the disclosure of DeFoor. (DeFoor, ¶ [0095]).
Regarding claim 20, Emrey and DeFoor disclose One or more non-transitory storage media storing instructions (“Software 450, which can be stored in storage 440 and executed by processor 410, can include, for example, the programming that embodies the functionality of the present disclosure (e.g., as embodied in the systems, computers, servers, and/or devices as described above)” and “can also be stored and/or transported within any computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described above, that can fetch and execute instructions associated with the software from the instruction execution system, apparatus, or device”; Emrey, ¶ [0086]-[0087]) which, when executed by one or more computing devices, cause performance of the method recited in Claim 13 (See mapping of limitations presented with reference to claim 13; Emrey and DeFoor, ¶ (see above)).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Barrett (U.S. Pat. App. Pub. No. 2010/0169309) discloses methods and systems for tagging documents with annotations and for extracting text from documents using such annotations.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Sean E. Serraguard whose telephone number is (313)446-6627. The examiner can normally be reached 07:00-17:00 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel C. Washburn can be reached at (571) 272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Sean E Serraguard/Patent Examiner, Art Unit 2657
Prosecution Insights