DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
All objections/rejections not mentioned in this Office Action have been withdrawn by the Examiner.
Examiner’s Note
Applicant’s Patent Application Fee Determination Record (SB06), as filed on 09 February 2026, indicates two (2) independent claims and 19 total claims including dependent claims. The listed independent claims appear to correspond to claims 1 and 13 as these are the only claims which are clearly in independent form and contain no statements indicating reliance on another claim. The remaining claims, claims 2-12 and 15-20 are understood, as indicated by the applicant, to be asserted as dependent claims.
Applicant is advised regarding the interpretation of dependency of the claims. The Federal Circuit has indicated that “A claim's status as dependent or independent depends on the substance of the claim in light of the language of § 112, ¶ 4, and not the form alone.” (Monsanto v. Syngenta Seeds, 503 F.3d 1352, 1357 (Fed. Cir. 2007)). The court further explained that, under pre-AIA 35 U.S.C § 112, ¶ 4 (currently 35 U.S.C § 112(d)), “[t]o establish whether a claim is dependent upon another, this court examines if the new claim both refers to an earlier claim and further limits that referent.” (Id., citing 35 U.S.C § 112, ¶ 4 (2000)). Specifically with reference to dependence from a process claim, the court held that whether a claim is properly read as dependent or independent turns on whether the asserted dependent claim “specifically requires …the performance of the steps” in the referent process claim. (Id. at 1358).
During prosecution, it falls to the examiner to determine status as dependent or independent with reference to 35 U.S.C § 112(d). (See MPEP 2173.05(f)). As such, we perform the same analysis here for claims 2-12 and 15-20.
Regarding claims 2-12, these claims both refer to claim 1 and provide further limitation to that claim. As well, under the broadest reasonable interpretation, claims 2-12 specifically require the performance of the steps in the referent process claim. As such, claims 2-12 are understood as dependent claims for the purposes of further prosecution.
Regarding claims 15-18, these claims both refer to claim 13 and provide further limitation to that claim. As well, under the broadest reasonable interpretation, claims 15-18 specifically require the performance of the steps in the referent process claim. As such, claims 15-18 are understood as dependent claims for the purposes of further prosecution.
With reference to the substance of claims 19 and 20, these claims “refer to an earlier claim” but the fail to “further limit that referent,” as the limitations are directed to changing the statutory class of the referent claim. Claims 1 and 13 recites a process which “consist[s] of a series of steps or acts to be performed.” (MPEP 2106). Claims 19 and 20 are asserted as dependent from claims 1 and 13, respectively, but act only to convert the previously cited process to another statutory class. Specifically, claims 19 and 20 are directed to a storage medium. Claims 19 and 20 are not understood as a result of, a continuation of, or a further limit to the process of claims 1 and 13, respectively.
Further, claims 19 and 20 do not specifically require the performance of the steps in the referent process claim. Though claims 1 and 13 recite a series of steps or acts to be performed, the broadest reasonable interpretation of claims 19 and 20 fails to require the actions defined by the process of claim 1 actually be performed. Each of claims 19 and 20 call for “performance of the method” recited claims 1 and 13, respectively, only “when executed by one or more computing devices.” (Instant Application, claims 19 and 20). The process of claims 1 and 13, respectively, is incorporated by reference into claims 19 and 20, as a computer code or instruction of some kind, which need not ever be executed. As such, though claims 19 and 20 refer back to claims 1 and 13 in an apparent dependent form, these claims are independent claims.
If the applicant wishes for these claims to be treated as dependent, applicant is advised to amend the claims, in light of specification support, such that the claims further limit the referenced independent claim.
Status of the Claims
Prior to entry of the amendment(s) and/or consideration of the argument(s), the status of the claims is as follows.
Claim(s) 1-20 is/are pending.
Claims 1, 3-14, and 16-18 are objected to because of informalities.
Claim 7 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite.
Claim(s) 1-11, and 13-20 are rejected under 35 U.S.C. §101 because the claimed invention is directed to an abstract idea without significantly more.
Claim(s) 1-3, 8-15, and 19-20 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Emrey (U.S. Pat. App. Pub. No. 2024/0386207, hereinafter Emrey) with further evidence from Non-Patent Literature to Jurafsky (Daniel Jurafsky and James H. Martin. 2020. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models, 3rd edition Draft. Published: 30 December 2020. <Retrieved on: 14 Nov 2025><URL: https://teaching.bb-ai.net/Student-Projects/Winograd-Challenge-Papers/Jurafsky-Martin-Speech-and-Language-Processing.pdf.>, hereinafter Jurafsky).
Claims 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey as applied to claim 1 above, and further in view of Salloum (U.S. Pat. App. Pub. No. 2019/0065462, hereinafter Salloum).
Claims 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey as applied to claim 1 above, and further in view of Mirhaji (U.S. Pat. No. 8,429,179, hereinafter Mirhaji).
Claims 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey as applied to claim 1 above, and further in view of Kelsey (U.S. Pat. App. Pub. No. 2018/0260472, hereinafter Kelsey).
Claims 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey as applied to claim 1 above, and further in view of Rahman (U.S. Pat. App. Pub. No. 2024/0330661, hereinafter Rahman).
Claims 16-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey as applied to claim 13 and 14 above, and further in view of Solomon (U.S. Pat. App. Pub. No. 2025/0272504, hereinafter Solomon).
Response to Amendments
Applicant’s amendment filed on 09 February 2026 has been entered.
In view of the amendment to the claim(s), the amendment of claim(s) 1-13 and 15-18 and the cancellation of claim(s) 14 have been acknowledged and entered.
In view of the amendment to claim(s) 1, 3-13, and 16-18 and the cancellation of claim(s) 14, the objection to claim(s) 1, 3-14, and 16-18 is withdrawn.
In view of the amendment to claim(s) 7, the rejection of claim(s) 7 under 35 U.S.C. §112 is withdrawn.
In view of the amendment of claim(s) 1-13 and 15-18 and the cancellation of claim(s) 14, the rejection of claims 14 under 35 U.S.C. §101 is withdrawn. The rejection(s) of claim(s) 1-13 and 15-18 under 35 U.S.C. §101 is/are maintained as modified in response to amendment, for the reasons provided in the action below.
In view of the amendment to claim(s) 1-13 and 15-18 and the cancellation of claim(s) 14, the rejection of claims 1-20 under 35 U.S.C. §102 and §103 is withdrawn.
In light of the amended claims, new grounds for rejection under 35 U.S.C. §103 are provided in the action below.
Response to Arguments
Applicant’s arguments regarding the subject matter rejections under 35 U.S.C. §101, see pages 12-13 of the Response to Non-Final Office Action dated 21 November 2025, which was received on 9 February 2026 (hereinafter Response and Office Action, respectively), have been fully considered.
Regarding the rejection of claims 1-11 and 13-20 under 35 U.S.C. §101, applicant asserts that claims 1 and 13, as amended, overcome the rejection under 101 for the following reasons. First, applicant asserts that the claims 1 and 13 “reflect an improvement in the functioning of a computer, or an improvement to other technology or technical field” and that the “specification describes the invention such that the improvement would be apparent to one of ordinary skill in the art.” These arguments are not persuasive.
Regarding Applicant's assertions regarding MPEP 2106.04(d)(1), it is respectfully noted that Applicant’s disclosed consideration of the requirements is incomplete. Regarding claim 1, applicant indicates that the specification at paragraphs [0006], [0024], and [0027], which applicant argues discloses “problems with processing output from a Q&A system that are addressed by the claimed invention.” Applicant argues further that, in light of the specification, it would be apparent to one of ordinary skill in the art that claim 1 “improves the operation of machine learned model scoring by stabilizing scoring through ordered text-normalization transforms, such as generating a claim based on introductory text and each item in a list, and producing machine-processable inputs that an alignment scorer model can consume to produce a reliable output.” Regarding claim 13, applicant indicates that the specification at paragraphs [0006] and [0039] disclose problems with processing output from Q&A and problems in combinations of documents, and that one skilled in the art would that claim 13 “improves the operation of machine learned model scoring by generating a plurality of groupings of a plurality of documents, generating a combination of a grouping of the plurality of documents, and generating a score for each grouping.” Respectfully, the above analysis does not correspond with the analysis required at MPEP 2106.04(d)(1) in light of the decision in Ex Parte Desjardins.
The MPEP indicates two key considerations: that “first the specification should be evaluated to determine if the disclosure provides sufficient details such that one of ordinary skill in the art would recognize the claimed invention as providing an improvement in the functioning of a computer, or an improvement to other technology or a technical field.” MPEP 2106.04(d)(1). The question is not whether there is an indicated problem, but whether there is a disclosed improvement with “sufficient details such that one of ordinary skill in the art would recognize the claimed invention as providing an improvement in the functioning of a computer, or an improvement to other technology or a technical field.” Second, if it is determined that the specification sets forth said improvement with “sufficient detail,” then “the claim must be evaluated to ensure that the claim itself reflects the disclosed improvement, i.e., that is, the claim includes the components or steps of the invention that provide the improvement described in the specification.” (Id.)
In presenting the above argument, it is understood that applicant is relying on the holding in Ex Parte Desjardins. In Ex Parte Desjardins, Director Squires explains the position of the panel, indicating that “the Federal Circuit held that the eligibility determination should turn on whether 'the claims are directed to an improvement to computer functionality versus being directed to an abstract idea'.” (Ex Parte Desjardins, Appeal No. 2024-000567 (PTAB September 26, 2025, Appeals Review Panel Decision) at 8 (citing Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1339 (Fed. Cir. 2016)). Desjardins further explains that “we discern at least the following limitation of independent claim 1 that reflects the improvement: 'adjust the first values of the plurality of parameters to optimize performance of the machine learning model on the second machine learning task while protecting performance of the machine learning model on the first machine learning task'.” which constituted “an improvement to how the machine learning model itself operates, and not, for example, the identified mathematical calculation,” the improvement being to “effectively learn new tasks in succession whilst protecting knowledge about previous tasks.”
As provided for in Desjardins and Enfish, there is a substantial connection between the indicated improvement and the claim language deemed to constitute the improvement, in determining that the claims constitute an improvement to computer functioning. In Enfish, the court relied on “step three of the 'means for configuring' algorithm” in determining that the claims were “directed to a self-referential table for a computer database,” where step three was determined based on claim construction by the District Court to require “For each column [of the table], store information about that column in one or more rows, rendering the table self-referential, the appending, to the logical table, of new columns that are available for immediate use being possible through the creation of new column definition records.” In Desjardins, the Office held that the improvement was directed to the machine learning itself, based on the claim language “adjust the first values of the plurality of parameters to optimize performance of the machine learning model on the second machine learning task while protecting performance of the machine learning model on the first machine learning task”.
Based on Desjardins and Enfish, an improvement in an existing technology can be distinguished based on (1) the claim language disclosing both the existing technology and limitations which are sufficiently specific to the technology, as “bolstered by the specification’s teachings”, (2) which leads to the conclusion that “the claims are directed to an improvement of an existing technology.”
In light of the above, the determination of whether a claim limitation is directed to an improvement in an existing technology turns on (1) the claim language providing sufficient detail such that the technology itself is the target of the limitation, and not the abstract idea (e.g., Desjardins found the target was machine learning based on the express recitation of machine learning in the limitation, Enfish found that the target was a “self-referential table” based on the claims being directed to a “logical table” in light of the claim construction of the district court); and (2) the limitations have a substantial connection with the described benefit to the technology, where the limitations result in the indicated improvement (as understood by the examiner, Desjardins relied on the limitation “adjust the first values of the plurality of parameters... on the second machine learning task while protecting performance… on the first machine learning task” for a “machine learning model” such that the machine learning model can “effectively learn new tasks in succession whilst protecting knowledge about previous tasks” which allows said models to “us[e] less of their storage capacity” and enables “reduced system complexity.”; Enfish distinguished “the claims here that are directed to a specific improvement to computer functionality” (emphasis added) from “patent ineligible claims at issue in other cases” which “recited use of an abstract mathematical formula on any general purpose computer” (emphasis added) based on “the self-referential table recited in the claims” being “directed to a specific implementation of a solution to a problem in the software arts”).
In light of the above, we first address whether the specification discloses an improvement “in the functioning of a computer, or an improvement to other technology or a technical field” with “sufficient details such that one of ordinary skill in the art would recognize the claimed invention as providing… [said] improvement.” Upon review of paragraphs [0006], [0024], [0027], and [0039], the asserted improvement is not apparent in the cited paragraphs. Paragraph [0006] discloses a problem with LLMs hallucinating, specifically indicating the shortcomings of prior art prompting techniques in avoiding those hallucinations, which “underscores the need for metrics that enable the detection of hallucinations and signal how well a response is grounded in the provided information”. This is not a problem with a “Q&A system” generally, but with large language models. It is noted that applicant defines hallucinations with relation to LLMs (see [0004]), and Q&A systems generally are not indicated by the applicant to have this problem outside of LLM-based Q&A systems (see [0005] and [0006]). However, as “the need for metrics” does not provide sufficient detail as to what the solution actually is, this paragraph only discloses a problem, not a solution. Applicant then moves to paragraph [0024], which describes an approach for processing output from a Q&A system. However, this approach is taught away from by the applicant as the described approach “does not respect the semantics of the text, which can potentially lead to sentences that are impossible to correctly evaluate for grounding purposes.” (Instant Application, [0024]). Paragraph [0027] describes a second “naïve approach to identifying claims in a list” which is described as deficient as well. Paragraph [0039] describes a problem regarding “some Q&A systems” having “input length limits.” However, the relationship between the described problem of paragraph [0039], and any specific improvement is unclear. The only information provided by paragraph [0039] is how the prior art is deficient, not how the applicant has succeeded in resolving that deficiency. As such, applicant’s indication of an improvement was not supported by the provided paragraphs.
Upon further review, there are numerous statements of benefit regarding applicant’s “embodiments”, such as paragraphs [0016], [0029], and [0030], which could be considered as an improvement. However, unfortunately, the specification fails to provide a clear connection in the specification between any specific embodiment and the described improvement, such that one skilled in the art would recognize the claimed invention, as described in claims 1 and 13, as resulting in the described improvement.
Assuming, arguendo, that the “disclosure provides sufficient details” that “the claimed invention” provides the above described improvement, we then ask if “the claim itself reflects the disclosed improvement.” Respectfully, it is noted that all statements of benefit or improvement appear to revolve around metrics for reduction of hallucinations in LLMs, more specifically LLM-based Q&A models. As understood in the context of paragraph [0016], the “system and method… for automatically evaluating how well responses … are grounded in a specific set of reference documents” which “generate a set of tangible and interpretable metrics” which “allows for the online evaluation of applications built using LLMs, including but not limited to Q&A systems, chatbots, etc.” (Instant Application, [0016]). It is noted that the recited improvement (“a set of…metrics”) and the disclosed benefit is described as broadly applicable to LLMs, rather than Q&A systems, which would be understood by one having ordinary skill in the art to indicate that the hallucination problem being addressed by the “metrics” is an LLM problem, not a generic “Q&A system” problem. However, claim 1 is neither directed at LLMs and/or LLM-based Q&A models.
Claims 1 and 13, as amended, are directed to text data which output or produced “by a question and answer computer system.” The question of an improvement is necessarily one of a deficiency in the art and the elements disclosed which overcome that deficiency. However, as discussed above, the overarching problem that this application appears to be addressing is not one of Q&A computer systems generally, but of LLMs generally. Applicant does not positively recite that the steps of the method are directed to LLMs until claim 2 which provides the limitation “wherein the question and answer computer system comprises an LLM.” Based on the principles of claim differentiation, claim 1 is understood as being broader than claim 2, such that claim 2 further limits claim 1. As such, the “question and answer computer system” defines a group of possible systems which can be something other than “an LLM”. This is further clarified at paragraph [0020] of the specification which indicates that “Q&A system 120 may comprise one or more LLMs, but embodiments are not limited to an LLM-based approach.” Respectfully, it is further noted that claims 1 and 13, as currently amended, don’t even require the text data to be from a machine learning model, as “question and answer computer system” is not limited to machine learning models in the specification or the claims. The specification does describe machine learning generally, but does not describe a deficiency or an improvement directed to “machine-learned models” generally. Though machine learned models are discussed, these models are described with reference to classification, not generation of the “text data.” (see Instant Application, [0031], [0034], [0036], and [0050]).
As explained above, both Enfish and Desjardins rely on the claim language disclosing both the existing technology and limitations which are sufficiently specific to the technology in concluding that “the claims are directed to an improvement of an existing technology.” In the specification of the present application, the various “embodiments” result in improvements which overcome deficiencies described with respect to a specific technology, “LLMs and/or LLM-based Q&A models.” At a bare minimum, any described improvements for overcoming a deficiency in a specific technology could not be reasonably understood by one having ordinary skill in the art to be reflected by a method which can operate without ever interacting with the specific technology. As such, applicants arguments that claims 1 and 13 reflect an improvement in the functioning of a computer, or an improvement to other technology or technical field which would be recognized by a person having ordinary skill in the art are not persuasive and the rejection under 35 USC 101 is maintained.
Applicant’s arguments regarding the prior art rejections under 35 U.S.C. §102/103, see pages 12-13 of the Response, have been fully considered.
With respect to the rejection(s) of claim(s) 1 under 35 U.S.C. 102(a)(2) as being anticipated by Emrey with further evidence from Jurafsky, applicant asserts that Emrey fails to teach or suggest all limitations of claim 1. Applicant’s arguments are persuasive. As such, the rejections of claim 1 under 35 U.S.C. §102 is withdrawn.
With respect to the rejection(s) of claim(s) 13 under 35 U.S.C. 102(a)(2) as being anticipated by Emrey with further evidence from Jurafsky, applicant asserts that Emrey fails to teach or suggest “generating a plurality of combinations of documents.” Applicant’s arguments are not persuasive.
As presented in the Office Action, Emrey discloses the generation of a plurality of combinations of a plurality of documents (“The ground truth information may include” a collection of “trusted or authoritative documents,” which can result in multiple “data structures 210, each of which can include multiple documents, [0049], [0060], [0066]) for a claim (e.g., for verification of “part of one or more NLG-generated claims in the output data 202”, [0055]). Emrey further explains that “The system may be configured to compare data structure 208 representing generative model output data against …all data points in one or more of the data structures,” which establishes both the existence of multiple data structures 210 and the comparison of data structure 208 {a claim} against the multiple data structures 210. (Emrey, [0067]).
However, as there are additional limitations which require revised mapping, the rejection as currently presented is withdrawn. The rejection as modified in response to the amendments is more fully explained in the rejection below.
Applicant further argues that the rejection(s) of dependent claims 2-12 and 15-18 and independent claims 19 and 20 should be withdrawn for at least the same reasons as independent claims 1 and 13. Applicant’s arguments in light of the amended claims are persuasive. As such, the rejections of claims 2-12 and 15-20 under 35 U.S.C. §102 and 35 U.S.C. §103 are withdrawn.
However, upon further consideration, new ground(s) of rejection under 35 U.S.C. §103 are made in light of combinations of Emrey, Jurafsky, Salloum, Mirhaji, Kelsey, Rahman, and newly cited references Hurst (U.S. Pat. App. Pub. No. 2002/0016796, hereinafter Hurst) and De Foor (U.S. Pat. App. Pub. No. 2025/0005301, hereinafter DeFoor).
The Applicant has not provided any further statement and therefore, the Examiner directs the Applicant to the below rationale.
Claim Rejections - 35 USC § 101
35 U.S.C. §101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 1-11, 13, 15, and 19-20 are rejected under 35 U.S.C. §101 because the claimed invention is directed to an abstract idea without significantly more.
To determine subject matter eligibility for each of the recited claims above, we turn to the subject matter eligibility test, also referred to as the Alice/Mayo test, described in MPEP 2106. Regarding step 1 of the subject matter eligibility test, we first determine if the claims are directed to a statutory category. The independent claim(s) 1, and mutatis mutandis claim(s) 19, recites “identifying text data, wherein the text data was automatically generated by a question and answer computer system; identifying, within the text data, a list of items and introductory text that is associated with the list of items, wherein the list of items comprises multiple items; for each item in the list of items: generating a claim that is based on the introductory text and said each item by appending the introductory text to each item; adding the claim to a set of claims that is associated with the text data; wherein the set of claims comprises multiple claims; for each claim in the set of claims: generating, by a machine learning model, a score that reflects a level of support of said each claim in a set of documents; adding the score to a set of scores for the set of claims; and wherein the set of scores comprises multiple scores; causing, to be presented on a screen of a computing device, data that is based on the set of scores; wherein the method is performed by one or more computing devices”
The independent claim(s) 13, and mutatis mutandis claim(s) 20, recites “identifying text data that was output by a question and answer computer system based on a prompt and a plurality of documents; identifying a set of claims within the text data; generating a plurality of combinations of the plurality of documents, wherein each combination of the plurality of combinations is based on two or more documents in the plurality of documents; for each claim in the set of claims: generating, by a machine learned model, for said each claim, a score that reflects a level of support of said each claim in the combination; wherein generating the score is performed for each combination of the plurality of combinations; wherein the score for said each claim reflects a level of support of said each claim in said each combination; adding the score to a set of scores for said each claim; and causing data to be presented on a screen of a computing device based on the set of scores; wherein the method is performed by one or more computing devices.” As the claims recites a process and an article of manufacture, the claim is directed to one of the statutory categories under step 1 of the subject matter eligibility test.
In Step 2A of the test, which is a Two Prong analysis, we then determine if the claim is directed to a judicial exception. For Step 2A, Prong One, we first ask if the claim recites an abstract idea, Law of Nature, or Natural Phenomena. Regarding claim(s) 1 and 19, the limitations of “identifying…”, “identifying…”, “generating…”, “adding…”, “generating…”, “adding…”, and “causing…” as drafted covers managing personal behavior or relationships or interactions between people, which is a method of organizing human activity. Respectively, regarding claim(s) 13 and 20, the limitations of “identifying…”, “identifying…”, “generating…”, “generating…”, “adding…”, and “causing…” as drafted covers managing personal behavior or relationships or interactions between people, which is a method of organizing human activity. Regarding claims 1, 13, and 19-20, as described with reference to a conversation between a first person and a second person, the first person receives text data from a second person (such as an unprompted statement of fact or in response to a question, and includes one or more asserted facts). The first person then, in attempting to verify the asserted facts, determines one or more specific details in the statement which relate to and define the asserted facts. For example, the text data “Abraham Lincoln was born in a log cabin in Illinois in February of 1809” results in a list of items based on a native speaker of a language understanding the statement made. In this example, the list of specific details includes Lincoln’s birthdate (“Abraham Lincoln was born… in February of 1809”) and his place of birth (“Abraham Lincoln was born in a log cabin in Illinois…”). Each of the asserted facts are interpreted by the first person as a factual claim from the second person (e.g., the first claim “Abraham Lincoln was born in February of 1809” and the second claim “Abraham Lincoln was born in a log cabin in Illinois.” As this step is generally performed as a mental activity by the first person and the second person, as participants in a conversation, the generation of a list and the determination of claims can be performed as multiple steps or as a single combined step, each of which can be written as a list (such as in the case of more complex statements). The first person can then review reliable documents to verify the first claim (e.g., According to Textbook 1, the first claim is completely accurate, as Abraham Lincoln was verifiably born on February 12, 1809) and the second claim (e.g., According to Primary Literature 1, the first claim is only partially accurate, as Abraham Lincoln was verifiably born in a log cabin, but the log cabin was in Kentucky, not Illinois) made by the second person. Further, it is well known to indicate the relative correctness of a factual assertion in a conversation using percentages (e.g., the first claim would be determined to be 100% accurate and the second claim might be considered to be 75% accurate, as Lincoln was born in a log cabin and later lived in Illinois, even though he was not born there). The first person can then respond to the second person based on the assessment (e.g., “The first part of that statement is right, but the second part is not accurate. He was born in a log cabin, but that was in Kentucky. He practiced law in Illinois, though.”), where the response can be provided in any number of ways based on types of conversation (e.g., email), convenience, or personal preference. Further, as indicated with respect to claim 13, data may also be grouped in a variety of ways, such as based on the number of documents in the group, content, or the like, as desired for digestibility of the information. The use of a threshold provides no further distinction over the above described human activity, beyond adding a clerical task of partitioning data based on relationship to the claims made. Therefore, the claims are directed to human activity, and, thus, directed to an abstract idea which is a judicial exception.
In Step 2A, Prong Two of the analysis, we next determine if the claim recites additional elements which integrate the judicial exception into a practical application. The judicial exception recited in claims 1, 13, and 19-20 is not integrated into a practical application. In particular, claim(s) 1, 13, and 19-20 recite additional elements of “identifying… a list of items and introductory text”, “for each item…generating a claim”, “generating a combination of the plurality of documents…”, “generating a score…”, “to be presented on a screen,” a “computing device,” and “non-transitory storage media,” as per the independent claims. The “identifying… a list of items and introductory text” is insignificant pre-solution activity. It merely describes the generic receipt of text data, which is a necessary antecedent for any computer-based information processing. The inclusion of “output by a question and answer computer system based on a prompt and a plurality of documents” merely recites a specific environment upon which the pre-solution activity is performed and provides not relevant further limitation of the additional element. The “for each item… generating a claim” is a generic computer implementation of the human activity. The “generating a claim” is a generic description of a mental process which is naturally and instantaneously performed by each participant in a conversation. Though the generic description is intended for implementation in a computer, there are no provisions for actually generating a claim using the computer components or further explanations for how a generated claim might differ from standard human interpretation of language. The “generating a combination of the plurality of documents” the further human activity of verifying the source facts through corroboration, which is related to the dialogue verification. Applicant has merely indicated specific steps which are taken by a human to corroborate facts in the process of factual verification. The generation of a score is the heart of the abstract idea itself, the determination based on verifiable facts, whether asserted facts are accurate or not. The claim does not specify what the score is beyond the general relationship between “the score” and “reflect[ing] a level of support”, or how the score is determined such that the score amounts to more than a numerical reflection of a human decision on verifiability/accuracy. The use of a screen, recited as “to be presented on a screen” is a recitation of the typical operation of a computer screen, where the screen, as the visual portion of the described computing device, performs “predictab[le]” functions based on “vague, functional descriptions”, such that the screen “simply provides the environment in which the abstract idea… is carried out.” (In re TLI Communications LLC Patent Litigation, 823 F.3d 607, 613-14 (Fed. Cir. 2016) quoting Alice, 134 S.Ct. at 2359, quoting Mayo, 132 S.Ct. at 1294)). As such, the presentation on a screen does not meaningfully contribute to a practical application of the abstract idea. Regarding the computer components individually, these are general-purpose computer components with no provisions for the practical application of the abstract idea. The “screen,” the “computing device,” and the “non-transitory storage media,” is/are not meaningfully integrated into the practical application of the abstract ideas recited in claim(s) 1, 13, and 19-20. Accordingly, the additional elements of the claims fail to integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
Regarding Step 2B of the analysis, we next determine if the claim recites additional elements which amount to substantially more than the judicial exception. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of using a “screen,” the “computing device,” and the “non-transitory storage media,” to perform the described dialogue receipt and verification amounts to no more than mere instructions to apply the exception using generic computer components. Mere instructions to apply an exception using a generic computing device or general purpose computer component cannot provide an inventive concept. (See Alice Corp. v. CLS Bank, 573 U.S. 208, 221, 223, 110 USPQ2d 1976, 1982-84 (2014) (quoting Mayo Collaborative Servs. V. Prometheus Labs., Inc., 566 U.S. 66, 72, 101 USPQ2d 1961, 1965). As well, the remaining claim limitations are well-known, routine, and conventional such as to not qualify as an inventive concept. Specifically, transmission of data over a network is well known, as is evidenced by OIP Techs, Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015). Further, “merely identifying a [graphic] user interface,” or more generally a screen, with relation to determinations which “can be performed in the human mind or using a pencil and paper” has been deemed insufficient to render otherwise abstract claims as non-abstract, where the court further indicated that the resulting device, system and/or method was “still missing” an “improved structure or function.” See Broadband iTV, Inc. v. Amazon.com, Inc., 113 F.4th 1359, 1367-68 (Fed. Cir. 2024). The court has consistently held that “[s]teps that do nothing more than spell out what it means to ‘apply it on a computer’ cannot confer patent-eligibility.” Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1371-72 (Fed.Cir. 2015)(citing Alice, 134 S.Ct. at 2359 (warning against a § 101 analysis that turns on the draftsman's art (citing Parker v. Flook, 437 U.S. 584, 593, 98 S.Ct. 2522, 57 L.Ed.2d 451 (1978))).
Further, the above discussion expressly considering the changes to 2106.04(d)(1) in light of the decision in Ex Parte Desjardins are incorporated here by reference without further recitation.
Therefore, and in light of the preceding analysis, the claims do not amount to significantly more than the judicial exception. For these reasons, claims 1, 13, and 19-20 are not patent eligible.
With respect to claim(s) 2, the claims relate to the use of a LLM as part of the question answer system. As performed by a person, these steps appear to refer to the further limitation of the above described human activity to being performed by a well-known generic computer implementation. No additional limitation is present.
With respect to claim(s) 3, the claims relate to the preprocessing of pronouns to their associated named entity. As performed by a person, these steps appear to refer to the mental process of understanding the meaning of a pronoun based on context. No additional limitation is present.
With respect to claim(s) 4, the claims relate to the preprocessing of a numbered list, including the removal of reference numerals. As performed by a person, these steps appear to refer to the mental process of recognizing important information in the conversation (e.g., If the second speaker presents a list with numerical organization, such as “Abraham Lincoln was born (1) in a log cabin (2) in Illinois (3) in February of 1809.”, it is understood that the numerical order doesn’t change the asserted facts and can be ignored for comprehension purposes). No additional limitation is present.
With respect to claim(s) 5, the claims relate to the preprocessing of a flattened list, which is a list which is implicitly incorporated into one or more sentences, based on the presence of punctuation or appropriate grammar elements. As performed by a person, these steps appear to refer to the mental process of recognizing listed items without an explicit list format based on language structure. No additional limitation is present.
With respect to claim(s) 6, the claims relate to the removal of sentences which have limited content value. As performed by a person, these steps appear to refer to the mental process of remembering information in a conversation based on the level of usefulness of the information. No additional limitation is present.
With respect to claim(s) 7, the claim is believed to relate to detection of a reference to a person in the second person point of view and the clarification of ambiguities related to the reference. As performed by a person, these steps appear to refer to the mental process of applying context from previous portions of dialogue to disambiguate a pronoun in the context of the question answer scheme. No additional limitation is present.
With respect to claim(s) 8, the claim relates to associating the scores with a label. As performed by a person, these steps appear to refer to the mental process of applying context from previous portions of dialogue to disambiguate a pronoun in the context of the question answer scheme. No additional limitation is present.
With respect to claim(s) 9, the claim relates to the determination of mathematical thresholds for determining reliability of a factual claim. As performed by a person, these steps appear to refer to the mathematical process of determining a threshold for the comparison score. No additional limitation is present.
With respect to claim(s) 10, the claim relates to the determination of average scores for the scores given to the factual claims. As performed by a person, these steps appear to refer to the mathematical process of calculating an average score for the related comparison scores. No additional limitation is present.
With respect to claim(s) 11, the claim relates to the determination of an associated label for a minimum value of the scores. As performed by a person, these steps appear to refer to the mental and clerical process of labeling claims based on their level of verifiability. No additional limitation is present.
With respect to claim(s) 15, the claim relates to limiting the organization of documents to being less than the total number of documents available. As performed by a person, these steps appear to refer to the mental process of limiting the documents to the most probative available documents. No additional limitation is present.
With respect to claim(s) 18, the claim relates to reviewing the document set based on the size of the set overall and moving forward with the analysis if the set is within a predefined threshold. As performed by a person, these steps appear to refer to the mental process of determining that a set of documents is too large and reducing the set prior to comparing with the factual claims. No additional limitation is present.
These claims further do not remedy the judicial exception being integrated into a practical application and further fail to include additional elements that are sufficient to amount to significantly more than the judicial exception. As such, for the same reasons as described above with reference to independent claim(s) 1, 13, and 19-20, dependent claim(s) 2-11, 15, and 18 are not patent eligible.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-12 and 19 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Regarding claim 1, and mutatis mutandis claim 19, the limitation “generating a claim that is based on the introductory text and said each item by appending the introductory text to each item” is not supported by the application as filed. Claim 1 recites “generating a claim that is based on the introductory text and said each item by appending the introductory text to each item” at lines 7-8. However, “appending the introductory text to each item” is not supported by the specification as filed.
The BRI of the word “append,” based on the ordinary meaning of the word, and as understood in the context of the specification, is to add or attach something to the very end of a piece of writing without replacing existing content. However, the specification as filed supports the exact opposite of this. See for example, paragraph [0029], which provides an explanation of the asserted process. Regarding the example introductory text “To qualify for the certificate, students must:” and the claim “submit a Certificate Form of Intent” as derived from the list at paragraph [0028], the specification explains that, “when constructing sentences, the colon in the introduction is removed, along with the bullets that precede the items.” Thus, paragraph [0029] explains that the result of the appending is “To qualify for the certificate, students must submit a Certificate Form of Intent.” However, in the context of the claim language as currently organized, this would be prepending, not appending. As currently drafted, the above example would be reversed, as follows: “submit a Certificate Form of Intent To qualify for the certificate, students must.” Such an embodiment is not disclosed in the specification as filed, in such a way that one skilled in the art would understand that the inventors had possession of the claimed invention.
Of note, as part of the Response, applicant amended paragraph [0028] to address a clear error related to this embodiment, by changing the word “prepended” in “the item is also prepended to the portion of the sentence that introduces the list” to “appended.” This amendment was not objected to, as, in light of paragraphs [0028] and [0029], the use of “prepended” was understood as clear error.
The current claim language of claim 1 is believed to be the result of a clerical error, given the relationship to the contemporaneous amendment. As such, the following proposed amendment, if accepted, would overcome the rejection of claim 1 under 35 USC 112(a): amend “…by appending the introductory text to each item” to “…by prepending the introductory text to each item”
Regarding claims 2-12, claims 2-12 depend from claim 1 and incorporate all limitations therein. Therefore, claims 2-12 are rejected for at least the same reasons as claim 1.
Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 13, 15, and 20 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Emrey.
Regarding claim 13, Emrey discloses A method comprising (Systems and methods described with reference to “detecting errors and hallucinations in generative model output data”; Emrey, ¶ [0046]): identifying text data that was output by a question and answer computer system (“One or more processors of fact extraction stage 206 may be configured to receive inputs such as generative model output data 202 and ground truth information 204” where the “outputs” are generated by question-answering models, including LLMs; Emrey, ¶ [0049], [0050]) based on a prompt and a plurality of documents (the output of the LLM is in response to a user input, and “ground truth information” which “may (or may not) be an input to the generative model” and “the ground truth information may include one or more (e.g., a collection of)…trusted or authoritative documents... factual information and/or information based on opinions... [and] “desired” truth representative of the user-curated set of documents”; Emrey, ¶ [0033]-[0034]); identifying a set of claims within the text data (“The one or more processors at fact extraction stage 206 may use the pre-processed data (e.g., generative model output data 202 and/or ground truth information 204) to identify factual information and/or assertations (e.g., NLG-generated claims)” which may be applied to “identify or extract entities from the output data that form part of one or more NLG-generated claims in the output data 202,” which may be either viewed as a single claim (e.g., one”) or a group (e.g., “or more”), which further, “may constitute all or part of data structure 208 representing generative model output data.”; Emrey, ¶ [0049], [0055], [0060]); generating a plurality of combinations of the plurality of documents, (Discloses “generat[ing]… data structure 210 representing fact data” which can be “a collection of documents (e.g. related documents). {a plurality of documents}” and “pairs of data structures 208 and 210” indicates a plurality of data structures 210 {a plurality of combinations}. It is noted that numerous portions of Emrey describe multiple data structures 210 (see also “one or more comparisons (e.g., as described below) using existing data structures 210” at [0062]); Emrey, ¶ [0049], [0060], [0062], [0066]) wherein each combination of the plurality of combinations is based on two or more documents in the plurality of documents (the generation of data structure 210 can be based on “a collection of documents” {based on two or more documents} which can be taken from the broad set of all documents which qualify as “ground truth documents”; Emrey, ¶ [0060], [0062]); for each claim in the set of claims: generating, by a machine learned model, for said each claim, a score that reflects a level of support of said each claim in the combination (“The data structure 208 representing generative model output data” may be provided to “the fact comparison stage 212 configured to compare and assess whether the data structure 208 representing generative model output data comprises hallucinations and/or errors” where the fact comparison stage can “compare data structure 208 representing generative model output data against data structure 210 representing fact data (or vice versa)” and assign “a comparison score based on the comparison to data structure 210 representing fact data” where the comparison score can represents the level of similarity or dissimilarity from the ground truth data. Further “The system may be configured to compare data structure 208 representing generative model output data against …all data points in one or more of the data structures,” and “the data structure 208 representing generative model output data may be compared to the data structure 210 representing fact data” based on both “one data point from the data structure 208 representing generative model output data compared to more than one data point from the data structure 210 representing fact data” and “multiple data points from a first one of the data structures may each be compared to multiple data points from the second one of the data structures” which is understood to result in a score for each comparison, and each claim in data structure 208 being compared to each data point in each of the data structures 210.; Emrey, ¶ [0063], [0067]-[0069]) wherein generating the score is performed for each combination of the plurality of combinations (Specifically discloses “Detecting the degree of similarity between information in each of the data structures may comprise determining differences between data points (e.g., data values, nodes, etc.) in the data structures.” As such, the comparison is understood as being performed against all relevant data individually (i.e., “each of the data structures...”); Emrey, ¶ [0066]); wherein the score for said each claim reflects a level of support of said each claim in said each combination (As stated above the “the degree of similarity” is determined for “information in each of the data structures”, where the individual degree of similarity for the “pairs of data structures 208 and 210” as compared in aa one to one, one to many and/or a many to many manner, reflects “whether generative model output data 202 is substantiated by ground truth information 204 {a level of support}.”; Emrey, ¶ [0066]-[0067]); adding the score to a set of scores for said each claim (The system assigns “a comparison score based on the comparison” each of the data points of the number of “data structure 210 representing fact data,” thus generating a set of scores for each of the data points.; Emrey, ¶ [0068]-[0069]); and causing data to be presented on a screen of a computing device based on the set of scores (The system can “distinguish whether one or more data points of data structure 208 representing generative model output data is very similar, moderately similar, moderately dissimilar, or very dissimilar from the data structure 210 representing fact data” and “Based on the comparison” the system “may generate an output 214 indicative of identified hallucinations and/or errors” which “may be provided in a machine-readable and/or user-readable (e.g., user-friendly) format.”; Emrey, ¶ [0069], [0072]); wherein the method is performed by one or more computing devices (“Software 450, which can be stored in storage 440 and executed by processor 410, can include, for example, the programming that embodies the functionality of the present disclosure (e.g., as embodied in the systems, computers, servers, and/or devices as described above)” and “can also be stored and/or transported within any computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described above, that can fetch and execute instructions associated with the software from the instruction execution system, apparatus, or device”; Emrey, ¶ [0086]-[0087]).
Regarding claim 15, Emrey discloses wherein each combination in the plurality of combinations comprises less than all of the plurality of documents (Though not expressly described as a determination that each combination comprises less than all of the documents which comprise the “ground truth information”, Discloses determining whether an “existing data structure 210” is “suitable for verification of generative model output data” and that such an existing data sturcture may have “insufficient information... to verify or refute a purported fact from the generative model output data.” As each of these data structures are generated from the ground truth information and may contain information which is not suitable for any specific “generative model output data”, the combinations of documents throughout all data structures 210” as generated for any specific data structure 208 necessarily contain less than the entirety of available ground truth information (i.e., only documents which are “suitable for verification” are included, and documents which are not suitable for verification are acknowledged to exist); Emrey, ¶ [0062]).
Regarding claim 20, Emrey discloses One or more non-transitory storage media storing instructions (“Software 450, which can be stored in storage 440 and executed by processor 410, can include, for example, the programming that embodies the functionality of the present disclosure (e.g., as embodied in the systems, computers, servers, and/or devices as described above)” and “can also be stored and/or transported within any computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described above, that can fetch and execute instructions associated with the software from the instruction execution system, apparatus, or device”; Emrey, ¶ [0086]-[0087]) which, when executed by one or more computing devices, cause performance of the method recited in Claim 13 (See mapping of limitations presented with reference to claim 13; Emrey, ¶ (see above)).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 8-12, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey in view of Hurst, with further support from Jurafsky.
Regarding claim 1, Emrey discloses A method comprising (Systems and methods described with reference to “detecting errors and hallucinations in generative model output data”; Emrey, ¶ [0046]): identifying text data, wherein the text data was automatically generated by a question and answer computer system (“One or more processors of fact extraction stage 206 may be configured to receive inputs such as generative model output data 202 and ground truth information 204.”; Emrey, ¶ [0049]); identifying, within the text data, a list of items and introductory text that is associated with the list of items… (Though not expressly recited in the context of a list of items, the “Fact extraction stage 206 may comprise one or more natural language processing (NLP) models configured to extract entities, relationships between entities, and constraints defining the semantic context of the generative model output data 202” and exemplary models which may be incorporated, can include “named entity recognition models, coreference resolution models, and/or syntactic/semantic parsing models,” where the syntactic/semantic parsing model performs semantic parsing to “to determine the meaning of an input (e.g., generative model output data 202), but with particular attention paid to the arrangement of words and the grammar in the input,” including lists, and performs syntactic parsing to analyze the grammar and structure of the sentence, where the outputs of all models in the fact extraction stage result in a “collection of outputs” which “may constitute all or part of data structure 208 representing generative model output data”. Together, these components generate the individual components of the text chunk, which represent the constituent factual assertions of the text chunk. It is further noted that the parsing of a list of items based on main clauses (syntactic) for a predicate argument structure (semantic) and subordinate clauses (syntactic) as corresponding arguments (semantic) is an implicit and native capability which is a fundamental function of the syntactic/semantic parsers, as known in the relevant art. See, for example, Jurafski at page 243, explaining the handing of conjunctions (lists) as a fundamental part of grammar and parsing, and page 390 which discusses semantic role labeling, as part of text normalization, which “is the task of assigning semantic role labels to the constituents of a sentence” where “Semantic roles are abstract models of the role an argument plays in the event described by the predicate.”; Emrey, ¶ [0053]); for each item in the list of items: generating a claim that is based on the introductory text and said each item… (“The one or more processors at fact extraction stage 206 may use the pre-processed data (e.g., generative model output data 202 and/or ground truth information 204) to identify factual information and/or assertations (e.g., NLG-generated claims)” which may be applied to “identify or extract entities from the output data that form part of one or more NLG-generated claims in the output data 202” where “A given data structure 210 representing fact data may characterize... a collection of facts (e.g., related facts)”; Emrey, ¶ [0055]); adding the claim to a set of claims that is associated with the text data; wherein the set of claims comprises multiple claims (“The one or more processors of fact extraction stage 206 may be configured to not only detect and extract entities from inputs, but also optionally to categorize the extracted entities,” where the extracted entities from inputs are associated with the same input, and the extracted entities {set of entities} is pluralized, thus corresponding to multiple claims.; Emrey, ¶ [0056]);for each claim in the set of claims: generating, by a machine learning model, a score that reflects a level of support of said each claim in a set of documents (“The data structure 208 representing generative model output data” may be provided to “the fact comparison stage 212 configured to compare and assess whether the data structure 208 representing generative model output data comprises hallucinations and/or errors” where the fact comparison stage can “compare data structure 208 representing generative model output data against data structure 210 representing fact data (or vice versa)” and assign “a comparison score based on the comparison to data structure 210 representing fact data” where the comparison score can represents the level of similarity or dissimilarity from the ground truth data, and the “fact comparison stage 212 may comprise one or more natural language understanding (NLU) models” examples of which include “BERT, RoBERTa, GPT-n, T5, transformer, and their derivatives {by a machine learning model}”; Emrey, ¶ [0055], [0063], [0066]-[0069]); adding the score to a set of scores for the set of claims; and wherein the set of scores comprises multiple scores (The system assigns “a comparison score based on the comparison” each of the data points of the “data structure 210 representing fact data,” thus generating a set of scores for each of the data points, and where in the context of multiple data points “individual data points may be assigned a score {comprises multiple scores}”; Emrey, ¶ [0068]-[0069]); causing, to be presented on a screen of a computing device, data that is based on the set of scores (The system can “distinguish whether one or more data points of data structure 208 representing generative model output data is very similar, moderately similar, moderately dissimilar, or very dissimilar from the data structure 210 representing fact data” and “Based on the comparison” the system “may generate an output 214 indicative of identified hallucinations and/or errors” which “may be provided in a machine-readable and/or user-readable (e.g., user-friendly) format.”; Emrey, ¶ [0069], [0072]); wherein the method is performed by one or more computing devices (“Software 450, which can be stored in storage 440 and executed by processor 410, can include, for example, the programming that embodies the functionality of the present disclosure (e.g., as embodied in the systems, computers, servers, and/or devices as described above)” and “can also be stored and/or transported within any computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described above, that can fetch and execute instructions associated with the software from the instruction execution system, apparatus, or device”; Emrey, ¶ [0086]-[0087]). However, Emrey fails to expressly recite wherein the list of items comprises multiple items, [and] appending the introductory text to each item.
Hurst teaches systems and methods for “extracting a meaningful text block from a document” which includes “itemized lists” such as “elliptical lists” as “preprocessing for text mining processing and natural language processing.” (Hurst, ¶ [0001], [0006], [0105]). Regarding claim 1, Hurst teaches identifying, within the text data, a list of items and introductory text that is associated with the list of items, wherein the list of items comprises multiple items (Discloses the processing of a list, such as “an elliptical list or table” where the system uses “concepts regarding a graph” such that “each node corresponds to a text block, while each arc corresponds to a link (i.e., connection relation) between text blocks” where “a link is formed” from a source text block {introductory text} “to all text blocks that exist in the next line of the text block t” also referred to as “sink nodes {a list of items}.” Further, as “text blocks” is pluralized, text blocks is understood to include multiple items. As this is described as applicable to a list as separated from a table, it is recognized that the list need not be in a table format (i.e., text blocks may refer simply to portions of text as derived from the list); Hurst, ¶ [0059]-[0060], [0089], [0105]); for each item in the list of items: generating a claim that is based on the introductory text and said each item by appending the introductory text to each item (“all text blocks in T′ that exist on the left side of t are assigned to a variable L (step 105). Then, a link is generated between t and each text block that composes L (step 106). That is, a link is formed to all the text blocks that exist in the next line of the text block t (if the next line is a blank line, the further next line) and are located on the left of t,” where “a document is clustered” based on a retrieved graph, the graph corresponding to the “set of text blocks combined by a link” where the “the validity of links included in each sub-cluster s is evaluated using a language model” and “if the evaluation result is reasonable, the text blocks of the source and sink of the link are merged”, where, as understood in the context of an “elliptical list,” a confirmed reasonable merged source and sink, is a generated claim (i.e., merged link) that is based on the source text block {introductory text} and performed for each sink text block {said each item}, by “appending” the source text block {introductory text} to each sink text block {each item} (e.g., as shown in the example of “a link between ‘number’ and ‘of’... merged to generate a new text block ‘number of’”).; Hurst, ¶ [0089], [0095], [0097], [0100], [0105]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the error and hallucination detection systems of Emrey to incorporate the teachings of Hurst to include identifying, within the text data, a list of items and introductory text that is associated with the list of items, wherein the list of items comprises multiple items, [and] for each item in the list of items: generating a claim that is based on the introductory text and said each item by appending the introductory text to each item. Emrey discloses a response verification system to prevent hallucinations, the system including a fact extraction stage with “syntactic/semantic parsing models.” However, Emrey is silent regarding the parsing of elliptical lists. Hurst discloses preprocessing techniques for “extracting a meaningful text block from a document where a table, an itemized list, a multiple column, etc., are arbitrarily laid out” including generating “a connection candidate between objects” as a link, determining the “validity of each link… using a language model” and “then objects are connected [appended] if it is determined that the connection candidate (link) is valid.” A person having ordinary skill in the art would be motivated to combine the hallucination prevention system of Emrey with the text preprocessing of Hurst, such that important information “included in… tables and lists” can be extracted “in such a form that is applicable to later semantics analysis,” rather than abandoning this information as previously done in the prior art, as disclosed by Hurst. (Hurst, ¶ [0008]).
Regarding claim 2, the rejection of claim 1 is incorporated. Emrey and Hurst disclose all of the elements of the current invention as stated above. Emrey further discloses wherein the text data was generated by a question and answer computer system (“Generative model output data 202 may comprise outputs produced by a generative model” including “question-answering models” where generative models includes LLMs; Emrey, ¶ [0029], [0050]).
Regarding claim 3, the rejection of claim 1 is incorporated. Emrey and Hurst disclose all of the elements of the current invention as stated above. Emrey further discloses further comprising: identifying, within the text data, a pronoun (“The coreference resolution models of fact extraction stage 206 may generate a data structure output that comprises an indication of various pronouns or other linguistic expressions in the generative model output data 202 related to a given extracted entity.”; Emrey, ¶ [0058]); identifying, within the text data, one or more nouns upon which the pronoun is based (The pronoun is identified with relation to named entity, as derived from “the output of one or more NER models that have identified entities” which, in the context of a pronoun, the recognized named entity is a noun.; Emrey, ¶ [0058]); and prior to dividing the text data into sentences, replacing the pronoun with the one or more nouns (The coreference resolution model replaces the pronoun with the noun in the generated “data structure output” which occurs prior to parsing the text data; Emrey, ¶ [0053], [0058]-[0059]).
Regarding claim 8, the rejection of claim 1 is incorporated. Emrey and Hurst disclose all of the elements of the current invention as stated above. Emrey further discloses further comprising: for each score of the set of scores: mapping said each score to a range of values from among a plurality of ranges of values (“system 100 may be configured to identify information including NLG-generated claims in generative model output data and compare these NLG-generated claims to ground truth information to determine whether the NLG-generated claims are grounded in trusted factual information” where “the one or more processors of fact comparison stage 212 may utilize one or more thresholds to distinguish whether one or more data points of data structure 208 representing generative model output data is very similar, moderately similar, moderately dissimilar, or very dissimilar from the data structure 210” As such, the scores for each claim are mapped based on the threshold to at least five sets of scores, which correspond to the above labels.”; Emrey, ¶ [0037], [0069]); identifying a label that is associated with the range of values (“Exemplary thresholds may be about 75% or greater similarity for data points which are very similar, about 50-75% similarity for data points which are moderately similar, about 25-50% similarity for data points which are moderately dissimilar, and about 0-25% similarity for data points which are very dissimilar.”; Emrey, ¶ [0069]); and assigning the label to the claim that corresponds to said each score, (“The overall comparison score may be measured against one or more thresholds to determine whether the data structure 208 representing generative model output data meets or exceeds a standard for similarity (or dissimilarity) relative to data structure 210 representing fact data.”; Emrey, ¶ [0069]) wherein the data that is based on the set of scores is also based on the label of each claim in the set of claims (The overall comparison score can be cumulative of the individual comparison scores, and “the one or more processors of fact comparison stage 212 may generate an output 214 indicative of identified hallucinations and/or errors” which “may be provided in a... user-readable (e.g., user-friendly) format,” where the overall comparison score, the individual comparison scores, and/or the various threshold determinations of similarity, are all understood as “indicative of identified hallucinations and/or errors”.; Emrey, ¶ [0068]-[0069], [0072]).
Regarding claim 9, the rejection of claim 1 is incorporated. Emrey and Hurst disclose all of the elements of the current invention as stated above. Emrey further discloses further comprising: based on the text data, identifying a plurality of claims that includes the set of claims (“Data points within data structure 208 representing generative model output data (e.g., representative of utterances, individual entities, etc.) may be assigned a comparison score based on the comparison to data structure 210 representing fact data.”; Emrey, ¶ [0068]); wherein each claim in the plurality of claims is associated with a different score of a plurality of scores that includes the set of scores (Each claim in the plurality of claims “Data points within data structure 208... individual entities... may be assigned a comparison score” where the comparison score is assigned individually.; Emrey, ¶ [0068]); identifying a minimum score from the plurality of scores (“a threshold of similarity vs. dissimilarity may be dichotomous, e.g., scores greater than or equal to about 50% may indicate similarity, whereas scores below about 50% may indicate dissimilarity.”; Emrey, ¶ [0068]); and assigning, to the text data, the minimum score as a grounding score (In a dichotomy, the threshold is the minimum score and the grounding score.; Emrey, ¶ [0068]).
Regarding claim 10, the rejection of claim 1 is incorporated. Emrey and Hurst disclose all of the elements of the current invention as stated above. Emrey further discloses further comprising: based on the text data, identifying a plurality of claims that includes the set of claims (“Data points within data structure 208 representing generative model output data (e.g., representative of utterances, individual entities, etc.) may be assigned a comparison score based on the comparison to data structure 210 representing fact data.”; Emrey, ¶ [0068]); wherein each claim in the plurality of claims is associated with a different score of a plurality of scores that includes the set of scores (Each claim in the plurality of claims “Data points within data structure 208... individual entities... may be assigned a comparison score” where the comparison score is assigned individually.; Emrey, ¶ [0068]); computing a mean score from the plurality of scores (“individual data points may be assigned a score that can be compiled (e.g., using a weighted or unweighted sum) with that of related data points to determine an overall comparison score for a given portion of data structure 208”; Emrey, ¶ [0068]); and assigning, to the text data, the mean score as a grounding score (“an overall comparison score for a given portion of data structure 208 representing generative model output data as compared to data structure 210 representing fact data.”; Emrey, ¶ [0068]).
Regarding claim 11, the rejection of claim 1 is incorporated. Emrey and Hurst disclose all of the elements of the current invention as stated above. Emrey further discloses further comprising: based on the text data, identifying a plurality of claims that includes the set of claims (“Data points within data structure 208 representing generative model output data (e.g., representative of utterances, individual entities, etc.) may be assigned a comparison score based on the comparison to data structure 210 representing fact data.”; Emrey, ¶ [0068]); wherein each claim in the plurality of claims is associated with a different score of a plurality of scores that includes the set of scores (Each claim in the plurality of claims “Data points within data structure 208... individual entities... may be assigned a comparison score” where the comparison score is assigned individually.; Emrey, ¶ [0068]); identifying a minimum score from the plurality of scores (“a threshold of similarity vs. dissimilarity may be dichotomous, e.g., scores greater than or equal to about 50% may indicate similarity, whereas scores below about 50% may indicate dissimilarity,” where In a dichotomy, the threshold is the minimum score and the grounding score.; Emrey, ¶ [0068]); mapping minimum score to a range of values from among a plurality of ranges of values (The minimum score is mapped to the range of values from 0-50%, which is from the two possible values in this example, and two constitutes a plurality.; Emrey, ¶ [0068]); identifying a label that is associated with the range of values (The label associated with the range of values is indicated “dissimilarity,” also referred to as ungrounded.; Emrey, ¶ [0068]); and assigning, to the text data, the label as a grounding label (“The overall comparison score may be measured against one or more thresholds to determine whether the data structure 208 representing generative model output data meets or exceeds a standard for similarity (or dissimilarity) relative to data structure 210 representing fact data.”, thus assigning both the value and the label to the text data.; Emrey, ¶ [0069]).
Regarding claim 12, the rejection of claim 1 is incorporated. Emrey and Hurst disclose all of the elements of the current invention as stated above. Emrey further discloses further comprising: based on the text data, identifying a plurality of claims that includes the set of claims (“Data points within data structure 208 representing generative model output data (e.g., representative of utterances, individual entities, etc.) may be assigned a comparison score based on the comparison to data structure 210 representing fact data.”; Emrey, ¶ [0068]); wherein each claim in the plurality of claims is associated with a different score of a plurality of scores that includes the set of scores (Each claim in the plurality of claims “Data points within data structure 208... individual entities... may be assigned a comparison score” where the comparison score is assigned individually.; Emrey, ¶ [0068]); for each score of the plurality of scores: mapping said each score to a range of values from among a plurality of ranges of values (“system 100 may be configured to identify information including NLG-generated claims in generative model output data and compare these NLG-generated claims to ground truth information to determine whether the NLG-generated claims are grounded in trusted factual information” where “the one or more processors of fact comparison stage 212 may utilize one or more thresholds to distinguish whether one or more data points of data structure 208 representing generative model output data is very similar, moderately similar, moderately dissimilar, or very dissimilar from the data structure 210” As such, the scores for each claim are mapped based on the threshold to at least five sets of scores, which correspond to the above labels.”; Emrey, ¶ [0037], [0069]); identifying a label that is associated with the range of values (“Exemplary thresholds may be about 75% or greater similarity for data points which are very similar, about 50-75% similarity for data points which are moderately similar, about 25-50% similarity for data points which are moderately dissimilar, and about 0-25% similarity for data points which are very dissimilar.”; Emrey, ¶ [0069]); assigning the label to the claim that corresponds to said each score (“The overall comparison score may be measured against one or more thresholds to determine whether the data structure 208 representing generative model output data meets or exceeds a standard for similarity (or dissimilarity) relative to data structure 210 representing fact data.”; Emrey, ¶ [0069]); including the label in a set of labels (For each of the thresholds and respective labels, “about 75% or greater similarity” for “very similar,” “about 50-75% similarity” for “moderately similar,” “about 25-50% similarity” for “moderately dissimilar,” and “about 0-25% similarity” for “very dissimilar,” these can be further classified in the described dichotomy, where “scores greater than or equal to about 50% may indicate similarity, whereas scores below about 50% may indicate dissimilarity,” thus each label is included in a set of labels; Emrey, ¶ [0069]); determining a number of labels, in the set of labels, that indicate that the claim that corresponds to the label is grounded (In the above labels sets, very similar and moderately similar labels for the claims, which is two labels, correspond to an indication of similarity which corresponds to grounded.; Emrey, ¶ [0069]); based on the number of labels, determining a ratio of the number of labels to a particular number of labels that are in the set of labels (Though not disclosed as a ratio of the number of labels in the set of labels that indicates that the claim that corresponds to the label is grounded. However, such a ratio implicitly exists in Emrey. It is further noted that the overall score, in one example, is described as a dichotomy. Further, individual claims, each of which have individual labels and can each correspond to 4 different labels each with an associated percentage of similarity required for said label. as such, there are a range of ratios for each of the 4 different labels, based on the number of individual claims, which will result in being above the dichotomous threshold for grounding or below it.; Emrey, ¶ [0068], [0069]); and assigning, to the text data, the ratio as a grounding score (Said ratios, as described above for the combination of individual claims and overall claim grounding, can be assigned as a grounding score, as each combination which produces an about 50% cumulative similarity results in grounding.; Emrey, ¶ [0068], [0069]).
Regarding claim 19, Emrey discloses One or more non-transitory storage media storing instructions (“Software 450, which can be stored in storage 440 and executed by processor 410, can include, for example, the programming that embodies the functionality of the present disclosure (e.g., as embodied in the systems, computers, servers, and/or devices as described above)” and “can also be stored and/or transported within any computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described above, that can fetch and execute instructions associated with the software from the instruction execution system, apparatus, or device”; Emrey, ¶ [0086]-[0087]) which, when executed by one or more computing devices, cause performance of the method recited in Claim 1cause performance of the method recited in Claim 1 (See mapping of limitations presented with reference to claim 1; Emrey, ¶ (See above)).
Claims 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey and Hurst as applied to claim 1 above, and further in view of Salloum, with further support from Jurafsky.
Regarding claim 4, the rejection of claim 1 is incorporated. Emrey and Hurst disclose all of the elements of the current invention as stated above. Emrey further discloses wherein the method further comprises: prior to generating the score for each claim in the set of claims, removing… [numbers] (describes “the input data received by the system may be pre-processed,” which includes “segmentation, tokenization, stemming, lemmatization, parts-of-speech (POS) tagging, and/or other NLP pre-processing techniques” where text normalization is a well-known NLP pre-processing technique (example, Jurafsky, pg. 425 “remove numeric quantities”; pg. 570 “a first pass of text normalization to deal with numbers and abbreviations and other non-standard words”), and includes removing numbers, and other non-standard or non-semantically informative elements, from text, and where the pre-processing can occur prior to “the input data”, which can include the ground truth data, being subject to “further (e.g., substantive) processing of the data,” thus the preprocessing, and the text normalization, necessarily occurs prior to processing for generating the comparison score, as the system must receive the input data and the ground truth document data, before processing the same.; Emrey, ¶ [0054]-[0055]). However, Emrey fails to expressly recite wherein the list of items is a numbered list, further comprising...removing each number that precedes an item in the list of items.
Salloum teaches systems and methods for automatic formatting of documents. (Salloum, ¶ [0002]). Regarding claim 4, Salloum teaches wherein the list of items is a numbered list, wherein the method further comprises...removing each number that precedes an item in the list of items (Specifically discloses the preprocessing of a numbered list, where “Numbered lists have line-initial numbers replaced with dummy tokens. The first item in a numbered list has one token (‘NUM_LIST_1’), and all subsequent numbers have another (‘NUM_LIST_8’).”; Salloum, ¶ [0036]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the error and hallucination detection systems of Emrey, as modified by the list preprocessing of Hurst, to incorporate the teachings of Salloum to include wherein the list of items is a numbered list, further comprising...removing each number that precedes an item in the list of items. Salloum discloses uniformly subjecting documents for “text preprocessing” including the removal of numerals from numbered lists, “to better enable the translation system to reproduce punctuation and other formatting elements as well as to combat problems of sparsity for numerals”, which gives better consideration to informative numbers than prior art systems, thus retaining information which might otherwise be lost from standard text normalization, as recognized by Salloum. (Salloum, ¶ [0036]).
Claims 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey and Hurst as applied to claim 1 above, and further in view of Mirhaji.
Regarding claim 5, the rejection of claim 1 is incorporated. Emrey and Hurst disclose all of the elements of the current invention as stated above. Emrey further discloses wherein the list of items is within a sentence of the text data and is a flattened list (Though not described in the context of a list of items, the “Fact extraction stage 206 may comprise one or more natural language processing (NLP) models configured to extract entities, relationships between entities, and constraints defining the semantic context of the generative model output data 202” and exemplary models which may be incorporated, can include “named entity recognition models, coreference resolution models, and/or syntactic/semantic parsing models,” where the syntactic/semantic parsing model performs semantic parsing to understand the meaning of the individual components of a sentence, including the parsing of conjunctions (flattened lists), and performs syntactic parsing to analyze the grammar and structure of the sentence. Together, these components generate the individual components of the sentence, including the parsing of lists, which represent the constituent factual assertions. It is noted that the parsing of a list of items is an implicit and native capability which is a fundamental function of the syntactic/semantic parsers, as known in the relevant art.; Emrey, ¶ [0053]-[0054]). However, Emrey fails to expressly recite further comprising: determining that the list of items is a flattened list based on a number of commas or based on a number of phrases separated by commas in the sentence.
Mirhaji teaches systems and methods for “collection, integration and contextualization of information.” (Mirhaji, ¶ [0002]). Regarding claim 5, Mirhaji teaches further comprising: determining that the list of items is a flattened list based on a number of commas or based on a number of phrases separated by commas in the sentence (Discloses that a syntax ontology 135 is a conventional tool used by a parser 182 to parse text by establishing a basis for “identifying certain linguistic expressions.” This includes using “syntactic cues that may be reliably used for segmentation of a sentence”, such as “punctuations (for example, “.”, ″, “;”).” Further, Mirhaji provides specific example of this native function in FIG. 32, which shows the “output of a syntactic parser” breaking a single sentence “Large Blister on Toes and Abdomen,” which would be a flattened list per applicant’s specification, into its component facts.; Mirhaji, ¶ Col. 15, lines 5-14; Col. 20, lines 6-14, Col. 33, lines 16-18; FIG. 32).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the error and hallucination detection systems of Emrey, as modified by the list preprocessing of Hurst, to incorporate the teachings of Mirhaji to include further comprising: determining that the list of items is a flattened list based on a number of commas or based on a number of phrases separated by commas in the sentence. Mirhaji discloses systems and methods which “aid in the collection, representation and mining of data,” which “enable complete, reliable and fast collection and validation of information throughout various research projects, and among different participating locations,” with recognized “implications in multiple different contexts (decision support, research, quality of care, etc.),” including in a question answering capacity, as recognized by Mirhaji. (Mirhaji, Col. 1, lines 62-67, Col. 9, lines 3-10, Col. 13, lines 1-22).
Claims 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey and Hurst as applied to claim 1 above, and further in view of Kelsey (U.S. Pat. App. Pub. No. 2018/0260472, hereinafter Kelsey).
Regarding claim 6, the rejection of claim 1 is incorporated. Emrey and Hurst disclose all of the elements of the current invention as stated above. However, Emrey fails to expressly recite further comprising: identifying one or more filler sentences in the text data; removing the one or more filler sentences from consideration when generating a particular score for the text data.
Kelsey teaches systems and methods for “natural language processing.” (Kelsey, ¶ [0002]). Regarding claim 6, Kelsey teaches further comprising: identifying one or more filler sentences in the text data (“identification of paragraphs and/or discrete text chunks of related sentences having low content value”; Kelsey, ¶ [0057]); removing the one or more filler sentences from consideration when generating a particular score for the text data (In response to the “identification of paragraphs and/or discrete text chunks of related sentences having low content value” the “received text can be pre-processed and filtered”; Kelsey, ¶ [0056]-[0057]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the error and hallucination detection systems of Emrey, as modified by the list preprocessing of Hurst, to incorporate the teachings of Kelsey to include further comprising: identifying one or more filler sentences in the text data; removing the one or more filler sentences from consideration when generating a particular score for the text data. “The disclosed technologies adopt an approach of successive refinement in stages to simplify text, identify and extract semantic content, and obtain digested fragments of text” which includes “identification of paragraphs and/or discrete text chunks of related sentences having low content value,” where removal of sentences having a low content value provides the known benefit of “quality improvement can arise from process refinement” for “a succession of source documents,” which provides the known benefit of better quality source material, such as for question and answer systems, as recognized by Kelsey. (Kelsey, ¶ [0029], [0031]).
Claims 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey and Hurst as applied to claim 1 above, and further in view of Rahman.
Regarding claim 7, the rejection of claim 1 is incorporated. Emrey and Hurst disclose all of the elements of the current invention as stated above. Emrey further discloses further comprising: identifying, within the text data, a sentence that refers to a prompter in a second person manner (“The coreference resolution models of fact extraction stage 206 may generate a data structure output that comprises an indication of various pronouns or other linguistic expressions in the generative model output data 202 related to a given extracted entity,” where a pronoun is a reference to a person, such as you, s/he, etc., and where “an indication of various pronouns or other linguistic expressions in the generative model output data 202” is an identification of the sentence from the “generative model output data 202” which contains said indication; Emrey, ¶ [0058]); in response to identifying the sentence, identifying... [relevant information] associated with the sentence (“The coreference resolution models may receive a pre-processed input as described above with respect to the input of the named entity recognition (NER) models. The pre-processed input (e.g., generative model output data 202 and/or ground truth information 204)” where ground truth information can include the user input.; Emrey, ¶ [0034], [0058]); generating a new sentence that is based on the sentence and the [relevant information] (The coreference resolution model replaces the pronoun, as derived from the sentence, with the noun, as derived from the relevant information, in the generated “data structure output”; Emrey, ¶ [0053], [0058]-[0059]) generating a particular score for the text data that is based on the new sentence (“Data points within data structure 208 representing generative model output data (e.g., representative of utterances, individual entities, etc.) may be assigned a comparison score based on the comparison to data structure 210 representing fact data.”; Emrey, ¶ [0068]). However, Emrey fails to expressly recite in response to identifying the sentence, identifying a question that is associated with the sentence; [and] generating a new sentence that is based on the sentence and the question.
Rahman teaches systems and methods for “generating and correcting language model outputs.” (Rahman, ¶ [0002]). Regarding claim 7, Rahman teaches in response to identifying the sentence, identifying a question that is associated with the sentence (“responses generated by language model 204 can include inherent ambiguities. For example, pronouns such as “he”, “she”, “it” can refer to multiple different nouns, and an entity can be described in various ways across a text (e.g., “president of the United States”, “he”),” and “Request processing module 202 provides the response generated by language model 204 to NER model 208 in order to find named entities that can be used by coreference resolution module 210 to resolve ambiguous references in the response.”; Rahman, ¶ [0029]); generating a new sentence that is based on the sentence and the question (“Given the response and name entities as inputs, coreference resolution module 210 generates a resolved response.”; Rahman, ¶ [0029]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the error and hallucination detection systems of Emrey, as modified by the list preprocessing of Hurst, to incorporate the teachings of Rahman to include in response to identifying the sentence, identifying a question that is associated with the sentence; [and] generating a new sentence that is based on the sentence and the question. “The disclosed techniques” of Rahman “can identify a response as being incomplete when not all relevant portions of a context are included in the response, “ where the “incomplete response can be modified to include additional portions of the context that are relevant to a user request”, providing the known and obvious benefit of a complete and accurate response to a prompt, as recognized by Rahman. (Rahman, ¶ [0065]).
Claims 16-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Emrey as applied to claim 13 above, and further in view of DeFoor.
Regarding claim 16, the rejection of claim 13 is incorporated. Emrey discloses all of the elements of the current invention as stated above. However, Emrey fails to expressly recite wherein generating the plurality of combinations comprises: determining that total size of the plurality of documents is greater than a predefined threshold; creating a plurality of new combinations from the plurality of documents, wherein a first new combination in the plurality of new combinations is created by removing one of the plurality of documents from the plurality of documents, wherein a second new combination in the plurality of new combinations is created by removing another one of the plurality of documents from the plurality of documents; for each new combination in the plurality of new combinations: determining whether said each new combination exceeds the predefined threshold; if said each new combination exceeds the predefined threshold, then adding said each new combination to an OVER_THE_LIMIT set; if said each new combination does not exceed the predefined threshold, then adding said each new combination to the plurality of combinations.
DeFoor teaches systems and methods for “the evaluation of large volumes of documents” by large language models. (DeFoor, ¶ [0029]-[0030]). Regarding claim 16, DeFoor teaches wherein generating the plurality of combinations comprises: determining that total size of the plurality of documents is greater than a predefined threshold (As part of the text sharding method 500, “a text generation modeling system implementing a large language model may specify a size threshold in terms of a number of tokens (e.g., words).” and a the system can then make a “determination... as to whether the length of the selected text portion exceeds the maximum text chunk size,” where the selected text portion “may be the entirety of the text” and each of the text portions may refer to “documents” and/or “lists of documents”; DeFoor, ¶ [0097]-[0098], [0100]-[0101]); creating a plurality of new combinations from the plurality of documents, (“If it is determined that the length of the selected text portion {the plurality of documents} exceeds the maximum text chunk size, then at 510 one or more domain-specific text chunking constraints are identified” and “An updated text portion that does not exceed the maximum text chunk size is identified at 512” resulting in a plurality of “text portions selected at 506 and identified at 512” {a plurality of new combinations}”; DeFoor, ¶ [0102]-[0103]) wherein a first new combination in the plurality of new combinations is created by removing one of the plurality of documents from the plurality of documents, (The disclosed plurality of updated text portions includes at least a first updated text portion, where in the context of a dividing a larger selected text portion into the plurality of “smaller” updated text portions, from the perspective of the first updated text portion, the documents of all other updated text portions including the second updated text portion are “removed” {removing one of the plurality of documents} from the selected text portion to generate the first updated text portion.; DeFoor, ¶ [0095], [0103]) wherein a second new combination in the plurality of new combinations is created by removing another one of the plurality of documents from the plurality of documents (The disclosed plurality of updated text portions includes at least a first updated text portion, where in the context of a dividing a larger selected text portion into the plurality of “smaller” updated text portions, from the perspective of the first updated text portion, the documents of all other updated text portions including the first updated text portion are “removed” {removing another one of the plurality of documents} from the selected text portion to generate the second updated text portion.; DeFoor, ¶ [0095], [0103]); for each new combination in the plurality of new combinations: determining whether said each new combination exceeds the predefined threshold (Further “two or more of the text portions resulting from the division at 512” can be above or below “the maximum text chunk size” and, as such, “each of these may be assigned to a text chunk or chunks at operation 514” where the assignment itself may be a provisional assignment or it may be omitted, subject to FIG. 6, where “A determination is made at 516 as to whether to select an additional portion of the text” each updated text portion, where the plurality of updated text portions, as provisionally assigned to the respective text chunk, is then reviewed at 508 in the same manner as the selected text portion, including determining at 508 “whether the length of the selected text portion exceeds the maximum text chunk size. {whether each new combination exceeds the predefined threshold}”; DeFoor, ¶ [0101], [0106]-[0108]; FIG. 5); if said each new combination exceeds the predefined threshold, then adding said each new combination to an OVER_THE_LIMIT set (If “the length of” any of the plurality of updated text portions “exceeds the maximum text chunk size,” said updated text portion is subject to division, as described with reference to the selected text portion, which is understood as being part of an “OVER THE_LIMIT” set.; DeFoor, ¶ [0101], [0106]-[0108]; FIG. 5); if said each new combination does not exceed the predefined threshold, then adding said each new combination to the plurality of combinations (Once the selection process is complete, each of the “updated text portions” {new combinations} is added to a respective text chunk of the set of text chunks {the plurality of combinations}.; DeFoor, ¶ [0104], [0106], [0108]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the error and hallucination detection systems of Emrey to incorporate the teachings of DeFoor to include wherein generating the plurality of combinations comprises: determining that total size of the plurality of documents is greater than a predefined threshold; creating a plurality of new combinations from the plurality of documents, wherein a first new combination in the plurality of new combinations is created by removing one of the plurality of documents from the plurality of documents, wherein a second new combination in the plurality of new combinations is created by removing another one of the plurality of documents from the plurality of documents; for each new combination in the plurality of new combinations: determining whether said each new combination exceeds the predefined threshold; if said each new combination exceeds the predefined threshold, then adding said each new combination to an OVER_THE_LIMIT set; if said each new combination does not exceed the predefined threshold, then adding said each new combination to the plurality of combinations. Emrey discloses a sophisticated hallucination detection system which includes the partitioning of ground truth information into a plurality of data structures 210. However, Emrey fails to expressly recite exactly how these data structures 210 are generated from the overall ground truth information set. DeFoor discloses an automatic system for query evaluation including text sharding of a large ground truth dataset, for the generation of a plurality of selected text portions which are under a token limit threshold. A person of ordinary skill in the art would be motivated to combine the hallucination detection system of Emrey with the text sharding of DeFoor to assure that the prompt and all received ground truth information falls within the token limit of the model, providing the well-known benefit of assuring that the model is capable of receiving and processing the entire input, as understood in light of the disclosure of DeFoor. (DeFoor, ¶ [0095]).
Regarding claim 17, the rejection of claim 16 is incorporated. Emrey and DeFoor disclose all of the elements of the current invention as stated above. However, Emrey fail(s) to expressly recite further comprising, for each combination in the OVER_THE_LIMIT set: creating one or more new particular combinations from said each combination; for each new particular combination in the one or more new particular combinations: determining whether said each new particular combination exceeds the predefined threshold; if said each new particular combination exceeds the predefined threshold, then adding said each new particular combination to the OVER_THE_LIMIT set; and if said each new particular combination does not exceed the predefined threshold, then adding said each new particular combination to the plurality of combinations.
The relevance of DeFoor is described above with relation to claim 16. Regarding claim 17, DeFoor teaches further comprising, for each combination in the OVER_THE_LIMIT set (all updated text portion which exceed the size threshold, and thus are part of an “OVER THE_LIMIT” set, are reprocessed as shown in FIG. 5.; DeFoor, ¶ [0101], [0106]-[0108]; FIG. 5): creating one or more new particular combinations from said each combination (“If it is determined that the length of the... [updated text portion] {the plurality of documents} exceeds the maximum text chunk size, then at 510 one or more domain-specific text chunking constraints are identified” and “An updated text portion that does not exceed the maximum text chunk size is identified at 512” resulting in a plurality of “text portions selected at 506 and identified at 512” {one or more new particular combinations}”; DeFoor, ¶ [0102]-[0103]); for each new particular combination in the one or more new particular combinations: determining whether said each new particular combination exceeds the predefined threshold (The process described in FIG. 5 is iterative, as such the “two or more of the text portions resulting from the division at 512” can each be determined as above or below “the maximum text chunk size” and, as such, “each of these may be assigned to a text chunk or chunks at operation 514” where the assignment itself may be a provisional assignment or it may be omitted, subject to FIG. 6, where “A determination is made at 516 as to whether to select an additional portion of the text” each updated text portion, where the plurality of updated text portions, as provisionally assigned to the respective text chunk, is then reviewed at 508 in the same manner as the selected text portion, including determining at 508 “whether the length of the selected text portion exceeds the maximum text chunk size. {whether each new particular combination exceeds the predefined threshold}”; DeFoor, ¶ [0101], [0106]-[0108]; FIG. 5); if said each new particular combination exceeds the predefined threshold, then adding said each new particular combination to the OVER_THE_LIMIT set (If “the length of” any of the plurality of updated text portions “exceeds the maximum text chunk size,” said updated text portion is subject to division, as described with reference to the selected text portion, which is understood as being part of an “OVER THE_LIMIT” set.; DeFoor, ¶ [0101], [0106]-[0108]; FIG. 5); and if said each new particular combination does not exceed the predefined threshold, then adding said each new particular combination to the plurality of combinations (Once the selection process is complete, each of the “updated text portions” {new particular combinations} is added to a respective text chunk of the set of text chunks {the plurality of combinations}.; DeFoor, ¶ [0104], [0106], [0108]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the error and hallucination detection systems of Emrey to incorporate the teachings of DeFoor to include further comprising, for each combination in the OVER_THE_LIMIT set: creating one or more new particular combinations from said each combination; for each new particular combination in the one or more new particular combinations: determining whether said each new particular combination exceeds the predefined threshold; if said each new particular combination exceeds the predefined threshold, then adding said each new particular combination to the OVER_THE_LIMIT set; and if said each new particular combination does not exceed the predefined threshold, then adding said each new particular combination to the plurality of combinations. Emrey discloses a sophisticated hallucination detection system which includes the partitioning of ground truth information into a plurality of data structures 210. However, Emrey fails to expressly recite exactly how these data structures 210 are generated from the overall ground truth information set. DeFoor discloses an automatic system for query evaluation including text sharding of a large ground truth dataset, for the generation of a plurality of selected text portions which are under a token limit threshold. A person of ordinary skill in the art would be motivated to combine the hallucination detection system of Emrey with the text sharding of DeFoor to assure that the prompt and all received ground truth information falls within the token limit of the model, providing the well-known benefit of assuring that the model is capable of receiving and processing the entire input, as understood in light of the disclosure of DeFoor. (DeFoor, ¶ [0095]).
Regarding claim 18, the rejection of claim 13 is incorporated. Emrey discloses all of the elements of the current invention as stated above. Emrey further discloses further comprising: prior to generating the score, …[performing preprocessing of the documents] (“the input data received by the system may be pre-processed, or fact extraction stage 206 may comprise one or more pre-processing models to process the received input data prior to further (e.g., substantive) processing of the data.”; Emrey, ¶ [0054]); wherein generating the score is only performed in response to…[preprocessing of the documents] (The pre-processing can occur prior to “the input data”, which can include the ground truth data, being “further (e.g., substantive) processing of the data,” thus the preprocessing, and the text normalization, necessarily occurs prior to processing for generating the comparison score, as the system must receive the input data and the ground truth document data, before processing the same.; Emrey, ¶ [0054]-[0055]). However, Emrey fail(s) to expressly recite [wherein preprocessing of the documents includes]... determining whether a size of the combination is less than a predefined threshold; and wherein [further processing steps are]… only performed in response to determining that the size of the combination is less than the predefined threshold.
The relevance of DeFoor is described above with relation to claim 16. Regarding claim 18, DeFoor teaches [wherein preprocessing of the documents includes]... determining whether a size of the combination is less than a predefined threshold (As part of the text sharding method 500, which is a preprocessing step, “a text generation modeling system implementing a large language model may specify a size threshold in terms of a number of tokens (e.g., words).” and a the system can then make a “determination... as to whether the length of the selected text portion exceeds the maximum text chunk size,” where the selected text portion “may be the entirety of the text” and each of the text portions may refer to “documents” and/or “lists of documents” and, as a preprocessing step, this is prior to generating a score in the context of Emrey.; DeFoor, ¶ [0097]-[0098], [0100]-[0101]); and wherein [further processing steps are]… only performed in response to determining that the size of the combination is less than the predefined threshold (DeFoor performs the above process until all text chunks are comprised of updated text segments which are below the size threshold. As such, preprocessing is not complete until the size is less than a predetermined threshold. In the context of Emrey, since the score is not determined until after preprocessing is complete, the score would only be performed in response to determining that each “updated text portion... does not exceed the maximum text chunk size is identified at 512”; DeFoor, ¶ [0102]-[0103]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the error and hallucination detection systems of Emrey to incorporate the teachings of DeFoor to include [wherein preprocessing of the documents includes]... determining whether a size of the combination is less than a predefined threshold; and wherein [further processing steps are]… only performed in response to determining that the size of the combination is less than the predefined threshold. Emrey discloses a sophisticated hallucination detection system which includes the partitioning of ground truth information into a plurality of data structures 210. However, Emrey fails to expressly recite exactly how these data structures 210 are generated from the overall ground truth information set. DeFoor discloses an automatic system for query evaluation including text sharding of a large ground truth dataset, for the generation of a plurality of selected text portions which are under a token limit threshold. A person of ordinary skill in the art would be motivated to combine the hallucination detection system of Emrey with the text sharding of DeFoor to assure that the prompt and all received ground truth information falls within the token limit of the model, providing the well-known benefit of assuring that the model is capable of receiving and processing the entire input, as understood in light of the disclosure of DeFoor. (DeFoor, ¶ [0095]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Corlatescu (U.S. Pat. App. Pub. No. 2024/0427807) discloses an LLM funnel chain in conjunction with hierarchically layered documentation to produce subgroups of documents based on prompts and document groups.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Sean E. Serraguard whose telephone number is (313)446-6627. The examiner can normally be reached 07:00-17:00 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel C. Washburn can be reached at (571) 272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Sean E Serraguard/Patent Examiner, Art Unit 2657
Prosecution Insights