DETAILED ACTION
Notice of AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 15 is objected to because of the following informalities:
In claim 15, line 3, “the likelihood parameter of the structured statement” lacks antecedent basis. Lines 1-2 introduce the concept of a likelihood parameter only for the “pre-validated structured statement” and not the structured statement extracted from the text generated by the LLM in response to an input. The examiner suggests amending this to recite “a likelihood parameter of the structured statement”
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-23 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Step 1 of the Alice/Mayo framework, Claims 1-21 are directed to a method (a process), Claim 22 is directed to a system (a machine), and Claim 23 is directed to a non-transitory medium storing program instructions (an article of manufacture), which each fall within one of the four statutory categories of inventions.
Regarding Claim 1
Step 2A, prong 1 (Is the claim directed to a law of nature, a natural phenomenon or an abstract idea).
Claim 1 recites the following mental processes, that in each case under the broadest reasonable interpretation, covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper but for the recitation of generic computer components (e.g., “computer implemented”, “large language model (LLM)”).
A … method of validating a text generated by a large language model (LLM), comprising: (under the broadest reasonable interpretation, a human can mentally validate a text generated by a LLM, such as if the LLM output is “The patient presented with fever and therefore may have Influenza”, the human can validate that Influenza is one potential cause of a fever in a patient)
extracting a structured statement from the text generated by the LLM in response to an input, the structured statement comprising a first concept, a second concept, and a relational term defining a relationship between the first concept and the second concept; (under the broadest reasonable interpretation, a human can mentally extract a logical statement from a natural language sentence, such as (patient, fever) with relation Influenza)
searching using the structured statement, a dataset including a plurality of pre-validated structured statements; (under the broadest reasonable interpretation, a human can mentally use the structured statement (c1: patient, c2: fever, r: Influenza) to search a dataset, such as a paper file with similarly-constructed structured statements)
validating the text generated by the LLM in response to a match between the structured statement and at least one of the plurality of pre-validated structured statements of the dataset. (under the broadest reasonable interpretation, a human can mentally validate the text if it finds a matching pre-validated structured statement)
Step 2A, prong 2 (Does the claim recite additional elements that integrate the judicial exception into a practical application?).
The judicial exception is not integrated into a practical application.
Regarding the “computer implemented” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of a computer. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (a computer). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Accordingly, at Step 2A, prong two, after considering all claim elements individually and as an ordered combination, it is determined that the claims do not integrate the judicial exception into a practical application.
Step 2B (Does the claim recite additional elements that amount to significantly more than the judicial exception?)
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
Regarding the “computer implemented” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Accordingly, at Step 2B after considering all claim elements individually and as an ordered combination, it is determined that the claims do not integrate the judicial exception into a practical application.
Regarding Claim 2
Step 2A, Prong 1
wherein the extracting, the searching, and the validating are iterated for each of a plurality of structured statements extracted from the text, wherein the text is validated when each of the plurality of structured statements is matched to a corresponding pre-validated structured statement in the dataset. (under the broadest reasonable interpretation, a human can mentally perform the extracting, searching, and validating steps as explained with respect to claim 1, with respect to a plurality of texts, and where each text is validated at the time the structured statement is matched to a corresponding pre-validated structured statement in the dataset)
Regarding Step 2A, Prong 2, the claim does not include any additional elements that integrate the judicial exception into a practical application and regarding Step 2B, there are no additional elements recited that amount to significantly more than the judicial exception.
Regarding Claim 3
Step 2A, Prong 1
wherein the extracting, the searching, and the validating are performed prior to providing the text generated by the LLM in response to the input, wherein the text and an indication of validation of the text are provided in response to the input. (under the broadest reasonable interpretation, a human can mentally perform the extracting, searching, and validating steps (as explained above with respect to claim 1) prior to the point in time where the text and an indication of validation are provided to the user)
Regarding Step 2A, Prong 2, the claim does not include any additional elements that integrate the judicial exception into a practical application and regarding Step 2B, there are no additional elements recited that amount to significantly more than the judicial exception.
Regarding Claim 4
Step 2A, Prong 1
identifying at least one of a mismatch indicating a contradiction between the structured statement and at least one of the plurality of pre-validated structured statements, no match between the structured statement and any of the plurality of pre-validated structured statements; (under the broadest reasonable interpretation, a human can mentally identify a mismatch or no match between the structured statement and a pre-validated structured statement)
non-validating the text in response to the identified mismatch and/or no match. (under the broadest reasonable interpretation, a human can mentally determine that text is not valid in view of an identified mismatch or mental determination of no match)
Regarding Step 2A, Prong 2, the claim does not include any additional elements that integrate the judicial exception into a practical application and regarding Step 2B, there are no additional elements recited that amount to significantly more than the judicial exception.
Regarding Claim 5
Step 2A, Prong 1
generating an indication for the structured statement indicating one of: confirmation in response to the match, contradiction in response to the mismatch, and no match. (under the broadest reasonable interpretation, a human can mentally determine one of these options when considering a structured statement with respect to a set of pre-validated structured statements on paper)
Regarding Step 2A, Prong 2, the claim does not include any additional elements that integrate the judicial exception into a practical application and regarding Step 2B, there are no additional elements recited that amount to significantly more than the judicial exception.
Regarding Claim 6
Step 2A, Prong 1
wherein when a pre-validated statement is “if A then B”, the structured statement comprising “A causes B” or “B because of A” is validated, and the structured statement comprising “Not B and A?” is identified as the contradiction. (under the broadest reasonable interpretation, a human can mentally evaluate these types of logical statements for equivalency or contradiction)
Regarding Step 2A, Prong 2, the claim does not include any additional elements that integrate the judicial exception into a practical application and regarding Step 2B, there are no additional elements recited that amount to significantly more than the judicial exception.
Regarding Claim 7
Step 2A, Prong 1
further comprising in response to the non-validation of the text, generating an adaptation of the input, (under the broadest reasonable interpretation, a human can mentally change the input to the LLM in response to the non-validation of the previous text output from the LLM)
iterating the extracting, the searching and the validating for the adapted text. (under the broadest reasonable interpretation, a human can mentally iterate these steps as explained with respect to claim 1)
Step 2A, Prong 2
Regarding the “feeding the adapted input into the LLM to obtain an adapted text” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of inputting data into a generic LLM. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (inputting data into a generic LLM). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Step 2B
Regarding the “feeding the adapted input into the LLM to obtain an adapted text” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding Claim 8
Step 2A, Prong 1
wherein the generating the adapted input, … the extracting, and the searching, are iterated until the text is validated. (under the broadest reasonable interpretation, a human can mentally iterate these steps until the text is validated)
Step 2A, Prong 2
Regarding the “feeding the adapted input” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of inputting data into a generic LLM. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (inputting data into a generic LLM). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Step 2B
Regarding the “feeding the adapted input” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding Claim 9
Step 2A, Prong 1
in response to the non-validation of the text, identifying a pre-validated structured statement correlated with the structured statement; (under the broadest reasonable interpretation, a human can mentally identify a matching pre-validated structured statement on paper)
and correcting context and/or other impacted content accordingly. (under the broadest reasonable interpretation, a human can mentally correct context and/or other impacted content to be input into a LLM accordingly)
Step 2A, Prong 2
Regarding the “instructing the LLM to re-write using the correlated pre-validated structured statement instead of the structured statement” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of inputting data into a generic LLM. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (inputting data into a generic LLM). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Step 2B
Regarding the “instructing the LLM to re-write using the correlated pre-validated structured statement instead of the structured statement” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding Claim 10
Step 2A, Prong 2
Regarding the “prompting the LLM to re-write the text according to the matching structured statement” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of inputting data into a generic LLM. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (inputting data into a generic LLM). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Step 2B
Regarding the “prompting the LLM to re-write the text according to the matching structured statement” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding Claim 11
Step 2A, Prong 1
wherein each pre-validated structured statement is associated with an indication of a validated source, (under the broadest reasonable interpretation, a human can mentally associate pre-validated structured statements in a dataset with an indication of a validated source, such as a check box or flag)
Step 2A, Prong 2
Regarding the “providing the text generated by the LLM and the indication of the validated source in response to the match” limitation, such additional element of a data transmitting step is recited at a high level of generality and amounts to extra-solution activity of transmitting data, i.e. post-solution activity of transmitting data from the claimed process (see MPEP 2106.05(g)).
Step 2B
Regarding the “providing the text generated by the LLM and the indication of the validated source in response to the match” limitation, as discussed above, the additional element of a data transmitting step is recited at a high level of generality and amounts to extra-solution activity of receiving data, i.e. post-solution activity of transmitting data from the claimed process. The courts have found limitations directed to transmitting information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (see MPEP 2106.05(d)(II), “receiving or transmitting data over a network”, "electronic record keeping," and "storing and retrieving information in memory").
Regarding Claim 12
Step 2A, Prong 1
wherein each of the plurality of structured statements is matched with a pre-validated structured statement associated with a respective source used for the validation (under the broadest reasonable interpretation, a human can mentally match pre-validated structured statements printed on paper with source attributions)
mapping a plurality of validated sources to the plurality of structured statements, and providing the mapping. (under the broadest reasonable interpretation, a human can mentally map source attributions to structured statements, and outputting such mapping such as by writing the mapping on paper)
Step 2A, Prong 2
Regarding the “wherein the text comprises a plurality of structured statements” limitation, such additional element of a data transmitting step is recited at a high level of generality and amounts to extra-solution activity of transmitting data, i.e. post-solution activity of transmitting data from the claimed process (see MPEP 2106.05(g)).
Step 2B
Regarding the “wherein the text comprises a plurality of structured statements” limitation, as discussed above, the additional element of a data transmitting step is recited at a high level of generality and amounts to extra-solution activity of receiving data, i.e. post-solution activity of transmitting data from the claimed process. The courts have found limitations directed to transmitting information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (see MPEP 2106.05(d)(II), “receiving or transmitting data over a network”, "electronic record keeping," and "storing and retrieving information in memory").
Regarding Claim 13
Step 2A, Prong 2
Regarding the “wherein the text comprises medical content, the first concept and/or the second concept of the pre-validated structured statements included in the dataset comprise medical parameters, the relational term comprises a clinical relationship, and the pre-validated structured statements are validated by medical literature” limitation, such limitation amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use. As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not integrate a judicial exception into a practical application.
Step 2B
Regarding the “wherein the text comprises medical content, the first concept and/or the second concept of the pre-validated structured statements included in the dataset comprise medical parameters, the relational term comprises a clinical relationship, and the pre-validated structured statements are validated by medical literature” limitation, such limitation amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use as explained above, which does not amount to significantly more than the judicial exception. MPEP 2106.05(h).
Regarding Claim 14
Step 2A, Prong 1
further comprising providing an indication of quality of the validation of the pre-validated structured statement according to a type of clinical evidence used for generation of the pre-validated structured statement, selected from: double blind randomized control trial, observational study, meta-analysis, case report, retrospective study, and expert opinion. (under the broadest reasonable interpretation, a human can mentally review the clinical evidence recited in this limitation and provide an indication of quality of the validation of the pre-validated structured statement”)
Regarding Step 2A, Prong 2, the claim does not include any additional elements that integrate the judicial exception into a practical application and regarding Step 2B, there are no additional elements recited that amount to significantly more than the judicial exception.
Regarding Claim 15
Step 2A, Prong 1
wherein the plurality of pre-validated structured statements are associated with a likelihood parameter (under the broadest reasonable interpretation, a human can mentally associate a pre-validated structured statement with a likelihood parameter, such as the 4-tuple (patient, fever, influenza, 50%))
wherein the match comprises a partial match when the likelihood parameter of the structured statement does not match the likelihood parameter of at least one of the plurality of pre-validated structured statements. (under the broadest reasonable interpretation, a human can mentally determine a partial match, rather than a full match, when the first and second concepts and relationship match, but the likelihood concept does not match)
Regarding Step 2A, Prong 2, the claim does not include any additional elements that integrate the judicial exception into a practical application and regarding Step 2B, there are no additional elements recited that amount to significantly more than the judicial exception.
Regarding Claim 16
Step 2A, Prong 1
in response to a match between the structured statement and the at least one of the plurality of pre-validated structured statements, identifying a contradiction by at least one of: (i) matching the first concept and the second concept and detecting an opposite relation of the relational term, (ii) detecting that the second concept is an opposite of the first concept; and providing an indication of the contradiction. (under the broadest reasonable interpretation, a human can mentally identify a contradiction in one of the methods recited herein logically)
Regarding Step 2A, Prong 2, the claim does not include any additional elements that integrate the judicial exception into a practical application and regarding Step 2B, there are no additional elements recited that amount to significantly more than the judicial exception.
Regarding Claim 17
Step 2A, Prong 1
in response to a match between the structured statement and the at least one of the plurality of pre-validated structured statements, identifying mismatch of the relational term; (under the broadest reasonable interpretation, a human can mentally identify a mismatch after evaluating a logical statement)
and at least one of:
…
using natural language processing approaches to extract a structure of the structured statement, and compare the structure to the matching at least one of the plurality of pre-validated structured statements to determine whether the matching at least one of the plurality of pre-validated structured statements confirms or contradicts the structured statement. (under the broadest reasonable interpretation, a human can mentally use natural language processing to extract a first-order logical form from a sentence and compare such first-order logical form to other first-order logical forms in a data structure to determine a confirmation or a contradiction)
Step 2A, Prong 2
Regarding the “querying the LLM if the matching at least one of the plurality of pre-validated structured statements confirms or contradicts the structured statement” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of inputting data into a generic LLM. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (inputting data into a generic LLM). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Step 2B
Regarding the “querying the LLM if the matching at least one of the plurality of pre-validated structured statements confirms or contradicts the structured statement” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding Claim 18
Step 2A, Prong 1
in response to a match between the structured statement and the at least one of the plurality of pre-validated structured statements, and at least one of: (under the broadest reasonable interpretation, a human can mentally determine a match between a structured statement, such as a first-order logic translation, and a similar statement in a plurality of pre-validated structured statements)
for each extracted statement: using natural language processing approaches or asking the LLM or another LLM to create a new structured statement from the extracted statement, (under the broadest reasonable interpretation, a human can mentally translate a natural language sentence into first-order logic)
comparing the new structured statement to the matching at least one of the plurality of pre-validated structured statements to determine whether the matching at least one of the plurality of pre-validated structured statements confirms or contradicts the structured statement; (under the broadest reasonable interpretation, a human can mentally compare first-order logic statements from a structured statement to a pre-validated structured statement to determine a confirmation or contradiction)
Step 2A, Prong 2
Regarding the “(i) querying the LLM if the matching at least one of the plurality of pre-validated structured statements confirms or contradicts the structured statement” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of inputting data into a generic LLM. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (inputting data into a generic LLM). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Regarding the “(ii) asking the LLM to extract at least one statement from the structured statement” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of inputting data into a generic LLM. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (inputting data into a generic LLM). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Regarding the “for each at least one of the plurality of pre-validated structured statements matched to the structured statement, asking the LLM or another LLM if the extracted statement is validated or contradicted by the respective pre-validated structured statement” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of inputting data into a generic LLM. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (inputting data into a generic LLM). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Step 2B
Regarding the “(i) querying the LLM if the matching at least one of the plurality of pre-validated structured statements confirms or contradicts the structured statement” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding the “(ii) asking the LLM to extract at least one statement from the structured statement” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding the “for each at least one of the plurality of pre-validated structured statements matched to the structured statement, asking the LLM or another LLM if the extracted statement is validated or contradicted by the respective pre-validated structured statement” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding Claim 19
Step 2A, Prong 1
wherein searching comprises searching for combinations of linked pre-validated structured statements, and the match is between the structured statement and a combination of two or more linked pre-validated structured statements. (under the broadest reasonable interpretation, a human can mentally search and match 2+ lined pre-validated structured statements)
Regarding Step 2A, Prong 2, the claim does not include any additional elements that integrate the judicial exception into a practical application and regarding Step 2B, there are no additional elements recited that amount to significantly more than the judicial exception.
Regarding Claim 20
Step 2A, Prong 1
creating the plurality of pre-validated structured statements by extracting structured statements from pre-validated text. (under the broadest reasonable interpretation, a human can mentally extract and pre-validate structured statements from pre-validated text)
Regarding Step 2A, Prong 2, the claim does not include any additional elements that integrate the judicial exception into a practical application and regarding Step 2B, there are no additional elements recited that amount to significantly more than the judicial exception.
Regarding Claim 21
Step 2A, Prong 1
creating a new pre-validated structured statement from a combination of two or more linked pre-validated structured statements. (under the broadest reasonable interpretation, a human can mentally create new pre-validated statements by joining 2 or more pre-validated structured statements)
Regarding Step 2A, Prong 2, the claim does not include any additional elements that integrate the judicial exception into a practical application and regarding Step 2B, there are no additional elements recited that amount to significantly more than the judicial exception.
Regarding Claim 22
Step 2A, Prong 1
creating at least one pre-validated structured statement by analyzing a plurality of records, and extracting the first concept, the second concept and the relational term from the plurality of records. (under the broadest reasonable interpretation, a human can mentally create a pre-validated structured statement in view of the first and second concepts and relational term as set forth herein.
Regarding Step 2A, Prong 2, the claim does not include any additional elements that integrate the judicial exception into a practical application and regarding Step 2B, there are no additional elements recited that amount to significantly more than the judicial exception.
Regarding Claim 23
Step 2A, Prong 1
Claim 23 recites a system that corresponds to the method of claim 1, and therefore the analysis under Step 2A, Prong 1 with respect to claim 1 also applies to this claim 23.
Step 2A, Prong 2
Claim 23 recites a system that corresponds to the method of claim 1, and therefore the analysis under Step 2A, Prong 2 with respect to claim 1 also applies to this claim 23. While claim 23 recites additional generic computing components (“LLM”, “processor”), such additional generic computing components do not change the analysis under Step 2A, Prong 2. These additional elements are recited at a high-level of generality and amount to no more than mere instructions to apply the exception using generic computer components. Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Step 2B
Claim 23 recites a system that corresponds to the method of claim 1, and therefore the analysis under Step 2B with respect to claim 1 also applies to this claim 23. While claim23 recites additional generic computing components (“LLM”, “processor”), such additional generic computing components do not change the analysis under Step 2B because the limitations merely provide instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding Claim 24
Step 2A, Prong 1
Claim 24 recites a non-transitory medium storing program instructions that corresponds to the method of claim 1, and therefore the analysis under Step 2A, Prong 1 with respect to claim 1 also applies to this claim 24.
Step 2A, Prong 2
Claim 24 recites a non-transitory medium storing program instructions that corresponds to the method of claim 1, and therefore the analysis under Step 2A, Prong 2 with respect to claim 1 also applies to this claim 24. While claim 24 recites additional generic computing components (“LLM”, “processor”, “non-transitory medium storing program instructions”), such additional generic computing components do not change the analysis under Step 2A, Prong 2. These additional elements are recited at a high-level of generality and amount to no more than mere instructions to apply the exception using generic computer components. Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Step 2B
Claim 24 recites a non-transitory medium storing program instructions that corresponds to the method of claim 1, and therefore the analysis under Step 2B with respect to claim 1 also applies to this claim 24. While claim 24 recites additional generic computing components (“LLM”, “processor”, “non-transitory medium storing program instructions”), such additional generic computing components do not change the analysis under Step 2B because the limitations merely provide instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2, 4-5, 17, 20, and 22-23 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by US 20250200392 A1, hereinafter referenced as WHALEN.
Regarding Claim 1
WHALEN teaches:
A computer implemented method of validating a text generated by a large language model (LLM), comprising: (WHALEN, para. 0015: “Disclosed herein are systems, methods, and non-transitory computer-readable media (generally, “techniques”) for large language model (LLM) verification. The techniques include verifying large language model responses by obtaining a query and its corresponding answer from a large language model. This conversational text is then fed into a second large language model, which translates the answer into first-order logic. The verification process uses an automated theorem prover. It checks the validity of this logic translation by determining the unsatisfiability of two scenarios: one where the negation of the logic translation and domain-specific logic formulas are combined, and another where the logic translation itself is combined with these formulas. Based on this analysis, the theorem prover ascertains whether the translated answer is valid, invalid, or neither. The final step is communicating this verification status through an appropriate output medium, such as a graphical user interface, a database, or a report, providing a structured and methodical approach to assessing the accuracy and reliability of language model responses.”)
extracting a structured statement from the text generated by the LLM in response to an input, the structured statement comprising a first concept, a second concept, and a relational term defining a relationship between the first concept and the second concept; (WHALEN, para. 0031: “To verify responses generated by the vanilla LLM 102 against domain logic model 118, the responses are transformed into formal logic for accuracy. This means that the responses, initially in a natural language format, are converted into a structured, logical form. This transformation is useful to assessing the accuracy of the responses. By expressing the responses in the language of formal logic, it becomes possible to rigorously compare them against the established principles and rules encapsulated in the domain logic model 118. The domain logic model 118 acts as a benchmark, embodying the specific logical constructs and knowledge pertinent to the domain in question. Therefore, when the vanilla LLM 102's responses are converted into this formal logical structure, it allows for a systematic and precise evaluation.”;
WHALEN, paras. 0079-0081: “For example, consider the following two first-order logic translations of the sentence: “The weight of the item is 0.5 pounds (eight ounces).”: [0080] Translation B1: weightInLbs(item)=0.5 [0081] Translation B2: weightInOz(item)=8”
Examiner’s Note: Paras. 0079-0081 provide an example of the structured, logical form used to represent a natural language sentence, where in this example, there is a first concept (item), a second concept (0.5 pounds, or 8 ounces), and a relationship between the first and second concepts (weight in lbs or oz))
searching using the structured statement, a dataset including a plurality of pre-validated structured statements; and (WHALEN, para. 0030: “The DMC 116 uses the documents 114 to create a domain model in logic (referred to as “domain logic model 118”) that is used by the answer verifier 106 to verify responses generated by the vanilla LLM 102 to prompts submitted to the vanilla LLM 102 by a customer 120. The domain logic model 118 is specifically tailored to enhance the functionality of the vanilla LLM 102. When the customer 120 submits prompts to the vanilla LLM 102, it generates responses based on its general knowledge and algorithms. However, the accuracy and relevance of these responses, especially in specialized or complex domains, may not always be optimal. This is where the domain logic model 118 created by the DMC 116 becomes useful. It acts as a verifier or a benchmark against which the responses from the vanilla LLM 102 are evaluated. The logic model 118, informed and structured by the critical information in the documents 114, embodies a logical and domain-specific understanding.”
WHALEN, para. 0031: “To verify responses generated by the vanilla LLM 102 against domain logic model 118, the responses are transformed into formal logic for accuracy. This means that the responses, initially in a natural language format, are converted into a structured, logical form. This transformation is useful to assessing the accuracy of the responses. By expressing the responses in the language of formal logic, it becomes possible to rigorously compare them against the established principles and rules encapsulated in the domain logic model 118. The domain logic model 118 acts as a benchmark, embodying the specific logical constructs and knowledge pertinent to the domain in question. Therefore, when the vanilla LLM 102's responses are converted into this formal logical structure, it allows for a systematic and precise evaluation.”;
WHALEN, para. 0033: “This process results in the transformation of the information in the documents 114 into a logic specification format, which essentially means converting the data into a structured, formal logic representation. This representation is what constitutes the domain logic model 118. By using pre-built prompt templates, the domain expert 112 can effectively harness the advanced capabilities of the logic-trained LLM 110, ensuring that the derived domain logic model 118 is both accurate and highly tailored to the specific requirements of the domain as represented in the documents 114.”
Examiner’s Note: the “domain logic model 118” corresponds to the recited “dataset” and the “established principes and rules encapsulated in the domain logic model 118” correspond to the recited “pre-validated structured statements”
validating the text generated by the LLM in response to a match between the structured statement and at least one of the plurality of pre-validated structured statements of the dataset. (WHALEN, para. 0030: “This is where the domain logic model 118 created by the DMC 116 becomes useful. It acts as a verifier or a benchmark against which the responses from the vanilla LLM 102 are evaluated. The logic model 118, informed and structured by the critical information in the documents 114, embodies a logical and domain-specific understanding. This allows it to assess the vanilla LLM 102's responses for their accuracy and alignment with the specialized knowledge encapsulated in the domain logic model 118. Essentially, the DMC 116, in conjunction with the domain logic model 118 and the answer verifier 106, imparts an additional layer of expertise and validation to the outputs of the vanilla LLM 102, ensuring that the information provided to the customer 120 is not only generated by advanced AI algorithms but is also vetted against the robust, logically constructed domain logic model 118.”;
WHALEN, para. 0031: “ To verify responses generated by the vanilla LLM 102 against domain logic model 118, the responses are transformed into formal logic for accuracy. This means that the responses, initially in a natural language format, are converted into a structured, logical form. This transformation is useful to assessing the accuracy of the responses. By expressing the responses in the language of formal logic, it becomes possible to rigorously compare them against the established principles and rules encapsulated in the domain logic model 118. The domain logic model 118 acts as a benchmark, embodying the specific logical constructs and knowledge pertinent to the domain in question. Therefore, when the vanilla LLM 102's responses are converted into this formal logical structure, it allows for a systematic and precise evaluation. This process ensures that the responses not only seem plausible in natural language but also adhere to the stringent criteria of logical coherence and factual accuracy as defined by the domain logic model 118. This verification step is useful for maintaining the integrity and reliability of the vanilla LLM 102's outputs, especially in scenarios where precision and correctness are paramount.”
WHALEN, para. 0057: “On the other hand, the answer verifier 106, designed to assess and ensure the accuracy of responses from a language model like the vanilla LLM 102, can function effectively as a standalone system for quality control in language processing. It can independently verify the relevance and correctness of answers generated by a LLM, making it valuable in applications where precision and reliability of language model outputs are critical.”;
Examiner’s Note: the broadest reasonable interpretation of “match” does not require a 100% precise match as explained by p. 14, line 28 – p. 15, 16, and therefore WHALEN meets this limitation by verifying accuracy and correctness of a formal logical structure based on the comparison of the logical structures of the domain logic model 118)
Regarding Claim 2
WHALEN teaches the method of claim 1 as explained above. WHALEN further teaches:
wherein the extracting, the searching, and the validating are iterated for each of a plurality of structured statements extracted from the text, wherein the text is validated when each of the plurality of structured statements is matched to a corresponding pre-validated structured statement in the dataset. (WHALEN, para. 0062: “Similarly, when the query converter 336 transforms the informal queries and the vanilla LLM 102's answers into formal assertions 338, it does so by use of the SMT-LIB language. This ensures that these assertions 338 are in a format that can be readily processed and analyzed by the system 100's logic solver 122. The uniform use of SMT-LIB across these components facilitates seamless integration and interoperability within the system 100, allowing for efficient and accurate verification of LLM responses against the domain logic models 118.”;
Examiner’s Note: As shown in Fig. 1, the process of taking queries and LLM responses (plural) and passing such information to answer verifier 106 can be done more than once, such that each pair of queries and LLM responses forwarded to answer verifier 106 is a single iteration)
Regarding Claim 4
WHALEN teaches the method of claim 1 as explained above. WHALEN further teaches:
further comprising: identifying at least one of a mismatch indicating a contradiction between the structured statement and at least one of the plurality of pre-validated structured statements, no match between the structured statement and any of the plurality of pre-validated structured statements; and non-validating the text in response to the identified mismatch and/or no match. (WHALEN, para. 0056: “The assertion prover 340 of the answer verifier system 106 employs a trusted solver 122 to assess the validity of the assertions 338 generated by the query converter 336. The assertion prover 340 determines whether each assertion 338—representing a query and its corresponding response from the vanilla LLM 102—is valid, invalid, or neither valid, not invalid. In this context, ‘valid’ means that a first query of the trusted solver 122 in terms of the domain logic model 118 and the assertion 338 is unsatisfiable. “Invalid” means that a second query of the trusted solver 122 in terms of the domain logic model 118 and the assertion 338 is unsatisfiable. If both the first query and the second query are satisfiable, then the assertion 338 is neither valid nor invalid. The assertion prover 340 in cooperation with the trusted solver 122 rigorously test each assertion 388 to arrive at this valid, invalid, or neither valid not invalid verdict.”)
Regarding Claim 5
WHALEN teaches the method of claim 4 as explained above. WHALEN further teaches:
further comprising generating an indication for the structured statement indicating one of: confirmation in response to the match, contradiction in response to the mismatch, and no match. (WHALEN, para. 0056: “The assertion prover 340 of the answer verifier system 106 employs a trusted solver 122 to assess the validity of the assertions 338 generated by the query converter 336. The assertion prover 340 determines whether each assertion 338—representing a query and its corresponding response from the vanilla LLM 102—is valid, invalid, or neither valid, not invalid. In this context, ‘valid’ means that a first query of the trusted solver 122 in terms of the domain logic model 118 and the assertion 338 is unsatisfiable. “Invalid” means that a second query of the trusted solver 122 in terms of the domain logic model 118 and the assertion 338 is unsatisfiable. If both the first query and the second query are satisfiable, then the assertion 338 is neither valid nor invalid. The assertion prover 340 in cooperation with the trusted solver 122 rigorously tests each assertion 388 to arrive at this valid, invalid, or neither valid not invalid verdict.”)
Regarding Claim 17
WHALEN teaches the method of claim 1 as explained above. WHALEN further teaches:
in response to a match between the structured statement and the at least one of the plurality of pre-validated structured statements, identifying mismatch of the relational term; (WHALEN, para. 0077: “In some embodiments, the LLM verifier system uses a theorem prover (e.g., an Satisfiability Modulo Theories (SMT) solver) to perform equivalence checks among all the translations. This component is specifically employed to conduct equivalence checks among the various translations obtained from the different LLMs. After the main answer from the conversation 440 is extracted and translated into first-order logic by the multiple LLMs, these translations, though structurally different, are intended to convey the same logical content. The theorem prover is used by the LLM verifier system to verify the logical equivalence of these translations.”;
Examiner’s Note: as explained by the examples in paras. 0072-0076, the LLM verifier system of WHALEN can determine if the concepts are the same, but the relation is a mismatch, such as in the case of Translation A3 in para. 0075)
and at least one of:
querying the LLM if the matching at least one of the plurality of pre-validated structured statements confirms or contradicts the structured statement; and
using natural language processing approaches to extract a structure of the structured statement, and compare the structure to the matching at least one of the plurality of pre-validated structured statements to determine whether the matching at least one of the plurality of pre-validated structured statements confirms or contradicts the structured statement. (WHALEN, para. 0077: “In some embodiments, the LLM verifier system uses a theorem prover (e.g., an Satisfiability Modulo Theories (SMT) solver) to perform equivalence checks among all the translations. This component is specifically employed to conduct equivalence checks among the various translations obtained from the different LLMs. After the main answer from the conversation 440 is extracted and translated into first-order logic by the multiple LLMs, these translations, though structurally different, are intended to convey the same logical content. The theorem prover is used by the LLM verifier system to verify the logical equivalence of these translations.”;
Examiner’s Note: because this is a method claim, only one of these contingent limitations needs to be met by the prior art (see MPEP 2111.04); in this case, WHALEN discloses using natural language techniques (translations) to extract a first-order logic structure (see paras. 0072-0075), and the LLM verifier system compares the first-order logic statements to confirm or contradict the first-order logic statement)
Regarding Claim 20
WHALEN teaches the method of claim 1 as explained above. WHALEN further teaches:
further comprising creating the plurality of pre-validated structured statements by extracting structured statements from pre-validated text. (WHALEN, para. 0033: “In some embodiments, the domain expert 112 uses the logic-trained LLM 110 through a series of pre-built prompt templates to derive the domain logic model 118 in a logic specification format from the set of documents 114. In this scenario, the domain expert 112 utilizes the logic-trained LLM 110, but rather than engaging in an open-ended or ad-hoc interaction, they use a series of pre-built prompt templates. These templates are designed to systematically extract and process information from the set of documents 114 in a highly structured and efficient manner. The pre-built prompts guide the interaction with the logic-trained LLM 110, ensuring that the queries and commands are precisely aligned with the goal of deriving the domain logic model 118. The logic-trained LLM 110, with its enhanced capabilities in understanding and applying logical constructs, interprets and analyzes the contents of the documents 114 under the framework provided by these templates. This process results in the transformation of the information in the documents 114 into a logic specification format, which essentially means converting the data into a structured, formal logic representation. This representation is what constitutes the domain logic model 118. By using pre-built prompt templates, the domain expert 112 can effectively harness the advanced capabilities of the logic-trained LLM 110, ensuring that the derived domain logic model 118 is both accurate and highly tailored to the specific requirements of the domain as represented in the documents 114.”;
Examiner’s Note: by relying on the documents 114 as representing the domain, WHALEN is requiring the initial documents 114 to be assumed to be accurate)
Regarding Claim 22
WHALEN teaches the method of claim 1 as explained above. WHALEN further teaches:
further comprising creating at least one pre-validated structured statement by analyzing a plurality of records, and extracting the first concept, the second concept and the relational term from the plurality of records. (WHALEN, para. 0033: “In some embodiments, the domain expert 112 uses the logic-trained LLM 110 through a series of pre-built prompt templates to derive the domain logic model 118 in a logic specification format from the set of documents 114. In this scenario, the domain expert 112 utilizes the logic-trained LLM 110, but rather than engaging in an open-ended or ad-hoc interaction, they use a series of pre-built prompt templates. These templates are designed to systematically extract and process information from the set of documents 114 in a highly structured and efficient manner. The pre-built prompts guide the interaction with the logic-trained LLM 110, ensuring that the queries and commands are precisely aligned with the goal of deriving the domain logic model 118. The logic-trained LLM 110, with its enhanced capabilities in understanding and applying logical constructs, interprets and analyzes the contents of the documents 114 under the framework provided by these templates. This process results in the transformation of the information in the documents 114 into a logic specification format, which essentially means converting the data into a structured, formal logic representation. This representation is what constitutes the domain logic model 118. By using pre-built prompt templates, the domain expert 112 can effectively harness the advanced capabilities of the logic-trained LLM 110, ensuring that the derived domain logic model 118 is both accurate and highly tailored to the specific requirements of the domain as represented in the documents 114.”;
WHALEN, paras. 0079-0081: “For example, consider the following two first-order logic translations of the sentence: “The weight of the item is 0.5 pounds (eight ounces).”: [0080] Translation B1: weightInLbs(item)=0.5 [0081] Translation B2: weightInOz(item)=8”
Examiner’s Note: Paras. 0079-0081 provide an example of the structured, logical form used to represent a natural language sentence, where in this example, there is a first concept (item), a second concept (0.5 pounds, or 8 ounces), and a relationship between the first and second concepts (weight in lbs or oz))
Regarding Claim 23
WHALEN teaches:
A system … comprising: at least one processor executing a code for: (WHALEN, para. 0192): “FIG. 11 illustrates an example of a programmable electronic device that processes and manipulates data to perform tasks and calculations disclosed herein for large language model (LLM) verification. Example programmable electronic device 1100 includes electronic components encompassing hardware or hardware and software including processor 1102, memory 1104, auxiliary memory 1106, input device 1108, output device 1110, mass data storage 1112, network interface 1114, and offload card 1124, all connected to bus 1116”
WHALEN, para. 0197: “Memory 1104 is an electronic component that stores data and instructions 1118 that processor 1102 processes.”
The remaining limitations correspond to the method of claim 1, and therefore this claim is rejected for the same reasons explained above with respect to claim 1.
Regarding Claim 23
WHALEN teaches:
A non-transitory medium storing program instructions … which when executed by at least one processor, cause the at least one processor to: (WHALEN, para. 0192): “FIG. 11 illustrates an example of a programmable electronic device that processes and manipulates data to perform tasks and calculations disclosed herein for large language model (LLM) verification. Example programmable electronic device 1100 includes electronic components encompassing hardware or hardware and software including processor 1102, memory 1104, auxiliary memory 1106, input device 1108, output device 1110, mass data storage 1112, network interface 1114, and offload card 1124, all connected to bus 1116”
WHALEN, para. 0197: “Memory 1104 is an electronic component that stores data and instructions 1118 that processor 1102 processes.”
The remaining limitations correspond to the method of claim 1, and therefore this claim is rejected for the same reasons explained above with respect to claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim 3, 7-11, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over WHALEN in view of Peng, Baolin, et al. "Check your facts and try again: Improving large language models with external knowledge and automated feedback." arXiv preprint arXiv:2302.12813 (2023), hereinafter referenced as PENG.
Regarding Claim 3
WHALEN teaches the method of claim 1 as explained above. However, WHALEN fails to explicitly teach:
wherein the extracting, the searching, and the validating are performed prior to providing the text generated by the LLM in response to the input, wherein the text and an indication of validation of the text are provided in response to the input.
However, in a related field of endeavor (augmenting a fixed LLM with external knowledge for question answering, see p. 1, section 1), PENG teaches and makes obvious:
wherein the extracting, the searching, and the validating are performed prior to providing the text generated by the LLM in response to the input, wherein the text and an indication of validation of the text are provided in response to the input. (PENG, pp. 1-2, section 1: “As illustrated by the example in Figure 1, given a user query (e.g., regarding a 2013 Los Angeles Galaxy player transfer), LLM-AUGMENTER first retrieves evidence from external knowledge (e.g., Web or task-specific datasets) and, if necessary, further consolidates evidence by linking retrieved raw evidence with related context (e.g., information of the entity “2013 Los Angeles Galaxy”) and performing reasoning to form evidence chains (e.g., table-passage in the figure). Then, LLM AUGMENTER queries a fixed LLM (i.e., ChatGPT in our study) using a prompt that contains the consolidated evidence for ChatGPT to generate a candidate response grounded in external knowledge (evidence). LLM-AUGMENTER then verifies the candidate response e.g., by checking whether it hallucinates evidence. If so, LLM-AUGMENTER generates a feedback message (e.g., about the team “C.S.D. Municipal”). The message is used to revise the prompt to query ChatGPT again. The process iterates until a candidate response passes the verification and is sent to the user.”;
Examiner’s Note: PENG discloses analyzing the user query and performing the process of collecting and analyzing evidence based on such query, and then evaluating LLM responses for verification before providing such response to a user; the WHALEN-PENG combination now modifies the LLM of WHALEN to require claim validation (as in WHALEN) prior to providing the LLM response to the user (as in PENG))
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of WHALEN with PENG as explained above. As disclosed by PENG, one of ordinary skill would have been motivated to do so in order to reduce LLM “hallucinations without sacrificing the fluency and informativeness of its generated responses.” (PENG, p. 2, section 1). One of ordinary skill would have been further motivated to do so because PENG teaches that the factuality score of ChatGPT is substantially improved “by grounding ChatGPT’s responses in consolidated external knowledge and automated feedback.” (PENG, p. 2, section 1).
Regarding Claim 7
WHALEN teaches the method of claim 4 as explained above. However, WHALEN fails to explicitly teach:
further comprising in response to the non-validation of the text, generating an adaptation of the input, feeding the adapted input into the LLM to obtain an adapted text, and iterating the extracting, the searching and the validating for the adapted text.
However, in a related field of endeavor (augmenting a fixed LLM with external knowledge for question answering, see p. 1, section 1), PENG teaches and makes obvious:
further comprising in response to the non-validation of the text, generating an adaptation of the input, feeding the adapted input into the LLM to obtain an adapted text, and iterating the extracting, the searching and the validating for the adapted text. (PENG, pp. 1-2, section 1: “As illustrated by the example in Figure 1, given a user query (e.g., regarding a 2013 Los Angeles Galaxy player transfer), LLM-AUGMENTER first retrieves evidence from external knowledge (e.g., Web or task-specific datasets) and, if necessary, further consolidates evidence by linking retrieved raw evidence with related context (e.g., information of the entity “2013 Los Angeles Galaxy”) and performing reasoning to form evidence chains (e.g., table-passage in the figure). Then, LLM AUGMENTER queries a fixed LLM (i.e., ChatGPT in our study) using a prompt that contains the consolidated evidence for ChatGPT to generate a candidate response grounded in external knowledge (evidence). LLM-AUGMENTER then verifies the candidate response e.g., by checking whether it hallucinates evidence. If so, LLM-AUGMENTER generates a feedback message (e.g., about the team “C.S.D. Municipal”). The message is used to revise the prompt to query ChatGPT again. The process iterates until a candidate response passes the verification and is sent to the user.”;
Examiner’s Note: the WHALEN-PENG combination now modifies the LLM of WHALEN to revise the LLM prompt if validation is not successful as in PENG, and continuing to iterate the claim verification process of WHALEN until a validated response from the LLM is available)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of WHALEN with PENG as explained above. As disclosed by PENG, one of ordinary skill would have been motivated to do so in order to reduce LLM “hallucinations without sacrificing the fluency and informativeness of its generated responses.” (PENG, p. 2, section 1). One of ordinary skill would have been further motivated to do so because PENG teaches that the factuality score of ChatGPT is substantially improved “by grounding ChatGPT’s responses in consolidated external knowledge and automated feedback.” (PENG, p. 2, section 1).
Regarding Claim 8
WHALEN and PENG teach the method of claim 7 as explained above. However, WHALEN fails to explicitly teach:
wherein the generating the adapted input, the feeding the adapted input, the extracting, and the searching, are iterated until the text is validated.
However, in a related field of endeavor (augmenting a fixed LLM with external knowledge for question answering, see p. 1, section 1), PENG teaches and makes obvious:
wherein the generating the adapted input, the feeding the adapted input, the extracting, and the searching, are iterated until the text is validated. (PENG, pp. 1-2, section 1: “As illustrated by the example in Figure 1, given a user query (e.g., regarding a 2013 Los Angeles Galaxy player transfer), LLM-AUGMENTER first retrieves evidence from external knowledge (e.g., Web or task-specific datasets) and, if necessary, further consolidates evidence by linking retrieved raw evidence with related context (e.g., information of the entity “2013 Los Angeles Galaxy”) and performing reasoning to form evidence chains (e.g., table-passage in the figure). Then, LLM AUGMENTER queries a fixed LLM (i.e., ChatGPT in our study) using a prompt that contains the consolidated evidence for ChatGPT to generate a candidate response grounded in external knowledge (evidence). LLM-AUGMENTER then verifies the candidate response e.g., by checking whether it hallucinates evidence. If so, LLM-AUGMENTER generates a feedback message (e.g., about the team “C.S.D. Municipal”). The message is used to revise the prompt to query ChatGPT again. The process iterates until a candidate response passes the verification and is sent to the user.”;
Examiner’s Note: the WHALEN-PENG combination now modifies the LLM of WHALEN to revise the LLM prompt if validation is not successful as in PENG, and continuing to iterate the claim verification process of WHALEN until a validated response from the LLM is available)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of WHALEN with PENG as explained above. As disclosed by PENG, one of ordinary skill would have been motivated to do so in order to reduce LLM “hallucinations without sacrificing the fluency and informativeness of its generated responses.” (PENG, p. 2, section 1). One of ordinary skill would have been further motivated to do so because PENG teaches that the factuality score of ChatGPT is substantially improved “by grounding ChatGPT’s responses in consolidated external knowledge and automated feedback.” (PENG, p. 2, section 1).
Regarding Claim 9
WHALEN teaches the method of claim 4 as explained above. However, WHALEN fails to explicitly teach:
further comprising: in response to the non-validation of the text, identifying a pre-validated structured statement correlated with the structured statement; and instructing the LLM to re-write using the correlated pre-validated structured statement instead of the structured statement, and correcting context and/or other impacted content accordingly.
However, in a related field of endeavor (augmenting a fixed LLM with external knowledge for question answering, see p. 1, section 1), PENG teaches and makes obvious:
further comprising: in response to the non-validation of the text, identifying a pre-validated structured statement correlated with the structured statement; and instructing the LLM to re-write using the correlated pre-validated structured statement instead of the structured statement, and correcting context and/or other impacted content accordingly. (PENG, pp. 1-2, section 1: “As illustrated by the example in Figure 1, given a user query (e.g., regarding a 2013 Los Angeles Galaxy player transfer), LLM-AUGMENTER first retrieves evidence from external knowledge (e.g., Web or task-specific datasets) and, if necessary, further consolidates evidence by linking retrieved raw evidence with related context (e.g., information of the entity “2013 Los Angeles Galaxy”) and performing reasoning to form evidence chains (e.g., table-passage in the figure). Then, LLM AUGMENTER queries a fixed LLM (i.e., ChatGPT in our study) using a prompt that contains the consolidated evidence for ChatGPT to generate a candidate response grounded in external knowledge (evidence). LLM-AUGMENTER then verifies the candidate response e.g., by checking whether it hallucinates evidence. If so, LLM-AUGMENTER generates a feedback message (e.g., about the team “C.S.D. Municipal”). The message is used to revise the prompt to query ChatGPT again. The process iterates until a candidate response passes the verification and is sent to the user.”;
Examiner’s Note: the WHALEN-PENG combination now utilizes the teachings of WHALEN to determine if a statement is valid or not (see para. 0056), and in such a case, an actually valid statement is taken from the domain logic model 118 and provided to the LLM to re-generate a new response as in PENG)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of WHALEN with PENG as explained above. As disclosed by PENG, one of ordinary skill would have been motivated to do so in order to reduce LLM “hallucinations without sacrificing the fluency and informativeness of its generated responses.” (PENG, p. 2, section 1). One of ordinary skill would have been further motivated to do so because PENG teaches that the factuality score of ChatGPT is substantially improved “by grounding ChatGPT’s responses in consolidated external knowledge and automated feedback.” (PENG, p. 2, section 1).
Regarding Claim 10
WHALEN teaches the method of claim 1 as explained above. However, WHALEN fails to explicitly teach:
further comprising prompting the LLM to re-write the text according to the matching structured statement.
However, in a related field of endeavor (augmenting a fixed LLM with external knowledge for question answering, see p. 1, section 1), PENG teaches and makes obvious:
further comprising prompting the LLM to re-write the text according to the matching structured statement. (PENG, pp. 1-2, section 1: “As illustrated by the example in Figure 1, given a user query (e.g., regarding a 2013 Los Angeles Galaxy player transfer), LLM-AUGMENTER first retrieves evidence from external knowledge (e.g., Web or task-specific datasets) and, if necessary, further consolidates evidence by linking retrieved raw evidence with related context (e.g., information of the entity “2013 Los Angeles Galaxy”) and performing reasoning to form evidence chains (e.g., table-passage in the figure). Then, LLM AUGMENTER queries a fixed LLM (i.e., ChatGPT in our study) using a prompt that contains the consolidated evidence for ChatGPT to generate a candidate response grounded in external knowledge (evidence). LLM-AUGMENTER then verifies the candidate response e.g., by checking whether it hallucinates evidence. If so, LLM-AUGMENTER generates a feedback message (e.g., about the team “C.S.D. Municipal”). The message is used to revise the prompt to query ChatGPT again. The process iterates until a candidate response passes the verification and is sent to the user.”;
Examiner’s Note: the WHALEN-PENG combination now modifies the teachings of WHALEN so that in response to a structured match being found, prompting the LLM (as in WHALEN) using the match to generate a new output response)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of WHALEN with PENG as explained above. As disclosed by PENG, one of ordinary skill would have been motivated to do so in order to reduce LLM “hallucinations without sacrificing the fluency and informativeness of its generated responses.” (PENG, p. 2, section 1). One of ordinary skill would have been further motivated to do so because PENG teaches that the factuality score of ChatGPT is substantially improved “by grounding ChatGPT’s responses in consolidated external knowledge and automated feedback.” (PENG, p. 2, section 1).
The computer implemented method of claim 11, wherein the text comprises a plurality of structured statements, wherein each of the plurality of structured statements is matched with a pre-validated structured statement associated with a respective source used for the validation, and further comprising mapping a plurality of validated sources to the plurality of structured statements, and providing the mapping.
Regarding Claim 11
WHALEN teaches the method of claim 1 as explained above. However, WHALEN fails to explicitly teach:
wherein each pre-validated structured statement is associated with an indication of a validated source, and further comprising providing the text generated by the LLM and the indication of the validated source in response to the match.
However, in a related field of endeavor (augmenting a fixed LLM with external knowledge for question answering, see p. 1, section 1), PENG teaches and makes obvious:
wherein each pre-validated structured statement is associated with an indication of a validated source, and further comprising providing the text generated by the LLM and the indication of the validated source in response to the match. (PENG, pp. 1-2, section 1: “As illustrated by the example in Figure 1, given a user query (e.g., regarding a 2013 Los Angeles Galaxy player transfer), LLM-AUGMENTER first retrieves evidence from external knowledge (e.g., Web or task-specific datasets) and, if necessary, further consolidates evidence by linking retrieved raw evidence with related context (e.g., information of the entity “2013 Los Angeles Galaxy”) and performing reasoning to form evidence chains (e.g., table-passage in the figure). Then, LLM AUGMENTER queries a fixed LLM (i.e., ChatGPT in our study) using a prompt that contains the consolidated evidence for ChatGPT to generate a candidate response grounded in external knowledge (evidence). LLM-AUGMENTER then verifies the candidate response e.g., by checking whether it hallucinates evidence. If so, LLM-AUGMENTER generates a feedback message (e.g., about the team “C.S.D. Municipal”). The message is used to revise the prompt to query ChatGPT again. The process iterates until a candidate response passes the verification and is sent to the user.”;
Examiner’s Note: the WHALEN-PENG combination now modifies the teachings of WHALEN so that in response to a structured match being found, providing the indication that verification has been passed to the LLM along with the revised message for the revised prompt)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of WHALEN with PENG as explained above. As disclosed by PENG, one of ordinary skill would have been motivated to do so in order to reduce LLM “hallucinations without sacrificing the fluency and informativeness of its generated responses.” (PENG, p. 2, section 1). One of ordinary skill would have been further motivated to do so because PENG teaches that the factuality score of ChatGPT is substantially improved “by grounding ChatGPT’s responses in consolidated external knowledge and automated feedback.” (PENG, p. 2, section 1).
Regarding Claim 18
WHALEN teaches the method of claim 1 as explained above. WHALEN further teaches:
in response to a match between the structured statement and the at least one of the plurality of pre-validated structured statements, and at least one of: WHALEN, para. 0031: “ To verify responses generated by the vanilla LLM 102 against domain logic model 118, the responses are transformed into formal logic for accuracy. This means that the responses, initially in a natural language format, are converted into a structured, logical form. This transformation is useful to assessing the accuracy of the responses. By expressing the responses in the language of formal logic, it becomes possible to rigorously compare them against the established principles and rules encapsulated in the domain logic model 118. The domain logic model 118 acts as a benchmark, embodying the specific logical constructs and knowledge pertinent to the domain in question. Therefore, when the vanilla LLM 102's responses are converted into this formal logical structure, it allows for a systematic and precise evaluation. This process ensures that the responses not only seem plausible in natural language but also adhere to the stringent criteria of logical coherence and factual accuracy as defined by the domain logic model 118. This verification step is useful for maintaining the integrity and reliability of the vanilla LLM 102's outputs, especially in scenarios where precision and correctness are paramount.”
WHALEN, para. 0057: “On the other hand, the answer verifier 106, designed to assess and ensure the accuracy of responses from a language model like the vanilla LLM 102, can function effectively as a standalone system for quality control in language processing. It can independently verify the relevance and correctness of answers generated by a LLM, making it valuable in applications where precision and reliability of language model outputs are critical.”;
Examiner’s Note: the broadest reasonable interpretation of “match” does not require a 100% precise match as explained by p. 14, line 28 – p. 15, 16, and therefore WHALEN meets this limitation by verifying accuracy and correctness of a formal logical structure based on the comparison of the logical structures of the domain logic model 118)
However, WHALEN does not explicitly teach:
(i) querying the LLM if the matching at least one of the plurality of pre-validated structured statements confirms or contradicts the structured statement; and
(ii) asking the LLM to extract at least one statement from the structured statement;
for each extracted statement:
using natural language processing approaches or asking the LLM or another LLM to create a new structured statement from the extracted statement, and comparing the new structured statement to the matching at least one of the plurality of pre-validated structured statements to determine whether the matching at least one of the plurality of pre-validated structured statements confirms or contradicts the structured statement;
for each at least one of the plurality of pre-validated structured statements matched to the structured statement, asking the LLM or another LLM if the extracted statement is validated or contradicted by the respective pre-validated structured statement.
However, in a related field of endeavor (augmenting a fixed LLM with external knowledge for question answering, see p. 1, section 1), PENG teaches and makes obvious:
(i) querying the LLM if the matching at least one of the plurality of pre-validated structured statements confirms or contradicts the structured statement; and (PENG, pp. 1-2, section 1: “As illustrated by the example in Figure 1, given a user query (e.g., regarding a 2013 Los Angeles Galaxy player transfer), LLM-AUGMENTER first retrieves evidence from external knowledge (e.g., Web or task-specific datasets) and, if necessary, further consolidates evidence by linking retrieved raw evidence with related context (e.g., information of the entity “2013 Los Angeles Galaxy”) and performing reasoning to form evidence chains (e.g., table-passage in the figure). Then, LLM AUGMENTER queries a fixed LLM (i.e., ChatGPT in our study) using a prompt that contains the consolidated evidence for ChatGPT to generate a candidate response grounded in external knowledge (evidence). LLM-AUGMENTER then verifies the candidate response e.g., by checking whether it hallucinates evidence. If so, LLM-AUGMENTER generates a feedback message (e.g., about the team “C.S.D. Municipal”). The message is used to revise the prompt to query ChatGPT again. The process iterates until a candidate response passes the verification and is sent to the user.”;
Examiner’s Note: the WHALEN-PENG combination now modifies the teachings of WHALEN so that the LLM is queried only after the first-order logic statement of WHALEN is confirmed by the domain logic model 118 of WHALEN)
(ii) asking the LLM to extract at least one statement from the structured statement;
for each extracted statement:
using natural language processing approaches or asking the LLM or another LLM to create a new structured statement from the extracted statement, and comparing the new structured statement to the matching at least one of the plurality of pre-validated structured statements to determine whether the matching at least one of the plurality of pre-validated structured statements confirms or contradicts the structured statement;
for each at least one of the plurality of pre-validated structured statements matched to the structured statement, asking the LLM or another LLM if the extracted statement is validated or contradicted by the respective pre-validated structured statement. (Examiner’s Note: under MPEP 2111.04, everything under (ii) is a contingent limitation that is not required to be performed under the broadest reasonable interpretation of this method claim).
Claims 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over WHALEN in view of US 20190303772 A1, hereinafter referenced as ZHU.
Regarding Claim 6
WHALEN teaches the method of claim 4 as explained above. However, WHALEN fails to explicitly teach:
wherein when a pre-validated statement is “if A then B?”, the structured statement comprising “A causes B” or “B because of A” is validated, and the structured statement comprising “Not B and A?” is identified as the contradiction.
However, in a related field of endeavor (logical expressions using predicates, see para. 0017), ZHU teaches and makes obvious:
wherein when a pre-validated statement is “if A then B?”, the structured statement comprising “A causes B” or “B because of A” is validated, and the structured statement comprising “Not B and A?” is identified as the contradiction. (ZHU, para. 0027: “Additionally, the (logical) inference engine 114 derives logical consequences from known truth values of the constraints, and detects one or more contradictions. Specifically, in one or more embodiments of the invention, the inference engine 114 can identify logical contradictions among the constraints, and can derive additional truth values for constraints with unknown values.”
ZHU, para. 0028: “The inference engine 114 can generate a graph from the constraints stored in the constraint store 112, wherein each logical expression and sub-expression corresponds to a node in the graph. Additionally, in such a graph, equivalent expressions are identified as the same node, and each pair of an expression and a direct sub-expression corresponds to an edge in the graph.”;
Examiner’s Note: the WHALEN-ZHU combination now maps equivalent expressions to one another, and further identifies contradictions as taught by ZHU)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of WHALEN with ZHU as explained above. As disclosed by ZHU, one of ordinary skill would have been motivated to do so in order to “validate[] the consistency of assumptions and constraints pertaining to the target system by performing a logical inference to derive truth values of predicates and to detect contradictions.” (para. 0015).
Regarding Claim 16
WHALEN teaches the method of claim 1 as explained above. However, WHALEN fails to explicitly teach:
further comprising: in response to a match between the structured statement and the at least one of the plurality of pre-validated structured statements, identifying a contradiction by at least one of: (i) matching the first concept and the second concept and detecting an opposite relation of the relational term, (ii) detecting that the second concept is an opposite of the first concept; and providing an indication of the contradiction. (ZHU, para. 0027: “Additionally, the (logical) inference engine 114 derives logical consequences from known truth values of the constraints, and detects one or more contradictions. Specifically, in one or more embodiments of the invention, the inference engine 114 can identify logical contradictions among the constraints, and can derive additional truth values for constraints with unknown values.”
ZHU, para. 0029: “The inference engine 114 can also update additional node values via the following inference procedure. For each node with an updated truth value, the inference engine 114 determines if any logical connective in the graph neighborhood is violated, which implies a contradiction in the set of known truth values. If there is no contradiction, the inference engine 114 examines all nodes in the graph neighborhood with unknown truth values. For each such node, the inference engine 114 determines if setting the node's truth value to true or false would cause a contradiction with the truth values of the node's neighbors. If neither a true setting nor a false setting would cause contradiction, the truth value of the node remains unknown. If one of a true setting or false setting would cause a contradiction, the truth value of the node is derived to be the opposite value. If both a true setting and a false setting would cause a contradiction, a contradiction is detected among the known truth values of the node's neighbors. In one or more embodiments of the invention, this procedure can be performed recursively (by the inference engine 114) until no additional updates to truth values are obtained.”
Examiner’s Note: the WHALEN-ZHU combination now identifies contradictions as taught by ZHU)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of WHALEN with ZHU as explained above. As disclosed by ZHU, one of ordinary skill would have been motivated to do so in order to “validate[] the consistency of assumptions and constraints pertaining to the target system by performing a logical inference to derive truth values of predicates and to detect contradictions.” (para. 0015).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over WHELAN in view of PENG and further in view of Do, Hyo Jin, et al. "Facilitating human-LLM collaboration through factuality scores and source attributions." arXiv preprint arXiv:2405.20434 (May 2024), hereinafter referenced as DO.
Regarding Claim 12
WHALEN and PENG teach the method of claim 11 as explained above. However, WHALEN and PENG fail to explicitly teach:
wherein the text comprises a plurality of structured statements, wherein each of the plurality of structured statements is matched with a pre-validated structured statement associated with a respective source used for the validation, and further comprising mapping a plurality of validated sources to the plurality of structured statements, and providing the mapping.
However, in a related field of endeavor (mitigating hallucinations by LLMs, see p. 1, section 1), DO teaches and makes obvious:
wherein the text comprises a plurality of structured statements, wherein each of the plurality of structured statements is matched with a pre-validated structured statement associated with a respective source used for the validation, and further comprising mapping a plurality of validated sources to the plurality of structured statements, and providing the mapping. (DO, p. 4, section 3.2.2: “Next, participants were introduced to the source attribution feature, in which the source was annotated to show which parts were used to generate the model’s response. There were two designs that were presented to users: reference numbers, in which each sentence of the source document was numbered, and propositions in the response were tagged with the number corresponding to the source sentence from which it was derived; and highlight gradients, in which sections of the source that provided the information for parts of the response were highlighted. The order of presentation of the two source attribution designs was randomized between participants. Fig. 1 presents the two designs that users evaluated.”;
Examiner’s Note: as shown in Fig. 1, DO teaches mapping source evidence to the output of the LLM; the WHALEN-PENG-DO combination now modifies the LLM of WHALEN to output a plain language response, in addition to outputting the actual logical rules used to formulate such response, where source evidence is mapped to the plain language response and the actual logical rules using the mapping techniques of DO)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of WHALEN with PENG and DO as explained above. As disclosed by DO, one of ordinary skill would have been motivated to do so in order to “identify the most effective and preferred strategy for communicating two pieces of information about an LLM’s response: (1) the factuality score … and (2) source attribution.” (pp. 1-2, section 1).
Claims 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over WHELAN in view of Hong, Shengxin, et al. "Argmed-agents: Explainable clinical decision reasoning with llm disscusion via argumentation schemes." https://arxiv.org/abs/2403.06294v3, (June 10, 2024), hereinafter referenced as HONG.
Regarding Claim 13
WHALEN teaches the method of claim 1 as explained above. However, WHALEN fails to explicitly teach:
wherein the text comprises medical content, the first concept and/or the second concept of the pre-validated structured statements included in the dataset comprise medical parameters, the relational term comprises a clinical relationship, and the pre-validated structured statements are validated by medical literature.
However, in a related field of endeavor (explainable clinical decision reasoning, see p. 1, section I), HONG teaches and makes obvious:
wherein the text comprises medical content, the first concept and/or the second concept of the pre-validated structured statements included in the dataset comprise medical parameters, the relational term comprises a clinical relationship, and the pre-validated structured statements are validated by medical literature. (HONG, p. 2, section 1: “Our approach effectively combines argumentation frame works and cognitive clinical medicine. Arguments are presented through LLM simulations of clinical discussions and reasoning results are solved through formal computational models, which avoids the cumulative errors of LLM logical reasoning and improves the safety of clinical reasoning.”
HONG, p. 2, section II.A:
PNG
media_image1.png
174
358
media_image1.png
Greyscale
Examiner’s Note: HONG teaches an abstract argumentation pair, which has 2 concepts and a relation R, each with respect to medical concepts; the WHALEN-HONG combination now applies the LLM of WHALEN to the medical field (see paras. 0029, 0061 of WHALEN) using the teachings of HONG, where the logical forms of WHALEN can be adapted to be for the medical field using the teachings of HONG.
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of WHALEN with HONG as explained above. As disclosed by HONG, one of ordinary skill would have been motivated to do so because HONG teaches methods for providing “safer decisions compared to other state-of-art methods” and will provide “healthcare professionals with powerful tools to enhance their decision-making process and ultimately improve patient outcomes.” (p. 7, section VI).
Regarding Claim 14
WHALEN and HONG teach the method of claim 13 as explained above. WHALEN further teaches:
further comprising providing an indication of quality of the validation of the pre-validated structured statement according to a type of clinical evidence used for generation of the pre-validated structured statement, selected from: double blind randomized control trial, observational study, meta-analysis, case report, retrospective study, and expert opinion. (WHALEN, para. 0036: “Thus, while the DMC 116 offers significant automation and efficiency in domain logic model 118 creation, the expertise and judgment of the domain expert 112 may be needed to ensure the highest quality and accuracy of the final domain logic model 118, especially in complex or nuanced domains.”
WHALEN, para. 0042: “The domain expert data network 124 is a specialized component of a larger system designed to integrate and leverage the expertise of domain experts. The domain expert data network 124 encompasses the domain expert 112, the logic-trained LLM 110, and the curated set of documents 114 containing detailed domain-specific information. The domain expert 112 is an individual or an entity with a profound understanding and experience in particular fields. The logic-trained LLM 110 is a variant of a standard LLM, but it is specifically trained and tailored to incorporate the nuances and complexities of the domain-specific knowledge. The logic-trained LLM 110 works in tandem with the domain expert 112, processing and analyzing the information contained in the set of documents 114. These documents 114 serve as the knowledge base for the domain expert data network 124, encompassing detailed, domain-specific data, rules, regulations, and other pertinent information.”
WHALEN, para. 0069: “On the other hand, if the translations vary widely, indicating a lack of consensus, it suggests that the original natural language input is too ambiguous or complex to be reliably translated into formal logic. In such cases, the verifier system flags the input as potentially problematic, indicating that a high-quality, reliable translation cannot be guaranteed.”
Examiner’s Note: WHALEN discloses flagging inputs as possibly being unreliable, where the matched statement is based on expert opinion for the domain logic model 118.)
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over WHALEN in view of US 20160247090 A1, hereinafter referenced as YU.
Regarding Claim 15
WHALEN teaches the method of claim 1 as explained above. However, WHALEN fails to explicitly teach:
wherein the plurality of pre-validated structured statements are associated with a likelihood parameter, and wherein the match comprises a partial match when the likelihood parameter of the structured statement does not match the likelihood parameter of at least one of the plurality of pre-validated structured statements.
However, in a related field of endeavor (using first-order logic to extract semantic relationships between entities, para. 0008), YU teaches and makes obvious:
wherein the plurality of pre-validated structured statements are associated with a likelihood parameter, and wherein the match comprises a partial match when the likelihood parameter of the structured statement does not match the likelihood parameter of at least one of the plurality of pre-validated structured statements. (YU, para. 0024: “A probabilistic graphical model 106 is defined by the first-order logic KB 104, the entity pairs 102, and a number of weights. Weights can be associated with the first-order logic formulas that define the first-order logic KB 104. One of the weights can be associated with each of the formulas. The weights can be used to jointly determine the probabilities of the first-order logic formulas via a log-linear model. In a similar fashion, the probabilities of the first-order logic formulas can be used to determine the weights that are associated with the first-order logic formulas.”;
YU, para. 0037: “For example, a relationship can be associated with the entities in a pair when the relationship is associated with a first-order logic formula and when the first-order logic formula is associated with a joint inference. In the previous example, the joint inference provides a higher probability for the first-order logic formula than the probabilities that are associated with the other first-order logic formulas.”;
Examiner’s Note: the WHALEN-YU combination now associates each logic rule with a probability as explained by YU, such that when the probabilities do not precisely match, the match is determined to be a partial match because only the logic rule portion matches)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of WHALEN with YU as explained above. As disclosed by YU, one of ordinary skill would have been motivated to do so in order to perform joint inference to “make simultaneous statistical judgments about a number of relations between the entities.” (para. 0008).
Claims 19 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over WHALEN in view of Wang, Haoran, et al. "Explainable claim verification via knowledge-grounded reasoning with large language models." Findings of the association for computational linguistics: EMNLP 2023, hereinafter referenced as WANG.
Regarding Claim 19
WHALEN teaches the method of claim 1 as explained above. However, WHALEN fails to explicitly teach:
wherein searching comprises searching for combinations of linked pre-validated structured statements, and the match is between the structured statement and a combination of two or more linked pre-validated structured statements.
However, in a related field of endeavor (using first-order logic to perform explainable claim verification, see p. 6289, section 1), WANG teaches and makes obvious:
wherein searching comprises searching for combinations of linked pre-validated structured statements, and the match is between the structured statement and a combination of two or more linked pre-validated structured statements. (WANG, p. 6291, section 3.1:
PNG
media_image2.png
484
458
media_image2.png
Greyscale
Examiner’s Note: WANG teaches the multiple sub-claims can be joined into a larger claim (corresponding to recited “two or more linked pre-validated structured statements”); the WHALEN-WANG combination now performs higher-order searches using linked first-order logic statements as in WANG)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of WHALEN with WANG as explained above. As disclosed by WANG, one of ordinary skill would have been motivated to do so in order to use symbolic languages to “effectively identify the crucial entities, relations, and facts within the claim” without being limited to 2 concepts and 1 relation. (p. 6291, section 3.1).
Regarding Claim 21
WHALEN teaches the method of claim 1 as explained above. However, WHALEN fails to explicitly teach:
further comprising creating a new pre-validated structured statement from a combination of two or more linked pre-validated structured statements.
However, in a related field of endeavor (using first-order logic to perform explainable claim verification, see p. 6289, section 1), WANG teaches and makes obvious:
further comprising creating a new pre-validated structured statement from a combination of two or more linked pre-validated structured statements. (WANG, p. 6291, section 3.1:
PNG
media_image2.png
484
458
media_image2.png
Greyscale
Examiner’s Note: WANG teaches the multiple sub-claims can be joined into a larger claim (corresponding to recited “two or more linked pre-validated structured statements”); the WHALEN-WANG combination now performs higher-order searches using linked first-order logic statements as in WANG)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of WHALEN with WANG as explained above. As disclosed by WANG, one of ordinary skill would have been motivated to do so in order to use symbolic languages to “effectively identify the crucial entities, relations, and facts within the claim” without being limited to 2 concepts and 1 relation. (p. 6291, section 3.1).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20250348759 A1 (Kumar). “The present disclosure describes a system and method to verify/determine correctness and consistency of Artificial Intelligence (AI) generated content. The system may include a layered verification system that comprises a plurality of large language models (LLMs) to verify the content. In some aspects, the plurality of LLMs may include a content generation LLM and a verifier LLM. The content generation LLM may be configured to receive a user prompt, and generate content using static internal knowledge of the content generation LLM. The verifier LLM may obtain the generated content from the content generation LLM, and may verify the content against external dynamic knowledge bases.” (para. 0010).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL C LEE whose telephone number is (571)272-4933. The examiner can normally be reached M-F 12:00 pm - 8:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas can be reached at 571-272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL C. LEE/Examiner, Art Unit 2128
Prosecution Insights