Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1
Step 1: Is the claim to a process, machine, manufacture, or composition of matter?
Yes. Claim 1 is drawn to a process.
Step 2A, Prong One: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes. Claim 1 recites the following abstract ideas:
“identifying…within the first output, a plurality of assertions” – This constitutes a concept performed in the human mind, such as an observation, evaluation, judgement, and/or opinion. Furthermore, the courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). See MPEP 2106.04(a)(2).
“determining…that a first assertion in the plurality of assertions is false” – This constitutes a concept performed in the human mind, such as an observation, evaluation, judgement, and/or opinion. Furthermore, the courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). See MPEP 2106.04(a)(2).
“generating…a prompt that indicates that the first assertion is false” – This constitutes a concept performed in the human mind, such as an observation, evaluation, judgement, and/or opinion. Furthermore, the courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). See MPEP 2106.04(a)(2).
Step 2A, Prong Two: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. Claim 1 recites the following additional elements:
“by a computing system” – This amounts to mere application of the exception on a computer, i.e. a recitation of the words “apply it”. Implementing an abstract idea on a generic computer does not integrate the abstract idea into a practical application in Step 2A Prong Two or add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea of intermediated settlement on a generic computer. See MPEP 2106.05(f).
“submitting…the prompt as input to the LLM” – This amounts to insignificant extra solution activity in the form of mere data gathering, see MPEP 2106.05(g) and MPEP 2106.05(d), II, (i), regarding receiving or transmitting data.
“accessing…second output generated by the LLM, wherein the second output includes a second assertion that is different than the first assertion and corresponds to the first assertion” – This amounts to insignificant extra solution activity in the form of mere data gathering, see MPEP 2106.05(g) and MPEP 2106.05(d), II, (i), regarding receiving or transmitting data.
“wherein the method is performed by one or more computing devices” – This amounts to mere application of the exception on a computer, i.e. a recitation of the words “apply it”. Implementing an abstract idea on a generic computer does not integrate the abstract idea into a practical application in Step 2A Prong Two or add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea of intermediated settlement on a generic computer. See MPEP 2106.05(f).
Step 2B: Does the claim recite additional elements that amount ot significantly more than the judicial exception?
No. Claim 1 recites the following additional elements:
“by a computing system” – This amounts to mere application of the exception on a computer, i.e. a recitation of the words “apply it”. Implementing an abstract idea on a generic computer does not integrate the abstract idea into a practical application in Step 2A Prong Two or add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea of intermediated settlement on a generic computer. See MPEP 2106.05(f).
“submitting…the prompt as input to the LLM” – This amounts to insignificant extra solution activity in the form of mere data gathering, see MPEP 2106.05(g) and MPEP 2106.05(d), II, (i), regarding receiving or transmitting data.
“accessing…second output generated by the LLM, wherein the second output includes a second assertion that is different than the first assertion and corresponds to the first assertion” – This amounts to insignificant extra solution activity in the form of mere data gathering, see MPEP 2106.05(g) and MPEP 2106.05(d), II, (i), regarding receiving or transmitting data.
“wherein the method is performed by one or more computing devices” – This amounts to mere application of the exception on a computer, i.e. a recitation of the words “apply it”. Implementing an abstract idea on a generic computer does not integrate the abstract idea into a practical application in Step 2A Prong Two or add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea of intermediated settlement on a generic computer. See MPEP 2106.05(f).
Claim 2
Step 2A, Prong One: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes. Claim 2 recites the following abstract ideas:
“determining…that the second output generated by the LLM is free from false assertions” – This constitutes a concept performed in the human mind, such as an observation, evaluation, judgement, and/or opinion. Furthermore, the courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). See MPEP 2106.04(a)(2).
Step 2A, Prong Two: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. Claim 2 recites the following additional elements:
“by the computing system” – This amounts to mere application of the exception on a computer, i.e. a recitation of the words “apply it”. Implementing an abstract idea on a generic computer does not integrate the abstract idea into a practical application in Step 2A Prong Two or add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea of intermediated settlement on a generic computer. See MPEP 2106.05(f).
“in response to determining that the second output is free from false assertions, (a) storing the second output in persistent memory” – This amounts to insignificant extra solution activity in the form of mere data gathering, see MPEP 2106.05(g) and MPEP 2106.05(d), II, (iv), regarding storing or retrieving information in memory.
“in response to determining that the second output is free from false assertions…(b) causing a portion of the first output and the second assertion to be transmitted to a requesting entity that provided, to the LLM, initial input that cause the LLM to generate the first output” - This amounts to insignificant extra solution activity in the form of mere data gathering, see MPEP 2106.05(g) and MPEP 2106.05(d), II, (i), regarding receiving or transmitting data.
Step 2B: Does the claim recite additional elements that amount ot significantly more than the judicial exception?
No. Claim 2 recites the following additional elements:
“by the computing system” – This amounts to mere application of the exception on a computer, i.e. a recitation of the words “apply it”. Implementing an abstract idea on a generic computer does not integrate the abstract idea into a practical application in Step 2A Prong Two or add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea of intermediated settlement on a generic computer. See MPEP 2106.05(f).
“in response to determining that the second output is free from false assertions, (a) storing the second output in persistent memory” – This amounts to insignificant extra solution activity in the form of mere data gathering, see MPEP 2106.05(g) and MPEP 2106.05(d), II, (iv), regarding storing or retrieving information in memory.
“in response to determining that the second output is free from false assertions…(b) causing a portion of the first output and the second assertion to be transmitted to a requesting entity that provided, to the LLM, initial input that cause the LLM to generate the first output” - This amounts to insignificant extra solution activity in the form of mere data gathering, see MPEP 2106.05(g) and MPEP 2106.05(d), II, (i), regarding receiving or transmitting data.
Claim 3
Step 2A, Prong One: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes. Claim 3 recites the following abstract ideas:
The abstract ideas of its parent claims.
Step 2A, Prong Two: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. Claim 3 recites the following additional elements:
“in response to determining that the second output is free from false assertions: replacing, in the first output, the first assertion with the second assertion to generate modified first output” – This amounts to insignificant extra solution activity in the form of mere data gathering, see MPEP 2106.05(g) and MPEP 2106.05(d), II, (i), regarding transmitting or receiving data and (iii) regarding keeping electronic records.
“causing the modified first output to be transmitted to the requesting entity” - This amounts to insignificant extra solution activity in the form of mere data gathering, see MPEP 2106.05(g) and MPEP 2106.05(d), II, (i), regarding receiving or transmitting data.
Step 2B: Does the claim recite additional elements that amount ot significantly more than the judicial exception?
No. Claim 3 recites the following additional elements:
“in response to determining that the second output is free from false assertions: replacing, in the first output, the first assertion with the second assertion to generate modified first output” – This amounts to insignificant extra solution activity in the form of mere data gathering, see MPEP 2106.05(g) and MPEP 2106.05(d), II, (i), regarding transmitting or receiving data and (iii) regarding keeping electronic records.
“causing the modified first output to be transmitted to the requesting entity” - This amounts to insignificant extra solution activity in the form of mere data gathering, see MPEP 2106.05(g) and MPEP 2106.05(d), II, (i), regarding receiving or transmitting data.
Claim 4
Step 2A, Prong One: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes. Claim 4 recites the following abstract ideas:
The abstract ideas of its parent claim.
Step 2A, Prong Two: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. Claim 4 recites the following additional elements:
“storing, in a database, an association between the first assertion and the second assertion” – This amounts to insignificant extra solution activity in the form of mere data gathering, see MPEP 2106.05(g) and MPEP 2106.05(d), II, (iv), regarding storing or retrieving information in memory.
“retraining the LLM based on the association” - This amounts to mere application of the exception on a computer, i.e. a recitation of the words “apply it”. Implementing an abstract idea on a generic computer does not integrate the abstract idea into a practical application in Step 2A Prong Two or add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea of intermediated settlement on a generic computer. See MPEP 2106.05(f).
Step 2B: Does the claim recite additional elements that amount ot significantly more than the judicial exception?
No. Claim 4 recites the following additional elements:
“storing, in a database, an association between the first assertion and the second assertion” – This amounts to insignificant extra solution activity in the form of mere data gathering, see MPEP 2106.05(g) and MPEP 2106.05(d), II, (iv), regarding storing or retrieving information in memory.
“retraining the LLM based on the association” - This amounts to mere application of the exception on a computer, i.e. a recitation of the words “apply it”. Implementing an abstract idea on a generic computer does not integrate the abstract idea into a practical application in Step 2A Prong Two or add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea of intermediated settlement on a generic computer. See MPEP 2106.05(f).
Claim 5
Step 2A, Prong One: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes. Claim 5 recites the following abstract ideas:
“quality check assertions generated by the LLM” – This constitutes a concept performed in the human mind, such as an observation, evaluation, judgement, and/or opinion. Furthermore, the courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). See MPEP 2106.04(a)(2).
“generating…a first prompt based on the user request” – This constitutes a concept performed in the human mind, such as an observation, evaluation, judgement, and/or opinion. Furthermore, the courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). See MPEP 2106.04(a)(2).
Step 2A, Prong Two: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. Claim 5 recites the following additional elements:
“by an LLM management application…wherein the LLM management application is configured to…wherein the generation of submission of the first input is executed by the LLM management application” – This amounts to mere application of the exception on a computer, i.e. a recitation of the words “apply it”. Implementing an abstract idea on a generic computer does not integrate the abstract idea into a practical application in Step 2A Prong Two or add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea of intermediated settlement on a generic computer. See MPEP 2106.05(f).
“receiving,…,a user request from a user application” – This amounts to insignificant extra solution activity in the form of mere data gathering, see MPEP 2106.05(g) and MPEP 2106.05(d), II, (i), regarding receiving or transmitting data.
“submitting,…,the first prompt to the LLM to obtain the first assertion…” – This amounts to insignificant extra solution activity in the form of mere data gathering, see MPEP 2106.05(g) and MPEP 2106.05(d), II, (i), regarding receiving or transmitting data.
“subsequent to determining that the second assertion is true: forwarding, …, the second assertion to the user application” – This amounts to insignificant extra solution activity in the form of mere data gathering, see MPEP 2106.05(g) and MPEP 2106.05(d), II, (i), regarding receiving or transmitting data.
Step 2B: Does the claim recite additional elements that amount ot significantly more than the judicial exception?
No. Claim 5 recites the following additional elements:
“by an LLM management application…wherein the LLM management application is configured to…wherein the generation of submission of the first input is executed by the LLM management application” – This amounts to mere application of the exception on a computer, i.e. a recitation of the words “apply it”. Implementing an abstract idea on a generic computer does not integrate the abstract idea into a practical application in Step 2A Prong Two or add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea of intermediated settlement on a generic computer. See MPEP 2106.05(f).
“receiving,…,a user request from a user application” – This amounts to insignificant extra solution activity in the form of mere data gathering, see MPEP 2106.05(g) and MPEP 2106.05(d), II, (i), regarding receiving or transmitting data.
“submitting,…,the first prompt to the LLM to obtain the first assertion…” – This amounts to insignificant extra solution activity in the form of mere data gathering, see MPEP 2106.05(g) and MPEP 2106.05(d), II, (i), regarding receiving or transmitting data.
“subsequent to determining that the second assertion is true: forwarding, …, the second assertion to the user application” – This amounts to insignificant extra solution activity in the form of mere data gathering, see MPEP 2106.05(g) and MPEP 2106.05(d), II, (i), regarding receiving or transmitting data.
Claim 6
Step 2A, Prong One: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes. Claim 6 recites the following abstract ideas:
“identifying a particular assertion in the second output” – This constitutes a concept performed in the human mind, such as an observation, evaluation, judgement, and/or opinion. Furthermore, the courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). See MPEP 2106.04(a)(2).
“determining,…, whether the particular assertion is false” – This constitutes a concept performed in the human mind, such as an observation, evaluation, judgement, and/or opinion. Furthermore, the courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). See MPEP 2106.04(a)(2).
“in response to determining,…, that the particular assertion is false, determining whether to generate a subsequent prompt to correct the particular assertion” – This constitutes a concept performed in the human mind, such as an observation, evaluation, judgement, and/or opinion. Furthermore, the courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). See MPEP 2106.04(a)(2).
Step 2A, Prong Two: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. Claim 6 recites the following additional elements:
“by the computing system” – This amounts to mere application of the exception on a computer, i.e. a recitation of the words “apply it”. Implementing an abstract idea on a generic computer does not integrate the abstract idea into a practical application in Step 2A Prong Two or add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea of intermediated settlement on a generic computer. See MPEP 2106.05(f).
Step 2B: Does the claim recite additional elements that amount ot significantly more than the judicial exception?
No. Claim 6 recites the following additional elements:
“by the computing system” – This amounts to mere application of the exception on a computer, i.e. a recitation of the words “apply it”. Implementing an abstract idea on a generic computer does not integrate the abstract idea into a practical application in Step 2A Prong Two or add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea of intermediated settlement on a generic computer. See MPEP 2106.05(f).
Claim 7
Step 2A, Prong One: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes. Claim 7 recites the following abstract ideas:
“determining whether to generate the subsequent prompt to correct the particular assertion comprises: determining a number of attempts the computing system has made to correct output from the LLM given an initial prompt that caused generation of the first output; comparing the number of attempts to a threshold number of attempts; generating a second prompt if it is determined that the number of attempts is less than or equal to the threshold number of attempts” – This constitutes a concept performed in the human mind, such as an observation, evaluation, judgement, and/or opinion. Furthermore, the courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). See MPEP 2106.04(a)(2).
Step 2A, Prong Two: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. Claim 7 does not recite additional elements.
Step 2B: Does the claim recite additional elements that amount ot significantly more than the judicial exception?
No. Claim 7 does not recite additional elements.
Claim 8
Step 2A, Prong One: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes. Claim 8 recites the following abstract ideas:
“in response to determining, …, to not correct the particular assertion, generating a response that (a) includes one or more assertions, in the plurality of assertions, that have been determined to be true and (b) excludes the particular assertion.” – This constitutes a concept performed in the human mind, such as an observation, evaluation, judgement, and/or opinion. Furthermore, the courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). See MPEP 2106.04(a)(2).
Step 2A, Prong Two: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. Claim 8 recites the following additional elements:
“by the computing system” – This amounts to mere application of the exception on a computer, i.e. a recitation of the words “apply it”. Implementing an abstract idea on a generic computer does not integrate the abstract idea into a practical application in Step 2A Prong Two or add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea of intermediated settlement on a generic computer. See MPEP 2106.05(f).
Step 2B: Does the claim recite additional elements that amount ot significantly more than the judicial exception?
No. Claim 8 recites the following additional elements:
“by the computing system” – This amounts to mere application of the exception on a computer, i.e. a recitation of the words “apply it”. Implementing an abstract idea on a generic computer does not integrate the abstract idea into a practical application in Step 2A Prong Two or add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea of intermediated settlement on a generic computer. See MPEP 2106.05(f).
Claim 9
Step 2A, Prong One: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes. Claim 9 recites the following abstract ideas:
“n response to determining, …, to not correct the particular assertion, generating a response that includes (1) one or more assertions, in the plurality of assertions, that have been determined to be true and (2) data that indicates that the particular assertion is false.” – This constitutes a concept performed in the human mind, such as an observation, evaluation, judgement, and/or opinion. Furthermore, the courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). See MPEP 2106.04(a)(2).
Step 2A, Prong Two: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. Claim 9 recites the following additional elements:
“by the computing system” – This amounts to mere application of the exception on a computer, i.e. a recitation of the words “apply it”. Implementing an abstract idea on a generic computer does not integrate the abstract idea into a practical application in Step 2A Prong Two or add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea of intermediated settlement on a generic computer. See MPEP 2106.05(f).
Step 2B: Does the claim recite additional elements that amount ot significantly more than the judicial exception?
No. Claim 9 recites the following additional elements:
“by the computing system” – This amounts to mere application of the exception on a computer, i.e. a recitation of the words “apply it”. Implementing an abstract idea on a generic computer does not integrate the abstract idea into a practical application in Step 2A Prong Two or add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea of intermediated settlement on a generic computer. See MPEP 2106.05(f).
Claim 10
Step 2A, Prong One: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes. Claim 10 recites the following abstract ideas:
“performing, …, a clustering technique to generate, from the plurality of assertions, a plurality of clusters of assertions; for each cluster in the plurality of clusters: selecting a strict subset of the assertions in said each cluster; determining whether each assertion in the strict subset of the assertions is true.” – This constitutes a concept performed in the human mind, such as an observation, evaluation, judgement, and/or opinion. Furthermore, the courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). See MPEP 2106.04(a)(2).
Step 2A, Prong Two: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. Claim 10 recites the following additional elements:
“by the computing system” – This amounts to mere application of the exception on a computer, i.e. a recitation of the words “apply it”. Implementing an abstract idea on a generic computer does not integrate the abstract idea into a practical application in Step 2A Prong Two or add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea of intermediated settlement on a generic computer. See MPEP 2106.05(f).
Step 2B: Does the claim recite additional elements that amount ot significantly more than the judicial exception?
No. Claim 10 recites the following additional elements:
“by the computing system” – This amounts to mere application of the exception on a computer, i.e. a recitation of the words “apply it”. Implementing an abstract idea on a generic computer does not integrate the abstract idea into a practical application in Step 2A Prong Two or add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea of intermediated settlement on a generic computer. See MPEP 2106.05(f).
Claim 11
The analysis for claim 11 follows that of claim 1
Step 2A, Prong Two: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. Claim 11 recites the following additional elements beyond those of claim 1:
“One or more non-transitory storage media storing instructions which, when executed by one or more computing devices, cause…” – This amounts to mere application of the exception on a computer, i.e. a recitation of the words “apply it”. Implementing an abstract idea on a generic computer does not integrate the abstract idea into a practical application in Step 2A Prong Two or add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea of intermediated settlement on a generic computer. See MPEP 2106.05(f).
Step 2B: Does the claim recite additional elements that amount ot significantly more than the judicial exception?
No. Claim 11 recites the following additional elements beyond those of claim 1:
“One or more non-transitory storage media storing instructions which, when executed by one or more computing devices, cause…” – This amounts to mere application of the exception on a computer, i.e. a recitation of the words “apply it”. Implementing an abstract idea on a generic computer does not integrate the abstract idea into a practical application in Step 2A Prong Two or add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea of intermediated settlement on a generic computer. See MPEP 2106.05(f).
Claims 12-20 are ineligible under the same analysis of their corresponding claims in claims 2-10.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-3, 5-6, and 8-9 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Peng (Peng et al., arXiv:2302.12813v3 [cs.CL]).
Regarding claim 1, Peng teaches a method comprising:
accessing, by a computing system, first output generated by a large language model (LLM) (§1, “As illustrated by the example in Figure 1, given a user query (e.g., regarding a 2013 Los Angeles Galaxy player transfer), LLM-AUGMENTER first retrieves evidence from external knowledge (e.g., Web or task-specific datasets) and, if necessary, further consolidates evidence by linking retrieved raw evidence with related context (e.g., information of the entity “2013 Los Angeles Galaxy”) and performing reasoning to form evidence chains (e.g., table-passage in the figure). Then, LLMAUGMENTER queries a fixed LLM (i.e., ChatGPT in our study) using a prompt that contains the consolidated evidence for ChatGPT to generate a candidate response [accessing, by a computing system, first output generated by a large language model (LLM)] grounded in external knowledge (evidence). LLM-AUGMENTER then verifies the candidate response e.g., by checking whether it hallucinates evidence. If so, LLM-AUGMENTER generates a feedback message (e.g., about the team “C.S.D. Municipal”). The message is used to revise the prompt to query ChatGPT again. The process iterates until a candidate response passes the verification and is sent to the user.”);
identifying, by the computing system, within the first output, a plurality of assertions (See Annotated Figure 1, below. The portions in red and green can be considered assertions, also see §1, “LLM-AUGMENTER then verifies the candidate response e.g., by checking whether it hallucinates evidence [identifying, by the computing system, within the first output, a plurality of assertions]. If so, LLM-AUGMENTER generates a feedback message (e.g., about the team “C.S.D. Municipal”). The message is used to revise the prompt to query ChatGPT again. The process iterates until a candidate response passes the verification and is sent to the user.”);
determining, by the computing system, that a first assertion in the plurality of assertions is false (§1, “If so, LLM-AUGMENTER generates a feedback message (e.g., about the team “C.S.D. Municipal”) [determining, by the computing system, that a first assertion in the plurality of assertions is false]. The message is used to revise the prompt to query ChatGPT again. The process iterates until a candidate response passes the verification and is sent to the user.”)
generating, by the computing system, a prompt that indicates that the first assertion is false (see Figure 1, “Feedback”, §1, “If so, LLM-AUGMENTER generates a feedback message (e.g., about the team “C.S.D. Municipal”) [generating, by the computing system, a prompt that indicates that the first assertion is false]. The message is used to revise the prompt to query ChatGPT again. The process iterates until a candidate response passes the verification and is sent to the user.”)
submitting, by the computing system, the prompt as input to the LLM (§1, “The message is used to revise the prompt to query ChatGPT again. [submitting, by the computing system, the prompt as input to the LLM]. The process iterates until a candidate response passes the verification and is sent to the user.”)
accessing, by the computing system, second output generated by the LLM (Figure 1, “Revised candidate response” [accessing, by the computing system, second output generated by the LLM])
wherein the second output includes a second assertion that is different than the first assertion and corresponds to the first assertion (Figure 1, “Revised candidate response”, “…Sao Paulo…” [a second assertion that is different than the first assertion and corresponds to the first assertion] – The correspondence is in that both assertions are the names of football teams); wherein
the method is performed by one or more computing devices (Figure 2 [one or more computing devices]).
PNG
media_image1.png
778
738
media_image1.png
Greyscale
Regarding claim 2, Peng teaches all of the limitations of claim 1, further comprising:
determining, by the computing system, that the second output generated by the LLM is free from false assertions (§1, “The process iterates until a candidate response passes the verification and is sent to the user [the second output generated by the LLM is free from false assertions]); and in response to determining that the second output is free from false assertions, (a) storing the second output in persistent memory or (b) causing a portion of the first output and the second assertion to be transmitted to a requesting entity that provided, to the LLM, initial input that caused the LLM to generate the first output (see Figure 1, which shows the output at the bottom but with C.S.D. Municipal replaced with Sao Paulo [causing a portion of the first output and the second assertion to be transmitted], furthermore Figure 1 shows a user prompt on top and the LLM response on bottom [transmitted to a requesting entity that provided, to the LLM, the initial input that caused the LLM to generate the first output]).
Regarding claim 3, Peng teaches all of the limitations of claim 2, further comprising: in response to determining that the second output is free from false assertions:
replacing, in the first output, the first assertion with the second assertion to generate modified first output; causing the modified first output to be transmitted to the requesting entity (see Figure 1, which shows the output at the bottom but with C.S.D. Municipal replaced with Sao Paulo [replacing, in the first output, the first assertion with the second assertion to generate modified first output], furthermore Figure 1 shows a user prompt on top and the LLM response on bottom [causing the modified first output to be transmitted to the requesting entity]).
Regarding claim 5, Peng teaches all of the limitations of claim 1, further comprising: prior to accessing the first output: receiving, by an LLM management application, a user request from a user application, wherein the LLM management application is configured to quality check assertions generated by the LLM; generating, by the LLM management application, a first prompt based on the user request; submitting, by the LLM management application, the first prompt to the LLM to obtain the first assertion, wherein the generation and submission of the first input is executed by the LLM management application; and subsequent to determining that the second assertion is true: forwarding, by the LLM management application, the second assertion to the user application (§1, “As illustrated by the example in Figure 1, given a user query [receiving, by an LLM management application, a user request from a user application] (e.g., regarding a 2013 Los Angeles Galaxy player transfer), LLM-AUGMENTER first retrieves evidence from external knowledge (e.g., Web or task-specific datasets) and, if necessary, further consolidates evidence by linking retrieved raw evidence with related context (e.g., information of the entity “2013 Los Angeles Galaxy”) and performing reasoning to form evidence chains (e.g., table-passage in the figure). Then, LLMAUGMENTER queries a fixed LLM (i.e., ChatGPT in our study) using a prompt that contains the consolidated evidence [generating, by the LLM management application, a first prompt based on the user request] for ChatGPT to generate a candidate response grounded in external knowledge (evidence) [submitting, by the LLM management application, the first prompt to the LLM to obtain the first assertion, wherein the generation and submission of the first input is executed by the LLM management application]. LLM-AUGMENTER then verifies the candidate response e.g., by checking whether it hallucinates evidence [the LLM management application is configured to quality check assertions generated by the LLM]. If so, LLM-AUGMENTER generates a feedback message (e.g., about the team “C.S.D. Municipal”). The message is used to revise the prompt to query ChatGPT again. The process iterates until a candidate response passes the verification and is sent to the user. [subsequent to determining that the second assertion is true: forwarding, by the LLM management application, the second assertion to the user application]”).
Regarding claim 6, Peng teaches all of the limitations of claim 1, further comprising: identifying a particular assertion in the second output; determining, by the computing system, whether the particular assertion is false; in response to determining, by the computing system, that the particular assertion is false, determining whether to generate a subsequent prompt to correct the particular assertion (§1, “LLM-AUGMENTER then verifies the candidate response e.g., by checking whether it hallucinates evidence [identifying a particular assertion in the second output; determining, by the computing system, whether the particular assertion is false]. If so, LLM-AUGMENTER generates a feedback message (e.g., about the team “C.S.D. Municipal”). The message is used to revise the prompt to query ChatGPT again. The process iterates until a candidate response passes the verification and is sent to the user. [in response to determining, by the computing system, that the particular assertion is false, determining whether to generate a subsequent prompt to correct the particular assertion]”).
Further regarding claim 6, Peng discloses “determining whether the particular assertion is false” as shown above. Per MPEP 2111.04, II, the broadest reasonable interpretation of a method claim does not include steps which are contingent or are not necessarily required/performed. In claim 6, “in response to determining, by the computing system, that the particular assertion is false, determining whether to generate a subsequent prompt to correct the particular assertion” is contingent on the determination that the assertion is false. In the case that the assertion is not false, i.e. true, the steps of claim 6 are not executed and are not required under the broadest reasonable interpretation of the claim. Therefore, by disclosing “determining whether the particular assertion is false”, Peng discloses all of claim 6 under its broadest reasonable interpretation.
Regarding claims 8-9, the limitations are contingent on determining to not correct the particular assertion. In the instant that the particular assertion is determined to be corrected, the steps of claims 8-9 are not performed and therefore not required under the broadest reasonable interpretation of the claim. See MPEP 2111.04, II. Therefore, Peng according to claim 6 discloses all of the required limitations of claims 8-9.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 4 and 11-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Peng (Peng et al., arXiv:2302.12813v3 [cs.CL]) in view of Gunasekara (US20230146979A1).
Regarding claim 4, Peng teaches all of the limitations of claim 1 but does not teach the method of Claim 1 further comprising: storing, in a database, an association between the first assertion and the second assertion; retraining the LLM based on the association.
Gunasekara teaches storing, in a database, an association between the first assertion and the second assertion (Figure 5, step 4, “Calculate the reward…by the similarity between AGa’ and AGa” [storing, in a database, an associated between the first assertion and the second assertion]) and retraining the LLM based on the association (Figure 4: 414 to 406 [retraining the LLM based on the association]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Peng to include storing, in a database, an association between the first assertion and the second assertion; retraining the LLM based on the association in order to improve the LLM’s answers/outputs.
Regarding claim 11, Peng according to claim 1 teaches the functionality and methodology of claim 11 but does not teach one or more non-transitory storage media storing instructions which, when executed by one or more computing devices, cause the methodology to be carried out.
Gunasekara teaches one or more non-transitory storage media storing instructions which, when executed by one or more computing devices, cause the methodology for an LLM to be carried out ([0073] Memory 802 and persistent storage 805 are computer readable storage media [non-transitory storage media]. In this embodiment, memory 802 includes random access memory (RAM). In general, memory 802 may include any suitable volatile or non-volatile computer readable storage media. Cache 803 is a fast memory that enhances the performance of processors 801 by holding recently accessed data, and data near recently accessed data, from memory 802. [0074] Program instructions and data used to practice embodiments of the present invention may be stored in persistent storage 805 and in memory 802 for execution by one or more of the respective processors 801 via cache 803 [storing instructions which, when executed by one or more computing devices]…”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to provide one or more non-transitory storage media storing instructions which, when executed by one or more computing devices, cause the methodology of Peng to be carried out in order to provide a means for realizing the methodology of Peng.
Regarding claims 12-13, 15, and 16, Peng as modified teaches all of the limitations of claims 12-13, 15, and 16 in the same manner as claims 2-3, 5, and 6 under Peng.
Regarding claim 14, Peng as modified teaches all of the limitations of claim 14 for the same reasons as claim 4.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Peng (Peng et al., arXiv:2302.12813v3 [cs.CL]) in view of Wang (Wang et al., arXiv:2306.06427v2 [cs.CL])
Regarding claim 7, Peng teaches all of the limitations of claim 6, but does not teach wherein determining whether to generate the subsequent prompt to correct the particular assertion comprises: determining a number of attempts the computing system has made to correct output from the LLM given an initial prompt that caused generation of the first output; comparing the number of attempts to a threshold number of attempts; generating a second prompt if it is determined that the number of attempts is less than or equal to the threshold number of attempts.
Wang teaches wherein determining whether to generate the subsequent prompt to correct the particular assertion comprises: determining a number of attempts the computing system has made to correct output from the LLM given an initial prompt that caused generation of the first output; comparing the number of attempts to a threshold number of attempts; generating a second prompt if it is determined that the number of attempts is less than or equal to the threshold number of attempts (§3, “This dynamic generate-evaluate procedure continues until all entries in U are considered reliable or the maximum number of iterations N [determining a number of attempts the computing system has made to correct output from the LLM given an initial prompt that caused generation of the first output] is reached [comparing the number of attempts to a threshold number of attempts]. For cases where the maximum number of iterations is reached without any triples’ reliability score surpassing θ, triples with the highest reliability scores will be selected for inference (Line 15-17, Algorithm 1) [generating a second prompt if it is determined that the number of attempts is less than or equal to the threshold number of attempts]” – To clarify, when θ is surpassed, that is a determination that the number of attempts is less than or equal to the threshold number of attempts because the action taken when the threshold is reached is not taken).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include determining a number of attempts the computing system has made to correct output from the LLM given an initial prompt that caused generation of the first output; comparing the number of attempts to a threshold number of attempts; generating a second prompt if it is determined that the number of attempts is less than or equal to the threshold number of attempts in Peng as taught by Wang in order to provide contingency for if the iterative procedure exceeds a threshold number of iterations.
Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Peng (Peng et al., arXiv:2302.12813v3 [cs.CL]) in view of Gunasekara (US20230146979A1), further in view of Wang (Wang et al., arXiv:2306.06427v2 [cs.CL]).
Regarding claim 17, Peng as modified teaches all of the limitations of claim 16, but does not teach wherein determining whether to generate the subsequent prompt to correct the particular assertion comprises: determining a number of attempts the computing system has made to correct output from the LLM given an initial prompt that caused generation of the first output; comparing the number of attempts to a threshold number of attempts; generating a second prompt if it is determined that the number of attempts is less than or equal to the threshold number of attempts.
Wang teaches wherein determining whether to generate the subsequent prompt to correct the particular assertion comprises: determining a number of attempts the computing system has made to correct output from the LLM given an initial prompt that caused generation of the first output; comparing the number of attempts to a threshold number of attempts; generating a second prompt if it is determined that the number of attempts is less than or equal to the threshold number of attempts (§3, “This dynamic generate-evaluate procedure continues until all entries in U are considered reliable or the maximum number of iterations N [determining a number of attempts the computing system has made to correct output from the LLM given an initial prompt that caused generation of the first output] is reached [comparing the number of attempts to a threshold number of attempts]. For cases where the maximum number of iterations is reached without any triples’ reliability score surpassing θ, triples with the highest reliability scores will be selected for inference (Line 15-17, Algorithm 1) [generating a second prompt if it is determined that the number of attempts is less than or equal to the threshold number of attempts]” – To clarify, when θ is surpassed, that is a determination that the number of attempts is less than or equal to the threshold number of attempts because the action taken when the threshold is reached is not taken).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include determining a number of attempts the computing system has made to correct output from the LLM given an initial prompt that caused generation of the first output; comparing the number of attempts to a threshold number of attempts; generating a second prompt if it is determined that the number of attempts is less than or equal to the threshold number of attempts in Peng as taught by Wang in order to provide contingency for if the iterative procedure exceeds a threshold number of iterations.
Allowable Subject Matter
Claims 10 and 18-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claim 10, the clustering technique described would not have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention absent impermissible hindsight. None of the prior art cited indicates a clustering technique and the prior art at-large does not provide a convincing prima facie case of obviousness.
Regarding claims 18-19, the prior art does not anticipate claims 18-19. Furthermore, the prior art does not render obvious the determination to not correct the particular assertion and the subsequent response generation indicated in claims 18-19. The prior art is generally concerned with correcting false assertions, and therefore arriving at claims 18-19 from the prior art would require impermissible hindsight reasoning.
Claim 20 contains allowable subject matter under the same rational as claim 10.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Jha, Susmit, et al. "Dehallucinating large language models using formal methods guided iterative prompting." 2023 IEEE International Conference on Assured Autonomy (ICAA). IEEE, 2023. → Disclosure of correcting hallucinations in LLMs.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SCHYLER S SANKS whose telephone number is (571)272-6125. The examiner can normally be reached 06:30 - 15:30 Central Time, M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SCHYLER S SANKS/Primary Examiner, Art Unit 2129
Prosecution Insights