DETAILED ACTION
This is responsive to the amendment filed 02 February 2026.
Claims 1-20 remain pending and are considered below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 02 February 2026 have been fully considered but they are not persuasive.
Applicant argues:
The Applicant respectfully submits that the Applicant's claims 1-20 cannot be practically performed in the mind and as such that claims 1-20 comply with the provisions of 35 U.S.C. §101 and are patentable thereunder. More specifically, the Applicant submits that it would be impractical for a human, with or without the assistance of pen and paper, to determine a conceptual consistency score for the LLM by analyzing a plurality of background knowledge scores for a plurality of respective anchor tasks to determine a relationship between the background knowledge scores and the respective anchor task scores that enables predicting an anchor task score from a respective background knowledge score of the LLM. More specifically, in at least paragraphs [0049] - [0055] and Figure 4 of the Applicant's Specification, the Applicant teaches that to determine a conceptual consistency score for an LLM, the conceptual consistency determination system of the present principles analyzes a plurality of background knowledge scores for a plurality of respective anchor tasks to determine a relationship between the background knowledge scores and the respective anchor task scores to learn how to predict an anchor task score from a known background knowledge score.
The Examiner respectfully disagrees. A person may practically determine a conceptual consistency score for the LLM by analyzing a plurality of background knowledge scores for a plurality of respective anchor tasks to determine a relationship between the background knowledge scores and the respective anchor task scores that enables predicting an anchor task score from a respective background knowledge score of the LLM (e.g. a human may determine a third score by determining a relationship between scores and their corresponding tasks which enables predicting a forth score from a fifth score). Further, although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims.
Applicant also argues:
Specifically, in paragraph [0049] the Applicant teaches that, in one embodiment, the determined relationship between the background knowledge scores and the respective anchor task scores for determining the conceptual consistency is an average precision of sets of background knowledge scores and anchor task scores which includes what anchor task scores result from respective background knowledge scores as reflected in Equation 3. In explaining the determination of the conceptual consistency for an LLM in accordance with the present principles, the Applicant in paragraph [0055] teaches that conceptual consistency (Equation 3) for each LLM is determined and the results are shown in Figure 4, which shows/measures the ability to predict whether an LLM will be correct from its knowledge of relevant background information. That is, the Applicant teaches the ability to determine an anchor task score from a respective background knowledge score. With respect to the results shown in Figure 4, the Applicant teaches that in the embodiment of Figure 4, the plurality of parameters used to determine conceptual consistency for an LLM is between 108 and 1011 parameters.
In response to applicant's argument regarding certain features of the invention, it is noted that the features upon which applicant relies (i.e., average precision, Equation 3, plurality of parameters) are not recited in the rejected claims. Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
Furthermore, the claimed methods are not rendered patent eligible by the fact that (using existing machine learning technology) they perform a task previously undertaken by humans with greater speed and efficiency than could previously be achieved. We have consistently held, in the context of computer-assisted methods, that such claims are not made patent eligible under § 101 simply because they speed up human activity. See, e.g., Content Extraction, 776 F.3d at 1347; DealerTrack, 674 F.3d at 1333. Whether the issue is raised at step one or step two, the increased speed and efficiency resulting from use of computers (with no improved computer techniques) do not themselves create eligibility. See, e.g., Trinity Info Media, LLC v. Covalent, Inc., 72 F.4th 1355, 1363 (Fed. Cir. 2023) (rejecting argument that “humans could not mentally engage in the ‘same claimed process' because they could not perform ‘nanosecond comparisons' and aggregate ‘result values with huge numbers of polls and members' ”) (internal citation omitted); Customedia Techs., LLC v. Dish Network Corp., 951 F.3d 1359, 1365 (Fed. Cir. 2020) (holding claims abstract where “[t]he only improvements identified in the specification are generic speed and efficiency improvements inherent in applying the use of a computer to any task”)
Applicant further argues:
Even further, the Applicant respectfully submits that the Applicant's claims provide significantly more, and as such are patentable under 35 USC 101. More specifically, the Applicant's claims add something beyond what is well-understood, routine, and conventional in the field. That is, the Applicant submits that at least the Applicant's technical features of at least the Applicant's independent claims including "determining a background knowledge score and an anchor task score based on the LLM's performance for the given anchor task" and "determining a conceptual consistency score for the LLM by analyzing a plurality of background knowledge scores for a plurality of respective anchor tasks to determine a relationship between the background knowledge scores and the respective anchor task scores that enables predicting an anchor task score from a respective background knowledge score of the LLM" add something beyond what is well-understood, routine, and conventional in the field. In fact, in the Office Action the Examiner does not cite any references for rejecting the Applicant's claims and as such concedes that the Examiner concedes that the Applicant's claims add something beyond what is well-understood, routine, and conventional in the field. Specifically, the Applicant in at least paragraph [0052] teaches that at least one advantage of determining a conceptual consistency score 126 in accordance with the present principles is that a large model, such as an LLM, will become explainable in a way that allows developers to use and steer them more precisely based on conceptual knowledge. That is, the Applicant claims a method, apparatus, and system for measuring conceptual consistency of an LLM that add something beyond what is well-understood, routine, and conventional in the field and improves the application of an LLM because the conceptual consistency of the present principles provides a measure of accuracy for the LLM when applied to specific anchor tasks.
However, novelty does not necessarily imply eligibility under 35 USC 101. Courts have held “that merely reciting an abstract idea by itself in a claim—even if the idea is novel and non-obvious—is not enough to save it from ineligibility.”
Therefore, all of Applicant’s arguments have been addressed and they are not persuasive.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
In line 12, claim 1 recites the limitation “the background knowledge scores”. It is unclear if this limitation refers back exclusively to the “plurality of background knowledge scores” of lines 10-11 or if it also includes the “background knowledge score” of line 8. Claims 11 and 20 recite similar limitations and are likewise rejected.
The dependent claims are rejected for depending upon a rejected claim without providing a remedy.
Further, claim 2 in line 2 recites the limitation “the anchor task score”. It is unclear if this limitation refers back to the anchor task score of line 8 of parent claim 1 or the one in line 13. Claims 10 and 19 recite similar limitations and are likewise rejected.
Also, claim 2 in line 3 recites the limitation “the background score”. It is unclear if this limitation refers back to the background score of line 8 of parent claim 1 or one of the background scores in line 12. Claims 9 and 19 recite similar limitations and are likewise rejected.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Further, the judicial exception is not integrated into a practical application.
In claims 1, 11 and 20, the limitations known background knowledge facts for a given anchor task associated with known answers with the extracted LLM background knowledge facts to determine an LLM performance; determining a background knowledge score and an anchor task score based on the LLM's performance for the given anchor task; and determining a conceptual consistency score for the LLM by analyzing a plurality of background knowledge scores for a plurality of respective anchor tasks to determine a relationship between the background knowledge scores and the respective anchor task scores that enables predicting an anchor task score from a respective background knowledge score of the LLM; and
That is, other than reciting a “large language model (LLM)” (claims 1, 9 and 20), a “system for measuring conceptual consistency of a large language model (LLM), the system comprising: a prompting system …; a LLM performance evaluation module …; and a LLM conceptual consistency evaluation module” (claim 11) and a “non-transitory computer readable storage medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations” (claim 20) nothing in the claims precludes the steps from practically being performed in the mind. For example, a person may compare known background knowledge facts for a given anchor task associated with known answers with the extracted LLM background knowledge facts to determine an LLM performance (e.g. a human may compare an LLM’s output with pre-stored information to determine an LLM performance); determine a background knowledge score and an anchor task score based on the LLM's performance (e.g. a human may compute a first score and a second score based on the LLM performance); and determine a conceptual consistency score for the LLM by analyzing a plurality of background knowledge scores for a plurality of respective anchor tasks to determine a relationship between the background knowledge scores and the respective anchor task scores that enables predicting an anchor task score from a respective background knowledge score of the LLM (e.g. a human may determine a third score by determining a relationship between other scores and their corresponding tasks which enables predicting a forth score from a fifth score).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claims recite an abstract idea.
This judicial exception is not integrated into a practical application. In particular, the claims recite the additional elements – a “large language model (LLM)” (claims 1, 9 and 20), a “system for measuring conceptual consistency of a large language model (LLM), the system comprising: a prompting system …; a LLM performance evaluation module …; and a LLM conceptual consistency evaluation module” (claim 11) and a “non-transitory computer readable storage medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations” (claim 20) which are recited at a high-level of generality (i.e., as generic processors performing generic computer functions) such that they amount to no more than mere instructions to apply the exception using a generic computer components.
The claims also recite the additional elements “prompting the LLM in order to extract LLM background knowledge facts to background queries and anchor tasks”, and “outputting an indication of the conceptual consistency score”. The claims do not impose any limits on how the LLM is prompted or how the indication is output. In other words, the claims recite only the idea of a solution or outcome i.e., the claims fail to recite details of how a solution to a problem is accomplished. These limitations therefore represent extra-solution activity because they are mere nominal or tangential addition to the claims. Accordingly, the additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claims are therefore directed to an abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as an ordered combination do not amount to significantly more than the abstract idea. As stated above, the claims recite the additional limitations of a “large language model (LLM)” (claims 1, 9 and 20), a “system for measuring conceptual consistency of a large language model (LLM), the system comprising: a prompting system …; a LLM performance evaluation module …; and a LLM conceptual consistency evaluation module” (claim 11) and a “non-transitory computer readable storage medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations” (claim 20). However, these are recited at a high level of generality and are recited as performing generic computer functions routinely used in computer applications (see Applicant’s specification [0062]-[0066] and [0069]-[0071]). Generic computer components recited as performing generic computer functions that are well-understood, routine and conventional activities amount to no more than implementing the abstract idea with a computerized system.
The claims also recite the additional elements “prompting the LLM in order to extract LLM background knowledge facts to background queries and anchor tasks”, and “outputting an indication of the conceptual consistency score”. The claims do not impose any limits on how the LLM is prompted or how the indication is output. In other words, the claims recite only the idea of a solution or outcome i.e., the claims fail to recite details of how a solution to a problem is accomplished. These limitations represent the extra-solution activity of querying a model to get an output and outputting data which are well-understood, routine and conventional activities. Thus, taken alone, the additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide conventional computer implementation. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claims are not patent eligible.
The dependent claims, when analyzed as a whole, are held to be patent ineligible under 35 U.S.C. 101 because the additional recited limitations fail to establish that the claims are not directed to an abstract idea.
The dependent claims recite:
wherein the conceptual consistency score is a measure of an average precision of an ability to predict the anchor task score based on the background score.
wherein the LLM background knowledge facts extracted from the LLM are each represented by a tuple including at least two concepts and a relation between those concepts;
wherein for each fact tuple, concepts and relation are transformed into a question using a natural language template of questions for the relation;
wherein for two different concepts in the LLM, a cloud of relational information is formed from all tuples from all paths length L or less which connect those concepts in the LLM and forms the background knowledge for the anchor query provided by the LLM;
wherein the facts extracted from the LLM includes positive background knowledge and negative background knowledge;
wherein prompting the LLM includes using a zero-shot prompting approach.
wherein prompting the LLM includes varying questions presented to the LLM by substituting the question generated into a plurality of meta-prompts, wherein meta-prompts are variations on how to ask the question;
wherein the background knowledge score is a measure of how good the LLM is at verifying whether the extracted facts are true or false;
wherein the anchor task score is a measure of how good the LLM is answering questions through zero shot prompting.
The additional recited limitations further narrow the steps of the independent claims without however providing “a practical application of” or "significantly more than" the underlying “Mental Processes” abstract idea. Therefore, the dependent claims are also not patent eligible.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SAMUEL G NEWAY whose telephone number is (571)270-1058. The examiner can normally be reached Monday-Friday 9:00am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached at 571-272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SAMUEL G NEWAY/ Primary Examiner, Art Unit 2657