CTNF 18/958,378 CTNF 101884 Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. This office action is sent in response to Applicant’s communication received on 11/25/2024 for the application number 18958378. The office hereby acknowledges receipt of the following placed of record in the file: Specification, Abstract, Oath/Declaration and claims. Status of the claims Claims 1-20 are presented for examination. Information Disclosure Statement 06-52 The information disclosure statement (IDS) submitted on 11/25/2024 was filed before the mailing date of the first office action. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner. Claim Rejections - 35 USC § 112 07-34-01 Claim 7, 8, 9, 10, 16, 17, 18, 19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Specifically claim 7 & 16 recite: “wherein the features include one or more of a word frequency similarity score, one or more ROUGE scores, an adherence score between the question and a context, an adherence score between the answer and the context, adherence scores between each sentence in the answer and each source in the context, adherence scores between each sentence and all sources in the context, textual entailment scores between the context and the answer.” It is unclear whether the recites features are intended to be grouped conjunctively or disjunctively because the claim lacks sufficient punctuation and/or conjunctions. For purposes of examination, claim 7 is interpreted as requiring any one of the recited features. Claims 8-10 are likewise rejected under 35 U.S.C. 112(b) because they depend directly or indirectly from claim 7 and therefore include the same indefinite language discussed above. Claims 17-19 are likewise rejected under 35 U.S.C. 112(b) because they depend directly or indirectly from claim 16 and therefore include the same indefinite language discussed above. Claim Rejections - 35 USC § 101 07-04-01 AIA 07-04 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because as explained below. Claim 1 recites a method comprising: receiving an answer to a question from a retrieval augmented generation system into a hallucination detection system; checking the answer with predefined rules calculating feature values for the answer and determining a score for the answer using a model, wherein the score is a probability of whether the answer is a hallucination. Step (a) comprises a mental process. This step can be performed by a human as a person can receive an answer from another person and consider the answer. Step (b) comprises a mental process. This step can be performed by a human as a person can apply a variety of predefined rules or criteria. Step (c) comprises a mental process. This step can be performed by a human as a person can calculate values related to specific features of an answer. Step (d) comprises a mental process. This step can be performed by a human as a person can evaluate information and assign a score, likelihood, or probability regarding whether an answer is incorrect, unsupported, or hallucinated based on characteristics of the answer. Step 1: This part of the eligibility analysis evaluates whether the claim falls within any statutory category. See MPEP 2106.03. The claim recites at least method. Thus, the claim is a method, which is one of the statutory categories of invention. (Step 1: YES). Step 2A, Prong One: This part of the eligibility analysis evaluates whether the claim recites a judicial exception. As explained in MPEP 2106.04, subsection II, a claim “recites” a judicial exception when the judicial exception is “set forth” or “described” in the claim. As discussed above, the broadest reasonable interpretation of steps (a)-(d) recites a mental process. Specifically, step (a) can be performed by a human as a person can receive an answer from another person. Step (b) can be performed by a human as a person can apply a variety of predefined rules or criteria. Step (c) can be performed by a human as a person can calculate values related to specific features of an answer. Step (d) can be performed by a human as a person can evaluate information and assign a score, likelihood, or probability regarding whether an answer is incorrect, unsupported, or hallucinated based on characteristics of the answer. Hence the claim encompasses mental processes practically performed in the human mind by observation, evaluation, judgement, and opinion. See MPEP 2106.04(a)(2), subsection III. (Step 2A, Prong One: YES). Step 2A, Prong Two: This part of the eligibility analysis evaluates whether the claim as a whole integrates the recited judicial exception into a practical application of the exception or whether the claim is “directed to” the judicial exception. This evaluation is performed by (1) identifying whether there are any additional elements recited in the claim beyond the judicial exception, and (2) evaluating those additional elements individually and in combination to determine whether the claim as a whole integrates the exception into a practical application. See MPEP 2106.04(d). The claim recited additional elements including a retrieval augmented generation system, and a hallucination detection system. However, these elements are recited at a high level of generality and perform generic computer functions, such as receiving data, processing data, generating content, and providing output. The use of these elements to receive an answer to a prompt, check the answer with predefined rules, calculate feature values, and determine a score merely automates the mental processes described above using generic computer components. Such implementation does not impose any meaningful limit on the judicial exception. Accordingly, these elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. (Step 2A, Prong Two: NO), and the claim is directed to the judicial exception. (Step 2A: YES) Step 2B: This part of the eligibility analysis evaluates whether the claim as a whole amounts to significantly more than the recited exception i.e., whether any additional element, or combination of additional elements, adds an inventive concept to the claim. As discussed with respect to Step 2A, Prong Two, the retrieval augmented generation system, and a hallucination detection system comprise additional elements that perform well-understood, routine, and conventional activities in the field such as receiving data, processing data, generating content, and providing output. See MPEP 2106.05(g). As known in the art these elements are well understood, routine, and conventional functions of a computing device. Even when considered in combination these additional elements merely implement the abstract idea using generic computer components and perform insignificant extra-solutional activity, which does not provide an inventive concept. The claim is not patent eligible. Claim 2 recites a mental process as a human can receive an answer and associated sources from another person. Claim 3 recites a mental process as a human can apply predefined rules to determine whether an answer is a hallucination. Claim 4 recites a mental process as a human can determine whether an answer has sufficient context to support the answer. Claim 5 recites a mental process as a human can identify generic or irrelevant text and disregard such text when evaluating an answer. Claim 6 recites a mental process as a human can determine feature values associated with an answer. Claim 7 recites a mental process as a human can compare an answer and contextual information and evaluate the relationship between them. Claim 8 recites a mental process as a human can evaluate feature values and generate an output based on those feature values. Claim 9 recites a mental process as a human can analyze examples, identify relevant features, and learn to classify answers based on those features. Claim 10 recites a mental process as a human can evaluate multiple configurations and select a preferred configuration based on performance. Claim 11 recites a mental process as a human can apply a threshold to categorize or assess a likelihood of hallucination. Claim 12 recites substantially the same limitation as claim 1. Accordingly, it is directed to the same abstract idea. Claim 13 recites substantially the same limitation as claim 2. Accordingly, it is directed to the same abstract idea. Claim 14 recites a mental process as a human can apply rules to determine whether an answer is a hallucination, evaluate whether sufficient context exists to support the answer, and identify generic or irrelevant text. Claim 15 recites substantially the same limitation as claim 6. Accordingly, it is directed to the same abstract idea. Claim 16 recites substantially the same limitation as claim 7. Accordingly, it is directed to the same abstract idea. Claim 17 recites substantially the same limitation as claim 8. Accordingly, it is directed to the same abstract idea. Claim 18 recites substantially the same limitation as claim 9. Accordingly, it is directed to the same abstract idea. Claim 19 recites substantially the same limitation as claim 10. Accordingly, it is directed to the same abstract idea. Claim 20 recites substantially the same limitation as claim 11. Accordingly, it is directed to the same abstract idea. Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-23-aia AIA The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or nonobviousness. 07-21-aia AIA Claim s 1, 3, 4, 6, 7, 8, 9, 10, 11, 12, 15, 16, 17, 18, 19, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Tiwari et al. (US 20260093931 A1) in view of Zhang et al. (US 20260072960 A1) . Regarding claim 1, Tiwari teaches receiving an answer to a question from a large language model into a hallucination detection system; checking the answer with predefined rules (Para 0016, “Determining … that one of the plurality of hallucination scores is above a hallucination score threshold; and in response … determining, by the one or more processors, the confidence score without performing remaining hallucination evaluation”); calculating feature values for the answer (Para 0014, “combining, by the one or more processors, (i) a probabilistic score … (ii) a factual validation score … (iii) a semantic context matching score… (iv) a chunk relatedness score… (v) a query-response distance score… (vi) a RAGAs score”); and determining a score for the answer using a model, wherein the score is a probability of whether the answer is a hallucination. (Para 0007, “to determine a probabilistic score based on a probability of the response to the query having a hallucination.”). Tiwari does not teach receiving an answer to a question from a retrieval augmented generation system into a hallucination detection system. However, Zhang teaches receiving an answer to a question from a retrieval augmented generation system into a hallucination detection system. (Para 0026, “One or more embodiments evaluate the effectiveness of the response generation function of a RAG system.” And “These questions may indicate if the response is grounded in the document, or if the RAG system is experiencing hallucination.”). It would have been obvious to a person of ordinary skill in the art to modify Tiwari before the effective filing date in such a way as to incorporate the teachings of Zhang in order to retrieve answers from a retrieval augmented generation system because Zhang teaches that retrieved contextual information improves the accuracy of generated responses. (Para 0026). Regarding claim 3, Tiwari teaches wherein the predefined rules comprise hard rules configured to determine whether the answer is a hallucination or not a hallucination (Tiwali 0072, “Instead, the query classification module 308 a evaluates features of the query using a set of rules to determine the probabilistic score.”), wherein the hallucination detection system outputs a score of 100% hallucination when the answer satisfies at least one of the hard rules. (Para 0072, “For example, the query classification module 308 a may assign a probability of 0 to queries that request numerical or factual information and may assign a probability of 1 to queries that do not” wherein the probability of 1 corresponds to a 100% probability that the output is a hallucination). Regarding claim 4 Tiwari teaches wherein the predefined rules comprise skipping rules configured to determine whether the answer has sufficient context ( hallucination score based on the response and factual information, Fig 5 the numerical and factual evaluation module 308 b may determine whether the numbers and specific facts in the response match with other reliable sources (e.g., sources in the database 154 ), Para 0073) , wherein the hallucination detection system outputs a score of 100% hallucination when the answer does not have sufficient context ( confidence score based on the reliability ( factual information), 0073) Although Tiwari teaches the hallucination confidence, it teaches the low confidence corresponds to hallucination. It would have been obvious to one of ordinary skill in the art to assign the maximum hallucination score when the evaluation indicated that there was insufficient context. Doing so would represent a predictable use of Tiwari’s scoring to indicate the highest level of hallucination risk. Regarding claim 6, Tiwari teaches determining feature values, wherein at least one of the feature values is determined by a model. (Para 0071 The query classification module 308 may identify different features of the query). Regarding claim 7, Tiwari teaches wherein the features include one or more of a word frequency similarity score, one or more ROUGE scores, an adherence score between the question and a context, an adherence score between the answer and the context, adherence scores between each sentence in the answer and each source in the context, adherence scores between each sentence and all sources in the context, textual entailment scores between the context and the answer. (Para 0128, “The RAGAs module 308 f may then determine a faithfulness score based on the number of statements in the response that are supported by the context.” Wherein the faithfulness score is analogous to an adherence score between the answer and the context). Regarding claim 8, Tiwari teaches inputting the features into the model to generate an output (Para 0068, “the query classification module 308 a may apply the features of the query to the query classification model to determine the probabilistic score for the query.”), wherein the output is a probability that the answer is a hallucination, wherein the model is a classification model. (Para 0068, “to determine a probabilistic score based on a probability of the response including a hallucination.”). Regarding claim 9, Tiwari teaches performing feature engineering prior to training the classification model (Para 0066, “the query classification module 308 a may analyze each of the queries in the first and second set to identify features of each query.”), and training the classification model with a hallucination dataset and the engineered features. (Para 0067, “can then train the query classification model using this dataset, so that the query classification model … can distinguish between queries that are likely to result in hallucinations and queries that are not.”). Regarding claim 10, Tiwari does not teach performing hyperparameter optimization using multiple configurations of features, models, metrics, and the hallucination dataset. However, Zhang teaches performing hyperparameter optimization using multiple configurations of features, models, metrics, and the hallucination dataset. (Para 0046, “evaluation and tuning module 128 performs continuous model tuning by using hyperparameter optimization. Evaluation and tuning module 128 performs an exploration of the hyperparameter space using algorithms, such as grid search, random search, or more sophisticated methods like Bayesian optimization.”). It would have been obvious to a person of ordinary skill in the art to modify Tiwari before the effective filing date in such a way as to incorporate the teachings of Zhang in order to improve model performance and optimize hallucination detection accuracy (Para 0046). Regarding claim 11, Tiwari teaches adjusting a threshold to change a level of hallucination. (Para 0158, “For example, if the confidence score is above a first threshold (0.85… but above a second threshold (e.g., between 0.7 and 0.85), the aggregation module 308 h may categorize the confidence score as medium.”). Claim 12 is analogous to claim 1 in that it recites substantially the same limitations. It is therefore rejected for the same reasons set forth above. Claims 15-20 are analogous to claims 6-11 in that they recite substantially the same limitations. They are therefore rejected for the same reasons set forth above . 07-21-aia AIA Claim 2, 13 are rejected under 35 U.S.C. 103 as being unpatentable over Tiwari (US 20260093931 A1) and Zhang (US 20260072960 A1) as above in claims 1, 3, 4, 6, 7, 8, 9, 10, 11, 12, 15, 16, 17, 18, 19, 20 and further in view of Cao et al. (US 20260099693 A1) . Regarding claim 2, Tiwari modified by Zhang does not teach wherein the answer includes an answer to the question and sources retrieved by the retrieval augmented generation system. However, Cao teaches wherein the answer includes an answer to the question and sources retrieved by the retrieval augmented generation system. (Para 0018, “the RAG response 128 is either verbatim identical to the LLM response 126 or modified somehow by the RAG assistant 106 (via re-formatting, addition of citations to source document(s)).” Where the response can include citations). It would have been obvious to a person of ordinary skill in the art to modify Tiwari before the effective filing date in such a way as to incorporate the teachings of Cao in order to improve transparency and verifiability of the response. (Para 0018). Claim 13 is analogous to claim 2 in that it recites substantially the same limitations. It is therefore rejected for the same reasons set forth above . 07-21-aia AIA Claim 5, 14 are rejected under 35 U.S.C. 103 as being unpatentable over Tiwari (US 20260093931 A1) and Zhang (US 20260072960 A1) as above in claims 1, 3, 4, 6, 7, 8, 9, 10, 11, 12, 15, 16, 17, 18, 19, 20 and further in view of Lok et al. (US 20250166026 A1) . Regarding claim 5, Tiwari modified by Zhang does not teach wherein the predefined rules comprise discarding rules configured to determine whether the answer includes generic text, wherein generic text is discarded from the answer when the discarding rules are satisfied. However, Lok teaches wherein the predefined rules comprise discarding rules configured to determine whether the answer includes generic text (Para 0087, “may insert a summary of the product review that LLM 110 may have generated … to … remove filler content” wherein Lok removes filler content from the output of a model), wherein generic text is discarded from the answer when the discarding rules are satisfied. (Para 0087). It would have been obvious to a person of ordinary skill in the art to modify Tiwari before the effective filing date in such a way as to incorporate the teachings of Lok in order to remove extraneous information which doesn’t contribute to the desired evaluation (Para 0087). Regarding claim 14, Tiwari teaches wherein the predefined rules comprise hard rules configured to determine whether the answer is a hallucination or not a hallucination (Para 0072, “Instead, the query classification module 308 a evaluates features of the query using a set of rules to determine the probabilistic score.”), wherein the hallucination detection system outputs a score of 100% hallucination when the answer satisfies at least one of the hard rules (Para 0072, “For example, the query classification module 308 a may assign a probability of 0 to queries that request numerical or factual information and may assign a probability of 1 to queries that do not”). wherein the predefined rules comprise skipping rules configured to determine whether the answer has sufficient context ( hallucination score based on the response and factual information, Fig 5 the numerical and factual evaluation module 308 b may determine whether the numbers and specific facts in the response match with other reliable sources (e.g., sources in the database 154 ), Para 0073) , wherein the hallucination detection system outputs a score of 100% hallucination when the answer satisfies at least one of the skipping rules ( confidence score based on the reliability ( factual information), 0073). Although Tiwari teaches the hallucination confidence, it teaches the low confidence corresponds to hallucination. It would have been obvious to one of ordinary skill in the art to assign the maximum hallucination score when the evaluation indicated that there was insufficient context. Doing so would represent a predictable use of Tiwari’s scoring to indicate the highest level of hallucination risk. Tiwari modified by Zhang does not teach wherein the predefined rules comprise discarding rules configured to determine whether the answer includes generic text, wherein generic text is discarded from the answer when the discarding rules are satisfied. However, Lok teaches wherein the predefined rules comprise discarding rules configured to determine whether the answer includes generic text (Para 0087, “may insert a summary of the product review that LLM 110 may have generated … to … remove filler content” wherein Lok removes filler content from the output of a model), wherein generic text is discarded from the answer when the discarding rules are satisfied. (Para 0087). It would have been obvious to a person of ordinary skill in the art to modify Tiwari before the effective filing date in such a way as to incorporate the teachings of Lok in order to remove extraneous information which doesn’t contribute to the desired evaluation (Para 0087). Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL ALAN FOSTER JR. whose telephone number is (571)272-8874. The examiner can normally be reached M - Th 8:00am - 6:00pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hai Phan can be reached at (571) 272-6338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /MICHAEL A FOSTER JR/Examiner, Art Unit 2654 /HAI PHAN/Supervisory Patent Examiner, Art Unit 2654 Application/Control Number: 18/958,378 Page 2 Art Unit: 2654 Application/Control Number: 18/958,378 Page 3 Art Unit: 2654 Application/Control Number: 18/958,378 Page 4 Art Unit: 2654 Application/Control Number: 18/958,378 Page 5 Art Unit: 2654 Application/Control Number: 18/958,378 Page 6 Art Unit: 2654 Application/Control Number: 18/958,378 Page 7 Art Unit: 2654 Application/Control Number: 18/958,378 Page 8 Art Unit: 2654 Application/Control Number: 18/958,378 Page 9 Art Unit: 2654 Application/Control Number: 18/958,378 Page 10 Art Unit: 2654 Application/Control Number: 18/958,378 Page 11 Art Unit: 2654 Application/Control Number: 18/958,378 Page 12 Art Unit: 2654 Application/Control Number: 18/958,378 Page 13 Art Unit: 2654 Application/Control Number: 18/958,378 Page 14 Art Unit: 2654 Application/Control Number: 18/958,378 Page 15 Art Unit: 2654 Application/Control Number: 18/958,378 Page 16 Art Unit: 2654
Prosecution Insights