DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This communication is responsive to the applicant’s amendments dated 8/5/2026. The applicant amended claims 1, 8, and 15.
Response to Arguments
Applicant's arguments with respect to 35 U.S.C. 101 (See Remarks pg. 9, line 11 – pg. 18, line 14) filed 8/5/2026 have been fully considered but they are not persuasive.
The applicant argues that “the claimed invention is directed to provide a practical application that includes a technical solution to overcome issues associated with overuse of computer resources when automatically mapping medical codes to extracted information from text in a narrative form”. The examiner fails to see how this alleged improvement is reflected in the claim language. Next, the applicant argues that the claimed invention “improves a method to obtain medical data which allows for how information can be used from a plurality of sources to build a complex network of nodes and relationships, thereby delivering a sorted list of potential paths of medical diagnosis codes and related procedural codes - in particular, main and/or secondary diagnosis codes, as well as, main procedure codes, as well as, secondary procedure codes - as a result of a query”. Once again, the examiner fails to see how this alleged improvement is reflected in the claim language. As shown below in the rejection, the examiner believes the recited claims are directed to an abstract idea. In particular a mental process which can all be done in the mind or with a pen and paper. Therefore, the 35 U.S.C. 101 rejection is maintained.
Applicant's arguments with respect to 35 U.S.C. 102 and 103 (See Remarks pg. 18, line 15 – pg. 21, line 15) filed 8/5/2026 have been fully considered but they are not persuasive.
The applicant argues hallucinations are not explored in the amended claims. While the exact term hallucination may not present be in the applicant’s claim, Bolcer uses hallucination detection as a metric to determine the accuracy of the output for LLM models, which is similar to the applicant’s claimed invention. Next, the applicant argues that Bolcer produces several outputs which are not required by the amended claims. In general, the examiner views this argument as irrelevant and insufficient. The applicant provides no citations from Bolcer to support this argument even if it was relevant. Lastly, the applicant argues that there is no real ranking in Bolcer. As shown in greater detail below, Bolcer discusses scoring and ranking in paragraphs [0030] and [0062]. Therefore, the examiner is not persuaded by the applicant’s arguments and the prior art rejections are maintained.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Independent claims 1, 8, and 15 recites “receiving a user query”, “obtaining a plurality of outputs generated by an LLM as a response to said user query”, “determining accuracy of said plurality of outputs generated by said LLM by: extracting domain context from said user query and said plurality of outputs generated by said LLM”, “generating at least a model input based on said extracted domain contexts”, “providing a semantic analysis and comparing said plurality of LLM outputs and said at least one model input”, “ranking said plurality of LLM outputs based on comparison and analysis made between said at least one model input and each of said plurality of LLM outputs; wherein a weight is associated to each of said plurality of LLM outputs based on said comparison”, “computing a rank adjusted index by examining ranking of the plurality of LLM outputs and appending the results for identifying a match for an obligation to risk assessment”, and “generating a final response output based on said ranking and said obligation to risk assessment and providing it to a user in so as to provide a response to said user query”.
The limitation of receiving a user query, as drafted, is a process, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. That is, other than reciting “one or more processors”, “computer-readable memories”, “one or more computer-readable tangible storage medium”, and “program instructions stored on at least one of the one or more tangible storage medium”, nothing in the claim precludes the step from practically being performed in the mind. For example, but for the elements listed above, “receiving” in the context of this claim encompasses receiving a query, which a human can do in the mind or with a pen and paper. Next, the limitation of obtaining a response from a query, as drafted, is a process, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. That is, other than reciting the elements listed above, nothing in the claim precludes the step from practically being performed in the mind. For example, “obtaining” in the context of this claim encompasses generating a response to a question, which a human can do in the mind or with a pen and paper. Next, the limitation of determining accuracy of responses, as drafted, is a process, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. That is, other than reciting the elements listed above, nothing in the claim precludes the step from practically being performed in the mind. For example, “determining” in the context of this claim encompasses analyzing text, which a human can do in the mind or with a pen and paper. Next, the limitation of generating input data, as drafted, is a process, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. That is, other than reciting the elements listed above, nothing in the claim precludes the step from practically being performed in the mind. For example, “generating” in the context of this claim encompasses organizing data, which a human can do in the mind or with a pen and paper. Next, the limitation of providing semantic analysis and comparing data, as drafted, is a process, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. That is, other than reciting the elements listed above, nothing in the claim precludes the step from practically being performed in the mind. For example, “providing”, in the context of this claim encompasses analyzing data, which a human can do in the mind or with a pen and paper. Next, the limitation of ranking outputs, as drafted, is a process, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. That is, other than reciting the elements listed above, nothing in the claim precludes the step from practically being performed in the mind. For example, “ranking” in the context of this claim encompasses analyzing data, which a human can do in the mind or with a pen and paper. Next, the limitation of computing a rank, as drafted, is a process, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. That is, other than reciting the elements listed above, nothing in the claim precludes the step from practically being performed in the mind. For example, “computing” in the context of this claim encompasses performing calculations which a human can do in the mind or with a pen and paper. Lastly, the limitation of generating a response, as drafted, is a process, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. That is, other than reciting the elements listed above, nothing in the claim precludes the step from practically being performed in the mind. For example, “generating” in the context of this claim encompasses proving a response, which a human can do in the mind or with a pen and paper.
The judicial exception is not integrated into a practical application. In particular, the claim only recites the additional elements, using “one or more processors”, “computer-readable memories”, “one or more computer-readable tangible storage medium”, and “program instructions stored on at least one of the one or more tangible storage medium” to perform the recited limitations. These elements in these steps are recited at a high-level of generality such that is amounts no more than mere instructions to apply the exception using generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of using “one or more processors”, “computer-readable memories”, “one or more computer-readable tangible storage medium”, and “program instructions stored on at least one of the one or more tangible storage medium” to perform the recited limitations amounts to no more than mere instructions to apply the exception using generic computer components. Mere instructions to apply an exception using generic computer components cannot provide an inventive concept. The claim is not patent eligible.
Dependent claims 2-7, 9-14, and 16-20 are also rejected for the same reasons provided in independent claim 1, 8, and 15 above. The dependent claim, including the further recited limitation, does not integrate the abstract idea into a practical application and the additional elements, taken individually and in combination do not contribute to an inventive concept. In other words, the dependent claim is directed to an abstract idea without significantly more.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2, 4-5, 8-9, 11-12, 15-16, and 18-19 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Bolcer et al. US 20250103818 A1 (hereinafter Bolcer).
Regarding independent claims 1, 8, and 15, Bolcer teaches a method for evaluating accuracy of a large language model (LLM) output / a computer system for evaluating accuracy of a large language model (LLM) / a computer program product for evaluating accuracy of a large language model (LLM) comprising:
one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage medium, and program instructions stored on at least one of the one or more tangible storage medium for execution by at least one of the one or more processors via at least one of the one or more memories, wherein the computer system is enabled to perform the steps (Claim 1)
one or more computer-readable storage media (Claim 1); and
program instructions stored on said one or more computer-readable storage media (Claim 1)
receiving a user query (FIG. 1, 107);
obtaining a plurality of outputs generated by an LLM as a response to said user query (FIG. 2, “output”, [0045] “determining the accuracy of responses from a large language model”)
determining accuracy of said plurality of outputs generated by said LLM by: extracting domain context from said user query and said plurality of outputs generated by said LLM ([0030] “OpenAI's GPT, LLAMA, Alpcaca, and dozens of others, both domain specific”; [0035] “Using a query language, content can be found using tag names, tag values, phrases, raw text, scores, similarity, categorization”);
generating at least a model input based on said extracted domain contexts (FIG. 1, 109, 119; [0026] “The automated data query 119 may return a number of real-time, timestamped, natural language text, semi-structured records, reference data, scores, metrics, or embeddings and similarity scores. This culmination of data can be used to fine-tune any LLM 123”);
providing a semantic analysis and comparing said plurality of LLM outputs and said at least one model input (FIG. 2, NLP analysis, [0034] “The resulting text may be analyzed using an NLP pipeline. Using that extracted and enriched information, similarity analysis may use a vector datastore 121 of known hallucinations and good answers”);
ranking said plurality of LLM outputs based on comparison and analysis made between said at least one model input and each of said plurality of LLM outputs; wherein a weight is associated to each of said plurality of LLM outputs based on said comparison ([0030] “an LLM 123 or rule-based system can convert the prompt 107 into a new prompt, which also can be scored and ranked according to similarity to other successful prompts”; [0061-0062] “This facilitates the extraction of metadata of the answers themselves, enabling robust entity and relationship comparisons, vectorization, and the elucidation of temporal contexts like”; [0046] “The sub-query may be used to generate a small fine-tuning set to adjust the model with increased sample weights”); and
computing a rank adjusted index by examining ranking of the plurality of LLM outputs ([0030] “LLM 123 or rule-based system can convert the prompt 107 into a new prompt, which also can be scored and ranked according to similarity to other successful prompts or in the second case, converted to a multi-database collection query to be used as a LLM fine tuning training set”, examiner interprets scoring and ranking based off similarity as a rank adjusted index; [0062] “This discrepancy can be evaluated by a human or a machine learning model and the scores can be ranked and stored for future use”)) and appending the results for identifying a match for an obligation to risk assessment ([0037] In this example system, signals or topics have their own taxonomy that cover business issues, environmental, social, and governance issues with a material impact on the entity. For instance, a Business Risk signal can be specified as: Business. Business Risk. Competitive Risk if the text is found to represent a competitive risk to the company)
generating a final response output based on said ranking and said obligation to risk assessment and providing it to a user in response to said user query (FIG. 2, “output”, [0045] “determining the accuracy of responses from a large language model”; [0037]).
Regarding claim 2, 9, and 16, Bolcer teaches all of the limitations of claim 1, 8, 15, upon which claims 2, 9, and 16 depend.
Additionally, Bolcer teaches wherein said domain context includes a glossary (FIG. 1, 121;), a lemmatization ([0038] “similarity clustering links to other articles (or collections of information)”), and a human in the loop (HITL) context data ([0033] “Human-in-the-loop AI systems are a well-known practice in data science and AI training. Sub-goals are not human-generated unless a human-in-the-loop requires ideas on how to solve a problem”)
Regarding claim 4, 11, and 18, Bolcer teaches all of the limitations of claim 2, 9, and 16, upon which claims 4, 11, and 18 depend.
Additionally, Bolcer teaches wherein said glossary comprises of a dictionary of terms or phrases pulled of similar meaning to each other, created for relevant terms in said LLM output (FIG. 1, 121, 117; [0026] “The automated data query 119 may use any number of various databases and datastores 121 for specific topics”; [0028] An augmented output may be cross-validated 125 across various datastores 121 to check for consistency with other LLM output having the same entities, topics and/or metrics).
Regarding claim 5, 12, and 19, Bolcer teaches all of the limitations of claim 2, 9, and 16, upon which claims 5, 12, and 19 depend.
Additionally, Bolcer teaches wherein said weight module factors in a plurality of semantic similarity data to output a ranking of the most relevant to least relevant answers to said user query ([0030] “an LLM 123 or rule-based system can convert the prompt 107 into a new prompt, which also can be scored and ranked according to similarity to other successful prompts or in the second case, converted to a multi-database collection query to be used as a LLM fine tuning training set.”; [0040] “Sentiment scores 113 are averaged or weighted averaged for every mention or article or can be calculated on a per-mention basis”).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3, 10, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Bolcer in view of Manandise et al. US 20250139367 A1 (Manandise).
Regarding claims 3, 10, and 17, Bolcer teaches all of the limitations of claim 2, 9, and 16, upon which claims 3, 10, and 17 depend.
Bolcer fails to teach wherein said domain context also includes errors obtained from previous mismatches that related to a similar server error or a debugging need recorded previously on another computer servers.
However, Manandise teaches wherein said domain context also includes errors obtained from previous mismatches that related to a similar server error or a debugging need recorded previously on another computer servers ([0068] If during domain constraint validation 340, validation component 322 identifies one or more domain constraint errors, domain constraint validation 340 fails, and validation component 322 generates feedback for validation-error-based prompt generation component 324).
Bolcer in view of Manandise are considered to be analogous to the claimed invention because both are the same field of processing text using a LLM. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the techniques detecting hallucination as a metric for determining the accuracy of responses from a large language model of Bolcer with the technique of using domain context to include mismatch errors taught by Manandise in order to improve techniques for translating a prompt (e.g., posed in natural language) into a structured input to resolve the prompt as a planning problem (see Manandise [0001]).
Claims 6 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Bolcer in view Mui (US 20240412000 A1).
Regarding claims 6 and 13, Bolcer teaches all of the limitations of claims 1 and 8, upon which claims 6 and 13 depend.
Bolcer fails to teach wherein said ranking is presented from a most relevant to least relevant in a descending order.
However, Mui teaches wherein said ranking is presented from a most relevant to least relevant in a descending order ([0043] “the system may rank the classifications 225 in order of descending stochastic qualities, so that a top-ranked classification 225 is the most stochastic”).
Bolcer in view of Mui are considered to be analogous to the claimed invention because both are the same field of processing text using a LLM. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the techniques detecting hallucination as a metric for determining the accuracy of responses from a large language model of Bolcer with the technique of ranking relevancy in descending order taught by Mui in order to improve database systems and data processing, and more specifically to large language model controller (see Mui [0002]).
Claims 7, 14, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Bolcer in view Bayless et al. US 20250111192 A1 (hereinafter Bayless).
Regarding claims 7, 14, and 20, Bolcer teaches all of the limitations of claim 1, 8, and 15 upon which claims 7, 14, and 20 depend.
Bolcer fails to teach wherein semantics comparison is made based on a cosine similarity of all combinations of a plurality of words and concepts and/or cluster embeddings of a computed RAND index.
However, Bayless teaches wherein semantics comparison is made based on a cosine similarity of all combinations of a plurality of words and concepts and/or cluster embeddings of a computed RAND index ([0066] “The query engine 204 may utilize semantic similarity and similarity metrics as well for formal-language queries 128 that may require finding nodes or edges that are similar to a given entity. The query engine 202 may use similarity metrics, such as cosine similarity or Jaccard index, to measure the similarity between nodes in the knowledge graphs 122/202 to identify semantic similarities”)
Bolcer in view of Bayless are considered to be analogous to the claimed invention because both are the same field of processing text using a LLM. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the techniques detecting hallucination as a metric for determining the accuracy of responses from a large language model of Bolcer with the technique of performing semantic comparisons with cosine similarity taught by Bayless in order to improve using large language models (LLMs) to build knowledge graphs to answer queries submitted to a chatbot by users (see Bayless [Abstract]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Malkiel et al. (US 20250238629 A1) teaches a language model forward traversal with a few-shot learning forward prompt yields a primary answer from a primary question. Then at least one backward traversal yields at least one candidate question using backward prompt(s) with answer-question pairs derived from the forward prompt's question-answer pairs. Each backward prompt also includes the primary answer but not the primary question. Each backward traversal is through one or more language models, not necessarily including the forward traversal's language model. Sometimes backward traversals vary model temperature, top-p, or top-k. A vector distance calculated between at least some candidate question vectors and a primary question vector indicates whether the primary answer includes hallucination content, and in some cases how much. Some embodiments withhold hallucinated answers from user interfaces and device control interfaces. Some embodiments also loop to obtain an answer with less hallucination content.
Subramaniam et al. (US 20260064964 A1) teaches a computer-implemented method for evaluating integration of Responsible Artificial Intelligence Operations (RAIOPS) and Large Language Model Operations (LLMOPS) is disclosed. A response respective to each of prompts is generated using an LLM, in response to receiving data associated with each of the prompts. The data associated with each of the prompts and data associated with the response respective to each of the prompts is stored as an association. Further, based on user-specified criteria and using the data associated with the prompts or the data associated with the responses respective to the prompts, one or more evaluation metrics are generated for evaluating the responses respective to each of the prompts for one or more aspects. In accordance with the at least one evaluation metric, a knowledge graph visualization or a numerical score is generated to display performance of the LLM and determine whether the LLM needs optimization or tuning.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZEESHAN SHAIKH whose telephone number is (703)756-1730. The examiner can normally be reached Monday-Friday 7:30AM-5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Richemond Dorvil can be reached at (571) 272-7602. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ZEESHAN MAHMOOD SHAIKH/Examiner, Art Unit 2658
/RICHEMOND DORVIL/Supervisory Patent Examiner, Art Unit 2658
Prosecution Insights