Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is sent in response to Applicant’s communication received on 11/01/2024 for the application number 18934655. The office hereby acknowledges receipt of the following placed of record in the file: Specification, Abstract, Oath/Declaration and claims.
Status of the claims
Claims 1-20 are presented for examination.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because as
explained below.
Claim 1 recites a method comprising
Receiving information representing ground truth
Detecting entities and relations from the information
Constructing a knowledge graph based on knowledge triplets
Receiving an input query
Generating, via a previously trained large language model, a response with respect to the input query
Identifying one or more response triplets from the response.
Determining semantic similarity between the response triplets and matching knowledge graph triplets
Evaluating the response based on the semantic similarities to generate an assessment; and
Providing the response with the assessment including an explanation.
Step (a) comprises a mental process. This step can be performed by a human as a person
can receive information representing ground truth.
Step (b) comprises a mental process. This step can be performed by a human as a person
Can identify entities and relationships from received information.
Step (c) comprises a mental process. This step can be performed by a human as a person
can organize identified entities and relationships into a knowledge graph or other structured representation.
Step (d) comprises a mental process. This step can be performed by a human as a person
Can receive an input query.
Step (e) comprises a mental process. This step can be performed by a human as a person
Can formulate a response to a received query based on previously acquired knowledge.
Step (f) comprises a mental process. This step can be performed by a human as a person
Can identify entities and relationships within a response and represent them as triplets.
Step (g) comprises a mental process. This step can be performed by a human as a person
Can compare relationships represented by response triplets to relationships represented by knowledge graph triplets and determine semantic similarities.
Step (h) comprises a mental process. This step can be performed by a human as a person
Can evaluate a response based on the determined similarities and generate an assessment.
Step (i) comprises a mental process. This step can be performed by a human as a person
Can provide the response together with the assessment and an explanation of the assessment.
Step 1: This part of the eligibility analysis evaluates whether the claim falls within any
statutory category. See MPEP 2106.03. The claim recites at least method. Thus, the claim is a
process, which is one of the statutory categories of invention. (Step 1: YES). Step 2A, Prong
One: This part of the eligibility analysis evaluates whether the claim recites a judicial exception. As explained in MPEP 2106.04, subsection II, a claim “recites” a judicial exception when the
judicial exception is “set forth” or “described” in the claim. As discussed above, the broadest
reasonable interpretation of steps (a)-(i) recites a mental process.
Specifically, step (a) comprises a mental process. This step can be performed by a human as a person can receive information representing ground truth.
Step (b) can be performed by a human as a person can identify entities and relationships from received information.
Step (c) can be performed by a human as a person can organize identified entities and relationships into a knowledge graph or other structured representation.
Step (d) can be performed by a human as a person can receive an input query.
Step (e) can be performed by a human as a person can formulate a response to a received query based on previously acquired knowledge.
Step (f) can be performed by a human as a person can identify entities and relationships within a response and represent them as triplets.
Step (g) can be performed by a human as a person can compare relationships represented by response triplets to relationships represented by knowledge graph triplets and determine semantic similarities.
Step (h) can be performed by a human as a person can evaluate a response based on the determined similarities and generate an assessment.
Step (i) can be performed by a human as a person can provide the response together with the assessment and an explanation of the assessment.
Hence the claim encompasses mental processes practically performed in the human mind
by observation, evaluation, judgement, and opinion. See MPEP 2106.04(a)(2), subsection III.
(Step 2A, Prong One: YES).
Step 2A, Prong Two: This part of the eligibility analysis evaluates whether the claim as a
whole integrates the recited judicial exception into a practical application of the exception or
whether the claim is “directed to” the judicial exception. This evaluation is performed by (1)
identifying whether there are any additional elements recited in the claim beyond the judicial
exception, and (2) evaluating those additional elements individually and in combination to
determine whether the claim as a whole integrates the exception into a practical application. See
MPEP 2106.04(d).
The claim recited additional elements including a knowledge graph, an LLM, and information representing ground truth. However, these elements are recited at a high level of generality and perform generic computer functions, such as receiving data, processing data, generating content, and providing output. The use of these elements to access a query, retrieve contextual information, generate a dense context, integrate the dense context with the query, generate a response, and communicate the response merely automates the mental processes described above using generic computer components. Such implementation does not impose any meaningful limit on the judicial exception. Accordingly, these elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. (Step 2A, Prong Two: NO), and the claim is directed to the judicial exception. (Step 2A: YES)
Step 2B: This part of the eligibility analysis evaluates whether the claim as a whole
amounts to significantly more than the recited exception i.e., whether any additional element, or combination of additional elements, adds an inventive concept to the claim. As discussed with
respect to Step 2A, Prong Two, knowledge graph, an LLM, and information representing ground truth comprise additional elements that perform well-understood, routine, and conventional activities in the field such as receiving data, processing data, generating content, and providing output. See MPEP 2106.05(g). As known in the art these elements are well understood, routine, and conventional functions of a computing device. Even when considered in combination these additional elements merely implement the abstract idea using generic computer components and perform insignificant extra - solutional activity, which does not provide an inventive concept. The claim is not patent eligible.
Claim 2 recites a mental process as a human can identify a subject entity, an object entity, and a relationship from information and organize the information into a structured representation.
Claim 3 recites a mental process as a human can analyze information within a particular domain and formulate a conclusion regarding that information.
Claim 4 recites a mental process as a human can compare entities and relationships from another representation to entities and relationships from another representation and determine whether they are similar.
Claim 5 recites a mental process as a human can compare information from multiple sources and aggregate the results of the comparisons to reach a conclusion.
Claim 6 recites a mental process as a human can evaluate information based on comparisons, generate an overall assessment, and provide an explanation for the assessment.
Claim 7 recites a mental process as a human can generate an explanation based on similarities between corresponding portions of two sets of information.
Claims 8 & 15 recite substantially the same limitations as claim 1, but in different
statutory categories (computer readable medium & system). Accordingly, they are directed to
the same abstract idea as claim 1.
Claims 9 & 16 recite substantially the same limitations as claim 2. Accordingly, they are
directed to the same abstract idea.
Claims 10 & 17 recite substantially the same limitations as claim 3. Accordingly, they are
directed to the same abstract idea.
Claims 11 & 18 recite substantially the same limitations as claim 4. Accordingly, they are
directed to the same abstract idea.
Claims 12 & 19 recite substantially the same limitations as claim 5. Accordingly, they are
directed to the same abstract idea.
Claims 13 & 20 recite substantially the same limitations as claim 6. Accordingly, they are
directed to the same abstract idea.
Claim 14 recites substantially the same limitations as claim 7. Accordingly, it is
directed to the same abstract idea.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 5, 8, 9, 12, 15, 16, 19 are rejected under 35 U.S.C. 103 as being unpatentable over Emrey (US 20240386207 A1) in view of Tiwari et al (US 20260093931 A1).
Regarding claim 1, Emrey teaches receiving information representing ground truth (Para 0061, “extracted from the input data (e.g., generative model output data 202 and/or ground truth information 204)”, Fig. 2 teaches the extraction from the ground truth data); detecting entities and relations from the information (Para 0061, “The system may be configured to generate an RDF triple for each utterance … The RDF triple may describe relationships between entities in a structured manner.”); constructing a knowledge graph (KG) based on knowledge triplets, each of which characterizes a relation connecting two of the entities in the information and represents a ground truth fact (Para 0061, “RDF triples may be connected to one another in a graph data structure of nodes and edges. For example, a knowledge graph representative of a given generative model output data 202 may be generated”); identifying, from the response, one or more response triplets, each of which includes two entities via a relation (Para 0061, “The system may be configured to generate an RDF triple for each utterance extracted from the input data (e.g., generative model output data 202”, wherein RDF triples comprise two entities and a relation or more specifically, a subject, object, and predicate); with respect to each of the one or more response triplets, determining, if at least one matching KG triplet exists in the knowledge graph, semantic similarity between the response triplet and the at least one matching KG triplet (Para 0066, “The models may be configured to determine a similarity measure that characterized similarity or dissimilarity of the data structure 208 representing generative model output data and the data structure 210 representing fact data”, See Para 0058, 0061 wherein an RDF triple comprises subject, object, and predicate and corresponding triples are compared); evaluating the response based on the semantic similarities between the one or more response triplets and respective matching KG triplets to generate an assessment (Para 0064, “the system may generate an output 214 indicating one or more hallucinations and/or errors”); and providing the response with the assessment including an explanation of the assessment obtained based on the semantic similarity between each response triplet and its matching knowledge triplet. (Para 0080, “The generated output may comprise an indication of the errors and/or hallucinations in the generative model output data”).
Emrey does not teach receiving an input query; generating, via a previously trained large language model (LLM), a response with respect to the input query.
However, Tiwari teaches receiving an input query (Para 0005, “obtaining, by one or more processors, a query provided to a language model”); generating, via a previously trained large language model (LLM), a response with respect to the input query (Para 0005, “and a response to the query generated by the language model”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Emrey in such a way as to incorporate the teachings of Tiwari in order to enable evaluation of language model outputs to improve factual consistency (Para 0005).
Regarding claim 2, Emrey teaches with respect to the entities and the relations detected from the information, recognizing a subject entity and an object entity from the detected entities that are related according to one of the detected relations (Para 0061, “For example, a knowledge graph representative of a given generative model output data 202 may be generated which informs extracted entities, relationships between entities”), and creating a KG triplet with the subject entity, the relation that connects the subject and object entities, and the object entity (Para 0061, “The RDF triple may describe relationships between entities in a structured manner.”); and forming the knowledge graph based on the KG triplets. (Para 0061, “For example, a knowledge graph representative of a given generative model output data 202 may be generated” wherein an RDF inherently contains subject, predicate, and object entities).
Regarding claim 5, Emrey teaches determining, with respect to each of the at least one matching KG triplet, semantic similarity between the response triplet and the matching KG triplet (Para 0066, “The models may be configured to determine a similarity measure that characterized similarity or dissimilarity of the data structure 208 representing generative model output data and the data structure 210 representing fact data.”); aggregating the semantic similarities determined with respect to different matching KG triplets to derive the semantic similarity between the response triplet and relevant ground truth facts represented by the at least one matching KG triplet. (Para 0066, “In some embodiments, individual data points may be assigned a score that can be compiled (e.g., using a weighted or unweighted sum) with that of related data points to determine an overall comparison score”).
Claim 8 & 15 are analogous to claim 1 in that they recite substantially the same
limitations. They are therefore rejected for the same reasons set forth above.
Claim 9 & 16 are analogous to claim 2 in that they recite substantially the same
limitations. They are therefore rejected for the same reasons set forth above.
Claim 12 & 19 are analogous to claim 5 in that they recite substantially the same
limitations. They are therefore rejected for the same reasons set forth above.
Claims 3, 10, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Emrey (US 20240386207 A1) and Tiwari (US 20260093931 A1) as above in claims 1, 2, 5, 8, 9, 12, 15, 16, 19 and further in view of Vasseur et al. (US 12407581 B1).
Regarding claim 3, Emrey does not teach wherein the LLM is trained based on network management data; the entities and relations are network entities and relations; and the generated response from the LLM is a network anomaly determination.
However, Vasseur teaches wherein the LLM is trained based on network management data (Col. 5 Ln 51-57, “generative approaches instead seek to generate new content … based on an existing body of training data. For instance, in the context of network assurance, network control process 248 may use a generative model to generate synthetic network traffic based on existing user traffic to test how the network reacts.” Where the agent is thus trained based on network data); the entities and relations are network entities and relations (Col. 16 Ln 25-28, “troubleshooting agent 502 may leverage an LLM to summarize the content of the Trigger_Troubleshooting( ) message (e.g., type of anomaly, involved entities, etc.)”); and the generated response from the LLM is a network anomaly determination. (Col. 15 Ln 4-5, “The troubleshooting agent uses a language model to determine a root cause of the anomaly.”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Emrey in such a way as to incorporate the teachings of Vasseur in order to improve network anomaly detection and troubleshooting using domain specific model training. (Col. 15).
Claim 10 & 17 are analogous to claim 3 in that they recite substantially the same
limitations. They are therefore rejected for the same reasons set forth above.
Claims 4, 11, 18 are rejected under 35 U.S.C. 103 as being unpatentable over Emrey (US 20240386207 A1) and Tiwari (US 20260093931 A1) as above in claims 1, 2, 5, 8, 9, 12, 15, 16, 19 and further in view of Song (US 20110047178 A1)
Regarding claim 4, Emrey teaches wherein each of the response triplets includes a subject entity, an object entity, and a connecting relation linking the subject and object entities according to the response (Para 0061, “The RDF triple may describe relationships between entities in a structured manner.”, wherein an RDF inherently contains subject, predicate, and object entities); wherein the similarity is determined based on feature representations of the subject and the object entities and the connecting relations in both the response triplet and the matching KG triplet. (Para 0066, “Exemplary NLU models that may be used to detect similarities between the data structures include but are not limited to DeBERTa, RoBERTa, BERT, T5, transformer, and/or derivative models.” Where in NLU models and these examples inherently are based on and trained via features).
Emrey does not teach wherein the matching KG triplet is identified when: the subject entity in the matching KG triplet is similar to the subject entity in the response triplet, the object entity in the matching KG triplet is similar to the object entity in the response triplet, and the connecting relation in the matching KG triplet is similar to the connecting relation in the response triplet, wherein the similarity is determined based on feature representations of the subject and the object entities and the connecting relations in both the response triplet and the matching KG triplet.
However Song teaches wherein the matching KG triplet is identified when: the subject entity in the matching KG triplet is similar to the subject entity in the response triplet (Para 0069, “RDF triples that have the exactly same terms of subject, predicate and object with the query triples QT and received from the triple comparison unit 610 at step S118.”), the object entity in the matching KG triplet is similar to the object entity in the response triplet (Para 0069), and the connecting relation in the matching KG triplet is similar to the connecting relation in the response triplet. (Para 0069).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Emrey in such a way as to incorporate the teachings of Song in order to improve identification of matching triplets through comparison of corresponding triplet components.
Claim 11 & 18 are analogous to claim 4 in that they recite substantially the same
limitations. They are therefore rejected for the same reasons set forth above.
Claims 6, 7, 13, 14, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Emrey (US 20240386207 A1) and Tiwari (US 20260093931 A1) as above in claims 1, 2, 4, 5, 8, 9, 12, 15, 16, 19 and further in view of Perez et al. (US 12511326 B2).
Regarding claim 6, Emrey teaches accessing one or more semantic similarities, each of which is determined between each of the one or more response triplets and at least one matching KG triplet (Para 0066, “The models may be configured to determine a similarity measure that characterized similarity or dissimilarity of the data structure 208 representing generative model output data and the data structure 210 representing fact data.”); aggregating the one or more semantic similarities to generate an overall semantic similarity representing the assessment of the response (Para 0066, “In some embodiments, individual data points may be assigned a score that can be compiled (e.g., using a weighted or unweighted sum) with that of related data points to determine an overall comparison score”); and the similarity between each of the one or more response triplets and each of its matching KG triplet. (Para 0066, “The models may be configured to determine a similarity measure that characterized similarity or dissimilarity of the data structure 208 representing generative model output data and the data structure 210 representing fact data.”).
Emrey does not teach creating the explanation for the assessment based on at least one of the one or more semantic similarities.
However, Perez does teach creating the explanation for the assessment based on at least one of the one or more semantic similarities. (Col. 8 Ln 5-15, “The AI model 200 can also generate, along with a recommendation to keep or remove a keyword from a list, an explanation for the recommendation.” And “the explanation can relate to the semantic similarity of the keyword with other keywords”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Emrey in such a way as to incorporate the teachings of Perez in order to improve user understanding of outputs by providing an explanation (Col. 8)
Regarding claim 7, Emrey does not teach wherein the explanation for the assessment is further created based on the semantic similarity.
However, Perez teaches wherein the explanation for the assessment is further created based on the semantic similarity. (Col. 8 Ln 5-15, “The AI model 200 can also generate, along with a recommendation to keep or remove a keyword from a list, an explanation for the recommendation.” And “the explanation can relate to the semantic similarity of the keyword with other keywords”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Emrey in such a way as to incorporate the teachings of Perez in order to improve user understanding of outputs by providing an explanation (Col. 8).
Claim 13 & 20 are analogous to claim 6 in that they recite substantially the same
limitations. They are therefore rejected for the same reasons set forth above.
Claim 14 is analogous to claim 7 in that they recite substantially the same
limitations. They are therefore rejected for the same reasons set forth above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL ALAN FOSTER JR. whose telephone number is (571)272-8874. The examiner can normally be reached M - F 8:00am - 6:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hai Phan can be reached at (571) 272-6338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL A FOSTER JR/Examiner, Art Unit 2654
/HAI PHAN/Supervisory Patent Examiner, Art Unit 2654
Prosecution Insights