Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claims 1-3, 5-11 and 13-22 are presented for examination.
Claims 1, 9 and 17 were amended.
Claims 4 and 12 were cancelled.
This is a Non-Final Action.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 06/17/2026 has been entered.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claims 1-3, 5, 9-11, 15, 17-18 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Tan et al. (US 11,270,077) in view of Sodhi et al. (US 2024/0386216) further in view of Kedzie et al. (US 2025/0200333)
1. Tan teaches, A method comprising:
receiving, by an agent, a prompt from a user (Fig 1:102 – conversational service; 102 – utterance 110; Col 3: lines 16-19 – teaches NLP based conversational service 102 receives user utterance and classifies an intent of user utterance, Tan);
inputting the prompt into a large language model (LLM) (Fig 1: 170 – routing service; Col 5: lines 18-24 – teaches a routing service 170 of conversational service 102 performs intent classification by performing off-topic instance detection and forwarding user utterance 110 to the classifier (instead of the LLM), Tan), the LLM trained using a taxonomy (Fig 9:902-908; Col 15:lines 29-41 – teaches replacing labels in subdomain samples with the identifier for the subdomain, further teaching, training the in-domain classifier (instead of LLM) for the domain with the positive examples from the sampled in-domain examples with original labels and the sampled subdomain examples with subdomain identifiers as labels, Tan) having a plurality of root classifications, and a set of child classifications under each root classification (Fig 1; Col 4: lines 27-37 – teaches multiple child nodes extend down from the root 112 in a first hierarchical level, illustrated as domain -1 114, domain-2 116, and domain-3 118, further teaching that hierarchical structure 100 may include additional or alternative number of levels of nodes each representing a domain or subdomain, Tan), wherein the LLM is configured to generate natural language text as output (Col 1: lines 15 – teaches input and output of natural language using a chatbot, which has the classifier (instead of LLM) incorporated, Tan);
receiving, from the classifier (instead of LLM), a classification corresponding to one of the child classifications based on the prompt (Fig 1: in domain classifiers and output labels, Col 5: lines 56-62 – teaches each IC represents a multi-class classifier that assigns a real in-domain label to user utterance 110 or identifies a signature of a subdomain. The real label includes a final prediction intent or a signature of a subdomain where the node has children nodes, Tan);
inputting, to a second LLM, a second prompt describing deployment of one or more skills in the hierarchical set of skills; and
responsive to receiving, from the second LLM, an indication that detecting an exception was detected based on the deployment of the one or more skills in the hierarchical set of skills, overriding the deployment by;
stopping one or more currently deployed skills from the hierarchical set; and
performing an exception activity associated with the detected exception.
Tan does not explicitly teach,
a large language model (LLM);
deploying a hierarchical set of skills to iteratively identify one or more LLM calls based on the classification, wherein each classification in the hierarchical set of skills is mapped to a respective skill and each skill is an LLM call that is configured to invoke the LLM to perform a task;
inputting, to a second LLM, a second prompt describing deployment of one or more skills in the hierarchical set of skills; and
responsive to receiving, from the second LLM, an indication that detecting an exception was detected based on the deployment of the one or more skills in the hierarchical set of skills, overriding the deployment by;
stopping one or more currently deployed skills from the hierarchical set; and
performing an exception activity associated with the detected exception.
However, Sodhi teaches,
a large language model (LLM) (Abstract – teaches a LLM model utilizing prompts for task automation, Sodhi);
deploying a hierarchical set of skills (Fig 10, Paragraphs 46-47 – teaches the complex task may be broken down into a main task and subtasks, also, further levels such as subsubtask and subsubsubtask are also possible, Sodhi) to iteratively identify one or more LLM calls based on the classification (Fig 10:1055,1060 & 1075; Paragraphs 123 & 129 – teaches at step 1055 a subtask prompt template is selected using the subtask specified by the language model at step 1030, where the subtask is not done, processing proceeds to step 1060 to create a prompt using subtask prompt template, Sodhi), wherein each classification in the hierarchical set of skills is mapped to a respective skill (Fig 10:1055; Paragraph 123 – teaches the subtask specified by the language model may include subtask ID, and the subtask prompt template may be selected using the subtask ID, such as by retrieving the subtask prompt template from a data store using the subtask, Sodhi) and each skill is an LLM call that is configured to invoke the LLM to perform a task (Fig 10: 1060-1065; Paragraphs 124 and 126 – teaches the subtask prompt template may provide instructions to the language model to provide next action to complete the subtask and at step 1065 a query is submitted to a language model using the prompt to obtain a response from the language model, Sodhi); and
a second prompt describing deployment of one or more skills in the hierarchical set of skills (Fig 10: 1060; Paragraph 125 – teaches the prompt is generated using the selected template, the subtask (e.g. one or more a subtask ID< a subtask attribute, or a subtask value), a representation of a web page, and information about previous actions performed during the automation process, Sodhi);
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which said subject matter pertains to incorporate Sodhi’s generative LLM and hierarchical task/subtask prompting into Tan’s conversational classification framework so that classified user requests direct corresponding task specific LLM processing. Tan teaches training classifiers using domain and subdomain labeled examples and routing user inputs independently of the particular classifier implementation (Fig 9, Col 15: lines 29-41; Col 6:lines 62-67, Tan). Sodhi teaches generative LLM processing, selecting subtask prompt templates using subtask identifiers, and iteratively submitting the resulting prompts (Paragraph 38-39, 123-129, Sodhi). A POSITA would have been motivated to apply Tan’s labeled-training approach to the LLM and associate the resulting classifications with Sodhi’s corresponding prompt templates, thereby linking learned intent distinctions to reuseable, multistep task execution. The proposed combination would use the classifications as the template selection identifier and the same LLM for classification and task specific calls, with the expected result that each classified request initiates the appropriate hierarchical sequence of prompted operations.
However, Kedzie teaches,
inputting, to a second LLM (Paragraph 27, - teaches some current systems, attempt to identify hallucinations (or under-grounded outputs) using secondary LLMs, as the output sequence 136 is generated, the series of tokens in the output sequence 136 is fed into the secondary model, Kedzie);
responsive to receiving, from the second LLM, an indication that detecting an exception was detected based on the deployment of the one or more skills in the hierarchical set of skills, overriding the deployment by (Paragraphs 27 & 28 – teaches the secondary model attempts to identify whether the most recent token (or another span of tokens) is a hallucination or undergrounded output, furthermore, action signal generator 141 can generate an action signal 143 based upon the probe output 140, Kedzie);
stopping one or more currently deployed skills from the hierarchical set (Fig 7:216 and 218; Paragraph 50 – teaches the action signal 143 can be an output to transformer 102 to stop the current generation process as indicated by block 218, Kedzie); and
performing an exception activity associated with the detected exception (Fig 7:220, Paragraph 50 – teaches the action signal can be an output to provide additional context information (such as to extract additional context data from a knowledge store, prompt the user to provide additional context, or obtain additional context in other ways, Kedzie).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which said subject matter pertains to incorporate the secondary-LLM hallucination checking described by Kedzie into the combination of Tan and Sodhi system to detect erroneous output during task performing LLM calls and prevent continued processing based on that output. Kedzie describes evaluating generated tokens using a secondary LLM (Paragraph 27) and separately discloses an action-signal mechanism for stopping generation and obtaining additional context (Fig 7, Paragraph 50). A POSITA would be motivated to adapt that controller to use the secondary LLM’s adverse determination in place of the probe output, stopping the affected skill’s generation and requesting additional context for corrective processing. For Sodhi’s API based model implementation (Paragraph 61), output-based checking would permit evaluation where generated text is available, but internal model states are not exposed. The reason for selecting this approach, despite its additional computational and latency costs, would therefore be to provide error responsive execution control without requiring access to the task performing model’s hidden states.
2. The combination of Tan, Sodhi and Kedzie teach, The method of claim 1, wherein inputting the prompt into the LLM comprises an auto-regressive input that iteratively progresses through hierarchical classifications until a final child node is encountered (claim 1 – teaches inputting the prompt into a multi-domain context-based hierarchy to a leave node by selecting a parent domain node then selecting a subdomain node until labels the natural language input with a classification label, Tan).
3. The combination of Tan, Sodhi and Kedzie teach, The method of claim 1, wherein the hierarchical set of skills comprises a chain of two or more skills performed based on the classification (Col: 5: lines 61-20 – teaches ICs may execute recursively in the IC of the subdomains, Tan).
5. The combination of Tan, Sodhi and Kedzie teach, The method of claim 1, wherein detecting the exception comprises detecting content in a response generated by the LLM (Paragraph 27 – teaches the identified token or span of tokens constitutes content in the generating LLM’s response; detection of hallucinated or ungrounded content supplies the exception condition based on the secondary LLM approach, Kedzie) that violates a guardrail (Paragraph 19; Fig 9:280 and 282 – teaches the restriction that generated content remain consistent with tis supplied knowledge source corresponds to the guardrail; content contradicting that source violates the restriction. Fig 9 corroborates the criterion used to identify hallucinations during training, Kedzie).
Claims 9 and 17 are similar to claim 1 hence rejected similarly.
Claims 10 and 18 are similar to claim 2 hence rejected similarly.
Claim 11 and 19 are similar to claim 3 hence rejected similarly.
Claim 13 is similar to claim 5 hence rejected similarly.
Claims 6-8, 14-16, 20-21 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Tan et al. (US 11,270,077) in view of Sodhi et al. (US 2024/0386216) and Kedzie et al. (US 2025/0200333) further in view of Huang et al. (US 11,423,235)
All the limitations of claim 1 are taught above.
6. The combination of Tan, Sodhi and Kedzie does not explicitly teach,
retrieving, from a database, a plurality of categories that classify conversations based on content of the conversations;
determining whether the prompt aligns with one of the plurality of categories;
responsive to determining that the prompt does not align with one of the plurality of categories, retrieving, from the database, a set of previously proposed categories;
determining whether the prompt aligns with one of the set of previously proposed categories;
responsive to determining that the prompt does not align with one of the set of previously proposed categories, providing the prompt to the LLM to generate a new category for classifying the prompt;
receiving output from the LLM, the output comprising at least one recommended category; and tagging the prompt with the at least one recommended category from the output.
However, Huang teaches,
retrieving, from a database, a plurality of categories that classify conversations based on content of the conversations (Abstract, Col 10: lines 12-18 – new intents and entities are discovered and used to update an existing dialogue path (which requires classification), Huang);
determining whether the prompt aligns with one of the plurality of categories (Col 10: lines 12-18 - teaches predictive model 157 applied to determine the most likely task or intent the user input, Huang);
responsive to determining that the prompt does not align with one of the plurality of categories, retrieving, from the database, a set of previously proposed categories (Abstract & Col 7: lines 49-51– additional data is continually collected and used to retain model 157 and expand the intent/entity datable, Huang);
determining whether the prompt aligns with one of the set of previously proposed categories (Col 10: lines 12-18 - teaches predictive model 157 applied to determine the most likely task or intent the user input, Huang);
responsive to determining that the prompt does not align with one of the set of previously proposed categories, providing the prompt to the LLM to generate a new category for classifying the prompt (Abstract, Col 10: lines 12-18 – new intents and entities are discovered and used to update an existing dialogue path (which requires classification), Huang;
receiving output from the LLM, the output comprising at least one recommended category; and tagging the prompt with the at least one recommended category from the output (Col 10: lines 1-18 - teaches new intents are used to update the dialogue path and stored for future classicization, Huang).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which said subject matter pertains to incorporate Huang’s stored feature comparison and new intent discovery into the combination of Tan, Sodhi and Kedzie system to reuse previously obtained semantic information and adapt conversational processing to newly encountered inputs. Huang compares input-derived features with stored feature vectors and, following an unsuccessful comparison, performs new intent/entity discovery (Fig 7: 730 & 741; Col 13: lines 6-17 & 45-67). A POSITA would have been motivated to apply those techniques to the combined system’s classification and task selection stages, with the expected result of reusing known semantic associations for familiar requests and updating the information used to select appropriate processing for new requests, while retaining the existing LLM execution and error checking functions.
7. The combination of Tan, Sodhi, Kedzie and Huang teach, The method of claim 6, wherein determining whether the user prompt aligns with one of the plurality of categories comprise:
generating a prompt embedding representing the user prompt (Claim 1 - teaches processing the user unput to determine intent data and entity data which is based on feature vector, Huang);
retrieving a plurality of category embeddings each represents a respective category (Col 10: lines 1-18– teaches processing moves to block 330, where any new intent and entity values discovered are used to update a master intent/entity set, wherein each intent and entity is combined together to generate a feature vector.. different chatbots have different intent and entity sets which are collectively from the feature space (i.e. teaching multiple category representations (intent/entity sets) from chatbots, Huang); and
determining a similarity between the prompt embedding with one or more of the plurality of category embeddings (Col 10: lines 32-39 – teaches from all the workspaces 1 through N the predictive model selects one of the workspaces as providing the best answer for the user’s input, Huang).
8. The combination of Tan, Sodhi, Kedzie and Huang teach, The method of claim 7, wherein the LLM is trained by (Fig 1:102; Paragraph 26 0 teaches training transformer 102, Kedzie):
accessing a plurality of training examples (Fig 6:610 – database; 620 – preprocessing; Col 12: lines 1-5 – teaches raw data for training the predictive model is stored in a database 610. The data includes a set of N questions and corresponding chatbot weights, Huang), each training example including a conversation between an agent and a user (Fig 2B; Col 7:20-25 – teaches displayed GUI includes an entire conversation between the user and an example multi-task dialogue system; Fig 4; Col 10: lines 45-52 – teaches the raw data illustrated is one example of a training set used to train the predictive model. As shown, database 410 stores a series of questions and corresponding chatbot weights, Huang), wherein the conversation is generated by an additional large language model (Fig 8-9; Paragraph 58 – teaches during training, the llama 2-13b model is used as the base model to generate responses to task in each domain; Paragraph 55 – teaches a synthetic response generated by an annotator or otherwise, Kedzie).
Claims 14 and 20 are similar to claim 6 hence rejected similarly.
Claim 15 is similar to claim 7 hence rejected similarly.
Claim 16 is similar to claim 8 hence rejected similarly.
21. The combination of Tan, Sodhi, Kedzie and Huang teach, The computer system of claim 15, wherein deploying the hierarchical set of skills further comprises deploying at least one of the identified LLM call to a third LLM (Fig 10: 1055-1065; Paragraph 87 and 126 – teaches a language model may be used to select a sequence of subtasks to be performed. The same language model or maybe a different language model may be used to implement the subtasks. At step 1065, a query is submitted to a language model using the prompt to obtain a response from the language model, Sodhi) and stopping one or more currently deployed skills from the hierarchical set further comprises: instructing the LLM and the third LLM to stop executing (Fig 7:216 & 218; Paragraph 50 – teaches the action signal 143 can be an output to transformer 102 to stop the current generation process as indicated by the block 218, Kedzie) tool calls associated with the deployed hierarchical set of skills (Fig 10:steps 1065-1070 – teaches at step 1070, the web page operation indicated by the next action of the response of the language model is implemented or executed to obtain a new web page and Paragraph 83 – teaches the next action of the response of the language model may be implemented using a web browser extension or plugin or using a headless browser, Sodhi).
22. The combination of Tan, Sodhi, Kedzie and Huang teach, The computer system of claim 15, wherein the exception comprises a hallucination in an output from the LLM, wherein overriding the deployment further comprises: determining the exception activity based on the hallucination (Fig 7:216-222; Paragraphs 27 and 50 – teaches hallucination detection in LLm generated output. The generated token or span constitutes cntent in the LLM’s output, and the detected hallucination corresponds to the proposed exception, Kedzie).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Yun et al. (US 12,282,793) – teaches loading model specific data into first and second model (Abstract, Fig 5, Col 1558-Col 16: lines 1-8)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AMRESH SINGH whose telephone number is (571)270-3560. The examiner can normally be reached Monday-Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ann J. Lo can be reached at (571) 272-9767. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AMRESH SINGH/Primary Examiner, Art Unit 2159
