DETAILED ACTION
This communication is in response to the Application filed on 07/27/2024. Claims 1-20 are pending and have been examined. . Hence, this Action has been made FINAL.
Any previous objection/rejection not mentioned in this Office Action has been withdrawn by the examiner.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 13, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 10/30/2024 and 4/29/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Priority
Applicant claims the benefit of US Provisional Application No. 63/583,164, 63/583,162, and 63/583, 159, filed September 15, 2023. Claims 1-20 have been afforded the benefit of this filing date.
Response to Arguments
With regards to the 35 U.S.C. 101 rejections for claims 1-20, the applicant has amended the claim language to more clearly show operations performed by skill bots that would make it unreasonable to consider a mental process. For example, a skill bot transitioning it’s state to another GenAI component state/workflow based on a specific response being detected. This process is inherently technical and thus, the abstract idea rejections have been withdrawn
With regards to the 35 U.S.C. 103 rejections for claims 1-20, the applicant asserts that The Applicant respectfully disagrees with this mapping. As described in the portion quoted above, Guo teaches a context retrieval operation to clarify missing information ("personalized context component 210 may process as described herein above to generate the personalized context data 215 (e.g., 'The user prefers [Music Streaming Service 1]"' - see Guo at 10:1-12)for a model so that the model can continue executing the same underlying task ("model output data representing 'Does the user prefer to use [Music Streaming Service 1] or [Music Streaming Service 2] for playing music,' 'I need to determine whether the user prefers [Music Streaming Service 1] or [Music Streaming Service 2] for playing music' or the like" - id., and see also id. 9:30-35), which does not teach or suggest transitioning GenAI component states (defined as "a multi-turn interaction with a GenAI model," in independent claim 1 as amended herein) or transitioning workflows (defined as "defin[ing] an interaction with a user to complete the task," in independent claim 1 as amended herein). Therefore, Guo does not teach or suggest "the skill bot ... configured to transition, responsive to detecting the invalid input variable in the response, from the GenAI component state to another state different from that of the GenAI component state or to another workflow different from the workflow associated with the action," as recited in the Applicant's independent claim 1 as amended herein (emphasis added).
Examiner respectfully disagrees, this quote was just meant to show an example of the process happening. A more accurate mapping of how the reference is being interpreted would be:
response
The reply from the user clarifying an ambiguity
Invalid input variable
The specific piece of context info in the response
GenAI component state (or workflow)
The LLM/skill currently performing a task in the list/plan created by the plan generator.
Another GenAI component state
The prompt provided to the plan generator being updated, thus altering the plan and associated LLMs/skills
transitioning
The shortlister component providing the user context info to the plan generator to alter the plan
The applicant further asserts that the combination of Guo, Zheng, and Sakaoglu also does not teach or suggest these features. Assuming, arguendo, that the combination teaches, at most, performing a predefined context-gathering operation to obtain additional information (e.g., a music streaming platform preference) as requested by a model output (Guo), the output of the model generated responsive to a model instruction prompt instructing the model how to respond and providing examples (Zheng), and the examples including created negative samples (Sakaoglu), the combination of the cited references still does not teach or suggest a "skill bot ... configured to transition, responsive to detecting the invalid input variable in the response, from the GenAI component state to another state different from that of the GenAI component state or to another workflow different from the workflow associated with the action," as recited in the Applicant's independent claim 1 as amended herein (emphasis added).
Examiner respectfully disagrees, in Guo et al. this process is not only from predefined context gathering and can happen in the middle of a plan (workflow) when the shortlister (skill bot) determines it’s necessary. (the LLM shortlister component 160) may be capable of determining whether an ambiguity exists in the user input or the current task, and may determine that additional information is needed.) (Col. 22, Lines 44-64). This causes the plan being currently executed to be altered by a user input. Further explanation and updated mappings can be found in the rejection below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over US Patent Publication US 12424209 B1 (Guo et al.) in view of “BIM-GPT: Prompt-Based Virtual Assistant Framework for BIM Information Retrieval” (Zheng et al.) and “KARTAL: Web Application Vulnerability Hunting Using Large Language Models” (Sakaoglu).
Regarding Claims 1, 9, and 16, Guo et al. teaches A computer-implemented method comprising:
(Aspects of the disclosed system may be implemented as a computer method or as an article of manufacture such as a memory device or non-transitory computer readable storage medium.) (Col. 42, Lines 11-13).
Claim 9 states the alternative A system comprising: one or more processors; and one or more computer-readable media storing instructions which, when executed by the one or more processors, cause the system to perform operations comprising:
(Computer instructions for operating each device (110/120/125) and its various components may be executed by the respective device's controller(s)/processor(s) (804/904), using the memory (806/906) as temporary “working” storage at runtime.) (Col. 40, Lines 9-13).
Claim 16 states the alternative One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
(Aspects of the disclosed system may be implemented as a computer method or as an article of manufacture such as a memory device or non-transitory computer readable storage medium.) (Col. 42, Lines 11-13).
routing an utterance to a skill bot, wherein the skill bot is configured to execute an action for completing a task associated with the utterance,
(A system may receive a user input as speech. For example, a user may speak an input to a device. The device may send audio data, representing the spoken input, to the system. The system may perform ASR processing on the audio data to generate ASR data (e.g., text data, token data, etc.) representing the user input. The system may perform processing on the ASR data to determine an action responsive to the user input.) (Col. 2, Lines 26-33).
(The system may determine one or more components (e.g., a skill component, a LLM agent component, etc.), etc.) configured to perform action(s) associated with a top-priority task of the one or more tasks and the language model(s) may generate an output indicating one or more requests (e.g., application programming interface (API) calls) that the one or more components return a description of the function(s) (e.g., action(s)) they are configured to/will perform with respect to the user input and/or current task) (Col. 2, Lines 50-59).
(The LLM shortlister component 160 may be configured to determine one or more components (e.g., APIs, skill component(s) 194, LLM agent component(s) 192, TTS component 196, etc.) configured to perform an action related to the user input or the current task. The LLM shortlister component 160 may further be configured to generate and cause the execution of a request(s) (e.g., an API call(s)) for the one or more components to provide an output(s) such as a description(s) representing the function(s) (e.g., action(s)) the components are configured to/will perform with respect to the user input or the current task.) (Col. 7, Line 60 – Col. 8, Line 3)
Guo et al. takes in user utterances and determines an action/tasks to be completed in response to it. It does so by forwarding the utterance to a relevant skill bot or LLM for the task. The LLM shortlister can be considered the skill bot in this reference as it organizes the skills and LLMs used to complete the task.
and a workflow associated with the action defines an interaction with a user to complete the task and includes a generative artificial intelligence (GenAI) component state configured to facilitate completion of at least part of the task;
(As illustrated in FIG. 1, the user input data 127 may be received by the LLM orchestrator component 130 at the plan generation component 135, which may be configured to generate (e.g., using the plan generation language model 145) a list (e.g., one or more) of tasks (e.g., steps/sub-actions) that are to be completed in order to perform an action responsive to the user input and select (e.g., using the task selection language model 155) a task of the list of the tasks that is to be completed first (e.g., in a current iteration of processing by the system 100), as described in detail herein below with respect to FIG. 2.) (Col. 7 Lines 14-24).
A plan is generated which involves a list of tasks that need to be completed to complete the action. This is equivalent to a workflow.
wherein the GenAI component state is a multi-turn interaction with a GenAI model;
(The LLM shortlister component 160 may be configured to determine one or more components (e.g., APIs, skill component(s) 194, LLM agent component(s) 192, TTS component 196, etc.) configured to perform an action related to the user input or the current task. The LLM shortlister component 160 may further be configured to generate and cause the execution of a request(s) (e.g., an API call(s)) for the one or more components to provide an output(s) such as a description(s) representing the function(s) (e.g., action(s)) the components are configured to/will perform with respect to the user input or the current task.) (Col. 7, Line 60 – Col. 8, Line 3).
(In instances where the plan generation component 135 (e.g., using the plan generation language model 145) generates more than one task to be completed in order to perform the action responsive to the user input, the plan generation component 135 may further maintain and prioritize the list of tasks as the processing of the system 100 with respect to the user input is performed. In other words, as the system 100 processes to complete the list of tasks, the plan generation component 135 may (1) incorporate the results of the processing performed to complete the tasks into data provided to other components of the system 100; (2) update the list of tasks to indicate completed (or attempted, in-progress, etc.) tasks; (3) generate an updated prioritization of the tasks remaining to be completed (or tasks to be attempted again); and/or (4) determine an updated current task to be completed.) (Col. 7, Lines 36-51).
In Guo et al. the plan is a list of tasks to be completed by the various LLM’s and skill component. The GenAI component state can be considered whatever is the current skill/LM being used in the plan.
communicating, by the GenAI component state, the prompt to a GenAI provider for processing by the GenAI model;
(The plan prompt generation component 140 processes the user input data 127 to generate prompt data 220 representing a prompt for input to the plan generation language model 145. In some embodiments, the plan prompt generation component 140 may further receive an indication of one or more remaining tasks to be completed with respect to the user input data 127.) (Col. 8, Lines 28-34)
A prompt is created for the LLM to instruct to complete the required task.
receiving, at the GenAI component state from the GenAI model provider, a response generated by the GenAI model processing the prompt,
(The action plan execution component 185 may identify the request(s) in the action plan data 182 and cause the corresponding components (e.g., the API provider component 190, the LLM agent component 192, the skill component 194, and/or the TTS component 196) to generate action response data 197a-n representing the requested output(s), where individual action response data 197a may be provided by/correspond to a particular responding component—one of the API provider component 190, the LLM agent component 192, the skill component 194, and/or the TTS component 196.) (Col. 8, Lines 5-15)
The LLM provides a response to the task after being prompted.
wherein the skill bot is further configured to transition, responsive to detecting the invalid input variablein the response, from the GenAI component state to another state different from that of the GenAI component state or to another workflow different from the workflow associated with the action.
(In such examples, the personalized context component 210 may receive the user input, the current task, and/or model output data indicating that an ambiguity exists/clarifying information should be determined (e.g., model output data representing “Does the user prefer to use [Music Streaming Service 1] or [Music Streaming Service 2] for playing music,” “I need to determine whether the user prefers [Music Streaming Service 1] or [Music Streaming Service 2] for playing music” or the like). The personalized context component 210 may process as described herein above to generate the personalized context data 215 (e.g., “The user prefers [Music Streaming Service 1].”)) (Col. 10, Lines 1-12)
(The LLM shortlister component 160 may be configured to determine one or more components (e.g., APIs, skill component(s) 194, LLM agent component(s) 192, TTS component 196, etc.) configured to perform an action related to the user input or the current task.) (Col. 7, Lines 60-66)
(Following such a template format, for example, and for a selected task of “turn on all of the lights except the garage light” and corresponding relevant API data, the shortlister prompt generation component 175 may generate example prompt data 320a: { You are an AI agent to find an execute an API to complete the task of turn on all of the lights except the garage light Here are a list of relevant API available: Let's chat API Classic NLU API Smart Home skill Use the following format: Thought: think about what to do API: API calls compatible with the task Observation: the result of the API call Summary: summarized results from the API call If no appropriate API is found, summarize as nothing is found.) (Col. 16, Lines 6-26)
(As discussed herein above, one or more of the components discussed herein (e.g., the plan generation component 135 and/or the LLM shortlister component 160) may be capable of determining whether an ambiguity exists in the user input or the current task, and may determine that additional information is needed. In response to such a determination, in some embodiments the component(s) may be further configured to send a request for such additional information to the response arbitration component 470, which may process as described herein above to generate a request for the additional information to be sent to the personalized context component 210 or to be sent to the action plan execution component 185, which may cause output of the request to the user to solicit the additional information.) (Col. 22, Lines 44-64)
(Thereafter, the output of the personalized context component 210 may be sent to the response arbitration component 470 such that once the response arbitration component 470 receives the output of the LLM shortlister component 160, the response arbitration component 470 may resolve the ambiguity that resulted in the request for additional information in order to generate the output data 490. For further example, if the user input data 127 is generated to include the natural language representation of the user input, but the processing required to determine the corresponding contextual signals (e.g., weather data, time of data, dialog history, device information, etc.) is yet to be completed, the plan generation component 135 may begin processing with respect to the natural language representation of the user input. Once the corresponding contextual signals have been generated, the plan generation component 135 may begin processing with respect to the contextual signals and may update downstream components with the result of the processing with respect to the contextual signals.) (Col. 24, Lines 12-57)
The shortlister (skill bot) determines if ambiguities exist and alters the plan (workflow) to have the user resolve the ambiguity. The system receives a response from the user (invalid input variable) which can further alter the plan (workflow) execution. The context information provided by the user alters the prompt being used to execute the plan. This altered prompt represents another GenAI component state as it can change the skills/LLMs being accessed and/or the information it receives. This can happen in the middle of a plan being generated at which point the plan generation component will generate things not requiring user input until it receives the input. This change in the plan generation represents a transition from the shortlister component (skill bot).
Guo et al. does not explicitly teach: generating, by the GenAI component state, a prompt to include the utterance and one or more scope-related elements based on a prompt template, wherein the one or more scope-related elements include: (i) one or more scenarios, (ii) one or more negative few-shot examples that are considered out of scope (OOS) or out of domain (OOD), and (iii) instructions that teach the GenAI model to output an invalid input variable when the GenAI model determines an utterance is OOS or OOD; wherein when the GenAI model determines the utterance is OOS or OOD as part of the processing the prompt, the response includes the invalid input variable;
However, Zheng et al. teaches generating, by the GenAI component state, a prompt to include the utterance and one or more scope-related elements based on a prompt template,
(Consequently, we designed a prompt library consisting of five types of prompts: Intent Prompt, Parameter Prompt, Value Prompt for NLU, Summary Prompt for NLG, and General Question Prompt for QA. Furthermore, we developed a shared template for these five prompts.) (Page 9, Paragraphs 2-3)
Zheng et al. constructs a prompt using the user input and scope related elements. An example of this can be seen in Fig. 4 where the user utterance is shown at the bottom of the prompt.
wherein the one or more scope-related elements include: (i) one or more scenarios,
(The Few-shot Examples component provides examples of the user input and system output pairs, which demonstrate an output pattern for the GPT model.) (Page 10, Paragraph 3)
(Specifically, the Task Instruction component stated: "The user may ask questions about the functions of building components and the definition of the components' properties in the BIM database; users may also search for general knowledge about buildings, BIM technology, architecture, engineering, construction, and facility management.) (Page 25, Paragraph 4)
Through the few-shot examples and task instructions a scenario is created for the input utterance.
(ii) one or more negative few-shot examples (that are considered out of scope (OOS) or out of domain (OOD),) (Taught by Sakaoglu)
(The Few-shot Examples component provides examples of the user input and system output pairs, which demonstrate an output pattern for the GPT model.) (Page 10, Paragraph 3)
Example Q&A pairs are included in the prompt template. These pairs are not explicitly stated to be negative, out-of-scope, or out-of-domain, however, this aspect is taught by Sakaoglu below.
and (iii) instructions that teach the GenAI model to output an invalid input variable when the GenAI model determines an utterance is OOS or OOD;
(For the General Question Prompt, we delineated the scope of general questions that the VA would address and utilized the GPT model as an external knowledge base. Specifically, the Task Instruction component stated: "The user may ask questions about the functions of building components and the definition of the components' properties in the BIM database; users may also search for general knowledge about buildings, BIM technology, architecture, engineering, construction, and facility management. For other questions, your answer should be 'Sorry, I do not know. Your question is out of my scope.'" This scope definition minimized the generation of irrelevant text by the GPT model, preventing potential user confusion about its functionality.) (Page 25, Paragraph 4)
Instructions are provided for how the LLM can output an invalid input variable for out-of-scope questions.
wherein when the GenAI model determines the utterance is OOS or OOD as part of the processing the prompt, the response includes the invalid input variable; and
(For the General Question Prompt, we delineated the scope of general questions that the VA would address and utilized the GPT model as an external knowledge base. Specifically, the Task Instruction component stated: "The user may ask questions about the functions of building components and the definition of the components' properties in the BIM database; users may also search for general knowledge about buildings, BIM technology, architecture, engineering, construction, and facility management. For other questions, your answer should be 'Sorry, I do not know. Your question is out of my scope.'" This scope definition minimized the generation of irrelevant text by the GPT model, preventing potential user confusion about its functionality.) (Page 25, Paragraph 4)
The LLM determines if the utterance is out-of-scope.
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the natural language task processing system using multiple specialized models as taught by Guo et al. to engineer prompts that will help produce more desirable outputs as taught by Zheng et al. This would have been an obvious improvement as it allows the AI models to handle a wide variety of natural language queries across different tasks (Zheng et al. Page 9, Paragraph 3).
Guo et al. in view of Zheng et al. does not explicitly teach: (ii) one or more negative few-shot examples that are considered out of scope (OOS) or out of domain (OOD)
However, Sakaoglu teaches (ii) one or more negative few-shot examples that are considered out of scope (OOS) or out of domain (OOD)
(We converted the request and response pairs to our minimized text format and prepended contextual information to f it our template. We also created negative samples based on the vulnerable examples with small changes to make the examples non-vulnerable. Next, we crafted a few-shot prompt that includes a task description, a template for the response, and a list of labeled examples for ICL.) (Page 43, Paragraph 2)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the natural language task processing system using multiple specialized models as taught by Guo et al. in view of Zheng et al. include negative examples within the prompt templates as taught by Sakaoglu This would have been an obvious improvement as this can allow the prompt to specifically target vulnerabilities in the LLM that it would like to avoid (Sakaoglu Page 43, Paragraph 2).
Regarding Claims 2, 10, and 17, Guo et al. in view of Zheng et al. and Sakaoglu teaches the system of claims 1, 9, and 16.
Furthermore, Guo et al. teaches receiving the utterance from a user during a session with a digital assistant;
(The wakeword detection component 520 may be configured to detect various wakewords. In at least some examples, each wakeword may correspond to a name of a different digital assistant.) (Col. 26, Lines 8-11).
Guo et al. receives a user utterance to a digital assistant.
determining, by one or more machine learning models, an intent of the user based on the utterance;
(Further, in such embodiments, the personalized context component 210 (or the system 100) may include a personalized context prompt generation component (not illustrated), which may be configured to generate a prompt including the user input data 127 (or a representation of an intent of the user input) to be input to the LLM. The prompt may be an instruction for the LLM to determine one or more portions of context data (e.g., the personalized context data 215) associated with the prompt.) (Col. 9, Lines 47-56)
A prompt is formed using the user input and an intent detected from it.
identifying the skill bot based on the intent; and
(The skill component(s) 790 may correspond to one or more domains that are used in order to determine how to act on a spoken input in a particular way, such as by outputting a directive that corresponds to the determined intent, and which can be processed to implement the desired operation.) (Col. 37, Lines 51-56).
Skill bots are chosen based on the determined intent.
identifying, by the skill bot, the action for completing the task associated with the utterance.
(The LLM shortlister component 160 may further be configured to generate and cause the execution of a request(s) (e.g., an API call(s)) for the one or more components to provide an output(s) such as a description(s) representing the function(s) (e.g., action(s)) the components are configured to/will perform with respect to the user input or the current task.) (Col. 7, Lines 64 to Col. 8, Line 3)
Actions and outputs are then performed by the selected components for the task.
Regarding Claims 3, 11, and 18, Guo et al. in view of Zheng et al. and Sakaoglu teaches the system of claims 1, 9, and 16.
Furthermore, Guo et al. teaches wherein when the GenAI model determines the utterance is in-scope or in-domain (not OOS or OOD) as part of the processing the prompt,
(An LLM agent component 192 may correspond to a custom instantiation of an LLM (and other components) that is configured to handle user inputs relating to a particular domain/functionality. In some embodiments, the LLM agent component 192 may be configured to handle specific use cases via particular prompt generation, fine-tuning of the LLM, etc.) (Col. 20, Lines 8-15)
In Guo et al. the LLM’s are specifically configured to handle input in certain domains and are passed inputs based on the detected intent.
the response does not include the invalid input variable,
(the plan prompt generation component 140 may further receive an indication of the remaining tasks to be completed. In such embodiments, the plan prompt generation component 140 may further receive an indication of the completed task(s) and/or result(s) of the processing performed to complete the task(s).) (Col. 8, Lines 40-45).
(As such, the model output data 198 may further include data representing a confidence of how relevant the action response data 197 a is to the current task.) (Col. 18, Lines 7-9).
The tasks receive the output from previous tasks which includes a confidence score signifying that the utterance is relevant to them.
and the computer-implemented method further comprises responsive to the response not including the invalid input variable, maintaining the GenAI component state.
(the plan prompt generation component 140 may further receive an indication of the remaining tasks to be completed. In such embodiments, the plan prompt generation component 140 may further receive an indication of the completed task(s) and/or result(s) of the processing performed to complete the task(s).) (Col. 8, Lines 40-45).
The plan continues to the next task in response to receiving a relevant response.
Regarding Claims 4, 12, and 19, Guo et al. in view of Zheng et al. and Sakaoglu teaches the system of claims 1, 9, and 16.
Furthermore, Zheng et al. teaches the prompt further includes: (i) a definition of a role or persona for the GenAI model and
(The System component specifies the VA role of the GPT model, enabling it to understand the background of the user’s query. For example, “You are a virtual assistant that helps users retrieve building information from the BIM database, and answer general questions about BIM technology, architecture, engineering, construction, and operation.”) (Page 10, paragraph 3).
In Zheng et al. the prompt includes an indication of the role the model is taking on.
Zheng et al. teaches (ii) a description of the task;
(The System component specifies the VA role of the GPT model, enabling it to understand the background of the user’s query. For example, “You are a virtual assistant that helps users retrieve building information from the BIM database, and answer general questions about BIM technology, architecture, engineering, construction, and operation.”) (Page 10, paragraph 3)
The task is described to the model within the prompt
Sakaoglu teaches the one or more scenarios comprise an invalid scenario; and
(To rapidly create a new labeled dataset, we used a few different techniques. At first, we gathered vulnerability samples from OWASP and MITRE resources. These included demonstrative examples as well as documented real-world application vulnerabilities.) (Page 43, Paragraph 2).
Sakaoglu creates a template using real-world vulnerabilities which represent invalid scenarios.
Sakaoglu teaches the one or more negative few-shot examples are associated with the invalid scenario.
(We also created negative samples based on the vulnerable examples with small changes to make the examples non-vulnerable. Next, we crafted a few-shot prompt that includes a task description, a template for the response, and a list of labeled examples for ICL.) (Page 43, Paragraph 2).
Negative examples and real-world vulnerabilities are used in the prompt template.
Regarding Claims 5, 13, and 20, Guo et al. in view of Zheng et al. and Sakaoglu teaches the system of claims 4, 12, and 19.
Furthermore, Zheng et al. teaches the prompt further includes one or more positive few-shot examples, which include:
(Lastly, the Few-shot Example includes relevant examples based on the identified category and parameter information. This dynamic design can enhance the performance of GPT models by aligning them more effectively with BIM IR contexts.) (Page 12, Paragraph 2)
In Zheng et al. positive few-shot examples are included within the prompt.
(i) one or more additional example utterances that are considered to be in-scope or in-domain (not OOS or OOD), and
Table 2 Shows the types of examples that are provided for different types of requests and their domains such as the Category in a parameter prompt. Also, Fig. 4 shows an example template containing 3 sample responses.
(ii) instructions that teach the GenAI model to output a response based on sample responses that enforce format and structure of the response to be generated when an utterance is determined to be in-scope or in-domain (not OOS or OOD);
(3) The Task Instruction component describes the task and the output requirements in detail, which explicitly guide the GPT model to generate desired responses. For example, “Your first task is to classify the ONE intent … among …; Your second task is to identify ONE category … from the list …; This is a classification task; Answer as concisely as possible; The output should follow this template …”) (Page 10, Paragraph 3)
(The Few-shot Examples component provides examples of the user input and system output pairs, which demonstrate an output pattern for the GPT model.) (Page 10, Paragraph 3)
Positive few-shot examples and task instructions enforce the format for the output of the LLM.
the one or more scenarios further comprise a valid scenario; and
This can be seen in Fig. 4 where Few-shot examples are generated corresponding to the input domain and description of the task.
the one or more positive few-shot examples are associated with the valid scenario.
This can be seen in Fig. 4 where Few-shot examples are generated corresponding to the input domain and description of the task.
Regarding Claims 6 and 14, Guo et al. in view of Zheng et al. and Sakaoglu teaches the system of claims 1 and 9.
Furthermore, Guo et al. teaches wherein the another state is another GenAI component state different from the GenAI component state, and wherein the computer-implemented method further comprises:
(In instances where the plan generation component 135 (e.g., using the plan generation language model 145) generates more than one task to be completed in order to perform the action responsive to the user input, the plan generation component 135 may further maintain and prioritize the list of tasks as the processing of the system 100 with respect to the user input is performed. In other words, as the system 100 processes to complete the list of tasks, the plan generation component 135 may (1) incorporate the results of the processing performed to complete the tasks into data provided to other components of the system 100; (2) update the list of tasks to indicate completed (or attempted, in-progress, etc.) tasks; (3) generate an updated prioritization of the tasks remaining to be completed (or tasks to be attempted again); and/or (4) determine an updated current task to be completed.) (Col. 7, Lines 36-51).
The plan generation component creates a task list to complete the action corresponding to the user utterance. It can also update this plan and alter it during the process, thus transition to another workflow.
Zheng et al. teaches generating, by the another GenAI component state, another prompt to include the utterance or another utterance and one or more scope-related elements based on another prompt template;
(Consequently, we designed a prompt library consisting of five types of prompts: Intent Prompt, Parameter Prompt, Value Prompt for NLU, Summary Prompt for NLG, and General Question Prompt for QA. Furthermore, we developed a shared template for these five prompts.) (Page 9, Paragraphs 2-3)
Zheng et al. constructs a prompt using the user input and scope related elements. An example of this can be seen in Fig. 4 where the user utterance is shown at the bottom of the prompt.
Guo et al. teaches communicating, by the another GenAI component state, the another prompt to another GenAI provider for processing by another GenAI model;
(The plan prompt generation component 140 processes the user input data 127 to generate prompt data 220 representing a prompt for input to the plan generation language model 145. In some embodiments, the plan prompt generation component 140 may further receive an indication of one or more remaining tasks to be completed with respect to the user input data 127.) (Col. 8, Lines 28-34)
A prompt is created for the LLM to instruct to complete the required task.
Guo et al. teaches receiving, at the another GenAI component state from the another GenAI model provider, another response generated by the another GenAI model processing the another prompt,
(The action plan execution component 185 may identify the request(s) in the action plan data 182 and cause the corresponding components (e.g., the API provider component 190, the LLM agent component 192, the skill component 194, and/or the TTS component 196) to generate action response data 197a-n representing the requested output(s), where individual action response data 197a may be provided by/correspond to a particular responding component—one of the API provider component 190, the LLM agent component 192, the skill component 194, and/or the TTS component 196.) (Col. 8, Lines 5-15)
The LLM provides a response to the task after being prompted.
Zheng et al. teaches wherein when the another GenAI model determines the another utterance is OOS or OOD as part of the processing the another prompt, the another response includes the invalid input variable;
(For the General Question Prompt, we delineated the scope of general questions that the VA would address and utilized the GPT model as an external knowledge base. Specifically, the Task Instruction component stated: "The user may ask questions about the functions of building components and the definition of the components' properties in the BIM database; users may also search for general knowledge about buildings, BIM technology, architecture, engineering, construction, and facility management. For other questions, your answer should be 'Sorry, I do not know. Your question is out of my scope.'" This scope definition minimized the generation of irrelevant text by the GPT model, preventing potential user confusion about its functionality.) (Page 25, Paragraph 4)
The LLM determines if the utterance is out-of-scope.
Guo et al. teaches and responsive to the another response including the invalid input variable, transitioning from the another GenAI component state to a different state or different workflow.
(In such examples, the personalized context component 210 may receive the user input, the current task, and/or model output data indicating that an ambiguity exists/clarifying information should be determined (e.g., model output data representing “Does the user prefer to use [Music Streaming Service 1] or [Music Streaming Service 2] for playing music,” “I need to determine whether the user prefers [Music Streaming Service 1] or [Music Streaming Service 2] for playing music” or the like). The personalized context component 210 may process as described herein above to generate the personalized context data 215 (e.g., “The user prefers [Music Streaming Service 1].”)) (Col. 10, Lines 1-12)
The above quote shows an example of in response to an input variable, changing the skill component or LLM that tries to complete the task. The invalid input variable is taught in more detail by Zheng et al. below.
Regarding Claim 7, Guo et al. in view of Zheng et al. and Sakaoglu teach the system of claim 1.
Furthermore, Guo et al. teaches wherein the workflow further includes the another state.
(In instances where the plan generation component 135 (e.g., using the plan generation language model 145) generates more than one task to be completed in order to perform the action responsive to the user input, the plan generation component 135 may further maintain and prioritize the list of tasks as the processing of the system 100 with respect to the user input is performed. In other words, as the system 100 processes to complete the list of tasks, the plan generation component 135 may (1) incorporate the results of the processing performed to complete the tasks into data provided to other components of the system 100; (2) update the list of tasks to indicate completed (or attempted, in-progress, etc.) tasks; (3) generate an updated prioritization of the tasks remaining to be completed (or tasks to be attempted again); and/or (4) determine an updated current task to be completed.) (Col. 7, Lines 36-51).
(In instances where the plan generation component 135 determined that more than one task is to be completed in order to perform the action responsive to the user input data 127, the LLM shortlister component 160may send the model output data 198 to the plan generation component 135, which may process as described herein above to maintain and prioritize the task list based on the model output data 198 and select a new task to be completed.) (Col. 18, Lines 21-29).
In Guo et al. the plan is a list of tasks to be completed by the various LLM’s and skill component. The plan can be modified while the action is being complete representing a change to another workflow.
Regarding Claim 8, Guo et al. in view of Zheng et al. and Sakaoglu teach the system of claim 1.
Furthermore, Guo et al. teaches wherein the another state is associated with the another workflow that is different from the workflow.
(In instances where the plan generation component 135 (e.g., using the plan generation language model 145) generates more than one task to be completed in order to perform the action responsive to the user input, the plan generation component 135 may further maintain and prioritize the list of tasks as the processing of the system 100 with respect to the user input is performed. In other words, as the system 100 processes to complete the list of tasks, the plan generation component 135 may (1) incorporate the results of the processing performed to complete the tasks into data provided to other components of the system 100; (2) update the list of tasks to indicate completed (or attempted, in-progress, etc.) tasks; (3) generate an updated prioritization of the tasks remaining to be completed (or tasks to be attempted again); and/or (4) determine an updated current task to be completed.) (Col. 7, Lines 36-51).
(In instances where the plan generation component 135 determined that more than one task is to be completed in order to perform the action responsive to the user input data 127, the LLM shortlister component 160may send the model output data 198 to the plan generation component 135, which may process as described herein above to maintain and prioritize the task list based on the model output data 198 and select a new task to be completed.) (Col. 18, Lines 21-29).
In Guo et al. the plan is a list of tasks to be completed by the various LLM’s and skill component. The plan can be modified while the action is being complete representing a change to another workflow.
Regarding Claim 15, Guo et al. in view of Zheng et al. and Sakaoglu teach the system of claim 9.
Furthermore, Guo et al. teaches wherein the workflow further includes the another state or the another state is associated with the another workflow that is different from the workflow.
(In instances where the plan generation component 135 (e.g., using the plan generation language model 145) generates more than one task to be completed in order to perform the action responsive to the user input, the plan generation component 135 may further maintain and prioritize the list of tasks as the processing of the system 100 with respect to the user input is performed. In other words, as the system 100 processes to complete the list of tasks, the plan generation component 135 may (1) incorporate the results of the processing performed to complete the tasks into data provided to other components of the system 100; (2) update the list of tasks to indicate completed (or attempted, in-progress, etc.) tasks; (3) generate an updated prioritization of the tasks remaining to be completed (or tasks to be attempted again); and/or (4) determine an updated current task to be completed.) (Col. 7, Lines 36-51).
(In instances where the plan generation component 135 determined that more than one task is to be completed in order to perform the action responsive to the user input data 127, the LLM shortlister component 160may send the model output data 198 to the plan generation component 135, which may process as described herein above to maintain and prioritize the task list based on the model output data 198 and select a new task to be completed.) (Col. 18, Lines 21-29).
In Guo et al. the plan is a list of tasks to be completed by the various LLM’s and skill component. The plan can be modified while the action is being complete representing a change to another workflow.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS DANIEL LOWEN whose telephone number is (571)272-5828. The examiner can normally be reached Mon-Fri 8:00am - 4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Paras D Shah can be reached at (571) 270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NICHOLAS D LOWEN/Examiner, Art Unit 2653
/Paras D Shah/Supervisory Patent Examiner, Art Unit 2653
10/01/2026
Prosecution Insights