DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in reply to the communications filed on February 24, 2026. The Applicant’s Amendment and Request for Reconsideration has been received and entered.
Claims 1-20 are currently pending and have been examined. Claims 1, 12, and 17-20 have been amended.
Response to Arguments
Applicant’s amendments necessitated the new grounds of rejection.
Regarding the rejection of claims 1-20 under 35 USC 101, Applicant’s arguments have been fully considered but they are not persuasive for the reasons set forth infra. The Examiner respectfully notes that while Applicant asserts that the claims reflect an improvement in chatbot testing, the amended claims have been rejected under 35 U.S.C. 112(a) and 35 U.S.C. 112(b) and interpreted accordingly, and therefore do not fully encompass the asserted improvement.
Applicant’s remaining arguments have been fully considered but they are not persuasive. Particularly, Applicant’s arguments are directed to the instantly amended claims, and are thus moot in view of the new grounds of rejection. The Examiner respectfully notes that the amended claims have been rejected under 35 U.S.C. 112(a) and 35 U.S.C. 112(b) and interpreted accordingly, as set forth infra.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre- AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for pre- AIA the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 1, and similarly claim 20, recites "initiating, by the processor, an automated conversation between a first machine learning model and the chatbot based on the test scenario, wherein a response provided by the first machine learning model in response to a query made by the chatbot during the automated conversation is limited to information in the request;" (emphasis added).
Additionally, claim 1, and similarly claim 20, recites “modifying, by the processor, the chatbot based on the label.”
The recited subject matter of claims 1 and 20 do not conform to the disclosure in such a manner that one of ordinary skill in the art would recognize as being adequately described as the invention or as subject matter which the Applicant actually had possession of at the time of the invention. A review of the disclosure does not reveal the manner in which the response provided by the first machine learning model in response to a query made by the chatbot during the automated conversation is limited to information in the request,
Further, a review of the disclosure does not reveal the manner in which the chatbot is modified, by the processor, based on the label.
It is noted that this is not an enablement rejection. Applicant's failure to disclose any meaningful structures/algorithms regarding these limitations raises questions concerning whether Applicant truly had possession of these features at the time of filing.
Claims 2-19 depend from claim 1 and thus inherit the deficiencies of claim 1.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA , the applicant, regards as the invention.
Claim 1, and similarly claim 20, recites "initiating, by the processor, an automated conversation between a first machine learning model and the chatbot based on the test scenario, wherein a response provided by the first machine learning model in response to a query made by the chatbot during the automated conversation is limited to information in the request;" (emphasis added).
As discussed above, the disclosure does not disclose any meaningful structure/algorithm explaining how one would provide a response by the first machine learning model in response to a query made by the chatbot during the automated conversation is limited to information in the request. While the specification recites “When engaging in conversation with the chatbot 200, the first machine learning model 110 may only use the inputs provided by the test scenario when responding to questions from the chatbot 200” (App. Spec. [0044]), the test scenario is a broader scope than the request, further the response being “consistent with” the information in the request, does not the same as limited to information in the request. Consequently, the metes and bounds of these limitations are unclear because a person having ordinary skill in the art cannot determine how to avoid infringement. For examination purposes, the Examiner has interpreted this limitation as merely initiating, by the processor, an automated conversation between a first machine learning model and the chatbot based on the test scenario.
Additionally, claim 1, and similarly claim 20, recites “modifying, by the processor, the chatbot based on the label.”
As discussed above, the disclosure does not disclose any meaningful structure/algorithm explaining how one would modify, by the processor, the chatbot based on the label. While the specification recites “the evaluation 140 may further include a reason as to why for a particular judgment was made the label and/or a suggestion for making improvements to the chatbot configuration. This extra information may then be aggregated and used as feedback to improve the chatbot’s future performance” (App. Spec. [0049]), using feedback to improve the chatbot’s future performance is not necessarily modifying, by the processor, the chatbot based on the label. Consequently, the metes and bounds of these limitations are unclear because a person having ordinary skill in the art cannot determine how to avoid infringement. For examination purposes, the Examiner has interpreted this limitation as merely providing a chatbot.
Claims 2-19 depend from claim 1 and thus inherit the deficiencies of claim 1.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Step 1. When considering subject matter eligibility under 35 U.S.C. 101, it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter.
Step 2A – Prong One. If the claims fall within one of the statutory categories, it must then be determined whether the claims recite an abstract idea, law of nature, or natural phenomenon.
Step 2A – Prong Two. If the claims recite an abstract idea, law of nature, or natural phenomenon, it must then be determined whether the claims recite additional elements that integrate the judicial exception into a practical application. If the claims do not recite additional elements that integrate the judicial exception into a practical application, then the claims are directed to a judicial exception.
Step 2B. If the claims are directed to a judicial exception, it must be evaluated whether the claims recite additional elements that amount to an inventive concept (i.e. “significantly more”) than the recited judicial exception.
In the instant case, claims 1-19 are directed to a process; claim 20 is directed to a machine.
A claim “recites” an abstract idea if there are identifiable limitations that fall within at least one of the groupings of abstract ideas enumerated in MPEP 2106. In the instant case, claim 20, and similarly claim 1, recites the steps of:
identifying a test scenario comprising a request and an expected outcome; initiating an conversation based on the test scenario, wherein a response provided in response to a query made during the conversation is limited to information in the request; storing a recording of the conversation; providing the recording of the conversation and the test scenario as inputs, wherein takes as an input the expected outcome and generates a label based on the expected outcome and the conversation; modifying based on the label; and receiving an evaluation of the automated conversation based on the recording of the conversation and the expected outcome -- these steps set forth mental processes, including, inter alia, the observation, evaluation, judgment, and opinion of information.
Further, the limitations of the claims are not indicative of integration into a practical application. Taking the independent claim elements separately, the additional elements of performing the steps by a processor, by the first machine learning model, by the chatbot, to a second machine learning model that includes a generative large language model; and via an automated conversation between a first machine learning model and the chatbot; and a second machine learning model -- merely implement the abstract idea on a computer environment. Additionally, taking the dependent claim elements separately, the additional elements of performing the steps via an application programming interface (API), a third machine learning model, a database, the chatbot being web-based, an interface, and a generative large language model (LLM) also merely implement the abstract idea on a computer environment. Considered in combination, the steps of Applicant’s method add nothing that is not already present when the steps are considered separately.
Thus, claims 1-20 are directed to an abstract idea.
Regarding the independent claims, the technical elements of performing the steps by a processor, by the first machine learning model, by the chatbot, to a second machine learning model that includes a generative large language model; and via an automated conversation between a first machine learning model and the chatbot; and a second machine learning model -- are recited at a high level of generality and thus does not amount to significantly more, and thus merely implements the abstract idea on a computer environment. Additionally, regarding the dependent claims, the technical elements of an application programming interface (API), a third machine learning model, a database, the chatbot being web-based, an interface, and a generative large language model (LLM) also merely implement the abstract idea on a computer environment.
When considering the elements and combinations of elements, the claim(s) as a whole, do not amount to significantly more than the abstract idea itself. This is because the claims do not amount to an improvement to another technology or technical field; the claims do not amount to an improvement to the functioning of a computer itself; the claims do not move beyond a general link of the use of an abstract idea to a particular technological environment; the claims merely amounts to the application or instructions to apply the abstract idea on a computer; or the claims amounts to nothing more than requiring a generic computer to perform generic computer functions that are well-understood, routine and conventional activities previously known to the industry.
The analysis above applies to all statutory categories of invention. Accordingly, claims 1-20 are rejected as ineligible for patenting under 35 USC 101 based upon the same rationale.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-8, 10, 12, 13, 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Mahindru (US PGP 2022/0245199) in view of Sheikh (US PGP 2023/0368284).
As per claim 1, Mahindru teaches method of evaluating performance of a chatbot, the method comprising:
identifying, by a processor, a test scenario comprising a request and an expected outcome; (Mahindru: [0027]-[0028] (. . . generating benchmarking data, also referred to herein as ground truth (GT) from a knowledge base to evaluate the automated virtual dialog agent. . . . The benchmark data generation functions as a venue to extract GT within the scope of the knowledge base. A simulated dialog interaction is carried out with the automated virtual dialog agent as supported with the GT. As shown and described herein, the automated virtual dialog agent is subject to a performance evaluation that involves a comparison of a corresponding simulation log in view of the GT.); [0033]-[0036](The GT manager (152) is configured to automatically generate GT from one or more knowledge sources, e.g. knowledge domains, shown herein by way of example in LibraryA (172 A). The generated GT may be content based, usage based, and/or curation based. Content based GT is automatically generated by leveraging a corresponding structured dataset to generate questions based on symptoms, question variants, and graph traversal, and to obtain related entities for the symptoms. . . . Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses.))
initiating, by the processor, an automated conversation between a first machine learning model and the chatbot based on the test scenario, wherein a response provided by the first machine learning model in response to a query made by the chatbot during the automated conversation is limited to information in the request; (Mahindru: Fig. 6; [0041] (The simulator (154) interfaces with the dialog system (160) to simulate one or more NL dialog interactions using the automated virtual dialog agent (162) of the dialog system (160). In an exemplary embodiment, the simulator (154) leverages an operatively coupled simulator application (154 A) to conduct a simulated interaction with the chatbot (162). . . . ); [0035]-[0037] (Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses. . . . In one or more embodiments, one initial NL query and one outcome are generated in view of the corresponding structured knowledge, e.g. KG.); [0053] (As shown and described in FIG. 1, the simulator (154) is provided to support simulation of a NL dialog interaction. Referring to FIG. 6, a flow chart (600) is provided to illustrate a process for simulating interaction with the dialog system (160). A disambiguation selection path counting variable, N, is initialized (602), GT data is leveraged as a source to drive interaction with the virtual dialog agent, and a query, e.g. queryN, is generated and sent to the automated virtual agent using the operatively coupled simulator application (604). . . . The automated virtual agent responds to the query with a solution or a set of disambiguation options (608).)
storing, by the processor, a recording of the automated conversation; (Mahindru: Fig. 6; [0041] (The simulator (154) interfaces with the dialog system (160) to simulate one or more NL dialog interactions using the automated virtual dialog agent (162) of the dialog system (160). . . . The simulation defines a set of test queries with respective answers as present in a corresponding knowledge domain, which in an exemplary embodiment is represented as a knowledge graph. Output from the simulation is referred to herein as simulation data and includes a log of all queries and corresponding responses, which in an exemplary embodiment includes a solution or one or more disambiguation options.); [0034]-[0035] (Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses. Such interactions, and specifically the data associated with the interactions, are recorded and populated in one or more of the libraries of the knowledge base (170)).
providing, by the processor, the recording of the automated conversation and the test scenario as inputs to a second machine learning mode, wherein the second machine learning model includes a . . . model that takes as an input the expected outcome and generates a label based on the expected outcome and the automated conversation; and (Mahindru: Figs. 6-7; [0041]-[0042] (As shown herein, the evaluation manager (156), which is operatively coupled to the simulator (154), is configured to evaluate performance of the automated virtual dialog agent (162). The evaluation manager (156) compares the simulated interaction that is represented as simulation data with GT for the corresponding knowledge domain. The GT employed in the comparison may include one or more of the GT types, including content, usage, and curation based GT.); [0054]-[0055] (The dialog system (160) and the corresponding automated virtual agent (162) are subject to a performance evaluation by leveraging the GT and the corresponding simulation log. Referring to FIG. 7, a flow chart (700) is provided to illustrate a process for conducting the virtual dialog system performance evaluation. As shown, the variable NTotal is assigned to the quantity of query-responses recorded in the simulation log (702), and a corresponding counting variable, N, is initialized (704). For each query-responseN, a corresponding entry in the GT is found (706). The query-response in the simulation log is compared with the query-response in the GT (708). A multi-dimensional output is generated from the comparison at step (708), including: 1. the quantity of disambiguation questions actually asked and the difference between the quantity in the GT, 2. whether or not the questions were asked in the same order, and 3. if the solution presented in the simulation log matches with the GT solution.)
modifying, by the processor, the chatbot based on the label. (Mahindru: Fig. 6; [0041] (The simulator (154) interfaces with the dialog system (160) to simulate one or more NL dialog interactions using the automated virtual dialog agent (162) of the dialog system (160). In an exemplary embodiment, the simulator (154) leverages an operatively coupled simulator application (154 A) to conduct a simulated interaction with the chatbot (162). . . . ); [0035]-[0037] (Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses. . . . In one or more embodiments, one initial NL query and one outcome are generated in view of the corresponding structured knowledge, e.g. KG.); [0053] (As shown and described in FIG. 1, the simulator (154) is provided to support simulation of a NL dialog interaction. Referring to FIG. 6, a flow chart (600) is provided to illustrate a process for simulating interaction with the dialog system (160). A disambiguation selection path counting variable, N, is initialized (602), GT data is leveraged as a source to drive interaction with the virtual dialog agent, and a query, e.g. queryN, is generated and sent to the automated virtual agent using the operatively coupled simulator application (604). . . . The automated virtual agent responds to the query with a solution or a set of disambiguation options (608).); [0054]-[0055] (The dialog system (160) and the corresponding automated virtual agent (162) are subject to a performance evaluation by leveraging the GT and the corresponding simulation log. Referring to FIG. 7, a flow chart (700) is provided to illustrate a process for conducting the virtual dialog system performance evaluation. As shown, the variable NTotal is assigned to the quantity of query-responses recorded in the simulation log (702), and a corresponding counting variable, N, is initialized (704). For each query-responseN, a corresponding entry in the GT is found (706). The query-response in the simulation log is compared with the query-response in the GT (708). A multi-dimensional output is generated from the comparison at step (708), including: 1. the quantity of disambiguation questions actually asked and the difference between the quantity in the GT, 2. whether or not the questions were asked in the same order, and 3. if the solution presented in the simulation log matches with the GT solution. Insights and recommendations are generated based on output generated for each of the multiple dimensions (710). . . . In an exemplary embodiment, the recommendation(s), also referred to herein as a recommendation plan, is directed at improving interaction overhead, which may be implemented by collecting additional real-time data, and reducing interaction length. In an embodiment, other recommendations may be implemented, and as such the examples provided herein should not be considered limiting. Accordingly, the remediation actions are directed at improving the performance of the dialog system (160) and the corresponding automated virtual dialog agent (162).)
Mahindru does not explicitly disclose the following known technique which is taught by Sheikh:
. . . the second machine learning model includes a generative large language model . . . (Sheikh: [0021] Optionally, the machine learning model agent (ML-Model AA) is at least one of: an internal Large Language Model (Internal-LLM) of the agent-device (AA or micro-AA), an external Large Language Model (Internal-LLM) associated with the agent-device (AA or micro-AA).
This known technique is applicable to the method of Mahindru as they both share characteristics and capabilities, namely, they are directed to virtual agents.
One of ordinary skill in the art at the time of filing would have recognized that applying the known technique of Sheikh would have yielded predictable results and resulted in an improved method. It would have been recognized that applying the technique of Sheikh to the teachings of Mahindru would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such generative large language model (LLM) features into similar methods. Further, applying the generative large language model (LLM) to the second machine learning models of Mahindru would have been recognized by those of ordinary skill in the art as resulting in an improved method that would substantially eliminate or at least partially address challenges in effectively solve complex problems and the development of versatile and flexible multi-agent systems. (Sheikh: [0002]-[0007]; [0043])
As per claim 2, Mahindru/Sheikh teaches wherein the request is a task to be performed or a question to be answered by the chatbot to generate the expected outcome. (Mahindru: Fig. 6; [0033]-[0037] (Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses. . . . In one or more embodiments, one initial NL query and one outcome are generated in view of the corresponding structured knowledge, e.g. KG.); [0053] (As shown and described in FIG. 1, the simulator (154) is provided to support simulation of a NL dialog interaction. Referring to FIG. 6, a flow chart (600) is provided to illustrate a process for simulating interaction with the dialog system (160). A disambiguation selection path counting variable, N, is initialized (602), GT data is leveraged as a source to drive interaction with the virtual dialog agent, and a query, e.g. queryN, is generated and sent to the automated virtual agent using the operatively coupled simulator application (604). . . . The automated virtual agent responds to the query with a solution or a set of disambiguation options (608).); [0027]-[0028] (. . . generating benchmarking data, also referred to herein as ground truth (GT) from a knowledge base to evaluate the automated virtual dialog agent . . . .The benchmark data generation functions as a venue to extract GT within the scope of the knowledge base. A simulated dialog interaction is carried out with the automated virtual dialog agent as supported with the GT. As shown and described herein, the automated virtual dialog agent is subject to a performance evaluation that involves a comparison of a corresponding simulation log in view of the GT.); [0041])
As per claim 3, Mahindru/Sheikh teach further comprising:
providing customer-specific data comprising at least one of knowledge base data and application programming interface (API) description to the chatbot, wherein (Mahindru: [0027] (A system, a computer program product, and a method evaluate performance of an automated virtual dialog agent, and in an exemplary embodiment, a multi-turn dialog system, by automatically generating benchmarking data, also referred to herein as ground truth (GT) from a knowledge base to evaluate the automated virtual dialog agent. In an exemplary embodiment, the GT is automatically generated from a user's knowledge base and not from a standard or generic dataset. The benchmark data generation functions as a venue to extract GT within the scope of the knowledge base. A simulated dialog interaction is carried out with the automated virtual dialog agent as supported with the GT.); [0048])
the chatbot is configured to engage in the automated conversation further based on the at least one of the customer-specific data and the API description. (Mahindru: [0027] (. . . In an exemplary embodiment, the GT is automatically generated from a user's knowledge base and not from a standard or generic dataset. The benchmark data generation functions as a venue to extract GT within the scope of the knowledge base. A simulated dialog interaction is carried out with the automated virtual dialog agent as supported with the GT.); Fig. 6; [0041]; [0035]-[0037]; [0053]; [0048])
As per claim 4, Mahindru/Sheikh teach wherein the chatbot is configured to make an API call based on the request and the API description. (Mahindru: [0048] (An Application Program Interface (API) is understood in the art as a software intermediary between two or more applications. With respect to the artificial intelligence platform (150) shown and described in FIG. 1, one or more APIs may be utilized to support one or more of the tools (152), (154), (156), and (158) and their associated functionality. Referring to FIG. 2, a block diagram (200) is provided illustrating the tools (152), (154), (156), and (158), and their associated APIs. As shown, a plurality of tools are embedded within the AI platform (205), with the tools including the GT manager (252) associated with API0 (212), the simulator (254) associated with API1 (222), the evaluation manager (256) associated with API2 (232), and the remediation manager (258) associated with API3 (242). Each of the APIs may be implemented in one or more languages and interface specifications. API0 (212) provides functional support to automatically generate GT from a knowledge source; API1 (222) provides functional support to simulate a NL dialog with the automated virtual agent leveraging the GT; API2 (232) provides functional support to evaluate performance of the automated virtual dialog agent based on the simulation; and API3 (242) provides functional support to selectively identify and implement one or more remediation actions directed at improving performance of the dialog system. As shown, each of the APIs (212), (222), (232), and (242) are operatively coupled to an API orchestrator (260), otherwise known as an orchestration layer, which is understood in the art to function as an abstraction layer to transparently thread together the separate APIs.))
As per claim 5, Mahindru/Sheikh teach wherein the knowledge base data comprises a plurality of articles, and the request is a question to be answered by the chatbot, and (Mahindru: [0031]-[0034] (The AI platform (150) is also shown herein operatively coupled to the knowledge base (170), also referred to herein as a corpus of information. As shown, the knowledge base (170) is configured with a plurality of libraries, shown herein by way of example as LibraryA (172 A) and LibraryB (172 B). While two libraries are shown in FIG. 1, it should be understood that the knowledge base (170) may include fewer or more libraries. Further, the libraries, e.g. LibraryA (172 A) and LibraryB (172 B) may be combined together. The libraries, LibraryA (172 A) and LibraryB (172 B) may exist across a plurality of knowledge domains, including knowledge base (170) and other knowledge domains (not shown). Each library is populated with data, either in a structured or unstructured form.); [0039] (. . . Additionally, the AI platform (150) serves as a back-end system that can make available a variety of knowledge extracted from or represented in documents, network accessible sources and/or structured data sources . . . ); [0027]; Fig. 6; [0033]-[0037] (Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses. . . . In one or more embodiments, one initial NL query and one outcome are generated in view of the corresponding structured knowledge, e.g. KG.); [0053] (As shown and described in FIG. 1, the simulator (154) is provided to support simulation of a NL dialog interaction. Referring to FIG. 6, a flow chart (600) is provided to illustrate a process for simulating interaction with the dialog system (160). A disambiguation selection path counting variable, N, is initialized (602), GT data is leveraged as a source to drive interaction with the virtual dialog agent, and a query, e.g. queryN, is generated and sent to the automated virtual agent using the operatively coupled simulator application (604). . . . The automated virtual agent responds to the query with a solution or a set of disambiguation options (608).); [0027]-[0028])
wherein the chatbot is configured to respond to the request based on the plurality of articles. (Mahindru: [0031]-[0037] (The AI platform (150) is also shown herein operatively coupled to the knowledge base (170), also referred to herein as a corpus of information. As shown, the knowledge base (170) is configured with a plurality of libraries, shown herein by way of example as LibraryA (172 A) and LibraryB (172 B). While two libraries are shown in FIG. 1, it should be understood that the knowledge base (170) may include fewer or more libraries. Further, the libraries, e.g. LibraryA (172 A) and LibraryB (172 B) may be combined together. The libraries, LibraryA (172 A) and LibraryB (172 B) may exist across a plurality of knowledge domains, including knowledge base (170) and other knowledge domains (not shown). Each library is populated with data, either in a structured or unstructured form. . . . The GT manager (152) is configured to automatically generate GT from one or more knowledge sources, e.g. knowledge domains, shown herein by way of example in LibraryA (172 A). . . . Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses); [0027]-[0028] (. . . As shown and described herein, the automated virtual dialog agent is subject to a performance evaluation that involves a comparison of a corresponding simulation log in view of the GT.); Fig. 6; [0041] (The simulator (154) interfaces with the dialog system (160) to simulate one or more NL dialog interactions using the automated virtual dialog agent (162) of the dialog system (160). In an exemplary embodiment, the simulator (154) leverages an operatively coupled simulator application (154 A) to conduct a simulated interaction with the chatbot (162). . . . ); [0053] (The automated virtual agent responds to the query with a solution or a set of disambiguation options (608).)
As per claim 6, Mahindru/Sheikh teach further comprising:
providing one or more articles of the plurality of articles to a third machine learning model to generate the question; (Mahindru: [0031]-[0037] (The AI platform (150) is also shown herein operatively coupled to the knowledge base (170), also referred to herein as a corpus of information. As shown, the knowledge base (170) is configured with a plurality of libraries, shown herein by way of example as LibraryA (172 A) and LibraryB (172 B). . . . . The GT manager (152) is configured to automatically generate GT from one or more knowledge sources, e.g. knowledge domains, shown herein by way of example in LibraryA (172 A). . . . Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses); Fig. 3; [0049]-[0051] (Referring to FIG. 3, a flow chart (300) illustrating a process for automatically generating ground truth (GT) from a corresponding knowledge source is provided. . . . For each symptom, a natural language query is generated (304), which in an exemplary embodiment employs variance generation and adding or removing entities.); [0039] (. . . Additionally, the AI platform (150) serves as a back-end system that can make available a variety of knowledge extracted from or represented in documents, network accessible sources and/or structured data sources . . . ); [0028]; [0053])
receiving the question from the third machine learning model based on the one or more articles; and (Mahindru: [0031]-[0037] (The GT manager (152) is configured to automatically generate GT from one or more knowledge sources, e.g. knowledge domains, shown herein by way of example in LibraryA (172 A). . . . Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses . . . . In one or more embodiments, one initial NL query and one outcome are generated in view of the corresponding structured knowledge, e.g. KG.);); Fig. 3; [0049]-[0051] (Referring to FIG. 3, a flow chart (300) illustrating a process for automatically generating ground truth (GT) from a corresponding knowledge source is provided. . . . For each symptom, a natural language query is generated (304), which in an exemplary embodiment employs variance generation and adding or removing entities.))
identifying the question as the request. (Mahindru: [0031]-[0037] (The GT manager (152) is configured to automatically generate GT from one or more knowledge sources, e.g. knowledge domains, shown herein by way of example in LibraryA (172 A). . . . Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses . . . . In one or more embodiments, one initial NL query and one outcome are generated in view of the corresponding structured knowledge, e.g. KG.);); Fig. 3; [0049]-[0051] (Referring to FIG. 3, a flow chart (300) illustrating a process for automatically generating ground truth (GT) from a corresponding knowledge source is provided. . . . For each symptom, a natural language query is generated (304), which in an exemplary embodiment employs variance generation and adding or removing entities.))
As per claim 7, Mahindru/Sheikh teach further comprising:
providing one or more articles of the plurality of articles to a third machine learning model to generate a synthetic question; (Mahindru: [0031]-[0037] (The AI platform (150) is also shown herein operatively coupled to the knowledge base (170), also referred to herein as a corpus of information. As shown, the knowledge base (170) is configured with a plurality of libraries, shown herein by way of example as LibraryA (172 A) and LibraryB (172 B). . . . . The GT manager (152) is configured to automatically generate GT from one or more knowledge sources, e.g. knowledge domains, shown herein by way of example in LibraryA (172 A). . . . Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses); Fig. 3; [0049]-[0051] (Referring to FIG. 3, a flow chart (300) illustrating a process for automatically generating ground truth (GT) from a corresponding knowledge source is provided. . . . For each symptom, a natural language query is generated (304), which in an exemplary embodiment employs variance generation and adding or removing entities.); [0039] (. . . Additionally, the AI platform (150) serves as a back-end system that can make available a variety of knowledge extracted from or represented in documents, network accessible sources and/or structured data sources . . . ); [0028]; [0053])
receiving the synthetic question from the third machine learning model based on the one or more articles; (Mahindru: [0031]-[0037] (The GT manager (152) is configured to automatically generate GT from one or more knowledge sources, e.g. knowledge domains, shown herein by way of example in LibraryA (172 A). . . . Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses . . . . In one or more embodiments, one initial NL query and one outcome are generated in view of the corresponding structured knowledge, e.g. KG.); Fig. 3; [0049]-[0051] (Referring to FIG. 3, a flow chart (300) illustrating a process for automatically generating ground truth (GT) from a corresponding knowledge source is provided. . . . For each symptom, a natural language query is generated (304), which in an exemplary embodiment employs variance generation and adding or removing entities.))
querying a database comprising historical customer questions based on the synthetic question; (Mahindru: Fig. 4; [0051] (As shown and described in FIG. 3, content based GT is automatically generated by leveraging a knowledge graph to generate questions and graph traversal to identify one or more related entities for one or more corresponding answers. As shown and described in FIG. 1, two other forms of GT are generated, including usage and curation based GT. Referring to FIG. 4, a flow chart (400) is provided to illustrate a process for generating usage based GT. A usage log that records or recorded an original query text and all follow-up questions presented to the user, and selections made by the user with an eventual solution or action plan, is provided (402). The usage log includes feedback as to whether the query as represented in the query text was satisfactorily answered. The variable XTotal is assigned to the quantity of queries in the usage log that have feedback indicative of at least a satisfactory resolution (404). A corresponding query counting variable, X, is initialized (406). For queryX, the query text is obtained from the usage log (408), a follow-up question to queryX is also obtained from the usage log (410), and a user selection is obtained (412).))
receive a historical question that has semantic similarity to the synthetic question; and (Mahindru: [0033]-[0037] (TThe GT manager (152) is configured to automatically generate GT from one or more knowledge sources, e.g. knowledge domains, shown herein by way of example in LibraryA (172 A). The generated GT may be content based, usage based, and/or curation based. Content based GT is automatically generated by leveraging a corresponding structured dataset to generate questions based on symptoms, question variants, and graph traversal, and to obtain related entities for the symptoms. In an exemplary embodiment, a symptom is a phrase that describes some problems or issue with a system or any of its components. . . . . Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses.); [0049]-[0051] (Referring to FIG. 3, a flow chart (300) illustrating a process for automatically generating ground truth (GT) from a corresponding knowledge source is provided. . . . A relevant knowledge source is identified and a set of symptoms from the knowledge source is obtained (302). In an exemplary embodiment, a sub-set of symptoms is identified using one or more selection criteria. For each symptom, a natural language query is generated (304), which in an exemplary embodiment employs variance generation and adding or removing entities. In an embodiment, variance generation is a natural language equivalent of a phrase, and is utilized herein to expand the scope of the query through identification of comparable or equivalent terms. The knowledge graph, e.g. structured representation of the knowledge domain, is searched for query text matching the symptom (306). In an exemplary embodiment, a text matching technique, such as universal sentence encoding, is utilized at step (306). Output is generated from the search at step (306) in the form of matching symptoms, also referred to herein as matches (308). Each matching symptom has a corresponding score or weight. In an exemplary embodiment, approximate matching of two or more phrases or sentences is a common operation in natural language processing. The set of matching symptoms from step (308) is subject to a threshold evaluation, which in an exemplary embodiment is directed at quality of the matching symptom. Each matching symptom has a constraint. A constraint node and nodes connected to the constraint node are fetched for each matching symptom (310). The constraint node is not connected to another constraint node. The fetching at step (310) is directed at identification of both the constraint node(s) and all other nodes connected to the constraint node(s). For example, the solution for the symptom “battery issue while charging” and the specific solution to this symptom may be constrained by a specific hardware model number and series. Accordingly, the constraint node(s) connects to all relevant nodes in the graphs as an indicator of the constraint. . . . Referring to FIG. 4, a flow chart (400) is provided to illustrate a process for generating usage based GT. A usage log that records or recorded an original query text and all follow-up questions presented to the user, and selections made by the user with an eventual solution or action plan, is provided (402). The usage log includes feedback as to whether the query as represented in the query text was satisfactorily answered. The variable XTotal is assigned to the quantity of queries in the usage log that have feedback indicative of at least a satisfactory resolution (404). A corresponding query counting variable, X, is initialized (406). For queryX, the query text is obtained from the usage log (408), a follow-up question to queryX is also obtained from the usage log (410), and a user selection is obtained (412)).
identifying the historical question as the request. (Mahindru: [0033]-[0037] (The GT manager (152) is configured to automatically generate GT from one or more knowledge sources, e.g. knowledge domains, shown herein by way of example in LibraryA (172 A). The generated GT may be content based, usage based, and/or curation based. Content based GT is automatically generated by leveraging a corresponding structured dataset to generate questions based on symptoms, question variants, and graph traversal, and to obtain related entities for the symptoms. In an exemplary embodiment, a symptom is a phrase that describes some problems or issue with a system or any of its components. . . . . Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses.); [0049]-[0051] (Referring to FIG. 3, a flow chart (300) illustrating a process for automatically generating ground truth (GT) from a corresponding knowledge source is provided. . . . A relevant knowledge source is identified and a set of symptoms from the knowledge source is obtained (302). In an exemplary embodiment, a sub-set of symptoms is identified using one or more selection criteria. For each symptom, a natural language query is generated (304), which in an exemplary embodiment employs variance generation and adding or removing entities. . . . The knowledge graph, e.g. structured representation of the knowledge domain, is searched for query text matching the symptom (306). . . . . Referring to FIG. 4, a flow chart (400) is provided to illustrate a process for generating usage based GT. . . . For queryX, the query text is obtained from the usage log (408), a follow-up question to queryX is also obtained from the usage log (410), and a user selection is obtained (412)).)
As per claim 8, Mahindru/Sheikh teach further comprising:
providing the plurality of articles and a plurality of historical questions to a third machine learning model; (Mahindru: [0031]-[0037] (The AI platform (150) is also shown herein operatively coupled to the knowledge base (170), also referred to herein as a corpus of information. As shown, the knowledge base (170) is configured with a plurality of libraries, shown herein by way of example as LibraryA (172 A) and LibraryB (172 B). . . . . The GT manager (152) is configured to automatically generate GT from one or more knowledge sources, e.g. knowledge domains, shown herein by way of example in LibraryA (172 A). The generated GT may be content based, usage based, and/or curation based.); [0039] (. . . Additionally, the AI platform (150) serves as a back-end system that can make available a variety of knowledge extracted from or represented in documents, network accessible sources and/or structured data sources . . . ); Fig. 3; [0028]; [0053]; Fig. 4; [0049]-[0051] (As shown and described in FIG. 3, content based GT is automatically generated by leveraging a knowledge graph to generate questions and graph traversal to identify one or more related entities for one or more corresponding answers. As shown and described in FIG. 1, two other forms of GT are generated, including usage and curation based GT. Referring to FIG. 4, a flow chart (400) is provided to illustrate a process for generating usage based GT. A usage log that records or recorded an original query text and all follow-up questions presented to the user, and selections made by the user with an eventual solution or action plan, is provided (402). The usage log includes feedback as to whether the query as represented in the query text was satisfactorily answered. The variable XTotal is assigned to the quantity of queries in the usage log that have feedback indicative of at least a satisfactory resolution (404). A corresponding query counting variable, X, is initialized (406). For queryX, the query text is obtained from the usage log (408), a follow-up question to queryX is also obtained from the usage log (410), and a user selection is obtained (412).))
receiving a non-KB-answerable question from the third machine learning model, the non-KB-answerable question being one among the plurality of historical questions that is not answerable based on the plurality of articles; and (Mahindru: [0031]-[0037] (The GT manager (152) is configured to automatically generate GT from one or more knowledge sources, e.g. knowledge domains, shown herein by way of example in LibraryA (172 A). The generated GT may be content based, usage based, and/or curation based. Content based GT is automatically generated by leveraging a corresponding structured dataset to generate questions based on symptoms, question variants, and graph traversal, and to obtain related entities for the symptoms. Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses.); [0049]-[0051] (As shown and described in FIG. 3, content based GT is automatically generated by leveraging a knowledge graph to generate questions and graph traversal to identify one or more related entities for one or more corresponding answers. As shown and described in FIG. 1, two other forms of GT are generated, including usage and curation based GT. Referring to FIG. 4, a flow chart (400) is provided to illustrate a process for generating usage based GT. . . . For queryX, the query text is obtained from the usage log (408), a follow-up question to queryX is also obtained from the usage log (410), and a user selection is obtained (412)).); Examiner notes that usage-based questions are distinct from content-based questions (KB-answerable questions)).
identifying the non-KB-answerable question as the request. (Mahindru: [0031]-[0037] (The GT manager (152) is configured to automatically generate GT from one or more knowledge sources, e.g. knowledge domains, shown herein by way of example in LibraryA (172 A). The generated GT may be content based, usage based, and/or curation based. Content based GT is automatically generated by leveraging a corresponding structured dataset to generate questions based on symptoms, question variants, and graph traversal, and to obtain related entities for the symptoms. Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses.); [0049]-[0051] (As shown and described in FIG. 3, content based GT is automatically generated by leveraging a knowledge graph to generate questions and graph traversal to identify one or more related entities for one or more corresponding answers. As shown and described in FIG. 1, two other forms of GT are generated, including usage and curation based GT. Referring to FIG. 4, a flow chart (400) is provided to illustrate a process for generating usage based GT. . . . For queryX, the query text is obtained from the usage log (408), a follow-up question to queryX is also obtained from the usage log (410), and a user selection is obtained (412)).)
As per claim 10, Mahindru/Sheikh teach wherein the chatbot is web-based, and the first machine learning model and the chatbot are configured to engage in the automated conversation over a text-based interface. (Mahindru: Fig. 1; Fig. 9; [0030]; [0070] (web browser); [0002] (Chatbots interact with the user through dialog, often either textual (e.g., online or by text) or auditory (e.g., by telephone)); [0026] (An automated virtual agent, referred to herein as a chatbot, is an Artificial Intelligence (AI) program that simulates interactive human conversation by using pre-calculated phrases and auditory or text-based signals.); Fig. 6; [0041] (The simulator (154) interfaces with the dialog system (160) to simulate one or more NL dialog interactions using the automated virtual dialog agent (162) of the dialog system (160). In an exemplary embodiment, the simulator (154) leverages an operatively coupled simulator application (154 A) to conduct a simulated interaction with the chatbot (162). . . . ); [0033]-[0037]; [0053])
As per claim 12, Mahindru/Sheikh teach wherein each of the first machine learning model comprises a second generative large language model (LLM). (Sheikh: [0021] Optionally, the machine learning model agent (ML-Model AA) is at least one of: an internal Large Language Model (Internal-LLM) of the agent-device (AA or micro-AA), an external Large Language Model (Internal-LLM) associated with the agent-device (AA or micro-AA).)
The motivation for applying the known techniques of Sheikh to the teachings of Mahindru is the same as that set forth above, in the rejection of Claim 1.
As per claim 13, Mahindru/Sheikh teach further comprising:
generating a prompt according to the test scenario, the prompt identifying a task or question for generating the expected outcome, (Mahindru: Fig. 6; [0041] (The simulator (154) interfaces with the dialog system (160) to simulate one or more NL dialog interactions using the automated virtual dialog agent (162) of the dialog system (160). In an exemplary embodiment, the simulator (154) leverages an operatively coupled simulator application (154 A) to conduct a simulated interaction with the chatbot (162). . . . ); [0035]-[0037] (Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses. . . . In one or more embodiments, one initial NL query and one outcome are generated in view of the corresponding structured knowledge, e.g. KG.); [0053] (As shown and described in FIG. 1, the simulator (154) is provided to support simulation of a NL dialog interaction. Referring to FIG. 6, a flow chart (600) is provided to illustrate a process for simulating interaction with the dialog system (160). A disambiguation selection path counting variable, N, is initialized (602), GT data is leveraged as a source to drive interaction with the virtual dialog agent, and a query, e.g. queryN, is generated and sent to the automated virtual agent using the operatively coupled simulator application (604). . . . The automated virtual agent responds to the query with a solution or a set of disambiguation options (608).)
wherein the initiating the automated conversation between the first machine learning model and the chatbot comprises: (Mahindru: Fig. 6; [0041]; [0035]-[0037]; [0053])
providing the prompt to the first machine learning model. (Mahindru: Fig. 6; [0041] (The simulator (154) interfaces with the dialog system (160) to simulate one or more NL dialog interactions using the automated virtual dialog agent (162) of the dialog system (160). In an exemplary embodiment, the simulator (154) leverages an operatively coupled simulator application (154 A) to conduct a simulated interaction with the chatbot (162). . . . ); [0035]-[0037] (Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses. . . . In one or more embodiments, one initial NL query and one outcome are generated in view of the corresponding structured knowledge, e.g. KG.); [0053] (As shown and described in FIG. 1, the simulator (154) is provided to support simulation of a NL dialog interaction. Referring to FIG. 6, a flow chart (600) is provided to illustrate a process for simulating interaction with the dialog system (160). A disambiguation selection path counting variable, N, is initialized (602), GT data is leveraged as a source to drive interaction with the virtual dialog agent, and a query, e.g. queryN, is generated and sent to the automated virtual agent using the operatively coupled simulator application (604). . . . The automated virtual agent responds to the query with a solution or a set of disambiguation options (608).)
As per claim 17, Mahindru/Sheikh teach wherein the label comprises one of a resolved label, a not-resolved label, and an unclear label, and (Mahindru: [0042]-[0043] (. . . Output from the evaluation manager is multi-dimensional, including the quantity of disambiguation questions asked and the difference with respect to the test data, whether or not the questions were asked in a particular order, and if the presented solution matches with an intended solution.); [0055]-[0054] (. . . A multi-dimensional output is generated from the comparison at step (708), including: 1. the quantity of disambiguation questions actually asked and the difference between the quantity in the GT, 2. whether or not the questions were asked in the same order, and 3. if the solution presented in the simulation log matches with the GT solution.); [0035] (. . . The generation of a follow-up query or follow-up queries is particularly useful where, for example, an initial response to the initial NL query does not provide a satisfactory response, whether due to an ambiguity in the initial response or another reason. In such instances, a first follow-up or disambiguation query is generated in view of the corresponding structured knowledge, e.g. KG.))
the method further includes identifying, by the processor, receiving by the processor, a reason for the label and an opportunity for improvement. (Mahindru: [0042]-[0043]; [0054]-[0055] (. . . A multi-dimensional output is generated from the comparison at step (708), including: 1. the quantity of disambiguation questions actually asked and the difference between the quantity in the GT, 2. whether or not the questions were asked in the same order, and 3. if the solution presented in the simulation log matches with the GT solution. Insights and recommendations are generated based on output generated for each of the multiple dimensions (710). . . . The output data at step (710) includes insights and recommendations based on various metrics collected. . . . Recommendations corresponding to the collected metrics are directed at an automatic comparison of a defined or pre-defined business goal against an actual metric reflecting performance and identification of one or more corresponding remediation actions. As shown herein, one or more remediation actions for application to the dialog system (160) based on the corresponding output is identified (712) and selectively implemented (714). For example, in an embodiment, the one or more remediation actions may be identified when the performance evaluation of the virtual dialog agent (162) fails to satisfy a performance threshold. In an exemplary embodiment, the recommendation(s), also referred to herein as a recommendation plan, is directed at improving interaction overhead, which may be implemented by collecting additional real-time data, and reducing interaction length.))
As per claim 18, Mahindru/Sheikh teach further comprising:
tracking a percentage of conversations comprising the automated conversation having the resolved label; and (Mahindru: [0003] (Businesses may place certain requirements like accuracy or interaction quality on virtual assistance that is expected to be satisfied before commercial deployment of the virtual system. For example, the virtual system might have a minimum performance requirement of, e.g., 50 percent accuracy for a support agent user base or 90 percent accuracy for an end user base. Accordingly, it is desirable to subject the dialog systems to benchmarking or quality testing before deployment.); [0042]-[0043]; Fig. 7; [0054]-[0055] (The dialog system (160) and the corresponding automated virtual agent (162) are subject to a performance evaluation by leveraging the GT and the corresponding simulation log. Referring to FIG. 7, a flow chart (700) is provided to illustrate a process for conducting the virtual dialog system performance evaluation. As shown, the variable NTotal is assigned to the quantity of query-responses recorded in the simulation log (702), and a corresponding counting variable, N, is initialized (704). For each query-responseN, a corresponding entry in the GT is found (706). The query-response in the simulation log is compared with the query-response in the GT (708). A multi-dimensional output is generated from the comparison at step (708), including: 1. the quantity of disambiguation questions actually asked and the difference between the quantity in the GT, 2. whether or not the questions were asked in the same order, and 3. if the solution presented in the simulation log matches with the GT solution. . . . The output data at step (710) includes insights and recommendations based on various metrics collected. Business goals may be pre-defined in terms of metrics and corresponding metric measurements, such as expected accuracy of the chatbot, along with an acceptable error range. Examples of such metrics includes, but is not limited to, accuracy, interaction overhead, interaction length, quality of follow-up questions, and response time. In an exemplary embodiment, the metrics may be prioritized, such as assignment of a priority to accuracy in place of response time.)
identifying suggestions for modifying the chatbot based on the at least one of the reason for the label and the opportunity for improvement. (Mahindru: [0003]; [0042]-[0043]; [0054]-[0055] (. . . A multi-dimensional output is generated from the comparison at step (708), including: 1. the quantity of disambiguation questions actually asked and the difference between the quantity in the GT, 2. whether or not the questions were asked in the same order, and 3. if the solution presented in the simulation log matches with the GT solution. Insights and recommendations are generated based on output generated for each of the multiple dimensions (710). . . . The output data at step (710) includes insights and recommendations based on various metrics collected. Business goals may be pre-defined in terms of metrics and corresponding metric measurements, such as expected accuracy of the chatbot, along with an acceptable error range. Examples of such metrics includes, but is not limited to, accuracy, interaction overhead, interaction length, quality of follow-up questions, and response time. In an exemplary embodiment, the metrics may be prioritized, such as assignment of a priority to accuracy in place of response time. Recommendations corresponding to the collected metrics are directed at an automatic comparison of a defined or pre-defined business goal against an actual metric reflecting performance and identification of one or more corresponding remediation actions. As shown herein, one or more remediation actions for application to the dialog system (160) based on the corresponding output is identified (712) and selectively implemented (714). For example, in an embodiment, the one or more remediation actions may be identified when the performance evaluation of the virtual dialog agent (162) fails to satisfy a performance threshold. In an exemplary embodiment, the recommendation(s), also referred to herein as a recommendation plan, is directed at improving interaction overhead, which may be implemented by collecting additional real-time data, and reducing interaction length.))
As per claim 19, Mahindru/Sheikh teach wherein the label comprising one of an attempted-to-answer label and a no-attempt-to-answer label. (Mahindru: [0042]-[0043] (. . . Output from the evaluation manager is multi-dimensional, including the quantity of disambiguation questions asked and the difference with respect to the test data, whether or not the questions were asked in a particular order, and if the presented solution matches with an intended solution.); [0051]-[0055] (. . . A multi-dimensional output is generated from the comparison at step (708), including: 1. the quantity of disambiguation questions actually asked and the difference between the quantity in the GT, 2. whether or not the questions were asked in the same order, and 3. if the solution presented in the simulation log matches with the GT solution.))
As per claim 20, this claim is substantially similar to claim 1 and is therefore rejected in the same manner as this claim, as set forth above. Additionally claim 20 recites receiving an evaluation of the automated conversation from the second machine learning model based on the recording of the automated conversation and the expected outcome. (Mahindru: Figs. 6-7; [0042]-[0043] (As shown herein, the evaluation manager (156), which is operatively coupled to the simulator (154), is configured to evaluate performance of the automated virtual dialog agent (162). The evaluation manager (156) compares the simulated interaction that is represented as simulation data with GT for the corresponding knowledge domain. The GT employed in the comparison may include one or more of the GT types, including content, usage, and curation based GT. Details of the simulation interaction evaluation are shown and described in FIG. 7. Output from the evaluation manager is multi-dimensional, including the quantity of disambiguation questions asked and the difference with respect to the test data, whether or not the questions were asked in a particular order, and if the presented solution matches with an intended solution); [0054]-[0055] (The dialog system (160) and the corresponding automated virtual agent (162) are subject to a performance evaluation by leveraging the GT and the corresponding simulation log. Referring to FIG. 7, a flow chart (700) is provided to illustrate a process for conducting the virtual dialog system performance evaluation. As shown, the variable NTotal is assigned to the quantity of query-responses recorded in the simulation log (702), and a corresponding counting variable, N, is initialized (704). For each query-responseN, a corresponding entry in the GT is found (706). The query-response in the simulation log is compared with the query-response in the GT (708). A multi-dimensional output is generated from the comparison at step (708), including: 1. the quantity of disambiguation questions actually asked and the difference between the quantity in the GT, 2. whether or not the questions were asked in the same order, and 3. if the solution presented in the simulation log matches with the GT solution. Insights and recommendations are generated based on output generated for each of the multiple dimensions (710).)
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Mahindru/Sheikh in view of Wu (US PGP 2022/0139384).
As per claim 9, Mahindru/Sheikh teach further comprising:
providing the plurality of articles to a third machine learning model to generate a synthetic question; (Mahindru: [0031]-[0037] (The AI platform (150) is also shown herein operatively coupled to the knowledge base (170), also referred to herein as a corpus of information. As shown, the knowledge base (170) is configured with a plurality of libraries, shown herein by way of example as LibraryA (172 A) and LibraryB (172 B). . . . . The GT manager (152) is configured to automatically generate GT from one or more knowledge sources, e.g. knowledge domains, shown herein by way of example in LibraryA (172 A). . . . Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses); Fig. 3; [0049]-[0051] (Referring to FIG. 3, a flow chart (300) illustrating a process for automatically generating ground truth (GT) from a corresponding knowledge source is provided. . . . For each symptom, a natural language query is generated (304), which in an exemplary embodiment employs variance generation and adding or removing entities.); [0039] (. . . Additionally, the AI platform (150) serves as a back-end system that can make available a variety of knowledge extracted from or represented in documents, network accessible sources and/or structured data sources . . . ); [0028]; [0053])
receiving a plurality of synthetic questions from the third machine learning model based on the plurality of articles; (Mahindru: [0031]-[0037] (The GT manager (152) is configured to automatically generate GT from one or more knowledge sources, e.g. knowledge domains, shown herein by way of example in LibraryA (172 A). . . . Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses . . . . In one or more embodiments, one initial NL query and one outcome are generated in view of the corresponding structured knowledge, e.g. KG. In one or more other embodiments, an initial NL query is generated and one or more follow-up NL queries are generated to arrive at the NL outcome as part of a multi-turn or multi-step conversation or interaction.); Fig. 3; [0049]-[0051] (Referring to FIG. 3, a flow chart (300) illustrating a process for automatically generating ground truth (GT) from a corresponding knowledge source is provided. . . . For each symptom, a natural language query is generated (304), which in an exemplary embodiment employs variance generation and adding or removing entities.))
querying a database comprising historical customer questions based on the synthetic question; (Mahindru: Fig. 4; [0051] (As shown and described in FIG. 3, content based GT is automatically generated by leveraging a knowledge graph to generate questions and graph traversal to identify one or more related entities for one or more corresponding answers. As shown and described in FIG. 1, two other forms of GT are generated, including usage and curation based GT. Referring to FIG. 4, a flow chart (400) is provided to illustrate a process for generating usage based GT. A usage log that records or recorded an original query text and all follow-up questions presented to the user, and selections made by the user with an eventual solution or action plan, is provided (402). The usage log includes feedback as to whether the query as represented in the query text was satisfactorily answered. The variable XTotal is assigned to the quantity of queries in the usage log that have feedback indicative of at least a satisfactory resolution (404). A corresponding query counting variable, X, is initialized (406). For queryX, the query text is obtained from the usage log (408), a follow-up question to queryX is also obtained from the usage log (410), and a user selection is obtained (412).))
identifying the historical question as the request. (Mahindru: [0033]-[0037] (The GT manager (152) is configured to automatically generate GT from one or more knowledge sources, e.g. knowledge domains, shown herein by way of example in LibraryA (172 A). The generated GT may be content based, usage based, and/or curation based. Content based GT is automatically generated by leveraging a corresponding structured dataset to generate questions based on symptoms, question variants, and graph traversal, and to obtain related entities for the symptoms. In an exemplary embodiment, a symptom is a phrase that describes some problems or issue with a system or any of its components. . . . . Interactions with the chatbot (162) are in the form of queries and corresponding responses, and a sequence of follow-up disambiguation questions and their responses.); [0049]-[0051] (Referring to FIG. 3, a flow chart (300) illustrating a process for automatically generating ground truth (GT) from a corresponding knowledge source is provided. . . . A relevant knowledge source is identified and a set of symptoms from the knowledge source is obtained (302). In an exemplary embodiment, a sub-set of symptoms is identified using one or more selection criteria. For each symptom, a natural language query is generated (304), which in an exemplary embodiment employs variance generation and adding or removing entities. . . . The knowledge graph, e.g. structured representation of the knowledge domain, is searched for query text matching the symptom (306). . . . . Referring to FIG. 4, a flow chart (400) is provided to illustrate a process for generating usage based GT. . . . For queryX, the query text is obtained from the usage log (408), a follow-up question to queryX is also obtained from the usage log (410), and a user selection is obtained (412)).)
Mahindru/Sheikh does not explicitly disclose the following known technique which is taught by Wu:
receive a historical question that is semantically distant from the synthetic question; and (Wu: [0035]-[0037] (. . . Several system responses may be randomly sampled from the corpus as negative samples.); [0023]-[0024] (. . . In some embodiments, the RCL may be formulated by applying a dual-encoder approach and simulating multiple negative samples.); [0050])
This known technique is applicable to the method of Mahindru/Sheikh as they both share characteristics and capabilities, namely, they are directed to dialogue systems.
One of ordinary skill in the art at the time of filing would have recognized that applying the known technique of Wu would have yielded predictable results and resulted in an improved method. It would have been recognized that applying the technique of Wu to the teachings of Mahindru/Sheikh would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such semantically distant features into similar methods. Further, applying semantic distance to the historical question of Mahindru/Sheikh would have been recognized by those of ordinary skill in the art as resulting in an improved method that would improve the pre-training of TOD language models. (Wu: [0015])
Claims 11 and 14-16 are rejected under 35 U.S.C. 103 as being unpatentable over Mahindru/Sheikh in view of Henderson (US PGP 2020/0152184).
As per claim 11, Mahindru/Sheikh teach wherein the chatbot is voice-based, and the first machine learning model and the chatbot are configured to engage in the automated conversation . . . (Mahindru: [0002] (Chatbots interact with the user through dialog, often either textual (e.g., online or by text) or auditory (e.g., by telephone)); [0026] (An automated virtual agent, referred to herein as a chatbot, is an Artificial Intelligence (AI) program that simulates interactive human conversation by using pre-calculated phrases and auditory or text-based signals.); [0029] (The computing devices (180), (182), (184), (186), (188), and (190) may be provided with a visual display, audio interface, an audio-video interface, or other types of interfaces configured to allow the user to interface with a representation of a virtual agent, e.g. chatbot, (162).)
While, Mahindru discloses text and speech interfaces, Mahindru/Sheikh does not explicitly disclose an text-to-speech and speech-to-text interface. Still, one of ordinary skill in the art would have recognized such features to be obvious, as they were well established at the time of invention.
For example, Henderson teaches . . . via a text-to-speech and speech-to-text interface. (Henderson: [0140] (In this method, at each dialogue turn, an input user utterance (spoken or text) is received. If the input is spoken, the utterance is converted to text using ASR.); [0348] The utterance may be directly output to the user as text (e.g. on a screen), or it may be converted to a speech signal using a text to speech system and outputted as audio. Any text to speech process may be used for this final step.))
This known technique is applicable to the method of Mahindru/Sheikh as they both share characteristics and capabilities, namely, they are directed to dialogue system simulations.
One of ordinary skill in the art at the time of filing would have recognized that applying the known technique of Henderson would have yielded predictable results and resulted in an improved method. It would have been recognized that applying the technique of Henderson to the teachings of Mahindru/Sheikh would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such text-to-speech and speech-to-text interface features into similar methods. Further, applying the text-to-speech and speech-to-text interface to the interface of Mahindru/Sheikh would have been recognized by those of ordinary skill in the art as resulting in an improved method that would improve the functioning of dialogue systems and the training of such systems and efficiently generate data for training such systems. (Henderson: [0003]-[0004])
As per claim 14, Mahindru/Sheikh teach the invention of claim 1 as set forth above.
Mahindru/Sheikh does not explicitly disclose the following known technique which is taught by Henderson:
identifying at least one feature associated with a simulated user; and (Henderson: [0379]-[0386] (Optionally, as well as comprising one or more combinations of dialogue slots and values, each dialogue scenario may also comprise sampled user profile information. The user profile helps simulate different types of users (e.g., more patient ones, more verbose ones), which leads to more varied dialogues. A user profile may be defined by one or more parameters. The parameters define the user behaviour in the dialogue. In S703 a, for each scenario, the parameters of the user profile are also sampled, to generate a user profile corresponding to the scenario. The sampling may be performed randomly or, for greater variance, sampled in a controlled way to ensure that each parameter value occurred at least with a certain number of dialogue scenarios for example. Thus the dialogue scenario may comprise a user profile sample (in which each user profile parameter has a sampled value) and a user goal (in which each slot has a sampled value). For example, in stage (a) of FIG. 8(b), a sampled dialogue scenario is shown in which there is a User profile and a User goal.); [0414] (. . . simulated user . . .)
providing the at least one of feature to the first machine learning model, (Henderson: [0379]-[0386] (Optionally, as well as comprising one or more combinations of dialogue slots and values, each dialogue scenario may also comprise sampled user profile information. The user profile helps simulate different types of users (e.g., more patient ones, more verbose ones), which leads to more varied dialogues. A user profile may be defined by one or more parameters. The parameters define the user behaviour in the dialogue. In S703 a, for each scenario, the parameters of the user profile are also sampled, to generate a user profile corresponding to the scenario. The sampling may be performed randomly or, for greater variance, sampled in a controlled way to ensure that each parameter value occurred at least with a certain number of dialogue scenarios for example. Thus the dialogue scenario may comprise a user profile sample (in which each user profile parameter has a sampled value) and a user goal (in which each slot has a sampled value). For example, in stage (a) of FIG. 8(b), a sampled dialogue scenario is shown in which there is a User profile and a User goal.); [0412]-[0414] (In S704, dialogue act sequences are generated from the dialogue scenarios. Thus in S703 a, many diverse scenarios are sampled, in S703 b these are converted into user agendas and then in S704 a dialogue simulation is performed for each scenario to generate the sequences, corresponding to the exchange of dialogue acts between the user and the system. These are generated using a simulated user, the user agenda and a system policy model (such as described in relation to S305 above).)
wherein the initiating the automated conversation between the first machine learning model and the chatbot is further based on the at least one feature. (Henderson: [0379]-[0386] (Optionally, as well as comprising one or more combinations of dialogue slots and values, each dialogue scenario may also comprise sampled user profile information. The user profile helps simulate different types of users (e.g., more patient ones, more verbose ones), which leads to more varied dialogues. A user profile may be defined by one or more parameters. The parameters define the user behaviour in the dialogue. In S703 a, for each scenario, the parameters of the user profile are also sampled, to generate a user profile corresponding to the scenario. The sampling may be performed randomly or, for greater variance, sampled in a controlled way to ensure that each parameter value occurred at least with a certain number of dialogue scenarios for example. Thus the dialogue scenario may comprise a user profile sample (in which each user profile parameter has a sampled value) and a user goal (in which each slot has a sampled value). For example, in stage (a) of FIG. 8(b), a sampled dialogue scenario is shown in which there is a User profile and a User goal.); [0412]-[0414] (In S704, dialogue act sequences are generated from the dialogue scenarios. Thus in S703 a, many diverse scenarios are sampled, in S703 b these are converted into user agendas and then in S704 a dialogue simulation is performed for each scenario to generate the sequences, corresponding to the exchange of dialogue acts between the user and the system. These are generated using a simulated user, the user agenda and a system policy model (such as described in relation to S305 above).)
This known technique is applicable to the method of Mahindru/Sheikh as they both share characteristics and capabilities, namely, they are directed to dialogue system simulations.
One of ordinary skill in the art at the time of filing would have recognized that applying the known technique of Henderson would have yielded predictable results and resulted in an improved method. It would have been recognized that applying the technique of Henderson to the teachings of Mahindru/Sheikh would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate the feature associated with a simulated user into similar methods. Further, applying the identifying at least one feature associated with a simulated user; and providing the at least one of feature to the first machine learning model, wherein the initiating the automated conversation between the first machine learning model and the chatbot is further based on the at least one feature to the teachings of Mahindru/Sheikh would have been recognized by those of ordinary skill in the art as resulting in an improved method that would lead to more varied dialogues and thus improve the functioning of dialogue systems and the training of such systems by (Henderson: [0379], [0003]-[0004]).
As per claim 15, Mahindru/Henderson teach wherein the at least one feature comprises at least one of a personality from among a plurality of personalities, a back-story from among a plurality of back-stories, an age from among a plurality of ages, and a job title from among a plurality of job titles. (Henderson: [0379]-[0386] (Optionally, as well as comprising one or more combinations of dialogue slots and values, each dialogue scenario may also comprise sampled user profile information. The user profile helps simulate different types of users (e.g., more patient ones, more verbose ones), which leads to more varied dialogues. A user profile may be defined by one or more parameters. The parameters define the user behaviour in the dialogue. . . . The different user profiles may account for one or more of the following factors: patient vs impatient users (some users might terminate the dialogue more quickly, this can be controlled by the parameters “max_number_repeats” (which may have a default value 2 for example) and “max_request_alternative” (which may have a default value 1 for example)), verbose vs laconic users (some users prefer to utter more information in a single dialogue turn), undecided users (some users might have a goal change in the middle of the conversation, controlled by a parameter “changed_goal_prob” (which may have a default value 0.1 for example)). Some or all of these and other behaviours are controlled by sets of hyper-parameters on the user side.)
The motivation for applying the known techniques of Henderson to the teachings of Mahindru/Sheikh is the same as that set forth above, in the rejection of Claim 14.
As per claim 16, Mahindru/Henderson teach wherein the identifying the at least one feature comprises at least one of:
randomly selecting the personality from among the plurality of personalities; (Henderson: [0379]-[0386] (Optionally, as well as comprising one or more combinations of dialogue slots and values, each dialogue scenario may also comprise sampled user profile information. The user profile helps simulate different types of users (e.g., more patient ones, more verbose ones), which leads to more varied dialogues. A user profile may be defined by one or more parameters. The parameters define the user behaviour in the dialogue. In S703 a, for each scenario, the parameters of the user profile are also sampled, to generate a user profile corresponding to the scenario. The sampling may be performed randomly or, for greater variance, sampled in a controlled way to ensure that each parameter value occurred at least with a certain number of dialogue scenarios for example. Thus the dialogue scenario may comprise a user profile sample (in which each user profile parameter has a sampled value) and a user goal (in which each slot has a sampled value). For example, in stage (a) of FIG. 8(b), a sampled dialogue scenario is shown in which there is a User profile and a User goal. . . . The different user profiles may account for one or more of the following factors: patient vs impatient users (some users might terminate the dialogue more quickly, this can be controlled by the parameters “max_number_repeats” (which may have a default value 2 for example) and “max_request_alternative” (which may have a default value 1 for example)), verbose vs laconic users (some users prefer to utter more information in a single dialogue turn), undecided users (some users might have a goal change in the middle of the conversation, controlled by a parameter “changed_goal_prob” (which may have a default value 0.1 for example)). Some or all of these and other behaviours are controlled by sets of hyper-parameters on the user side.)
randomly selecting the back-story from among the plurality of back-stories;
randomly selecting the age from among the plurality of ages; and
randomly selecting the job title from among a plurality of job titles.
The motivation for applying the known techniques of Henderson to the teachings of Mahindru/Sheikh is the same as that set forth above, in the rejection of Claim 14.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Mcgann (US PGP 2021/0160373) – disclosing speech data may be converted to text and vice versa.
Braz (US PGP 2018/0293973) -- testing a dialog personalization by running scenarios to determine if a conversational agent is capable of changing its tone according to a name, a gender, an age, etc. of the user.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JENNIFER V LEE whose telephone number is (571)272-4778. The examiner can normally be reached Monday - Friday 9AM - 5PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JEFFREY A. SMITH can be reached at (571)272-6763. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JENNIFER V LEE/Examiner, Art Unit 3688
/Jeffrey A. Smith/Supervisory Patent Examiner, Art Unit 3688
Prosecution Insights