DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendments filed 05/10/2024 have been accepted and considered in this office action. Claims 1 and 13 have been amended.
Response to Arguments
Applicant’s arguments with respect to claims 1-20 have been considered but are moot in view of new grounds of rejection necessitated by the applicant’s amendments to the claims.
Examiner Notes on Patent Subject Matter Eligibility under 35 U.S.C. 101
Independent claims 1 and 13 define a method that evaluates the intent-recognition of a chatbot through a health metric that can later be utilized by a developer. Under step 2A prong 1, the mental process category of abstract idea would not be appropriate as a human could not practically perform the recited process involving utterance embeddings alongside their complex mathematical transformations and high-dimensional vector spaces. The remaining dependent claims inherit the patent eligible subject matter of their parent claims, and thus, are found to be directed towards patent eligible subject matter under 35 U.S.C. 101 by virtue of their dependency.
Furthermore, the claims are patent-eligible under 35 U.S.C. § 101 because they are directed toward a technological improvement to a machine learning model, rather than a mere abstract idea. The specifications clearly details specific "improvement reasoning" inherent to the NLU domain. For example, the stated goal of the method is "for enhancing intent recognition" (P[0075]), and the described dynamic feedback process "may allow a user to improve intent health in a highly efficient manner" (P[0081]). This addresses a technical problem within the field of AI and NLU conversational bots, specifically improving model performance and efficiency. Furthermore, the claims are specific to the "LLM itself" and are not abstract management techniques. The method requires concrete technical steps such as utilizing a "machine learning model trained for natural language understanding (NLU)" (P[0075]), and "generating an utterance embedding for each of the retrieved utterances" (P[0075]). The core of the invention lies in the specific calculation of metrics within the complex, technical N-dimensional embedding space, utilizing "anomaly detection algorithm that computes the local density of a given utterance embedding" (P[0077]), thereby providing a non-abstract, technical solution tied to the functionality of the conversational bot system itself.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1 is/are rejected under 35 U.S.C. 103 as being unpatentable over Temraz et al. (hereinafter Temraz) (US 20240330597 A1) in view of Duong et al. (hereinafter Duong_1) (US 20210303798 A1) in further view of Libert et al. (hereinafter Libert) (US 20200193312 A1).
Regarding claim 1 Temaraz teaches: A method for evaluating an intent health related to a conversational bot for enhancing intent recognition, wherein the conversational bot comprises a machine learning model trained for natural language understanding (NLU) within a NLU domain that is defined by a collection of intents and sets of associated utterances (P[0150]: “Some intents may have significantly less (e.g., three standard deviations from a mean value) training utterances than other intents which may affect their performance.”), wherein the conversational bot is configured to select responses to provide to a customer during a conversation based on identifying a correct intent from among the collection of intents given an utterance made by the customer (P[0002]: “more specifically, to identifying patterns within conversations between intent-based chatbots and chatbot users for optimizing the chatbots.”), wherein each intent comprises a different intention of the customer and is defined by the set of utterances associated therewith (P[0150]: “Some intents may have significantly less (e.g., three standard deviations from a mean value) training utterances than other intents which may affect their performance.”), the method comprising the steps of:
retrieving, for the conversational bot, the collection of intents and associated utterances (P[0150]: “Some intents may have significantly less (e.g., three standard deviations from a mean value) training utterances than other intents which may affect their performance. This may be acceptable for some less relevant intents such as “goodbye” or “welcome.” However, for more relevant intents, a warning may be generated indicating training utterances below a training utterance threshold. According to an implementation, based on such a warning, training data up to the training utterance threshold (e.g., average size) of intents may be automatically generated. Such training data may be generated using an LLM/generative model, as discussed herein, to augment existing training data associated with respective intents.”);
generating an utterance embedding for each of the retrieved utterances (P[0201]: “Sentences within each concept may be embedded using, for example, a bag-of-words model, sentence encoders, or the like or a combination thereof. The embedding may be visualized with PCA, T-SNE, UMAP, and/or any other technique for visualization of high dimensional data, as discussed herein. Labels in the visualization may be concepts. Data points may be embedded sentences mapped into 2D or 3D vector space.”);
calculating scores for utterance-level health indicators for each intent of the collection of intents (P[0076] – P[0077]: “Semantic similarity may be computed using cosine similarity if sentences are embedded with deep neural network”, “A confusion matrix may be used to determine whether a trained classifier is unable to distinguish between intents (e.g., beyond a minimum threshold”);
wherein, when described in relation to a first intent of the collection of intents, the utterance-level health indicators comprise (P[0102]: “For each intent, a list of similarity scores with other intents may be extracted, if the similarity scores are above the similarity threshold (e.g., 50% in this example)”):
an utterance in conflict indicator that calculates a score based on a percentage of the utterances associated with the first intent comprising an utterance embedding that is computed to have a computed semantic similarity with the utterance embeddings of the utterances associated with the other intents that exceeds a first predetermined similarity threshold (P[0050]-P[0052]: “"Interstices result where these candidate intents overlap and more than one candidate intent matches the utterance of the user. For example, Interstice 3,4 is the region where the utterance is ambiguous” (regions of interstices and ambiguity and utterance of user read on utterance in conflict indicator), “The threshold can be expressed as an arbitrary score, such as a numeric value represented by 'Y.'" and "...multiple candidate intents have scores that exceed the threshold Y.” (reads on predetermined similarity threshold), “As shown in FIG. 5b, the threshold can be expressed as a percentage, such as 0.65 (i.e., 65%).”, (reads on percentage of utterances linked to threshold, P[0102]: “For each intent, a list of similarity scores with other intents may be extracted, if the similarity scores are above the similarity threshold (e.g., 50% in this example)”); and
Temraz does not teach:
an utterance outlier indicator that calculates a score based on a percentage of the utterances associated with the first intent comprising an utterance embedding that is computed to have a local density that is less than local densities computed for neighboring utterance embeddings beyond an acceptable threshold level of deviation.
However, Duong_1 teaches:
an utterance outlier indicator (Duong_1: P[0006]: "an outlier detection model constructed with a distance or density algorithm for outlier detection;" reads on the indicator) that calculates a score (P[0006]: "predicting, using the outlier detection model, the second probability as to whether the utterance belongs to the target domain based on the determined distance or density deviation;" reads on calculating a score, where the probability is the score) based on a percentage of the utterances associated with the first intent comprising an utterance embedding (P[0006]: "generating a sentence embedding for the utterance; outlier detection model constructed with a distance or density algorithm for outlier detection;" reads on the embedding and the use of a density algorithm, which inherently uses data points like utterances) that is computed to have a local density that is less than local densities computed for neighboring utterance embeddings (P[0006]: "determining, using the outlier detection model, a distance or density deviation between the sentence embedding for the utterance and embedding representations for neighboring clusters;" reads on computing and comparing the density/deviation to neighbors) beyond an acceptable threshold level of deviation (P[0006]: "evaluating the first probability and the second probability to determine a final probability as to whether the utterance belongs to the target domain; and classifying the utterance as in-domain or out-of-domain for the chatbot based on the final probability;" reads on the use of a threshold, implied in the evaluation and final classification step).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Temraz in view of Duong_1. Doing so would have provided the chatbot data generation methods of Temraz (Temraz, Abstract) and the advanced out-of-domain detection methods of Duong_1 (Duong_1, Abstract) thus improving the self-diagnostic capabilities for future improvements of the chatbot’s intent classification systems.
The combination of Tehraz and Duong_1 do not teach: calculating an overall intent health score for each intent of the collection of intents, wherein the overall intent health score is based on a weighted combination of the calculated scores for the utterance-level health indicators for the intent, the weighted combination of the calculated scores uses respective weights for each of the utterance-level health indicators, and the weights represent a relative importance of each utterance-level health indicator;
However, Libert teaches:
calculating an overall intent health score for each intent of the collection of intents, wherein the overall intent health score is based on a weighted combination of the calculated scores for the utterance-level health indicators for the intent, the weighted combination of the calculated scores uses respective weights for each of the utterance-level health indicators, and the weights represent a relative importance of each utterance-level health indicator (Libert, P[0042]: “In another, it develops a composite business model profile, which represents each company as a weighted combination of the four classes, for example a linear combination, with each class weighted according to its proportionate or percentage presence” (discloses combining multiple components (classes) with respective weights that reflect their significance (proportionate presence) which is structurally analogous to the claim), P[0044]: “Scoring module 126 includes data and machine-executable instructions for processing classifications of business entities in combination with other financial metrics to produce various scores that are useful in analyzing businesses.” (teaches weighted composite scoring explicitly with the use of weighted combinations of multiple classification components for an overall score));
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Temraz in view of Duong_1 to incorporate the teachings of Libert. Doing so would have provided the weighted scoring system’s accuracy and flexibility of Libert (Libert, Abstract, P[0042]) with the chatbot data generation methods of Temraz (Temraz, Abstract) and the advanced out-of-domain detection methods of Duong_1 (Duong_1, Abstract) thus improving the self-diagnostic capabilities for future improvements based on the most important weighted parameters of the chatbot’s intent classification systems through known weighted scoring mechanisms.
Claim(s) 2, 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Temraz et al. (hereinafter Temraz) (US 20240330597 A1) in view of Duong et al. (hereinafter Duong_1) (US 20210303798 A1), in further view of Libert et al. (hereinafter Libert) (US 20200193312 A1) and Jacob et al. (hereinafter Jacob_1) (US 20220309247 A1).
Regarding claim 2, the combination of Temraz and Duong_1, and Libert teach the methods according to claim 1.
The combination of Temraz, Duong_1, and Libert do not teach:
The method of claim 1, further comprising the step of: calculating an intent level health indicator comprising an intent-level static validation indicator, wherein, when described in relation to the first intent of the collection of intents, the intent-level static validation indicator comprises:
a too many utterances indicator that determines whether a number of utterances associated with the first intent exceeds a maximum threshold and
a too few utterances indicator that determines whether the number of utterances associated with the first intent less than a minimum threshold
However, Jacob_1 teaches:
The method of claim 1, further comprising the step of: calculating an intent level health indicator comprising an intent-level static validation indicator, wherein, when described in relation to the first intent of the collection of intents, the intent-level static validation indicator comprises:
a too many utterances indicator that determines whether a number of utterances associated with the first intent exceeds a maximum threshold (Jacob_1, P[0033] – P[0034]: The "data analysis unit 118 is configured to evaluate data-imbalances" [P] and checking if a count is "greater than the utterance count (U) combined with the tolerance limit (t)," such as 20 < 10+5 [P], reads on the concept of a maximum threshold for "too many" utterances); and
a too few utterances indicator that determines whether the number of utterances associated with the first intent less than a minimum threshold (Jacob_1, P[0033] – P[0034]: The "predefined utterance count (U) is selected as 10" [P] which provides a "minimum of 10 utterances" for a uniform distribution, reads on the concept of a minimum threshold for "too few" utterances).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Temraz in view of Duong_1 to incorporate the teachings of Libert and Jacob_1. Doing so would have provided the error correction methods of Jacob_1 (Jacob_1, Abstract) with the weighted scoring system’s accuracy and flexibility of Libert (Libert, Abstract, P[0042]) chatbot data generation methods of Temraz (Temraz, Abstract) and the advanced out-of-domain detection methods of Duong_1 (Duong_1, Abstract) thus improving the self-diagnostic capabilities for future improvements based on the most important weighted parameters of the chatbot’s intent classification systems through known weighted scoring mechanisms and fine-tuning methods.
Regarding claim 3, Temraz, Duong_1, Libert, and Jacob_1 teach the methods described in claim 2.
Temraz, Duong_1, Libert, and Jacob_1 do not explicitly teach:
wherein the overall intent health score is based on a weighted combination of the calculated scores for the utterance-level health indicators and the intent-level static validation indicator.
Claim 3 differs from the prior art combination of Temraz and Duong_1, in that Temraz and Duong_1 fail to explicitly disclose incorporating the specific "intent-level static validation indicator" of Claim 2 (which is disclosed by the combination of Temraz, Duong_1, Libert, and Jacob_1) into the existing "overall intent health score" weighted combination taught by Claim 1. Using different, known metrics (such as the quantity metrics disclosed by Jacob_1) to contribute to a comprehensive system health score is known in the art, in particular, such as the overall health score disclosed by Temraz, Duong_1, and Libert. Therefore, it would have been obvious to one having ordinary skill in the art to modify the health evaluation method taught by the combination of Temraz, Duong_1, and Libert to include the metrics from Jacob_1, as it merely constitutes the combination of known elements to achieve the predictable result of obtaining a more comprehensive evaluation of intent health.
Claim(s) 4, 13, 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Temraz et al. (hereinafter Temraz) (US 20240330597 A1) in view of Duong et al. (hereinafter Duong_1) (US 20210303798 A1), in further view of Libert et al. (hereinafter Libert) (US 20200193312 A1), Jacob et al. (hereinafter Jacob_1) (US 20220309247 A1), and He et al. (hereinafter He) (CN 109284371 B).
Regarding claim 4, Temraz, Duong_1, Libert, and Jacob_1 teach the methods according to claim 3
Temraz, Duong_1, Libert, and Jacob_1 do not teach:
wherein, in calculating the utterance outlier indicator, the local densities are each calculated via an anomaly detection algorithm that computes the local density of a given utterance embedding with respect to neighboring utterance embeddings in N-dimensional space wherein N is a dimension of an embedding vector of the utterance embeddings.
However, He teaches:
wherein, in calculating the utterance outlier indicator , the local densities are each calculated via an anomaly detection algorithm that computes the local density of a given utterance embedding with respect to neighboring utterance embeddings in N-dimensional space wherein N is a dimension of an embedding vector of the utterance embeddings (He, Page 8: Euclidean distance to determine proximity to the first vector in each classification category reads on comparing to neighbors; The LOF score itself functions as this "outlier indicator," highlighting vectors with a local density significantly lower than their neighbors (reads on utterance embedding local density comparisons beyond an acceptable threshold); "DBSCAN algorithm with noise" and "LOF value" (reads on the anomaly detection algorithms used for this density-based calculation for outliers); the use of vectors and distance metrics inherently takes place in N-dimensional space in these algorithms.).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Temraz in view of Duong_1, Libert, and Jacob_1 to incorporate the teachings of He. Doing so would have provided the thorough outlier detection methods of He (He, Abstract, Page 8), with the weighted scoring system’s accuracy and flexibility of Libert (Libert, Abstract, P[0042]), the error correction methods of Jacob_1 (Jacob_1, Abstract), the chatbot data generation methods of Temraz (Temraz, Abstract) and the advanced out-of-domain detection methods of Duong_1 (Duong_1, Abstract) thus improving the self-diagnostic capabilities for future improvements based on the most important weighted parameters of the chatbot’s intent classification systems through known weighted scoring mechanisms and fine-tuning methods including dealing with outliers.
Regarding claim 13, claim 13 recites the system corresponding to the methods presented in claim 1 and claim 4 and is rejected under the same grounds as above
Regarding claim 14, claim 14 recites the system corresponding to the methods presented in claim 2 and claim 3 and is rejected under the same grounds as above
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Temraz et al. (hereinafter Temraz) (US 20240330597 A1) in view of Duong et al. (hereinafter Duong_1) (US 20210303798 A1), and in further view of Libert et al. (hereinafter Libert) (US 20200193312 A1), Jacob et al. (hereinafter Jacob_1) (US 20220309247 A1), and Duong et al. (hereinafter Duong_2) (US 20210304075 A1).
Regarding claim 5, Temraz, Duong_1, Libert, and Jacob_1 teach the methods according to claim 3
Temraz, Duong_1, Libert, and Jacob_1 do not teach:
wherein, when the number of utterances associated with the first intent either exceeds or is found to be less than the maximum threshold or minimum threshold, respectively, the step of calculating the overall intent health score for each intent further comprises subtracting a predetermined constant from the weighted combination of the calculated scores for the utterance-level health indicators for the intent.
However, Duong_2 teaches:
wherein, when the number of utterances associated with the first intent either exceeds or is found to be less than the maximum threshold or minimum threshold, respectively, the step of calculating the overall intent health score for each intent further comprises subtracting a predetermined constant from the weighted combination of the calculated scores for the utterance-level health indicators for the intent (Duong_2 P[0110] – P[0112]: "training data is provided for six intents a-f The represented intents may distributed across a raw plurality of utterances" (reads on training data for intents), "the proportional representation of the batch is less than a threshold amount" (reads on less than the minimum threshold), "overly represented utterances" (reads on exceeds the maximum threshold), "proportional balancing techniques to reduce bias at the intent level" (reads on "bias" as the constant to be subtracted), "including minimum training utterances in a batch" (reads on applying the correction), "setting the above constraints on selection of training data would eliminate bias toward any particular intent" (reads on "eliminate bias" as subtracting/neutralizing the constant)).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Temraz in view of Duong_1, Libert, and Jacob_1 to incorporate the teachings of Duong_2. Doing so would have provided the batch bias eliminating methods of Duong_2 (Duong_2, Abstract) with the error correction methods of Jacob_1 (Jacob_1, Abstract), the weighted scoring system’s accuracy and flexibility of Libert (Libert, Abstract, P[0042]), the chatbot data generation methods of Temraz (Temraz, Abstract) and the advanced out-of-domain detection methods of Duong_1 (Duong_1, Abstract) thus improving the self-diagnostic capabilities for future improvements based on the most important weighted parameters of the chatbot’s intent classification systems through known weighted scoring mechanisms and fine-tuning methods such as batch-bias elimination.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Temraz et al. (hereinafter Temraz) (US 20240330597 A1) in view of Duong et al. (hereinafter Duong_1) (US 20210303798 A1), and in further view of Libert et al. (hereinafter Libert) (US 20200193312 A1), Jacob et al. (hereinafter Jacob_1) (US 20220309247 A1), and Hao et al. (hereinafter Hao) (US 11,847,565 B1).
Regarding claim 6, Temraz, Duong_1, Libert, and Jacob_1 teach the methods according to claim 3
Temraz, Duong_1, Libert, and Jacob_1 do not teach:
wherein, when described in relation to the first intent of the collection of intents, the utterance-level health indicators further comprise:
a similar utterance indicator that calculates a score based on a percentage of the utterances associated with the first intent comprising an utterance embedding that is computed to have a semantic similarity with the other utterance embeddings of the utterances associated with the first intent that exceeds a second predetermined similarity threshold.
However, Hao teaches:
wherein, when described in relation to the first intent of the collection of intents, the utterance-level health indicators further comprise:
a similar utterance indicator that calculates a score based on a percentage of the utterances associated with the first intent comprising an utterance embedding that is computed to have a semantic similarity with the other utterance embeddings of the utterances associated with the first intent that exceeds a second predetermined similarity threshold (Hao, Page 14, column 8, lines 20-46: "Other selection methods could also be used, such as building utterance groups through extracting utterances after clustering the utterances based upon one or more similarity metrics (e.g., K-means clustering)" (reads on using "utterance embedding" to compute "semantic similarity" via "one or more similarity metrics" like K-means clustering, which uses thresholds implicitly in its algorithm), "using clusters as utterance groups directly" (reads on selecting utterances for a "first intent" based on the clustering result), "FIG. 3 is a diagram of two exemplary utterance groups as generated by message grouping module 107. As shown in FIG. 3, the utterances in Group 1 relate to the user intent 'Account Balance' and the utterances in Group 2 relate to the user intent 'VIP Access.'" (reads on specific intent groups), Hao, Page 15, column 10, lines 55-63: "Because the percentage of utterances associated with a particular intent ('light gray') is 0.6, classifier 108b determines that this intent should be assigned to the utterance group." (reads on "calculates a score based on a percentage of the utterances associated with the first intent" and using a threshold to make a determination)).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Temraz in view of Duong_1, Libert, and Jacob_1 to incorporate the teachings of Hao. Doing so would have provided the thorough intent classification checking of Hao (Hao, Abstract) with the error correction methods of Jacob_1 (Jacob_1, Abstract), the weighted scoring system’s accuracy and flexibility of Libert (Libert, Abstract, P[0042]), the chatbot data generation methods of Temraz (Temraz, Abstract), and the advanced out-of-domain detection methods of Duong_1 (Duong_1, Abstract) thus improving the self-diagnostic capabilities for future improvements based on the most important weighted parameters of the chatbot’s intent classification systems through known weighted scoring mechanisms and fine-tuning methods such as checking intent classification.
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Temraz et al. (hereinafter Temraz) (US 20240330597 A1) in view of Duong et al. (hereinafter Duong_1) (US 20210303798 A1), and in further view of Libert et al. (hereinafter Libert) (US 20200193312 A1), Jacob et al. (hereinafter Jacob_1) (US 20220309247 A1), He et al. (hereinafter He) (CN 109284371 B) and Hao et al. (hereinafter Hao) (US 11,847,565 B1).
Regarding claim 15, claim 15 recites the system corresponding to the methods presented in claim 6 and is rejected under the same grounds as above
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Temraz et al. (hereinafter Temraz) (US 20240330597 A1) in view of Duong et al. (hereinafter Duong_1) (US 20210303798 A1), Libert et al. (hereinafter Libert) (US 20200193312 A1), Jacob et al. (hereinafter Jacob_1) (US 20220309247 A1), Hao et al. (hereinafter Hao) (US 11,847,565 B1) and Basil et al. (hereinafter Basil) (US 20220101839 A1).
Regarding claim 7, Temraz, Duong_1, Libert, Jacob_1, and Hao teach the methods according to claim 6
Temraz, Duong_1, Libert, Jacob_1, and Hao do not teach:
wherein the first predetermined similarity threshold comprises a level of semantic similarity that is less than a level of semantic similarity of the second predetermined similarity threshold; and
wherein semantic similarity is calculated using cosine similarity.
However, Basil teaches:
wherein the first predetermined similarity threshold comprises a level of semantic similarity that is less than a level of semantic similarity of the second predetermined similarity threshold; and
wherein semantic similarity is calculated using cosine similarity (Basil, P[0107]: ("Once the word-embeddings for the text of the salient intents is obtained, the word-embeddings may be used to calculate a semantic similarity between pairs of the salient intents." (reads on "semantic similarity is calculated"), "As an example, cosine similarity can be used to provide a measure of semantic closeness between word-embeddings in the higher dimensional space." (reads on "wherein semantic similarity is calculated using cosine similarity"), "With this obtained, the salient intents can then be group in accordance to those pairs having a cosine similarity of embeddings greater than a predetermined similarity threshold, which may be set between a range of 0 and 1." (reads on "first predetermined similarity threshold" or "second predetermined similarity threshold" being "predetermined"), "As will be appreciated, the higher this threshold is, the less salient intents get grouped together, thereby producing groups that are more homogenous, whereas a lower threshold value would result in more semantically diverse intents being grouped together, producing a less homogenous group." (reads on having different thresholds with different levels of semantic similarity))).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Temraz in view of Libert, Duong_1, Jacob_1 and Hao to incorporate the teachings of Basil. Doing so would have provided the intense utterance intent analysis of Basil (Basil, Abstract), with the thorough intent classification checking of Hao (Hao, Abstract), the weighted scoring system’s accuracy and flexibility of Libert (Libert, Abstract, P[0042]), the error correction methods of Jacob_1 (Jacob_1, Abstract), the chatbot data generation methods of Temraz (Temraz, Abstract) and the advanced out-of-domain detection methods of Duong_1 (Duong_1, Abstract) thus improving the self-diagnostic capabilities for future improvements based on the most important weighted parameters of the chatbot’s intent classification systems through known weighted scoring mechanisms and fine-tuning methods such as checking intent classification and utterance intent analysis.
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Temraz et al. (hereinafter Temraz) (US 20240330597 A1) in view of Duong et al. (hereinafter Duong_1) (US 20210303798 A1), and in further view of Libert et al. (hereinafter Libert) (US 20200193312 A1), Jacob et al. (hereinafter Jacob_1) (US 20220309247 A1), Hao et al. (hereinafter Hao) (US 11,847,565 B1) and Jacob et al. (hereinafter Jacob_2) (US 20220415326 A1).
Regarding claim 8, Temraz, Duong_1, Libert, and Jacob_1 and Hao teach the methods according to claim 6
Temraz, Duong_1, Libert, Jacob_1, and Hao do not teach:
wherein, when described in relation to the first intent of the collection of intents, the utterance-level health indicators further comprise:
an utterance-level static validation indicator that calculates a score based on a percentage of the utterances associated with the first intent found to have a total number of words or characters that either exceeds a maximum threshold or is less than a minimum threshold.
However, Jacob_2 teaches:
wherein, when described in relation to the first intent of the collection of intents, the utterance-level health indicators further comprise:
an utterance-level static validation indicator that calculates a score based on a percentage of the utterances associated with the first intent found to have a total number of words or characters that either exceeds a maximum threshold or is less than a minimum threshold.
(Jacob_2, P[0111]: "However, batching using techniques such as uniform proportional balancing would assign the same selection probability to intent 'f' as intent 'a', despite intent 'a' being represented one hundred times more in the raw utterance data." (reads on the problem of imbalanced data associated with a "first intent"), "To eliminate bias to overly represented utterances without reducing efficiency... a tailed batching technique will allow for including minimum training utterances in a batch when the proportional representation of the batch is less than a threshold amount." (reads on addressing an "utterance-level static validation indicator" when data is less than a "minimum threshold"), "For example, a particular utterance falling below a certain threshold for probability of selection will be automatically included in a batch in a minimum amount." (reads on applying the validation based on a threshold), "Techniques may specify that any intent falling below the threshold should cause at least one corresponding utterance to be included in any batch of training data." (reads on using the threshold as the criterion for identifying the problematic utterances), Jacob_2: P[0115], (scores written as a middle percentage of the range)).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Temraz in view of Duong_1, Libert, Jacob_1, and Hao to incorporate the teachings of Jacob_2. Doing so would have provided the multi-intent handling of Jacob_2 (Jacob_2, Abstract) with the thorough intent classification checking of Hao (Hao, Abstract), the weighted scoring system’s accuracy and flexibility of Libert (Libert, Abstract, P[0042]), the error correction methods of Jacob_1 (Jacob_1, Abstract), the chatbot data generation methods of Temraz (Temraz, Abstract) and the advanced out-of-domain detection methods of Duong_1 (Duong_1, Abstract) thus improving the self-diagnostic capabilities for future improvements based on the most important weighted parameters of the chatbot’s intent classification systems through known weighted scoring mechanisms and fine-tuning methods such as checking intent and robust multi-intent handling methods.
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Temraz et al. (hereinafter Temraz) (US 20240330597 A1) in view of Duong et al. (hereinafter Duong_1) (US 20210303798 A1), and in further view of Libert et al. (hereinafter Libert) (US 20200193312 A1), Jacob et al. (hereinafter Jacob_1) (US 20220309247 A1), He et al. (hereinafter He) (CN 109284371 B), Hao et al. (hereinafter Hao) (US 11,847,565 B1) and Jacob et al. (hereinafter Jacob_2) (US 20220415326 A1).
Regarding claim 16, claim 16 recites the system corresponding to the methods presented in claim 8 and is rejected under the same grounds as above
Claim(s) 9, 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Temraz et al. (hereinafter Temraz) (US 20240330597 A1) in view of Duong et al. (hereinafter Duong_1) (US 20210303798 A1), and in further view of Libert et al. (hereinafter Libert) (US 20200193312 A1), Jacob et al. (hereinafter Jacob_1) (US 20220309247 A1) and Bhowal et al. (hereinafter Bhowal) (US 20200327197 A1).
Regarding claim 9, Temraz, Duong_1, Libert, and Jacob_1 teach the methods according to claim 3
Temraz, Duong_1, Libert, and Jacob_1 do not teach:
The method of claim 3, further comprising the step of generating a user interface that displays the overall intent health score for a select intent of the collection of intents for communication to a user.
However, Bhowal teaches: The method of claim 3, further comprising the step of generating a user interface that displays the overall intent health score for a select intent of the collection of intents for communication to a user (Bhowal: P[0048], "The weighted summarization response 142 is a customized response" (reads on a piece of data generated by the system, which we assume is the "overall intent health score"), "The weighted summarization response 142 is output to the user 140" (reads on communicating the score to the user), "sent to the user device 116 for display or presentation to the user 140 via the communications interface component 114" (reads on the step of "generating a user interface that displays" the score), P[0070]: “In other non-limiting examples, similarity scores for each document or sub-document associated with an utterance is also computed. A response is created based on the aggregation of all the above outputs, coupled with intelligent summarization to make the response short and crisp.” (aggregation of all above outputs reads on overall intent health score)).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Temraz in view of Duong_1, Libert, and Jacob_1 to incorporate the teachings of Bhowal. Doing so would have provided the user interface structure of Bhowal (Bhowal, P0002]) with the error correction methods of Jacob_1 (Jacob_1, Abstract), the weighted scoring system’s accuracy and flexibility of Libert (Libert, Abstract, P[0042]), the chatbot data generation methods of Temraz (Temraz, Abstract) and the advanced out-of-domain detection methods of Duong_1 (Duong_1, Abstract) thus improving the self-diagnostic capabilities through a user-interface structure for future improvements based on the most important weighted parameters of the chatbot’s intent classification systems through known weighted scoring mechanisms and fine-tuning methods such as checking intent.
Regarding claim 10, Temraz, Duong_1, Libert, and Jacob_1 teach the methods according to claim 9
Temraz, Duong_1, Libert, and Jacob_1 do not teach:
The method of claim 9, wherein the generated user interface further displays the calculated scores for the utterance-level health indicators and the intent-level static validation indicator for the select intent that are used to calculate the overall intent health score for the select intent.
However, Bhowal teaches: The method of claim 9, wherein the generated user interface further displays the calculated scores for the utterance-level health indicators and the intent-level static validation indicator for the select intent that are used to calculate the overall intent health score for the select intent ((Bhowal, P[0103]: “A pre-trained machine learning model using an exhaustive list of intents and entities" (reads on having "intents" and associated data, which is the context for the indicators), "to identify the top 'k' number of entities having the same intent at a probability greater than point three percent (k>0.3)" (reads on calculation of a probability/score related to an "intent-level indicator"), "The system filters the utterances to select those closest to the user-provided utterance having the higher probability (k>0.6)" (reads on calculation of an underlying "utterance-level indicator" score using a threshold), "The system computes the similarity index of the user-provided utterance (query) with the filtered utterances" (reads on the computation of a specific score), "The response is sent back to the user" (reads on the display portion via the "generated user interface" as established in the base claim 9 mapping)).
Claim(s) 17, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Temraz et al. (hereinafter Temraz) (US 20240330597 A1) in view of Duong et al. (hereinafter Duong_1) (US 20210303798 A1), and in further view of Libert et al. (hereinafter Libert) (US 20200193312 A1), Jacob et al. (hereinafter Jacob_1) (US 20220309247 A1), He et al. (hereinafter He) (CN 109284371 B) and Bhowal et al. (hereinafter Bhowal) (US 20200327197 A1).
Regarding claim 17, claim 17 recites the system corresponding to the methods presented in claim 9 and is rejected under the same grounds as above
Regarding claim 18, claim 18 recites the system corresponding to the methods presented in claim 10 and is rejected under the same grounds as above
Claim(s) 11, 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Temraz et al. (hereinafter Temraz) (US 20240330597 A1) in view of Duong et al. (hereinafter Duong_1) (US 20210303798 A1), and in further view of Libert et al. (hereinafter Libert) (US 20200193312 A1), Jacob et al. (hereinafter Jacob_1) (US 20220309247 A1), Bhowal et al. (hereinafter Bhowal) (US 20200327197 A1), and Saeki et al. (hereinafter Saeki) (US 20210104240 A1)
Regarding claim 11,
Temraz, Duong_1, Jacob_1, Libert, and Bhowal teach the methods according to claim 9
Temraz, Duong_1, Jacob_1, Libert, and Bhowal do not teach:
wherein:
in proximity to the score displayed for the utterance in conflict indicator, the generated user interface further displays one or more utterances found to exceed the first predetermined threshold; and
in proximity to the score displayed for the utterance outlier indicator, the generated user interface further displays one or more utterances found to exceed the acceptable threshold level of deviation ().
However, Saeki teaches:
The method of claim 9, wherein:
in proximity to the score displayed for the utterance in conflict indicator, the generated user interface further displays one or more utterances found to exceed the first predetermined threshold; and
in proximity to the score displayed for the utterance outlier indicator, the generated user interface further displays one or more utterances found to exceed the acceptable threshold level of deviation (Saeki, Fig.4, P[0064]-P[0068], P[0092]-P[0100]: "The utterance list 55 displays information on utterance sentences sequentially from the latest voice recognition result" (reads on a user interface that displays "one or more utterances"), "“card application guide [99%]” in the remarks column 58 of FIG. 4 indicates that the likelihood of the topic C1 “card application guide” for the corresponding utterance sentence 53 is “0.99”" (reads on displaying a "score" or "indicator" (likelihood) "in proximity to the" utterance/topic (in the remarks column)), "The verifying result by the user 4 can be utilized for improving the detecting accuracy of the system 1" (reads on the purpose of using this displayed information to verify/improve the system, functionally equivalent to using health indicators to fix training data issues), "detection threshold V1" (reads on a "predetermined threshold"), "display threshold V2" (reads on a "predetermined threshold"), "When determining that the likelihood of the selected topic C is not greater than the detection threshold V1" (reads on using a score/indicator in comparison to the "acceptable threshold level of deviation" or similarity threshold), "When determining that the likelihood of the selected topic C is greater than the display threshold V2" (reads on using the threshold as a criterion), "Even when the likelihood for an utterance sentence, which is greater than the display threshold V2, does not reach the detection threshold V1, the likelihood is displayed in the remarks column 58" (reads on displaying information based on exceeding specific thresholds)).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Temraz in view of Duong_1, Libert, Jacob_1, and Bhowal to incorporate the teachings of Saeki. Doing so would have provided the detailed utterance output of Saeki (Saeki, Fig. 5) with the dynamic user interface structure of Bhowal (Bhowal, P0002]) with the error correction methods of Jacob_1 (Jacob_1, Abstract), the weighted scoring system’s accuracy and flexibility of Libert (Libert, Abstract, P[0042]), the chatbot data generation methods of Temraz (Temraz, Abstract) and the advanced out-of-domain detection methods of Duong_1 (Duong_1, Abstract) thus improving the self-diagnostic capabilities through a user-interface structure with detailed utterance output for future improvements based on the most important weighted parameters of the chatbot’s intent classification systems through known weighted scoring mechanisms and fine-tuning methods such as checking intent.
Regarding claim 12,
Temraz, Duong_1, Jacob_1, Bhowal, and Saeki teach the methods according to claim 11.
Bhowal further teaches:
comprising the steps of: receiving input from the user modifying at least one of the one or more utterances found to exceed the first predetermined threshold or the one or more utterances found to exceed the acceptable threshold level of deviation; and dynamically updating the user interface by recalculating the overall intent health score for the select intent for communication to the user (Bhowal, Fig. 2, P[0051]: "The system further incorporates feedback from the user to retrain itself automatically for improved relevancy of generated responses." (reads on "receiving input from the user modifying at least one of the one or more utterances found to exceed the first predetermined threshold or the one or more utterances found to exceed the acceptable threshold level of deviation," where feedback is the input for modification, addressing the identified issues), "to retrain itself automatically for improved relevancy" (reads on the system effectively recalculating and improving its internal representation/score/health in response to the modification, functionally "dynamically updating the user interface by recalculating the overall intent health score for the select intent for communication to the user")).
Claim(s) 19, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Temraz et al. (hereinafter Temraz) (US 20240330597 A1) in view of Duong et al. (hereinafter Duong_1) (US 20210303798 A1), and in further view of Libert et al. (hereinafter Libert) (US 20200193312 A1), Jacob et al. (hereinafter Jacob_1) (US 20220309247 A1 in further view of He et al. (hereinafter He) (CN 109284371 B) in further view of Bhowal et al. (hereinafter Bhowal) (US 20200327197 A1), and Saeki et al. (hereinafter Saeki) (US 20210104240 A1).
Regarding claim 19, claim 19 recites the system corresponding to the methods presented in claim 11 and is rejected under the same grounds as above
Regarding claim 20, claim 20 recites the system corresponding to the methods presented in claim 12 and is rejected under the same grounds as above
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHASHIDHAR S MANOHARAN whose telephone number is (571)272-6772. The examiner can normally be reached M-F 8:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at 571-272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHASHIDHAR SHANKAR MANOHARAN/Examiner, Art Unit 2655
/ANDREW C FLANDERS/Supervisory Patent Examiner, Art Unit 2655