Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA
Information Disclosure Statement
2. The information disclosure statement (IDS) submitted on December 12, 2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Amendment and Arguments
3. The amendment filed on April 02, 2026 has been entered. Claims 1-20 remain pending in the application. Claims 1, 14, and 18 as well as various dependent claims are amended.
Applicant’s arguments with respect to the 35 U.S.C. 101 rejections for claims 18-20 have been considered and are persuasive. Accordingly, these rejections have been withdrawn.
Applicant’s arguments with respect to the 35 U.S.C. 102 rejections for claims 1-12 and 14-20 have been considered but are moot because the arguments are directed towards amended claim language, addressed on new grounds of rejection below.
Claim Rejections - 35 USC § 103
4. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically taught as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
5. Claims 1-12 and 14-20 are rejected under 35 U.S.C. 103 as being unpatentable over Mallinar (U.S. Publication No. 20200142959) in view of Katz (U.S. Publication No. 20200167429).
Regarding claim 1, Mallinar discloses a computing system comprising:
a processor ([0057] - …processor…);
and computer storage memory having computer-executable instructions stored thereon which, when executed by the processor, configure the computing system to perform operations comprising ([0057] - which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks):
obtain training data including data requests for information ([0012] - When training a classifier for use in a Natural Language Processing (NLP) model, such as a chatbot, to identify various intents that a user may express, a developer should provide a large and varied training dataset to the classifier);
generate a model prompt to be input into a large language model, the model prompt including an instruction to generate an intent taxonomy structure, an indication of the training data to use for generating the intent taxonomy, and a taxonomy attribute desired to be used as criteria to generate a quality intent taxonomy ([0040] - The dataset builder 365 identifies the example utterances 510 that are most anomalous compared to the set of example utterances 510 being evaluated based on various scoring schema. The scoring schema identify how different an individual utterance's vector is from the other vectors for the other utterances being evaluated in the multidimensional vector space. In some embodiments, the dataset builder 365 uses a cosine distance, a trained forest model, or a one-class support vector machine to determine the anomaly score, which may be understood as a distance in the multidimensional space away from the inlier utterances).
However, Mallinar does not disclose how to obtain, as output from the large language model, the intent taxonomy structure including a plurality of user intent classes categories and associated user intent category descriptions;
analyze the intent taxonomy structure to determine whether the plurality of user intent categories and associated user intent category descriptions are valid, wherein analyzing the intent taxonomy comprises:
generating, via the large language model, one or more user intent category labels for one or more samples,
and performing validation of the one or more intent category labels by prompting the large language model to analyze the one or more user intent category labels in view of the plurality of user intent categories from the intent taxonomy structure,
and wherein when the intent taxonomy structure is determined as valid, providing the intent taxonomy structure for use in identifying user intent, and when the intent taxonomy structure is determined as invalid, refining the intent taxonomy structure.
Katz does teach how to obtain, as output from the large language model, the intent taxonomy structure including a plurality of user intent classes categories and associated user intent category descriptions ([0013] - A text classification system assigns probability values to each of a set of classification labels for a given text based on an analysis of the individual words in the text. Each of the text classification labels indicates one possible categorization for the text, such as an intent of the text);
analyze the intent taxonomy structure to determine whether the plurality of user intent categories and associated user intent category descriptions are valid ([0066] - the probability score determination module 312 calculates probability scores for a set of classification labels. The probability score determination module 312 determines the probability score for each classification label based on the sets of coefficient values determined for each word in the text. Specifically, the probability score determination module 312 determines the probability score for a classification label based on the coefficient value from each of the sets of coefficient values that corresponds to the given classification label. That is, the probability score determination module 312 gathers the coefficient value from each set of coefficient values that represents how strongly the word indicates the intent corresponding to the given classification label and then calculates the probability score for the given classification label based on the gathered coefficient values. For example, the probability score determination module 312 may sum the gathered coefficient values to determine the probability score for the given classification label), wherein analyzing the intent taxonomy comprises:
generating, via the large language model, one or more user intent category labels for one or more samples ([0013] - A text classification system assigns probability values to each of a set of classification labels for a given text based on an analysis of the individual words in the text. Each of the text classification labels indicates one possible categorization for the text, such as an intent of the text),
and performing validation of the one or more intent category labels by prompting the large language model to analyze the one or more user intent category labels in view of the plurality of user intent categories from the intent taxonomy structure ([0013] - That is, each text classification label indicates one possible intent of the text. The probability values assigned to each text classification label indicates the likelihood that the intent corresponding to the text classification label properly classifies the text. The text classification label with the highest probability value is determined to have the highest probability of properly indicating the intent of the text),
and wherein when the intent taxonomy structure is determined as valid, providing the intent taxonomy structure for use in identifying user intent, and when the intent taxonomy structure is determined as invalid, refining the intent taxonomy structure ([0013] - That is, each text classification label indicates one possible intent of the text. The probability values assigned to each text classification label indicates the likelihood that the intent corresponding to the text classification label properly classifies the text. The text classification label with the highest probability value is determined to have the highest probability of properly indicating the intent of the text).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mallinar to incorporate the teachings of Katz in order to implement the above limitations. Doing so allows for technical improvement over existing systems (Katz [0043]).
Regarding claim 2, Mallinar discloses the computing system, wherein the data requests include search queries or chat inquiries ([0015] - Using existing chat logs, the training dataset is built to include a pool of examples that are particularly unique to provide the model being trained with the most-difficult to handle responses in the training dataset).
Regarding claim 3, Mallinar discloses the computing system, wherein the training data further includes data responses, user interactions, or a combination thereof ([0030] - One or more chat log databases 371 are maintained in the storage 370, which include various historical natural language utterances and responses (i.e., conversations) received from users. The conversations may include human-to-human interactions as well as human-to-chatbot interactions).
Regarding claim 4, Mallinar discloses the computing system, wherein the taxonomy attribute comprises an accuracy criterion, a completeness criterion, a conciseness criterion, a clarity criterion, or a consistency criterion ([0040] - the dataset builder 365 uses a cosine distance, a trained forest model, or a one-class support vector machine to determine the anomaly score, which may be understood as a distance in the multidimensional space away from the inlier utterances).
Regarding claim 5, Mallinar discloses the computing system, wherein the model prompt further includes an indication of a desired taxonomy structure ([0044] - Each vector encoding schema 610 creates a vector representation for each historic utterance, which enables the dataset builder 365 to numerically analyze the historic utterances. In embodiments that use multiple vector encoding schemas 610, the vector encoding schemas 610 votes on whether a given historical utterance is anomalous, such that at more than half of the encoding schemas 610 agree on an individual utterance being anomalous).
Regarding claim 6, Mallinar discloses the computing system, wherein the desired taxonomy structure specifies information to provide in the intent taxonomy structure, a number of user intent categories, a number of examples to provide in association with each user intent category, a number of hierarchical levels in the intent taxonomy structure, or a combination thereof ([0045] - The vector encoding schemas 610 output an ordered list of the anomalous utterances, ranked accord according to the values of the anomaly scores. In embodiments using multiple scoring schemas 615 or vector encoding schemas 610, the order of the anomalous utterances may be based on a highest anomaly score reported, a lowest anomaly score reported, or an average anomaly score. The dataset builder 365 may select the n anomalous examples 620 from the n most-anomalous utterances, or may select all utterances having an anomaly score greater than an anomaly threshold).
Regarding claim 7, Mallinar discloses the computing system, wherein the intent taxonomy structure includes a description associated with each user intent category of the plurality of user intent categories and an example associated with each user intent category of the plurality of user intent categories ([0041] - the dataset builder 365 updates an existing training dataset with newly analyzed example utterances 510, while in other embodiments, the dataset builder 365 creates a new training set with the example utterances 510 analyzed. In various embodiments, the dataset builder 365 may analyze the example utterances included in a current training dataset as well as the utterances received within the last d days to update or create a new training dataset).
Regarding claim 8, Mallinar discloses the computing system, wherein refining the intent taxonomy structure includes updating the intent taxonomy structure with a new user intent category, a new example, or a new description or updating the intent taxonomy structure by removing a user intent category ([0046] - When the classifier 363 has incorrectly determined the intent of the anomalous example 620, the developer responds with a new label 430 that matches the correct intent for the anomalous example 620, and the dataset builder 365 optionally adds the anomalous example 620 with the new label 430 to one or more of an out-intent pool of utterances of the training dataset 630 for the intent associated with initial label 430 (Block 750) and the in-intent pool of the corrected intent in the training dataset 630 (Block 740)).
Regarding claim 9, Mallinar discloses the computing system, wherein analyzing the intent taxonomy structure to determine whether the intent taxonomy is valid comprises verifying comprehensiveness of the intent taxonomy by:
generating a prompt to annotate the one or more samples using the intent taxonomy structure ([0036] - When an NLP assistant generates the conversation responses 420 in a live conversation 400, the NLP assistant first determines an intent for the conversation input 410 using a trained classifier 363. Similarly, in a historic conversation 400, between a human user and a human or NLP assistant, the trained classifier 363 analyzes the conversation inputs 410 as historic utterances to determine the intent of the human user);
providing the prompt as input to the large language model ([0036] - When an NLP assistant generates the conversation responses 420 in a live conversation 400, the NLP assistant first determines an intent for the conversation input 410 using a trained classifier 363. Similarly, in a historic conversation 400, between a human user and a human or NLP assistant, the trained classifier 363 analyzes the conversation inputs 410 as historic utterances to determine the intent of the human user);
obtaining, as output from the large language model, the one or more user intent category labels for the one or more samples ([0036] - The classifier 363 may assign one or more labels 430 to one or more conversation inputs 410 that reflect the determined intent for the particular conversation inputs 410);
and based on the one or more user intent category labels, determining a proportion of the one or more samples that have a user intent category label corresponding with a user intent category of the intent taxonomy structure ([0038] - Therefore, the dataset builder 365 identifies which examples selected from the root nodes 440 (whether correctly or incorrectly initially classified) to include in a training dataset, and may select a predefined number of examples to include in the training dataset. The examples may include a predefined number or percentage of outliers, inliers, and mandated examples).
Regarding claim 10, Mallinar discloses the computing system, wherein analyzing the intent taxonomy structure to determine whether the intent taxonomy structure is valid comprises verifying consistency of the intent taxonomy structure by:
generating a prompt to annotate the one or more samples using the intent taxonomy structure ([0036] - When an NLP assistant generates the conversation responses 420 in a live conversation 400, the NLP assistant first determines an intent for the conversation input 410 using a trained classifier 363. Similarly, in a historic conversation 400, between a human user and a human or NLP assistant, the trained classifier 363 analyzes the conversation inputs 410 as historic utterances to determine the intent of the human user);
providing the prompt as input to the large language model ([0036] - When an NLP assistant generates the conversation responses 420 in a live conversation 400, the NLP assistant first determines an intent for the conversation input 410 using a trained classifier 363. Similarly, in a historic conversation 400, between a human user and a human or NLP assistant, the trained classifier 363 analyzes the conversation inputs 410 as historic utterances to determine the intent of the human user);
obtaining, as output from the large language model, the one or more user intent category labels for the one or more samples ([0036] - The classifier 363 may assign one or more labels 430 to one or more conversation inputs 410 that reflect the determined intent for the particular conversation inputs 410);
and based on the one or more user intent category labels, determining if the large language model consistently applied definitions associated with the plurality of user intent categories ([0038] - The examples may include a predefined number or percentage of outliers, inliers, and mandated examples. For example, a training dataset may include x examples provided by or selected by a developer (i.e., mandated examples), y examples that fall within a predefined range of similarity to one another as vector representations or fall within a predicted pattern (i.e., inlier examples), and z examples that fall outside of a predefined range of similarity to one another in a vector representations or fall outside of a predicted pattern (i.e., outlier examples). For example, the vectorizations of the first, second, and third example utterances 510a-c are inliers, as the values fall within the predefined range 520 shown in the vector plot 500, while the fourth, fifth, and sixth example utterances 50d-f are outliers as the values fall outside of the predefined range 520. Any of the example utterances 510a-f may be mandated or historic examples).
Regarding claim 11, Mallinar discloses the computing system, wherein analyzing the intent taxonomy structure to determine whether the intent taxonomy structure is valid comprises verifying conciseness of the intent taxonomy structure by:
generating a prompt to annotate test samples using the intent taxonomy structure ([0036] - When an NLP assistant generates the conversation responses 420 in a live conversation 400, the NLP assistant first determines an intent for the conversation input 410 using a trained classifier 363. Similarly, in a historic conversation 400, between a human user and a human or NLP assistant, the trained classifier 363 analyzes the conversation inputs 410 as historic utterances to determine the intent of the human user);
providing the prompt as input to the large language model ([0036] - When an NLP assistant generates the conversation responses 420 in a live conversation 400, the NLP assistant first determines an intent for the conversation input 410 using a trained classifier 363. Similarly, in a historic conversation 400, between a human user and a human or NLP assistant, the trained classifier 363 analyzes the conversation inputs 410 as historic utterances to determine the intent of the human user);
obtaining, as output from the large language model, user intent category labels for the test samples ([0036] - The classifier 363 may assign one or more labels 430 to one or more conversation inputs 410 that reflect the determined intent for the particular conversation inputs 410);
and based on the user intent category labels, determining if a threshold number of test samples have a user intent category label associated with each user intent category of the plurality of user intent categories ([0038] - The examples may include a predefined number or percentage of outliers, inliers, and mandated examples. For example, a training dataset may include x examples provided by or selected by a developer (i.e., mandated examples), y examples that fall within a predefined range of similarity to one another as vector representations or fall within a predicted pattern (i.e., inlier examples), and z examples that fall outside of a predefined range of similarity to one another in a vector representations or fall outside of a predicted pattern (i.e., outlier examples). For example, the vectorizations of the first, second, and third example utterances 510a-c are inliers, as the values fall within the predefined range 520 shown in the vector plot 500, while the fourth, fifth, and sixth example utterances 50d-f are outliers as the values fall outside of the predefined range 520. Any of the example utterances 510a-f may be mandated or historic examples).
Regarding claim 12, Mallinar discloses the computing system, wherein analyzing the intent taxonomy to determine whether the intent taxonomy structure is valid comprises verifying accuracy of the intent taxonomy structure by:
generating a prompt to annotate test samples using the intent taxonomy structure ([0036] - When an NLP assistant generates the conversation responses 420 in a live conversation 400, the NLP assistant first determines an intent for the conversation input 410 using a trained classifier 363. Similarly, in a historic conversation 400, between a human user and a human or NLP assistant, the trained classifier 363 analyzes the conversation inputs 410 as historic utterances to determine the intent of the human user);
providing the prompt as input to the large language model ([0036] - When an NLP assistant generates the conversation responses 420 in a live conversation 400, the NLP assistant first determines an intent for the conversation input 410 using a trained classifier 363. Similarly, in a historic conversation 400, between a human user and a human or NLP assistant, the trained classifier 363 analyzes the conversation inputs 410 as historic utterances to determine the intent of the human user);
obtaining, as output from the large language model, user intent category labels for the test samples ([0036] - The classifier 363 may assign one or more labels 430 to one or more conversation inputs 410 that reflect the determined intent for the particular conversation inputs 410);
comparing the user intent category labels output from the large language model to human-annotated user intent category labels for the test samples ([0040] - The dataset builder 365 identifies the example utterances 510 that are most anomalous compared to the set of example utterances 510 being evaluated based on various scoring schema);
and based on the comparison, determining accuracy of the intent taxonomy structure ([0040] - The scoring schema identify how different an individual utterance's vector is from the other vectors for the other utterances being evaluated in the multidimensional vector space).
Regarding claim 14, Mallinar discloses a computer-implemented method comprising:
generating a first model prompt to input into a large language model, the first model prompt including an instruction to generate an intent taxonomy structure and an indication of training data, including data requests, to use for generating the intent taxonomy structure ([0040] - The dataset builder 365 identifies the example utterances 510 that are most anomalous compared to the set of example utterances 510 being evaluated based on various scoring schema. The scoring schema identify how different an individual utterance's vector is from the other vectors for the other utterances being evaluated in the multidimensional vector space. In some embodiments, the dataset builder 365 uses a cosine distance, a trained forest model, or a one-class support vector machine to determine the anomaly score, which may be understood as a distance in the multidimensional space away from the inlier utterances);
based on validating the intent taxonomy, storing the intent taxonomy for subsequent use in identifying user intent ([0046] - When the classifier 363 has correctly determined the intent of the anomalous example 620, the developer responds with the original label 430, and the dataset builder 365 adds the anomalous example 620 with the original label 430 to an in-intent pool of utterances of the training dataset 630 (Block 740). When the classifier 363 has incorrectly determined the intent of the anomalous example 620, the developer responds with a new label 430 that matches the correct intent for the anomalous example 620…).
obtaining an intent identification request to identify user intent associated with a new data request ([0013] - In a training dataset used a supervised (or semi-supervised) learning process, individual data points are labeled with a correspondence to various intents. These labels are metadata that inform the model being trained how the data point should be classified relative to a given intent),
generating a second model prompt to input into the large language model, the second model prompt including the intent taxonomy structure and an instruction to identify user intent associated with the new data request ([0040] - The dataset builder 365 identifies the example utterances 510 that are most anomalous compared to the set of example utterances 510 being evaluated based on various scoring schema. The scoring schema identify how different an individual utterance's vector is from the other vectors for the other utterances being evaluated in the multidimensional vector space. In some embodiments, the dataset builder 365 uses a cosine distance, a trained forest model, or a one-class support vector machine to determine the anomaly score, which may be understood as a distance in the multidimensional space away from the inlier utterances);
obtaining, as output from the large language model, a label of a user intent category associated with the new data request ([0012] - When training a classifier for use in a Natural Language Processing (NLP) model, such as a chatbot, to identify various intents that a user may express, a developer should provide a large and varied training dataset to the classifier);
and providing the label of the user intent category associated with the new data request for display via a user interface or for analysis of the user intent corresponding with the new data request ([0046] - ([0046] - When the classifier 363 has correctly determined the intent of the anomalous example 620, the developer responds with the original label 430, and the dataset builder 365 adds the anomalous example 620 with the original label 430 to an in-intent pool of utterances of the training dataset 630 (Block 740). When the classifier 363 has incorrectly determined the intent of the anomalous example 620, the developer responds with a new label 430 that matches the correct intent for the anomalous example 620…A user interface may present the n next-most-likely intents to the developer to choose from when selecting a new intent).
However, Mallinar does not disclose obtaining, as output from the large language model, the intent taxonomy structure including a plurality of user intent categories and associated user intent category description;
validating the intent taxonomy by generating one or more user intent category labels for one or more samples and performing validation of the one or more intent category labels by prompting the large language model to analyze the one or more user intent category labels in view of the plurality of user intent categories from the intent taxonomy structure.
Katz does teach obtaining, as output from the large language model, the intent taxonomy structure including a plurality of user intent categories and associated user intent category description ([0013] - A text classification system assigns probability values to each of a set of classification labels for a given text based on an analysis of the individual words in the text. Each of the text classification labels indicates one possible categorization for the text, such as an intent of the text);
validating the intent taxonomy by generating one or more user intent category labels for one or more samples and performing validation of the one or more intent category labels by prompting the large language model to analyze the one or more user intent category labels in view of the plurality of user intent categories from the intent taxonomy structure ([0013] - That is, each text classification label indicates one possible intent of the text. The probability values assigned to each text classification label indicates the likelihood that the intent corresponding to the text classification label properly classifies the text. The text classification label with the highest probability value is determined to have the highest probability of properly indicating the intent of the text [0066] - the probability score determination module 312 calculates probability scores for a set of classification labels. The probability score determination module 312 determines the probability score for each classification label based on the sets of coefficient values determined for each word in the text. Specifically, the probability score determination module 312 determines the probability score for a classification label based on the coefficient value from each of the sets of coefficient values that corresponds to the given classification label. That is, the probability score determination module 312 gathers the coefficient value from each set of coefficient values that represents how strongly the word indicates the intent corresponding to the given classification label and then calculates the probability score for the given classification label based on the gathered coefficient values. For example, the probability score determination module 312 may sum the gathered coefficient values to determine the probability score for the given classification label).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mallinar to incorporate the teachings of Katz in order to implement the above limitations. Doing so allows for technical improvement over existing systems (Katz [0043]).
Regarding claim 15, Mallinar discloses the method, wherein validating the intent taxonomy includes using human-labeled user intent associated with data samples ([0037] - the classifier 363 identifies a root node 440 in the conversation 400. The root node 440 identifies the first indication from the user of a main purpose for the conversation 400…).
Regarding claim 16, Mallinar discloses the method further comprising:
obtaining the new data request from a user device, the new data request comprising a search query or a chat inquiry ([0015] - Using existing chat logs, the training dataset is built to include a pool of examples that are particularly unique to provide the model being trained with the most-difficult to handle responses in the training dataset);
and using the label of the user intent category associated with the new data request to generate a recommendation associated with the new data request to provide to the user device ([0034] - The conversations 400 may be between a human user and a human assistant or a human user and an NLP assistant, such as a chatbot, using an NLP model 362 to determine how to respond to the human user. The conversations 400 includes one or more utterances received from the human user as conversation inputs 410, and one or more conversation responses 420 from the assistant. Other example conversations 400 may include more or fewer utterances with different dialog).
Regarding claim 17, Mallinar discloses the method, wherein the intent taxonomy structure includes descriptions of the one or more user intent categories and examples associated with the one or more user intent categories ([0041] - the dataset builder 365 updates an existing training dataset with newly analyzed example utterances 510, while in other embodiments, the dataset builder 365 creates a new training set with the example utterances 510 analyzed. In various embodiments, the dataset builder 365 may analyze the example utterances included in a current training dataset as well as the utterances received within the last d days to update or create a new training dataset).
Regarding claim 18, Mallinar discloses one or more non-transitory computer storage media having computer-executable instructions embodied thereon that, when executed by one or more processors, cause the one or more processors to perform a method ([0057] - which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks), the method comprising:
obtaining training data including data requests for information ([0012] - When training a classifier for use in a Natural Language Processing (NLP) model, such as a chatbot, to identify various intents that a user may express, a developer should provide a large and varied training dataset to the classifier);
generating a model prompt to be input into a large language model, the model prompt including an instruction to generate an intent taxonomy, an indication of the training data to use for generating the intent taxonomy, and a taxonomy attribute desired to be used as criteria to generate a quality intent taxonomy ([0040] - The dataset builder 365 identifies the example utterances 510 that are most anomalous compared to the set of example utterances 510 being evaluated based on various scoring schema. The scoring schema identify how different an individual utterance's vector is from the other vectors for the other utterances being evaluated in the multidimensional vector space. In some embodiments, the dataset builder 365 uses a cosine distance, a trained forest model, or a one-class support vector machine to determine the anomaly score, which may be understood as a distance in the multidimensional space away from the inlier utterances);
and providing the intent taxonomy structure for subsequent use to identify, via the large language model, user intent associated with a new data request ([0046] - When the classifier 363 has correctly determined the intent of the anomalous example 620, the developer responds with the original label 430, and the dataset builder 365 adds the anomalous example 620 with the original label 430 to an in-intent pool of utterances of the training dataset 630 (Block 740). When the classifier 363 has incorrectly determined the intent of the anomalous example 620, the developer responds with a new label 430 that matches the correct intent for the anomalous example 620…).
However, Mallinar does not disclose obtaining, as output from the large language model, the intent taxonomy structure including a plurality of user intent classes categories and associated user intent category descriptions;
analyze the intent taxonomy structure to determine whether the plurality of user intent categories and associated user intent category descriptions are valid, wherein analyzing the intent taxonomy comprises:
generating, via the large language model, one or more user intent category labels for one or more samples,
and performing validation of the one or more intent category labels by prompting the large language model to analyze the one or more user intent category labels in view of the plurality of user intent categories from the intent taxonomy structure.
Katz does teach obtaining, as output from the large language model, the intent taxonomy structure including a plurality of user intent classes categories and associated user intent category descriptions ([0013] - A text classification system assigns probability values to each of a set of classification labels for a given text based on an analysis of the individual words in the text. Each of the text classification labels indicates one possible categorization for the text, such as an intent of the text);
analyze the intent taxonomy structure to determine whether the plurality of user intent categories and associated user intent category descriptions are valid ([0066] - the probability score determination module 312 calculates probability scores for a set of classification labels. The probability score determination module 312 determines the probability score for each classification label based on the sets of coefficient values determined for each word in the text. Specifically, the probability score determination module 312 determines the probability score for a classification label based on the coefficient value from each of the sets of coefficient values that corresponds to the given classification label. That is, the probability score determination module 312 gathers the coefficient value from each set of coefficient values that represents how strongly the word indicates the intent corresponding to the given classification label and then calculates the probability score for the given classification label based on the gathered coefficient values. For example, the probability score determination module 312 may sum the gathered coefficient values to determine the probability score for the given classification label), wherein analyzing the intent taxonomy comprises:
generating, via the large language model, one or more user intent category labels for one or more samples ([0013] - A text classification system assigns probability values to each of a set of classification labels for a given text based on an analysis of the individual words in the text. Each of the text classification labels indicates one possible categorization for the text, such as an intent of the text),
and performing validation of the one or more intent category labels by prompting the large language model to analyze the one or more user intent category labels in view of the plurality of user intent categories from the intent taxonomy structure ([0013] - That is, each text classification label indicates one possible intent of the text. The probability values assigned to each text classification label indicates the likelihood that the intent corresponding to the text classification label properly classifies the text. The text classification label with the highest probability value is determined to have the highest probability of properly indicating the intent of the text).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mallinar to incorporate the teachings of Katz in order to implement the above limitations. Doing so allows for technical improvement over existing systems (Katz [0043]).
Regarding claim 19, Mallinar discloses the non-transitory computer storage media, further comprising using human input to valide or refine the intent taxonomy, wherein the human input comprises recommendations for modifying the one or more user intent categories, descriptions associated with the one or more user intent categories, or examples associated with the one or more user intent categories ([0041] - the dataset builder 365 updates an existing training dataset with newly analyzed example utterances 510, while in other embodiments, the dataset builder 365 creates a new training set with the example utterances 510 analyzed. In various embodiments, the dataset builder 365 may analyze the example utterances included in a current training dataset as well as the utterances received within the last d days to update or create a new training dataset).
Regarding claim 20, Mallinar discloses the non-transitory computer storage media, wherein the human input comprises human-labeled user intent associated with data samples, and wherein the human-labeled user intent is compared with machine-labeled user intent generated via the large language model ([0037] - the classifier 363 identifies a root node 440 in the conversation 400. The root node 440 identifies the first indication from the user of a main purpose for the conversation 400…).
Allowable Subject Matter
6. Claim 13 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: The prior art could not overcome or render obvious the limitation of “wherein comparing the user intent class labels output from the large language model to the human-annotated user intent class labels comprises measuring an inter-coder reliability therebetween” as claimed.
The closest prior art, Mallinar (U.S. Publication No. 20200142959) either singularly or in combination fail to anticipate or render obvious the above described limitations. Mallinar discloses driving intent expansion via anomaly detection in a modular conversational system.
Conclusion
7. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Katz (U.S. Publication No. 2020167429) teaches efficient use of word embeddings for text classification. Weston (U.S. Publication No. 20250054493) teaches methods and systems for training and deploying natural language understanding models.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ETHAN DANIEL KIM whose telephone number is (571) 272-1405. The examiner can normally be reached on Monday - Friday 9:00 - 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Richemond Dorvil can be reached on (571) 272-7602. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ETHAN DANIEL KIM/
Examiner, Art Unit 2658
/RICHEMOND DORVIL/Supervisory Patent Examiner, Art Unit 2658
Prosecution Insights