Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA
Information Disclosure Statement
2. The information disclosure statement (IDS) submitted on December 12, 2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
3. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
4. Claims 18-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Claim 18 is directed to a “computer storage media” that is not limited to only include non-transitory hardware embodiments, and can thus be interpreted to include software embodiments per se. Therefore, it does not fall within at least one of the four categories of patent eligible subject matter.
Claims 19-20 are rejected for their dependence on claims 18, because they do not contain additional limitations that overcome the present rejection.
Claim Rejections - 35 USC § 102
5. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
6. Claims 1-12 and 14-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Mallinar (U.S. Publication No. 20200142959).
Regarding claim 1, Mallinar discloses a computing system comprising:
a processor ([0057] - …processor…);
and computer storage memory having computer-executable instructions stored thereon which, when executed by the processor, configure the computing system to perform operations comprising ([0057] - which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks):
obtain training data including data requests for information ([0012] - When training a classifier for use in a Natural Language Processing (NLP) model, such as a chatbot, to identify various intents that a user may express, a developer should provide a large and varied training dataset to the classifier);
generate a model prompt to be input into a large language model, the model prompt including an instruction to generate an intent taxonomy, an indication of the training data to use for generating the intent taxonomy, and a taxonomy attribute desired to be used as criteria to generate a quality intent taxonomy ([0040] - The dataset builder 365 identifies the example utterances 510 that are most anomalous compared to the set of example utterances 510 being evaluated based on various scoring schema. The scoring schema identify how different an individual utterance's vector is from the other vectors for the other utterances being evaluated in the multidimensional vector space. In some embodiments, the dataset builder 365 uses a cosine distance, a trained forest model, or a one-class support vector machine to determine the anomaly score, which may be understood as a distance in the multidimensional space away from the inlier utterances);
obtain, as output from the large language model, the intent taxonomy that includes one or more user intent classes ([0012] - When training a classifier for use in a Natural Language Processing (NLP) model, such as a chatbot, to identify various intents that a user may express, a developer should provide a large and varied training dataset to the classifier);
analyze the intent taxonomy to determine whether the intent taxonomy is valid, wherein analyzing the intent taxonomy comprises ([0044] - The dataset builder 365 ranks vector representations of the historic utterances returned from the chat log database based on an anomaly score for the historic utterances (Block 710)):
generating, via the large language model, one or more user intent class labels for one or more samples ([0013] - In a training dataset used a supervised (or semi-supervised) learning process, individual data points are labeled with a correspondence to various intents. These labels are metadata that inform the model being trained how the data point should be classified relative to a given intent),
and performing validation by prompting the large language model to analyze the one or more user intent class labels ([0046] - The dataset builder 365 sends the anomalous examples 620 to the node 54B used by the developer for the developer to verify the label 430 associated with the anomalous examples 620 (Block 730)),
and wherein when the intent taxonomy is determined as valid, providing the intent taxonomy for use in identifying user intent, and when the intent taxonomy is determined as invalid, refining the intent taxonomy ([0046] - When the classifier 363 has correctly determined the intent of the anomalous example 620, the developer responds with the original label 430, and the dataset builder 365 adds the anomalous example 620 with the original label 430 to an in-intent pool of utterances of the training dataset 630 (Block 740). When the classifier 363 has incorrectly determined the intent of the anomalous example 620, the developer responds with a new label 430 that matches the correct intent for the anomalous example 620…).
Regarding claim 2, Mallinar discloses the computing system, wherein the data requests include search queries or chat inquiries ([0015] - Using existing chat logs, the training dataset is built to include a pool of examples that are particularly unique to provide the model being trained with the most-difficult to handle responses in the training dataset).
Regarding claim 3, Mallinar discloses the computing system, wherein the training data further includes data responses, user interactions, or a combination thereof ([0030] - One or more chat log databases 371 are maintained in the storage 370, which include various historical natural language utterances and responses (i.e., conversations) received from users. The conversations may include human-to-human interactions as well as human-to-chatbot interactions).
Regarding claim 4, Mallinar discloses the computing system, wherein the taxonomy attribute comprises an accuracy criterion, a completeness criterion, a conciseness criterion, a clarity criterion, or a consistency criterion ([0040] - the dataset builder 365 uses a cosine distance, a trained forest model, or a one-class support vector machine to determine the anomaly score, which may be understood as a distance in the multidimensional space away from the inlier utterances).
Regarding claim 5, Mallinar discloses the computing system, wherein the model prompt further includes an indication of a desired taxonomy structure ([0044] - Each vector encoding schema 610 creates a vector representation for each historic utterance, which enables the dataset builder 365 to numerically analyze the historic utterances. In embodiments that use multiple vector encoding schemas 610, the vector encoding schemas 610 votes on whether a given historical utterance is anomalous, such that at more than half of the encoding schemas 610 agree on an individual utterance being anomalous).
Regarding claim 6, Mallinar discloses the computing system, wherein the desired taxonomy structure specifies information to provide in the intent taxonomy, a number of user intent classes, a number of examples to provide in association with each user intent class, a number of hierarchical levels in the intent taxonomy, or a combination thereof ([0045] - The vector encoding schemas 610 output an ordered list of the anomalous utterances, ranked accord according to the values of the anomaly scores. In embodiments using multiple scoring schemas 615 or vector encoding schemas 610, the order of the anomalous utterances may be based on a highest anomaly score reported, a lowest anomaly score reported, or an average anomaly score. The dataset builder 365 may select the n anomalous examples 620 from the n most-anomalous utterances, or may select all utterances having an anomaly score greater than an anomaly threshold).
Regarding claim 7, Mallinar discloses the computing system, wherein the intent taxonomy includes a description associated with each user intent class of the one or more user intent classes and an example associated with each user intent class of the one or more user intent classes ([0041] - the dataset builder 365 updates an existing training dataset with newly analyzed example utterances 510, while in other embodiments, the dataset builder 365 creates a new training set with the example utterances 510 analyzed. In various embodiments, the dataset builder 365 may analyze the example utterances included in a current training dataset as well as the utterances received within the last d days to update or create a new training dataset).
Regarding claim 8, Mallinar discloses the computing system, wherein refining the intent taxonomy includes updating the intent taxonomy with a new user intent class, a new example, or a new description or updating the intent taxonomy by removing a user intent class ([0046] - When the classifier 363 has incorrectly determined the intent of the anomalous example 620, the developer responds with a new label 430 that matches the correct intent for the anomalous example 620, and the dataset builder 365 optionally adds the anomalous example 620 with the new label 430 to one or more of an out-intent pool of utterances of the training dataset 630 for the intent associated with initial label 430 (Block 750) and the in-intent pool of the corrected intent in the training dataset 630 (Block 740)).
Regarding claim 9, Mallinar discloses the computing system, wherein analyzing the intent taxonomy to determine whether the intent taxonomy is valid comprises verifying comprehensiveness of the intent taxonomy by:
generating a prompt to annotate the one or more samples using the intent taxonomy ([0036] - When an NLP assistant generates the conversation responses 420 in a live conversation 400, the NLP assistant first determines an intent for the conversation input 410 using a trained classifier 363. Similarly, in a historic conversation 400, between a human user and a human or NLP assistant, the trained classifier 363 analyzes the conversation inputs 410 as historic utterances to determine the intent of the human user);
providing the prompt as input to the large language model ([0036] - When an NLP assistant generates the conversation responses 420 in a live conversation 400, the NLP assistant first determines an intent for the conversation input 410 using a trained classifier 363. Similarly, in a historic conversation 400, between a human user and a human or NLP assistant, the trained classifier 363 analyzes the conversation inputs 410 as historic utterances to determine the intent of the human user);
obtaining, as output, the one or more user intent class labels for the one or more samples ([0036] - The classifier 363 may assign one or more labels 430 to one or more conversation inputs 410 that reflect the determined intent for the particular conversation inputs 410);
and based on the one or more user intent class labels, determining a proportion of the one or more samples that have a user intent class label corresponding with a user intent class of the intent taxonomy ([0038] - Therefore, the dataset builder 365 identifies which examples selected from the root nodes 440 (whether correctly or incorrectly initially classified) to include in a training dataset, and may select a predefined number of examples to include in the training dataset. The examples may include a predefined number or percentage of outliers, inliers, and mandated examples).
Regarding claim 10, Mallinar discloses the computing system, wherein analyzing the intent taxonomy to determine whether the intent taxonomy is valid comprises verifying consistency of the intent taxonomy by:
generating a prompt to annotate the one or more samples using the intent taxonomy ([0036] - When an NLP assistant generates the conversation responses 420 in a live conversation 400, the NLP assistant first determines an intent for the conversation input 410 using a trained classifier 363. Similarly, in a historic conversation 400, between a human user and a human or NLP assistant, the trained classifier 363 analyzes the conversation inputs 410 as historic utterances to determine the intent of the human user);
providing the prompt as input to the large language model ([0036] - When an NLP assistant generates the conversation responses 420 in a live conversation 400, the NLP assistant first determines an intent for the conversation input 410 using a trained classifier 363. Similarly, in a historic conversation 400, between a human user and a human or NLP assistant, the trained classifier 363 analyzes the conversation inputs 410 as historic utterances to determine the intent of the human user);
obtaining, as output, the one or more user intent class labels for the one or more samples ([0036] - The classifier 363 may assign one or more labels 430 to one or more conversation inputs 410 that reflect the determined intent for the particular conversation inputs 410);
and based on the one or more user intent class labels, determining if the large language model consistently applied definitions associated with the one or more user intent classes ([0038] - The examples may include a predefined number or percentage of outliers, inliers, and mandated examples. For example, a training dataset may include x examples provided by or selected by a developer (i.e., mandated examples), y examples that fall within a predefined range of similarity to one another as vector representations or fall within a predicted pattern (i.e., inlier examples), and z examples that fall outside of a predefined range of similarity to one another in a vector representations or fall outside of a predicted pattern (i.e., outlier examples). For example, the vectorizations of the first, second, and third example utterances 510a-c are inliers, as the values fall within the predefined range 520 shown in the vector plot 500, while the fourth, fifth, and sixth example utterances 50d-f are outliers as the values fall outside of the predefined range 520. Any of the example utterances 510a-f may be mandated or historic examples).
Regarding claim 11, Mallinar discloses the computing system, wherein analyzing the intent taxonomy to determine whether the intent taxonomy is valid comprises verifying conciseness of the intent taxonomy by:
generating a prompt to annotate test samples using the intent taxonomy ([0036] - When an NLP assistant generates the conversation responses 420 in a live conversation 400, the NLP assistant first determines an intent for the conversation input 410 using a trained classifier 363. Similarly, in a historic conversation 400, between a human user and a human or NLP assistant, the trained classifier 363 analyzes the conversation inputs 410 as historic utterances to determine the intent of the human user);
providing the prompt as input to the large language model ([0036] - When an NLP assistant generates the conversation responses 420 in a live conversation 400, the NLP assistant first determines an intent for the conversation input 410 using a trained classifier 363. Similarly, in a historic conversation 400, between a human user and a human or NLP assistant, the trained classifier 363 analyzes the conversation inputs 410 as historic utterances to determine the intent of the human user);
obtaining, as output, user intent class labels for the test samples ([0036] - The classifier 363 may assign one or more labels 430 to one or more conversation inputs 410 that reflect the determined intent for the particular conversation inputs 410);
and based on the user intent class labels, determining if a threshold number of test samples have a user intent class label associated with each user intent class of the one or more user intent classes ([0038] - The examples may include a predefined number or percentage of outliers, inliers, and mandated examples. For example, a training dataset may include x examples provided by or selected by a developer (i.e., mandated examples), y examples that fall within a predefined range of similarity to one another as vector representations or fall within a predicted pattern (i.e., inlier examples), and z examples that fall outside of a predefined range of similarity to one another in a vector representations or fall outside of a predicted pattern (i.e., outlier examples). For example, the vectorizations of the first, second, and third example utterances 510a-c are inliers, as the values fall within the predefined range 520 shown in the vector plot 500, while the fourth, fifth, and sixth example utterances 50d-f are outliers as the values fall outside of the predefined range 520. Any of the example utterances 510a-f may be mandated or historic examples).
Regarding claim 12, Mallinar discloses the computing system, wherein analyzing the intent taxonomy to determine whether the intent taxonomy is valid comprises verifying accuracy of the intent taxonomy by:
generating a prompt to annotate test samples using the intent taxonomy ([0036] - When an NLP assistant generates the conversation responses 420 in a live conversation 400, the NLP assistant first determines an intent for the conversation input 410 using a trained classifier 363. Similarly, in a historic conversation 400, between a human user and a human or NLP assistant, the trained classifier 363 analyzes the conversation inputs 410 as historic utterances to determine the intent of the human user);
providing the prompt as input to the large language model ([0036] - When an NLP assistant generates the conversation responses 420 in a live conversation 400, the NLP assistant first determines an intent for the conversation input 410 using a trained classifier 363. Similarly, in a historic conversation 400, between a human user and a human or NLP assistant, the trained classifier 363 analyzes the conversation inputs 410 as historic utterances to determine the intent of the human user);
obtaining, as output, user intent class labels for the test samples ([0036] - The classifier 363 may assign one or more labels 430 to one or more conversation inputs 410 that reflect the determined intent for the particular conversation inputs 410);
comparing the user intent class labels output from the large language model to human-annotated user intent class labels for the test samples ([0040] - The dataset builder 365 identifies the example utterances 510 that are most anomalous compared to the set of example utterances 510 being evaluated based on various scoring schema);
and based on the comparison, determining accuracy of the intent taxonomy ([0040] - The scoring schema identify how different an individual utterance's vector is from the other vectors for the other utterances being evaluated in the multidimensional vector space).
Regarding claim 14, Mallinar discloses a computer-implemented method comprising:
generating a first model prompt to input into a large language model, the first model prompt including an instruction to generate an intent taxonomy and an indication of training data, including data requests, to use for generating the intent taxonomy ([0040] - The dataset builder 365 identifies the example utterances 510 that are most anomalous compared to the set of example utterances 510 being evaluated based on various scoring schema. The scoring schema identify how different an individual utterance's vector is from the other vectors for the other utterances being evaluated in the multidimensional vector space. In some embodiments, the dataset builder 365 uses a cosine distance, a trained forest model, or a one-class support vector machine to determine the anomaly score, which may be understood as a distance in the multidimensional space away from the inlier utterances);
obtaining, as output from the large language model, the intent taxonomy that includes one or more user intent classes based on the training data ([0012] - When training a classifier for use in a Natural Language Processing (NLP) model, such as a chatbot, to identify various intents that a user may express, a developer should provide a large and varied training dataset to the classifier);
based on validating the intent taxonomy, storing the intent taxonomy for subsequent use in identifying user intent ([0046] - When the classifier 363 has correctly determined the intent of the anomalous example 620, the developer responds with the original label 430, and the dataset builder 365 adds the anomalous example 620 with the original label 430 to an in-intent pool of utterances of the training dataset 630 (Block 740). When the classifier 363 has incorrectly determined the intent of the anomalous example 620, the developer responds with a new label 430 that matches the correct intent for the anomalous example 620…).
obtaining an intent identification request to identify user intent associated with a new data request ([0013] - In a training dataset used a supervised (or semi-supervised) learning process, individual data points are labeled with a correspondence to various intents. These labels are metadata that inform the model being trained how the data point should be classified relative to a given intent),
generating a second model prompt to input into the large language model, the second model prompt including the intent taxonomy and an instruction to identify user intent associated with the new data request ([0040] - The dataset builder 365 identifies the example utterances 510 that are most anomalous compared to the set of example utterances 510 being evaluated based on various scoring schema. The scoring schema identify how different an individual utterance's vector is from the other vectors for the other utterances being evaluated in the multidimensional vector space. In some embodiments, the dataset builder 365 uses a cosine distance, a trained forest model, or a one-class support vector machine to determine the anomaly score, which may be understood as a distance in the multidimensional space away from the inlier utterances);
obtaining, as output from the large language model, a label of a user intent class associated with the new data request ([0012] - When training a classifier for use in a Natural Language Processing (NLP) model, such as a chatbot, to identify various intents that a user may express, a developer should provide a large and varied training dataset to the classifier);
and providing the label of the user intent class associated with the new data request for display via a user interface or for analysis of the user intent corresponding with the new data request ([0046] - ([0046] - When the classifier 363 has correctly determined the intent of the anomalous example 620, the developer responds with the original label 430, and the dataset builder 365 adds the anomalous example 620 with the original label 430 to an in-intent pool of utterances of the training dataset 630 (Block 740). When the classifier 363 has incorrectly determined the intent of the anomalous example 620, the developer responds with a new label 430 that matches the correct intent for the anomalous example 620…A user interface may present the n next-most-likely intents to the developer to choose from when selecting a new intent).
Regarding claim 15, Mallinar discloses the method, wherein validating the intent taxonomy includes using human-labeled user intent associated with data samples ([0037] - the classifier 363 identifies a root node 440 in the conversation 400. The root node 440 identifies the first indication from the user of a main purpose for the conversation 400…).
Regarding claim 16, Mallinar discloses the method further comprising:
obtaining the new data request from a user device, the new data request comprising a search query or a chat inquiry ([0015] - Using existing chat logs, the training dataset is built to include a pool of examples that are particularly unique to provide the model being trained with the most-difficult to handle responses in the training dataset);
and using the label of the user intent class associated with the new data request to generate a recommendation associated with the new data request to provide to the user device ([0034] - The conversations 400 may be between a human user and a human assistant or a human user and an NLP assistant, such as a chatbot, using an NLP model 362 to determine how to respond to the human user. The conversations 400 includes one or more utterances received from the human user as conversation inputs 410, and one or more conversation responses 420 from the assistant. Other example conversations 400 may include more or fewer utterances with different dialog).
Regarding claim 17, Mallinar discloses the method, wherein the intent taxonomy includes descriptions of the one ore more user intent classes and examples associated with the one or more user intent classes ([0041] - the dataset builder 365 updates an existing training dataset with newly analyzed example utterances 510, while in other embodiments, the dataset builder 365 creates a new training set with the example utterances 510 analyzed. In various embodiments, the dataset builder 365 may analyze the example utterances included in a current training dataset as well as the utterances received within the last d days to update or create a new training dataset).
Regarding claim 18, Mallinar discloses one or more computer storage media having computer-executable instructions embodied thereon that, when executed by one or more processors, cause the one or more processors to perform a method ([0057] - which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks), the method comprising:
obtaining training data including data requests for information ([0012] - When training a classifier for use in a Natural Language Processing (NLP) model, such as a chatbot, to identify various intents that a user may express, a developer should provide a large and varied training dataset to the classifier);
generating a model prompt to be input into a large language model, the model prompt including an instruction to generate an intent taxonomy, an indication of the training data to use for generating the intent taxonomy, and a taxonomy attribute desired to be used as criteria to generate a quality intent taxonomy ([0040] - The dataset builder 365 identifies the example utterances 510 that are most anomalous compared to the set of example utterances 510 being evaluated based on various scoring schema. The scoring schema identify how different an individual utterance's vector is from the other vectors for the other utterances being evaluated in the multidimensional vector space. In some embodiments, the dataset builder 365 uses a cosine distance, a trained forest model, or a one-class support vector machine to determine the anomaly score, which may be understood as a distance in the multidimensional space away from the inlier utterances);
obtaining, as output from the large language model, the intent taxonomy that includes one or more user intent classes ([0012] - When training a classifier for use in a Natural Language Processing (NLP) model, such as a chatbot, to identify various intents that a user may express, a developer should provide a large and varied training dataset to the classifier);
using human input to validate or refine the intent taxonomy ([0040] - The dataset builder 365 identifies the example utterances 510 that are most anomalous compared to the set of example utterances 510 being evaluated based on various scoring schema);
and providing the intent taxonomy for subsequent use to identify, via the large language model, user intent associated with a new data request ([0046] - When the classifier 363 has correctly determined the intent of the anomalous example 620, the developer responds with the original label 430, and the dataset builder 365 adds the anomalous example 620 with the original label 430 to an in-intent pool of utterances of the training dataset 630 (Block 740). When the classifier 363 has incorrectly determined the intent of the anomalous example 620, the developer responds with a new label 430 that matches the correct intent for the anomalous example 620…).
Regarding claim 19, Mallinar discloses the media, wherein the human input comprises recommendations for modifying the one or more user intent classes, descriptions associated with the one or more user intent classes, or examples associated with the one or more user intent classes ([0041] - the dataset builder 365 updates an existing training dataset with newly analyzed example utterances 510, while in other embodiments, the dataset builder 365 creates a new training set with the example utterances 510 analyzed. In various embodiments, the dataset builder 365 may analyze the example utterances included in a current training dataset as well as the utterances received within the last d days to update or create a new training dataset).
Regarding claim 20, Mallinar discloses the media, wherein the human input comprises human-labeled user intent associated with data samples, and wherein the human-labeled user intent is compared with machine-labeled user intent generated via the large language model samples ([0037] - the classifier 363 identifies a root node 440 in the conversation 400. The root node 440 identifies the first indication from the user of a main purpose for the conversation 400…).
Allowable Subject Matter
7. Claim 13 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: The prior art could not overcome or render obvious the limitation of “wherein comparing the user intent class labels output from the large language model to the human-annotated user intent class labels comprises measuring an inter-coder reliability therebetween” as claimed.
The closest prior art, Mallinar (U.S. Publication No. 20200142959) either singularly or in combination fail to anticipate or render obvious the above described limitations. Mallinar discloses driving intent expansion via anomaly detection in a modular conversational system.
Conclusion
8. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Katz (U.S. Publication No. 2020167429) teaches efficient use of word embeddings for text classification. Weston (U.S. Publication No. 20250054493) teaches methods and systems for training and deploying natural language understanding models.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ETHAN DANIEL KIM whose telephone number is (571) 272-1405. The examiner can normally be reached on Monday - Friday 9:00 - 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Richemond Dorvil can be reached on (571) 272-7602. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ETHAN DANIEL KIM/
Examiner, Art Unit 2658
/RICHEMOND DORVIL/ Supervisory Patent Examiner, Art Unit 2658