DETAILED ACTION
This Office Action is in response to the correspondence filed by the applicant on 12/04/2024.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The Information Statements (IDS) filed on 11/6/2025 have been accepted and considered in this office action and are in compliance with the provisions of 37 CFR 1.97.
Claim Objections
Claims 20 is objected to under 37 CFR 1.75 as being a substantial duplicate of claim 18. When two claims in an application are duplicates or else are so close in content that they both cover the same thing, despite a slight difference in wording, it is proper after allowing one claim to object to the other as being a substantial duplicate of the allowed claim. See MPEP § 608.01(m).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-14 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. Claim 10 recites, “One or more computer-storage media having computer-executable instructions embodied thereon that, when executed by a computing system having a processor and memory, cause the processor to perform operations, the operations comprising: …" The specification describes ([00114]) the computer-storage medium, which is not specifically limited to non-transitory propagating signals. Examiner suggests adding the limitation "non-transitory" to the claim to avoid a rejection under 35 U.S.C. 101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2 and 5-7 are rejected under 35 U.S.C. 103 as being unpatentable over PAULI (US 2024/0338532 A1), and in further view of FAYYAZ (US 2025/0053748 A1).
REGARDING CLAIM 1, PAULI discloses a computerized system comprising:
one or more computer processors; and computer memory storing computer-useable instructions that, when used by the one or more computer processors, cause the one or more computer processors to perform operations, the operations (PAULI Par 118 – “Computing device 1200 includes a bus 1210 that directly or indirectly couples the following devices: computer storage memory 1212, one or more processors 1214, one or more presentation components 1216, input/output (I/O) ports 1218, I/O components 1220, a power supply 1222, and a network component 1224. While computing device 1200 is depicted as a seemingly single device, multiple computing devices 1200 may work together and share the depicted device resources. For example, memory 1212 may be distributed across multiple devices, and processor(s) 1214 may be housed with different devices.”) comprising:
accessing a first dataset associated with a training dataset, wherein the first dataset comprises a first plurality of data items that are accurately labeled based on an established criteria (PAULI Par 31 – “The user 102 is prompted for label inputs 136 for a subset of samples, thus identifying an initial set of ground truth labels 138 for some of the samples. These ground truth labels 138 also identify a set of categories of interest to the user 102 which form the foundation of training for the student model 130.”; Par 45 – “During manual sample annotation of a particular sample, the user 102 is presented with data about that sample, including the text of the sample, a current label (e.g., category) assigned to the sample (if any), and a suggested category or label 122 for that sample (as generated by the LLM 120 using the sample text as input). The user 102 can use the suggested label 122 for the sample, or may define a new category or assign the sample to an existing category. This labeling becomes a ground truth 138 for that sample.”);
using an example selection machine learning model, identifying few-shot learning data from the first dataset (PAULI Par 49 – “At operation 440, the AI assistant 110 trains a teacher model 132 that is configured to identify samples from the dataset 104 that, if annotated (either through soft-labeling by the LLM 120 or manual labeling by the user 102), are likely to improve the student model 130.”), the few-shot learning data is a second dataset comprising a second plurality of data items (PAULI Par 47 – “Once the initial manual sample annotation is complete, the AI assistant enters a training loop. This training loop begins with training of the student model 130 at operation 420. The AI assistant 110 identifies a set of training samples to use in this current iteration of training of the student model 130. The student model 130 is exclusively trained on soft-labeled samples (soft-labeled by the LLM engine 130). Ground truth labels are only used for evaluating the student model 130. Evaluation involves exclusively ground truth labels.”);
using a few-shot learning Large Language Model (LLM) and a plurality of few-shot learning prompts (PAULI Par 62 – “At operation 520, the AI assistant generates a prompt that includes a request to categorize the sample 510 to be labeled, as well as a list of the current categories, the text of the sample 510 to be labeled, and the text for each of the nearby sampled labels 514.”; Par 70 – “In operation 1006, assistant 110 generates few-shot learning prompts for the LLM 120, where the learning prompts include labeled samples that a student model determines to be similar to a current training example.”), generating a first set of labels for the second plurality of data items in the few-shot learning data (PAULI Par 46 – “The assistant 110 then uses the user-annotated samples to generate soft labels for the samples to train the student model 130 (e.g., start with zero shot learning and then move into few-shot learning). Samples may be shown to the student model 130 to identify a set of top categories. Sentences can be selected from these top categories and provided as context to the LLM engine 120 to train the student model 130, test the student model 130 against human annotated samples, and loop repeatedly through model retraining until improvement diminishes.”; Par 47 – “Once the initial manual sample annotation is complete, the AI assistant enters a training loop. This training loop begins with training of the student model 130 at operation 420. The AI assistant 110 identifies a set of training samples to use in this current iteration of training of the student model 130. The student model 130 is exclusively trained on soft-labeled samples (soft-labeled by the LLM engine 130). Ground truth labels are only used for evaluating the student model 130. Evaluation involves exclusively ground truth labels.”);
using an error analysis engine, generating a first error analysis output associated with the first set of labels for the second plurality data items (PAULI Par 48 – “Once initially trained, the AI assistant 110 is configured to evaluate the performance of the current build of the student model 130 at operation 430. This evaluation includes testing the current training samples with ground truth labels 138 with the student model 130 to determine an overall accuracy percentage.”) based on evaluating the first set of labels for the second plurality of data items to corresponding accurate labels for the second plurality of data items in the first dataset (PAULI Par 32 –“… evaluating the current performance of the student model 130 until improvement diminishes. This student model 130 is analyzed by the assistant 110 using pre-labeled data (e.g., a few human-labeled data samples for each category, such as the ground truth labels 138) to test how consistent the soft labels 126 are performing.”);
based on the first error analysis output, generating one or more updated few-shot learning prompts using the plurality of few-shot learning prompts (PAULI Par 38 – “As part of an AI-assistance experience, the AI assistant 110 uses the LLM 120 to generate suggestions to the user 102 about how to categorize a datapoint. The assistant 110 is context-aware, as the assistant 110 creates few-shot learning prompts for LLMs 120 in real time. For example, the assistant 110 dynamically re-engineers the few-shot learning prompt. Each time a new sample is sent to the LLM 120 for generating a soft label 126 or label suggestion 122 for the user 102, the assistant 110 includes reference sentences that the student model 130 identifies as similar (e.g., based on cosine similarity between category probabilities). These prompts thus contextualize what the assistant 110 has already learned about the dataset 104 and the intent of the user 102.”; Par 62 – “At operation 520, the AI assistant generates a prompt that includes a request to categorize the sample 510 to be labeled, as well as a list of the current categories, the text of the sample 510 to be labeled, and the text for each of the nearby sampled labels 514. At decision 530, if the current iteration is an automatic iteration (e.g., at operations 456-458), then this prompt 522 may be submitted to the LLM 120 to generate a soft label 126 for this sample 510 at operation 532.”);
using the few-shot learning LLM and the updated few-shot learning prompts, generating a second set of labels [for the second plurality of data items in the few-shot learning data] (PAULI Par 62 – “At operation 520, the AI assistant generates a prompt that includes a request to categorize the sample 510 to be labeled, as well as a list of the current categories, the text of the sample 510 to be labeled, and the text for each of the nearby sampled labels 514. At decision 530, if the current iteration is an automatic iteration (e.g., at operations 456-458), then this prompt 522 may be submitted to the LLM 120 to generate a soft label 126 for this sample 510 at operation 532.”);
generating a second error analysis output for the second set of labels for the second plurality of data items in the few-shot learning data (PAULI Par 48 – “Once initially trained, the AI assistant 110 is configured to evaluate the performance of the current build of the student model 130 at operation 430. This evaluation includes testing the current training samples with ground truth labels 138 with the student model 130 to determine an overall accuracy percentage. The AI assistant 110 may track model performance data through several automatic iterations of this training loop and may compare prior performance data to the current performance data of the student model 130 to, for example, determine whether the prior iteration of additional samples have improved the model performance. This performance data may be used to determine whether the upcoming training will continue with automatic model training at operations 452-458 (e.g., when performance is still improving under automatic model training) or branch out to collect additional manual annotation data from the user 102 at operations 460-462 (e.g., when automatic model training has ceased to yield performance improvements using only soft labels 126 from the LLM 120).”);
based on the second error analysis output and the few-shot learning LLM, training an LLM-based labeler (PAULI Par 34 – “Upon concluding a round of user annotation, the AI assistant 110 may similarly perform another round of automatic training, now retraining the student model 130 with a larger set of samples with ground truth labels 138 provided by the user 102. Accordingly, the AI assistant 110 performs iterations of automatic labeling and manual labeling until a performance threshold is reached (e.g., a pre-determined correct categorization percentage) or until the user 102 is content at the current performance of the student model 130. At such time, the AI assistant 110 may perform a full index 140 of the dataset 104 using the student model 130.”; Par 32 – “The assistant 110 trains a teacher model 132 to identify samples within the student model 130 that can help improve the student model 130 with additional human annotation. The assistant 110 prompts the user 102 for label inputs 136 and uses those new label inputs 136 to improve and test 134 the student model 130. This cycle can continue for many iterations until improvement of the student model 130 has peaked.”); and
deploying the LLM-based labeler (PAULI Par 22 – “Once the user is confident that the AI assistant has identified all relevant categories, understood the user's intent, and can reliably assign samples according to the user's instructions, this information is distilled into a light-weight student model (e.g., a conventional ML classifier) that can categorize the entire dataset at a low performance cost (e.g., performable by a conventional central processing unit (CPU) without necessarily needing a graphics processing unit (GPU), and with very high throughput (e.g., greater than 10,000 sentences per second)).”).
PAULI does not explicitly teach the [square-bracketed] limitation. In other words, PAULI generates a first set of labels associated with second data items and evaluates the first set of labels against the ground-truth data. In the next iteration, PAULI generates a second set of labels associated with other selected data items which are not necessarily the second data used for generating the first set of labels.
FAYYAZ discloses a method/system for training machine learning model for natural language processing comprising: using an error analysis engine (FAYYAZ Fig. 5 – “Prediction Loss Logic 514”), generating a first error analysis output associated with the first set of labels for the second plurality data items based on evaluating the first set of labels for the second plurality of data items to corresponding accurate labels for the second plurality of data items in the first dataset (FAYYAZ Par 56 – “As a first part, prediction loss logic 514 assesses prediction loss associated with the student model 508. Prediction loss (also referred to as reconstruction loss) measures the difference between the ground-truth responses generated by the teacher model 506 and model-generated responses produced by the student model 508, given the current state of its weights 512.”);
based on the first error analysis output, generating one or more updated few-shot learning prompts using the plurality of few-shot learning prompts (FAYYAZ Par 36 – “With respect to the abstract-token-generating operations, the prompt-generating component 104 first generates a second prompt that includes: a) abstract-token-prompting information; b) the current input information (IN); c) the response RN generated by the model 106 to the current input information; and d) the abstract token information AN-1 produced in one or more prior dialogue turns.”);
using the few-shot learning LLM and the updated few-shot learning prompts, generating a second set of labels [for the second plurality of data items in the few-shot learning data] (FAYYAZ Pars 81-83 – “The training system 502 repeats this process M times. At each iteration, the student model 508 produces a new abstract token and a new model-generated response ŷn. … The summation in FIG. 1 is over M stages of iteration that were used to produce the abstract token information, with n referring to a particular pass number of those iterations. The first part of Equation (2) expresses the prediction loss, and the second part of Equation (2) (after the summation) expresses sparsity loss. With respect to the first part of the equation, L(y,ŷn) measures the difference between the model-generated response ŷn and a ground-truth response y (produced by the teacher model 506). More specifically, different implementations can use different measures to express loss, such as cross entropy. In other cases, the training component 502 calculates prediction loss using any type of reinforcement learning algorithm.”; In other words, FAYYAZ teaches using the same data for mearsing the error each iteration.);
generating a second error analysis output for the second set of labels for the second plurality of data items in the few-shot learning data (FAYYAZ Pars 81-83 – “The training system 502 repeats this process M times. At each iteration, the student model 508 produces a new abstract token and a new model-generated response ŷn. … The summation in FIG. 1 is over M stages of iteration that were used to produce the abstract token information, with n referring to a particular pass number of those iterations. The first part of Equation (2) expresses the prediction loss, and the second part of Equation (2) (after the summation) expresses sparsity loss. With respect to the first part of the equation, L(y,ŷn) measures the difference between the model-generated response ŷn and a ground-truth response y (produced by the teacher model 506). More specifically, different implementations can use different measures to express loss, such as cross entropy. In other cases, the training component 502 calculates prediction loss using any type of reinforcement learning algorithm.”);
based on the second error analysis output and the few-shot learning LLM, training an LLM-based labeler (FAYYAZ Par 73 – “Overall, the training process described above iteratively improves the ability of the student model 508 to generate both accurate responses and succinct abstract token information. A response is accurate insofar as it matches a counterpart ground-truth response. The succinctness objective of the loss function forces the student model 508 to generate succinct abstract token information. At the same time, the accuracy objective of the loss function prevents the compression of the abstract token information from negatively impacting its ability to accurately capture the semantics of the full dialogue history. This combination of objectives results in abstract token information that is maximally expressive and small in size. Viewed from another perspective, the dual objectives of the loss function result in the elimination of redundant information in the full dialogue history.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method/system of PAULI to include performing an error analysis using the same data at each iteration, as taught by FAYYAZ.
One of ordinary skill would have been motivated to include performing an error analysis using the same data at each iteration, in order to evaluate a model more objectively.
REGARDING CLAIM 2, PAULI in view of FAYYAZ discloses the system of claim 1, wherein the example selection machine learning model selects the few-shot learning data based on both an active learning technique (PAULI Par 61 – “Example solutions take advantage of active learning. Active learning approaches aim to identify those data points that are most critical for training a model to understand and categorize a dataset. Here, active learning is used for at least two purposes, namely for selecting those samples that require feedback from the domain expert, and to select samples to be annotated by the LLM 120, to further reduce computing resource usage, training time, and cost. Several sampling strategies are implemented, and the strategy is dynamically selected which is most likely to be successful, given characteristics of the dataset and what has already been learned about it.”) and a clustering technique (PAULI Par 46 – “In some implementations, the assistant 110 performs cluster analysis of the embeddings 124 and, for each cluster, may sample a few points to show the user 102. This approach of clustering at the early stage, rather than letting the teacher model 132 choose, is because there is not enough data yet to train the teacher model 132. For example, the assistant 110 identifies 25 clusters and, from within each cluster, selects a centered sample, one or more fringe or outlier samples (e.g., samples within the cluster but somewhat distant from the center), and a few random samples within the cluster region. These cluster selections can be shown to the user 102 to create initial annotations (e.g., two samples per category).”).
REGARDING CLAIM 5, PAULI in view of FAYYAZ discloses the system of claim 1, wherein the plurality of few-shot learning prompts are associated with prompt templates that include the following (PAULI Par 62 – “At operation 520, the AI assistant generates a prompt that includes a request to categorize the sample 510 to be labeled, as well as a list of the current categories, the text of the sample 510 to be labeled, and the text for each of the nearby sampled labels 514.”; Par 70 – “generates few-shot learning prompts for the LLM 120, where the learning prompts include labeled samples that a student model determines to be similar to a current training example.”): an objective (PAULI Fig. 5 – “Please categorize the following samples”), a labeling structure (PAULI Fig. 5 – “into either of (labeled sample categories)”), an example format, few-shot examples (PAULI Fig. 5 – “list of samples w/ labels”), and a prompt for new data (PAULI Fig. 5 – “samples w/o label”).
REGARDING CLAIM 6, PAULI in view of FAYYAZ discloses the system of claim 1, further comprising a prompt engineering LLM that supports using the error analysis output to automatically update few-shot learning prompts for prompt-based learning using the updated few-shot learning prompts (PAULI Par 76 – “dynamically alter a few-shot learning prompt for the LLM, including labeled samples that a student model determines to be similar to a current training sample;”; Par 38 – “As part of an AI-assistance experience, the AI assistant 110 uses the LLM 120 to generate suggestions to the user 102 about how to categorize a datapoint. The assistant 110 is context-aware, as the assistant 110 creates few-shot learning prompts for LLMs 120 in real time. For example, the assistant 110 dynamically re-engineers the few-shot learning prompt. Each time a new sample is sent to the LLM 120 for generating a soft label 126 or label suggestion 122 for the user 102, the assistant 110 includes reference sentences that the student model 130 identifies as similar (e.g., based on cosine similarity between category probabilities). These prompts thus contextualize what the assistant 110 has already learned about the dataset 104 and the intent of the user 102.”; Par 62 – “At operation 520, the AI assistant generates a prompt that includes a request to categorize the sample 510 to be labeled, as well as a list of the current categories, the text of the sample 510 to be labeled, and the text for each of the nearby sampled labels 514. At decision 530, if the current iteration is an automatic iteration (e.g., at operations 456-458), then this prompt 522 may be submitted to the LLM 120 to generate a soft label 126 for this sample 510 at operation 532.”).
REGARDING CLAIM 7, PAULI in view of FAYYAZ discloses the system of claim 1, wherein the example selection machine learning model (PAULI Par 49 – “At operation 450, the AI assistant 110 applies the teacher model 132 to identify samples for further annotation.”), the few-shot learning LLM (PAULI Par 46 –“ The assistant 110 then uses the user-annotated samples to generate soft labels for the samples to train the student model 130 (e.g., start with zero shot learning and then move into few-shot learning).”; Par 57 – “For example, when the AI assistant 110 determines to continue with automatic labeling, the AI assistant 110 uses the LLM 120 to generate soft labels 126 for each of the newly selected samples at operations 456-458 and these samples and their soft labels are subsequently used to retrain the student model 130 at operation 420.”) and the error analysis engine (PAULI Par 48 – “Once initially trained, the AI assistant 110 is configured to evaluate the performance of the current build of the student model 130 at operation 430. This evaluation includes testing the current training samples with ground truth labels 138 with the student model 130 to determine an overall accuracy percentage.”) are part of a labeler development engine that iteratively develops the LLM-based labeler based on a plurality of iterative refinement operations (PAULI Fig. 4; Par 48 – “The AI assistant 110 may track model performance data through several automatic iterations of this training loop and may compare prior performance data to the current performance data of the student model 130 to, for example, determine whether the prior iteration of additional samples have improved the model performance.”).
Claims 3-4 and 8-9 are rejected under 35 U.S.C. 103 as being unpatentable over PAULI (US 2024/0338532 A1) in view of FAYYAZ (US 2025/0053748 A1), and in further view of GHOCHE (US 2024/0386214 A1).
REGARDING CLAIM 3, PAULI in view of FAYYAZ discloses the system of claim 1, wherein the few-shot learning data is identified to support a [hierarchical] categorization of data items in the few-shot learning data (PAULI Par 38 – “As part of an AI-assistance experience, the AI assistant 110 uses the LLM 120 to generate suggestions to the user 102 about how to categorize a datapoint. The assistant 110 is context-aware, as the assistant 110 creates few-shot learning prompts for LLMs 120 in real time. For example, the assistant 110 dynamically re-engineers the few-shot learning prompt.”).
PAUL in view of FAYYAZ does not explicitly teach the [square-bracketed] limitation.
GHOCHE discloses the [square-bracketed] limitation. GHOCHE discloses a method/system for automatically generating a natural language workflow for customer support of emails wherein the training data is identified to support a [hierarchical] categorization of data items (GHOCHE Par 98 – “A ticket covers the entire lifecycle of an issue. A dataset of historic tickets would conventionally be manually labelled for routing to agents. For example, a ticket might include fields for category and subcategory. It may also include fields identifying the queue the ticket was sent to. In some cases the agent who answered the ticket may be included. The priority level associated with the ticket may also be included.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method/system of PAULI in view of FAYYAZ to include supporting a hierarchical categorization of data, as taught by GHOCHE.
One of ordinary skill would have been motivated to include supporting a hierarchical categorization of data, in order to accurately categorize incoming emails and text tickets for customer service (Par 99).
REGARDING CLAIM 4, PAULI in view of FAYYAZ discloses the system of claim 1, wherein the plurality of few-shot learning prompts comprises hierarchical labeling prompts that are designed to guide the few-shot learning LLM to assign labels to data items (PAULI Par 62 – “At operation 520, the AI assistant generates a prompt that includes a request to categorize the sample 510 to be labeled, as well as a list of the current categories, the text of the sample 510 to be labeled, and the text for each of the nearby sampled labels 514.”; Par 70 – “In operation 1006, assistant 110 generates few-shot learning prompts for the LLM 120, where the learning prompts include labeled samples that a student model determines to be similar to a current training example.”) based on a predefined [hierarchical] structure (PAULI Par 31 – “The user 102 is prompted for label inputs 136 for a subset of samples, thus identifying an initial set of ground truth labels 138 for some of the samples. These ground truth labels 138 also identify a set of categories of interest to the user 102 which form the foundation of training for the student model 130.”).
PAUL in view of FAYYAZ does not explicitly teach the [square-bracketed] limitation.
GHOCHE discloses the [square-bracketed] limitation. GHOCHE discloses a method/system for automatically generating a natural language workflow for customer support of emails comprising hierarchical training data such that a model is trained to assign labels to data items based on a predefined [hierarchical] structure (GHOCHE Par 98 – “A ticket covers the entire lifecycle of an issue. A dataset of historic tickets would conventionally be manually labelled for routing to agents. For example, a ticket might include fields for category and subcategory. It may also include fields identifying the queue the ticket was sent to. In some cases the agent who answered the ticket may be included. The priority level associated with the ticket may also be included.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method/system of PAULI in view of FAYYAZ to include supporting a hierarchical categorization of data, as taught by GHOCHE.
One of ordinary skill would have been motivated to include supporting a hierarchical categorization of data, in order to accurately categorize incoming emails and text tickets for customer service (Par 99).
REGARDING CLAIM 8, PAULI in view of FAYYAZ discloses the system of claim 1, the operations further comprising:
using the LLM-based labeler, labeling the training dataset (PAULI Fig. 4; Par 58 – “In some implementations, the AI assistant 110 may use the current student model 130 to determine a soft label 138 for one or more of the selected samples and a confidence score for that soft label 138. If the confidence score of a particular soft label is above a predetermined threshold for that sample (e.g., if the student model 130 seems to indicate, with a degree of certainty, that the sample falls into one of the defined categories), then that soft label is automatically added to the sample at operations 452-454.”) comprising a plurality [email] data items (PAULI Par 29 – “For example, an organization may wish to analyze customer churn based on a dataset 104 of text-based customer complaints, where each complaint contains one or more sentences provided by the submitting customer. However, it should be understood that other types of data and use cases are possible”; Par 40 – “In some implementations, the AI assistant 110 is configured to support multiple types of media or modalities of data (multimodal), such as a combination of audio and text (e.g., customer voice complaint calls and online text-based complaints to classify types of complaints, or joint vision-language models), or images, video, and text (e.g., professional images of people, video interviews, and their text-based biographies, to classify occupation types), or other multi-modal deep learning models.”);
training [a machine learning model] the labeler with the training dataset (PAULI Par 28 –“In architecture 100, a user 102 at a user computing device 101 interacts with an AI assistant 110 to train a student model 130 that helps give the user 102 insights they need within a dataset 104 of structured or unstructured data from their organization.”; Par 34 – “Upon concluding a round of user annotation, the AI assistant 110 may similarly perform another round of automatic training, now retraining the student model 130 with a larger set of samples with ground truth labels 138 provided by the user 102. Accordingly, the AI assistant 110 performs iterations of automatic labeling and manual labeling until a performance threshold is reached (e.g., a pre-determined correct categorization percentage) or until the user 102 is content at the current performance of the student model 130. At such time, the AI assistant 110 may perform a full index 140 of the dataset 104 using the student model 130.”; Par 32 – “The assistant 110 trains a teacher model 132 to identify samples within the student model 130 that can help improve the student model 130 with additional human annotation. The assistant 110 prompts the user 102 for label inputs 136 and uses those new label inputs 136 to improve and test 134 the student model 130. This cycle can continue for many iterations until improvement of the student model 130 has peaked.”), wherein the machine learning model is associated with a customer service email management system (PAULI Par 28 –“In architecture 100, a user 102 at a user computing device 101 interacts with an AI assistant 110 to train a student model 130 that helps give the user 102 insights they need within a dataset 104 of structured or unstructured data from their organization.”; Par 29 – “For example, an organization may wish to analyze customer churn based on a dataset 104 of text-based customer complaints, where each complaint contains one or more sentences provided by the submitting customer. However, it should be understood that other types of data and use cases are possible”; Par 40 – “… such as a combination of audio and text (e.g., customer voice complaint calls and online text-based complaints to classify types of complaints, or joint vision-language models), …”); and
deploying the machine learning model to predict customer service actions for electronic communications (PAULI Par 21 –“The example solutions have applications across various industries including, for example, support ticket routing, insurance claim risk assessment, content moderation, medical record classification, Securities Exchange Commission (SEC) compliance assessment, classification of scientific response documents, categorization of upstream data for exploration, and customer account classification.”; Par 40 – “In some implementations, the AI assistant 110 is configured to support multiple types of media or modalities of data (multimodal), such as a combination of audio and text (e.g., customer voice complaint calls and online text-based complaints to classify types of complaints, or joint vision-language models), or images, video, and text (e.g., professional images of people, video interviews, and their text-based biographies, to classify occupation types), or other multi-modal deep learning models.”; Par 68 – “Use of example solutions can have a significant impact on top-level business key performance indicators (KPIs) the users care about (e.g., quickly identifying and responding to trends in customer feedback). Further, example solutions offer a persistent presence in assisting the user to make sense of their data, allowing the state of a project can be saved and restored from memory and learning from its cooperation with the user to improve its accuracy over time. Elements of a user interface provide intuitive visualizations of the model and its understanding of the data and the users' interest in it, allowing the user to achieve state of the art accuracy with minimal effort in terms of time and upskilling.”).
PAULI does not explicitly teach the [square-bracketed] limitation and teaches the underlined feature instead. In other words, PAULI teaches training the labeler (e.g., student model) to classify customer complaints and to respond the complaints, but does not explicitly teach training another model beside the labeler for predicting customer service actions. PAULI teaches online-based text communication, but does not explicitly teach [email] communication.
GHOCHE discloses the [square-bracketed] limitations. GHOCHE discloses a method/system for automatically generating a natural language workflow for customer support of emails comprising:
using the LLM-based labeler, labeling the training dataset (GHOCHE Par 161 – “In block 1620, the ticket data that was structured data is clustered. The clustering algorithm may include a rule or an algorithm to assign a text description to the cluster. The clusters are used to label the customer support tickets and generate training data to train the classifier. This corresponds to training the classifier on weakly supervised training data.”) comprising a plurality [email] data items (GHOCHE Par 156 – “Customer support tickets may include emails, chats, and other information that is unstructured text generated asynchronously. For example, a customer support chat UI may include a general subject field and unstructured text field for a customer to enter their question and initiate a chat with an agent.”; Par 198 – “In one implementation, the solve module 215 is designed to support handling long-form email tickets by detecting intent implementing a complete workflow to resolve a customer's query/concern.”);
training [a machine learning model] with the training dataset (GHOCHE Par 138 – “FIG. 10 is a high level flow chart of a method of training a ML classifier model to identify a category/subcategory of a customer question to perform routing of tickets to agents. In block 1005, historic ticket is ingested, which may include manually labelled category/subcategory routing information, as well as a priority level. In block 1010, a ML model is trained to identify category/subcategory of a customer question for routing purposes. …”; Par 161 – “In block 1620, the ticket data that was structured data is clustered. The clustering algorithm may include a rule or an algorithm to assign a text description to the cluster. The clusters are used to label the customer support tickets and generate training data to train the classifier. This corresponds to training the classifier on weakly supervised training data.”), wherein the machine learning model is associated with a customer service email management system (GHOCHE Par 156 – “Customer support tickets may include emails, chats, and other information that is unstructured text generated asynchronously. For example, a customer support chat UI may include a general subject field and unstructured text field for a customer to enter their question and initiate a chat with an agent.”; Par 198 – “In one implementation, the solve module 215 is designed to support handling long-form email tickets by detecting intent implementing a complete workflow to resolve a customer's query/concern.”); and
deploying the machine learning model to predict customer service actions for electronic communications (GHOCHE Par 138 – “FIG. 10 is a high level flow chart of a method of training a ML classifier model to identify a category/subcategory of a customer question to perform routing of tickets to agents. In block 1005, historic ticket is ingested, which may include manually labelled category/subcategory routing information, as well as a priority level. In block 1010, a ML model is trained to identify category/subcategory of a customer question for routing purposes. This may include, for example, identifying a category/subcategory for identifying an escalation category/subcategory. For example, a customer may be complaining about a repeat problem, or that they want a refund, or that customer service is no good, etc. A ticket corresponding to an escalation risk may be routed to a human agent with training or experience in handling escalation risks. In some implementations, an escalation risk is predicted based in part on other data, such as customer survey data as previously discussed. More generally, escalation risk can be predicted using a model that prioritizes tickets based on past escalations and ticket priority, with customer survey data being still yet another source of data used to train the model.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method/system of PAULI in view of FAYYAZ to include training a model for customer service actions prediction, as taught by GHOCHE.
One of ordinary skill would have been motivated to include training a model for customer service actions prediction, in order to accurately identify intent of customer emails and to generate workflows to resolve tickets (GHOCHE Par 213).
REGARDING CLAIM 9, PAULI in view of FAYYAZ discloses the system of claim 1, wherein the customer service actions are associated with a [hierarchical] classification of customer service actions (PAULI Par 38 – “As part of an AI-assistance experience, the AI assistant 110 uses the LLM 120 to generate suggestions to the user 102 about how to categorize a datapoint. The assistant 110 is context-aware, as the assistant 110 creates few-shot learning prompts for LLMs 120 in real time. For example, the assistant 110 dynamically re-engineers the few-shot learning prompt.”; Par 28 –“In architecture 100, a user 102 at a user computing device 101 interacts with an AI assistant 110 to train a student model 130 that helps give the user 102 insights they need within a dataset 104 of structured or unstructured data from their organization.”; Par 29 – “For example, an organization may wish to analyze customer churn based on a dataset 104 of text-based customer complaints, where each complaint contains one or more sentences provided by the submitting customer. However, it should be understood that other types of data and use cases are possible”; Par 40 – “… such as a combination of audio and text (e.g., customer voice complaint calls and online text-based complaints to classify types of complaints, or joint vision-language models), …”).
PAUL in view of FAYYAZ does not explicitly teach the [square-bracketed] limitation.
GHOCHE discloses the [square-bracketed] limitation. GHOCHE discloses a method/system for automatically generating a natural language workflow for customer support of emails wherein the customer service actions are associated with a [hierarchical] classification of customer service actions (GHOCHE Par 98 – “A ticket covers the entire lifecycle of an issue. A dataset of historic tickets would conventionally be manually labelled for routing to agents. For example, a ticket might include fields for category and subcategory. It may also include fields identifying the queue the ticket was sent to. In some cases the agent who answered the ticket may be included. The priority level associated with the ticket may also be included.”; Par 99 – “In one implementation, the ML system predicts the category and subcategory. The category and subcategory may determine, for example, a department or a subset of agents who can solve a ticket.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method/system of PAULI in view of FAYYAZ to include training a model for customer service actions prediction, as taught by GHOCHE.
One of ordinary skill would have been motivated to include training a model for customer service actions prediction, in order to accurately identify intent of customer emails and to generate workflows to resolve tickets (GHOCHE Par 213).
Claims 10-20 are rejected under 35 U.S.C. 103 as being unpatentable over PAULI (US 2024/0338532 A1), and in further view of GHOCHE (US 2024/0386214 A1).
REGARDING CLAIM 10, PAULI discloses one or more computer-storage media having computer-executable instructions embodied thereon that, when executed by a computing system having a processor and memory, cause the processor to perform operations (PAULI Par 118 – “Computing device 1200 includes a bus 1210 that directly or indirectly couples the following devices: computer storage memory 1212, one or more processors 1214, one or more presentation components 1216, input/output (I/O) ports 1218, I/O components 1220, a power supply 1222, and a network component 1224. While computing device 1200 is depicted as a seemingly single device, multiple computing devices 1200 may work together and share the depicted device resources. For example, memory 1212 may be distributed across multiple devices, and processor(s) 1214 may be housed with different devices.”), the operations comprising:
accessing email data associated with a customer service [email] management system (PAULI Par 29 – “For example, an organization may wish to analyze customer churn based on a dataset 104 of text-based customer complaints, where each complaint contains one or more sentences provided by the submitting customer. However, it should be understood that other types of data and use cases are possible”; Par 40 – “In some implementations, the AI assistant 110 is configured to support multiple types of media or modalities of data (multimodal), such as a combination of audio and text (e.g., customer voice complaint calls and online text-based complaints to classify types of complaints, or joint vision-language models), or images, video, and text (e.g., professional images of people, video interviews, and their text-based biographies, to classify occupation types), or other multi-modal deep learning models.”);
using a machine learning model associated with a Large Language Model (LLM)-based labeler (PAULI Par 46 –“ The assistant 110 then uses the user-annotated samples to generate soft labels for the samples to train the student model 130 (e.g., start with zero shot learning and then move into few-shot learning).”; Par 57 – “For example, when the AI assistant 110 determines to continue with automatic labeling, the AI assistant 110 uses the LLM 120 to generate soft labels 126 for each of the newly selected samples at operations 456-458 and these samples and their soft labels are subsequently used to retrain the student model 130 at operation 420.”), generating a predicted customer service action for an email in the email data (PAULI Par 21 –“The example solutions have applications across various industries including, for example, support ticket routing, insurance claim risk assessment, content moderation, medical record classification, Securities Exchange Commission (SEC) compliance assessment, classification of scientific response documents, categorization of upstream data for exploration, and customer account classification.”; Par 40 – “In some implementations, the AI assistant 110 is configured to support multiple types of media or modalities of data (multimodal), such as a combination of audio and text (e.g., customer voice complaint calls and online text-based complaints to classify types of complaints, or joint vision-language models), or images, video, and text (e.g., professional images of people, video interviews, and their text-based biographies, to classify occupation types), or other multi-modal deep learning models.”; Par 68 – “Use of example solutions can have a significant impact on top-level business key performance indicators (KPIs) the users care about (e.g., quickly identifying and responding to trends in customer feedback).”), wherein the LLM-based labeler is trained using an LLM-based labeler development engine that iteratively develops the LLM-based labeler based on a plurality of iterative refinement operations (PAULI Fig. 4; Par 48 – “The AI assistant 110 may track model performance data through several automatic iterations of this training loop and may compare prior performance data to the current performance data of the student model 130 to, for example, determine whether the prior iteration of additional samples have improved the model performance. This performance data may be used to determine whether the upcoming training will continue with automatic model training at operations 452-458 (e.g., when performance is still improving under automatic model training) or branch out to collect additional manual annotation data from the user 102 at operations 460-462 (e.g., when automatic model training has ceased to yield performance improvements using only soft labels 126 from the LLM 120).”); and
communicating the predicted customer service action (PAULI Figs, 7 and 8; Par 21 –“The example solutions have applications across various industries including, for example, support ticket routing, …”; Par 68 – “Use of example solutions can have a significant impact on top-level business key performance indicators (KPIs) the users care about (e.g., quickly identifying and responding to trends in customer feedback). Further, example solutions offer a persistent presence in assisting the user to make sense of their data, allowing the state of a project can be saved and restored from memory and learning from its cooperation with the user to improve its accuracy over time. Elements of a user interface provide intuitive visualizations of the model and its understanding of the data and the users' interest in it, allowing the user to achieve state of the art accuracy with minimal effort in terms of time and upskilling.”; Par 66 – “After several training iterations, the graph 710 now shows a snaking structure in the data. Each category is represented by a distinct color, both within the dots of the graph 710 and within the categories frame 712, where a particular point on the graph 710 is colored based on its soft- or human-annotated label. The categories frame 712 displays a pie chart of the three categories and associated statistics (e.g., 63 total samples, 27 of which are Orange officeholders, 19 of which are Athletes, and 17 of which are Artists).”).
PAULI does not explicitly teach the [square-bracketed] limitation. In other words, PAULI teaches online-based text communication, but does not explicitly teach [email] communication.
GHOCHE discloses the [square-bracketed] limitation. GHOCHE discloses a method/system for automatically generating a natural language workflow for customer support of emails comprising:
accessing email data associated with a customer service [email] management system (GHOCHE Par 156 – “Customer support tickets may include emails, chats, and other information that is unstructured text generated asynchronously. For example, a customer support chat UI may include a general subject field and unstructured text field for a customer to enter their question and initiate a chat with an agent.”; Par 198 – “In one implementation, the solve module 215 is designed to support handling long-form email tickets by detecting intent implementing a complete workflow to resolve a customer's query/concern.”);
using a machine learning model associated with a Large Language Model (LLM)-based labeler (GHOCHE Par 139 – “As illustrated in FIG. 11 , incoming tickets may be analyzed using the trained ML model to detect category/subcategory/priority in block 1105 and route 1110 a ticket to an agent based on the detected category/subcategory/priority.”; Par 175 – “Large language models (LLMs) can be used to aid in providing customer support. Generative AI models, such as ChatGPT may be used.”), generating a predicted customer service action for an email in the email data (GHOCHE Par 138 – “FIG. 10 is a high level flow chart of a method of training a ML classifier model to identify a category/subcategory of a customer question to perform routing of tickets to agents. In block 1005, historic ticket is ingested, which may include manually labelled category/subcategory routing information, as well as a priority level. In block 1010, a ML model is trained to identify category/subcategory of a customer question for routing purposes. This may include, for example, identifying a category/subcategory for identifying an escalation category/subcategory. For example, a customer may be complaining about a repeat problem, or that they want a refund, or that customer service is no good, etc. A ticket corresponding to an escalation risk may be routed to a human agent with training or experience in handling escalation risks. In some implementations, an escalation risk is predicted based in part on other data, such as customer survey data as previously discussed. More generally, escalation risk can be predicted using a model that prioritizes tickets based on past escalations and ticket priority, with customer survey data being still yet another source of data used to train the model.”);
communicating the predicted customer service action (GHOCHE Par 138 – “FIG. 10 is a high level flow chart of a method of training a ML classifier model to identify a category/subcategory of a customer question to perform routing of tickets to agents.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method/system of PAULI to include email communication, as taught by GHOCHE.
One of ordinary skill would have been motivated to include email communication, in order to provide more flexible and efficient interaction with a customer.
REGARDING CLAIM 11, PAULI in view of GHOCHE discloses the media of claim 10, wherein the LLM-based labeler development engine comprises an example selection machine learning model (PAULI Par 49 – “At operation 450, the AI assistant 110 applies the teacher model 132 to identify samples for further annotation.”) and a few-shot learning LLM (PAULI Par 46 –“ The assistant 110 then uses the user-annotated samples to generate soft labels for the samples to train the student model 130 (e.g., start with zero shot learning and then move into few-shot learning).”; Par 57 – “For example, when the AI assistant 110 determines to continue with automatic labeling, the AI assistant 110 uses the LLM 120 to generate soft labels 126 for each of the newly selected samples at operations 456-458 and these samples and their soft labels are subsequently used to retrain the student model 130 at operation 420.”).
REGARDING CLAIM 12, PAULI in view of GHOCHE discloses the media of claim 10, wherein the LLM-based labeler development engine comprises an error analysis engine (PAULI Par 48 – “Once initially trained, the AI assistant 110 is configured to evaluate the performance of the current build of the student model 130 at operation 430. This evaluation includes testing the current training samples with ground truth labels 138 with the student model 130 to determine an overall accuracy percentage.”) and an autonomous prompt-updating engine (PAULI Par 38 – “As part of an AI-assistance experience, the AI assistant 110 uses the LLM 120 to generate suggestions to the user 102 about how to categorize a datapoint. The assistant 110 is context-aware, as the assistant 110 creates few-shot learning prompts for LLMs 120 in real time. For example, the assistant 110 dynamically re-engineers the few-shot learning prompt. Each time a new sample is sent to the LLM 120 for generating a soft label 126 or label suggestion 122 for the user 102, the assistant 110 includes reference sentences that the student model 130 identifies as similar (e.g., based on cosine similarity between category probabilities). These prompts thus contextualize what the assistant 110 has already learned about the dataset 104 and the intent of the user 102.”; Par 62 – “At operation 520, the AI assistant generates a prompt that includes a request to categorize the sample 510 to be labeled, as well as a list of the current categories, the text of the sample 510 to be labeled, and the text for each of the nearby sampled labels 514. At decision 530, if the current iteration is an automatic iteration (e.g., at operations 456-458), then this prompt 522 may be submitted to the LLM 120 to generate a soft label 126 for this sample 510 at operation 532.”).
REGARDING CLAIM 13, PAULI in view of GHOCHE discloses the media of claim 10.
PAULI does not explicitly teach routing an email to a predefined email management service.
GHOCHE further discloses the operations further comprising:
based on the predicted customer service action, routing the email to a predefined email management service associated with the predicted customer service action (GHOCHE Fig. 11; Par 139 – “As illustrated in FIG. 11 , incoming tickets may be analyzed using the trained ML model to detect category/subcategory/priority in block 1105 and route 1110 a ticket to an agent based on the detected category/subcategory/priority. The routing may, for example, be based in part on the training and skills of human agents. For example, the category/subcategory may indicate that some agents are more capable of handling the question than other agents. For example, if there is indication of an escalation risk, the ticket may be routed to an agent with training and/or experience to handle escalation risk, such as a manager or a supervisor.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method/system of PAULI to include email communication, as taught by GHOCHE.
One of ordinary skill would have been motivated to include email communication, in order to provide more flexible and efficient interaction with a customer.
REGARDING CLAIM 14, PAULI in view of GHOCHE discloses the media of claim 10.
PAULI does not explicitly teach generating a response to a sender associated with email.
GHOCHE further discloses the operations further comprising generating an automated response to a sender associated with the email based on the predicted customer service action (GHOCHE Fig. 28B; Par 184 – “FIG. 28B is a high-level flowchart of an example of a method of empathy customization. In block 2820, there is the automatic determination of the topic/intent of a customer question. In block 2822, a template answer/workflow is automatically selected for topic that was previously automated. In block 2824, empathy customization is performed for selected template answer/workflow. This may include adding an empathic statement while retaining the substantive aspects of a template answer. In block 2826, the customer question is automatically responded to, including the empathy customization.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method/system of PAULI to include email communication, as taught by GHOCHE.
One of ordinary skill would have been motivated to include email communication, in order to provide more flexible and efficient interaction with a customer.
REGARDING CLAIM 15, PAULI discloses a computer-implemented method, the method comprising:
accessing a dataset associated with an LLM-based labeler development engine (PAULI Par 29 – “For example, an organization may wish to analyze customer churn based on a dataset 104 of text-based customer complaints, where each complaint contains one or more sentences provided by the submitting customer. However, it should be understood that other types of data and use cases are possible”; Par 40 – “In some implementations, the AI assistant 110 is configured to support multiple types of media or modalities of data (multimodal), such as a combination of audio and text (e.g., customer voice complaint calls and online text-based complaints to classify types of complaints, or joint vision-language models), or images, video, and text (e.g., professional images of people, video interviews, and their text-based biographies, to classify occupation types), or other multi-modal deep learning models.”);
using a few-shot learning LLM and few-shot learning prompts of the LLM-based labeler development engine (PAULI Par 62 – “At operation 520, the AI assistant generates a prompt that includes a request to categorize the sample 510 to be labeled, as well as a list of the current categories, the text of the sample 510 to be labeled, and the text for each of the nearby sampled labels 514.”; Par 70 – “In operation 1006, assistant 110 generates few-shot learning prompts for the LLM 120, where the learning prompts include labeled samples that a student model determines to be similar to a current training example.”), generating a set of labels for a plurality of data items in the dataset (PAULI Par 46 – “The assistant 110 then uses the user-annotated samples to generate soft labels for the samples to train the student model 130 (e.g., start with zero shot learning and then move into few-shot learning). Samples may be shown to the student model 130 to identify a set of top categories. Sentences can be selected from these top categories and provided as context to the LLM engine 120 to train the student model 130, test the student model 130 against human annotated samples, and loop repeatedly through model retraining until improvement diminishes.”; Par 47 – “Once the initial manual sample annotation is complete, the AI assistant enters a training loop. This training loop begins with training of the student model 130 at operation 420. The AI assistant 110 identifies a set of training samples to use in this current iteration of training of the student model 130. The student model 130 is exclusively trained on soft-labeled samples (soft-labeled by the LLM engine 130). Ground truth labels are only used for evaluating the student model 130. Evaluation involves exclusively ground truth labels.”);
generating an error analysis output for the set of labels for the plurality of data items in the dataset (PAULI Par 48 – “Once initially trained, the AI assistant 110 is configured to evaluate the performance of the current build of the student model 130 at operation 430. This evaluation includes testing the current training samples with ground truth labels 138 with the student model 130 to determine an overall accuracy percentage.”; Par 32 –“… evaluating the current performance of the student model 130 until improvement diminishes. This student model 130 is analyzed by the assistant 110 using pre-labeled data (e.g., a few human-labeled data samples for each category, such as the ground truth labels 138) to test how consistent the soft labels 126 are performing.”);
based on the error analysis output and the few-shot learning LLM, training an LLM-based labeler (PAULI Par 34 – “Upon concluding a round of user annotation, the AI assistant 110 may similarly perform another round of automatic training, now retraining the student model 130 with a larger set of samples with ground truth labels 138 provided by the user 102. Accordingly, the AI assistant 110 performs iterations of automatic labeling and manual labeling until a performance threshold is reached (e.g., a pre-determined correct categorization percentage) or until the user 102 is content at the current performance of the student model 130. At such time, the AI assistant 110 may perform a full index 140 of the dataset 104 using the student model 130.”; Par 32 – “The assistant 110 trains a teacher model 132 to identify samples within the student model 130 that can help improve the student model 130 with additional human annotation. The assistant 110 prompts the user 102 for label inputs 136 and uses those new label inputs 136 to improve and test 134 the student model 130. This cycle can continue for many iterations until improvement of the student model 130 has peaked.”);
using the LLM-based labeler, labeling a training dataset (PAULI Fig. 4; Par 58 – “In some implementations, the AI assistant 110 may use the current student model 130 to determine a soft label 138 for one or more of the selected samples and a confidence score for that soft label 138. If the confidence score of a particular soft label is above a predetermined threshold for that sample (e.g., if the student model 130 seems to indicate, with a degree of certainty, that the sample falls into one of the defined categories), then that soft label is automatically added to the sample at operations 452-454.”) comprising a plurality [email] data items (PAULI Par 29 – “For example, an organization may wish to analyze customer churn based on a dataset 104 of text-based customer complaints, where each complaint contains one or more sentences provided by the submitting customer. However, it should be understood that other types of data and use cases are possible”; Par 40 – “In some implementations, the AI assistant 110 is configured to support multiple types of media or modalities of data (multimodal), such as a combination of audio and text (e.g., customer voice complaint calls and online text-based complaints to classify types of complaints, or joint vision-language models), or images, video, and text (e.g., professional images of people, video interviews, and their text-based biographies, to classify occupation types), or other multi-modal deep learning models.”);
training [a machine learning model] the labeler using the labeled training dataset (PAULI Par 28 –“In architecture 100, a user 102 at a user computing device 101 interacts with an AI assistant 110 to train a student model 130 that helps give the user 102 insights they need within a dataset 104 of structured or unstructured data from their organization.”; Par 34 – “Upon concluding a round of user annotation, the AI assistant 110 may similarly perform another round of automatic training, now retraining the student model 130 with a larger set of samples with ground truth labels 138 provided by the user 102. Accordingly, the AI assistant 110 performs iterations of automatic labeling and manual labeling until a performance threshold is reached (e.g., a pre-determined correct categorization percentage) or until the user 102 is content at the current performance of the student model 130. At such time, the AI assistant 110 may perform a full index 140 of the dataset 104 using the student model 130.”; Par 32 – “The assistant 110 trains a teacher model 132 to identify samples within the student model 130 that can help improve the student model 130 with additional human annotation. The assistant 110 prompts the user 102 for label inputs 136 and uses those new label inputs 136 to improve and test 134 the student model 130. This cycle can continue for many iterations until improvement of the student model 130 has peaked.”), wherein the machine learning model is associated with a customer service email management system (PAULI Par 28 –“In architecture 100, a user 102 at a user computing device 101 interacts with an AI assistant 110 to train a student model 130 that helps give the user 102 insights they need within a dataset 104 of structured or unstructured data from their organization.”; Par 29 – “For example, an organization may wish to analyze customer churn based on a dataset 104 of text-based customer complaints, where each complaint contains one or more sentences provided by the submitting customer. However, it should be understood that other types of data and use cases are possible”; Par 40 – “… such as a combination of audio and text (e.g., customer voice complaint calls and online text-based complaints to classify types of complaints, or joint vision-language models), …”); and
deploying the machine learning model in the customer service email management system to predict customer service actions for electronic communications at the customer service email management system (PAULI Par 21 –“The example solutions have applications across various industries including, for example, support ticket routing, insurance claim risk assessment, content moderation, medical record classification, Securities Exchange Commission (SEC) compliance assessment, classification of scientific response documents, categorization of upstream data for exploration, and customer account classification.”; Par 40 – “In some implementations, the AI assistant 110 is configured to support multiple types of media or modalities of data (multimodal), such as a combination of audio and text (e.g., customer voice complaint calls and online text-based complaints to classify types of complaints, or joint vision-language models), or images, video, and text (e.g., professional images of people, video interviews, and their text-based biographies, to classify occupation types), or other multi-modal deep learning models.”; Par 68 – “Use of example solutions can have a significant impact on top-level business key performance indicators (KPIs) the users care about (e.g., quickly identifying and responding to trends in customer feedback). Further, example solutions offer a persistent presence in assisting the user to make sense of their data, allowing the state of a project can be saved and restored from memory and learning from its cooperation with the user to improve its accuracy over time. Elements of a user interface provide intuitive visualizations of the model and its understanding of the data and the users' interest in it, allowing the user to achieve state of the art accuracy with minimal effort in terms of time and upskilling.”).
PAULI does not explicitly teach the [square-bracketed] limitation and teaches the underlined feature instead. In other words, PAULI teaches training the labeler (e.g., student model) to classify customer complaints and to respond the complaints, but does not explicitly teach training another model beside the labeler for predicting customer service actions. PAULI teaches online-based text communication, but does not explicitly teach [email] communication.
GHOCHE discloses the [square-bracketed] limitations. GHOCHE discloses a method/system for automatically generating a natural language workflow for customer support of emails comprising:
using the LLM-based labeler, labeling a training dataset (GHOCHE Par 161 – “In block 1620, the ticket data that was structured data is clustered. The clustering algorithm may include a rule or an algorithm to assign a text description to the cluster. The clusters are used to label the customer support tickets and generate training data to train the classifier. This corresponds to training the classifier on weakly supervised training data.”) comprising a plurality [email] data items (GHOCHE Par 156 – “Customer support tickets may include emails, chats, and other information that is unstructured text generated asynchronously. For example, a customer support chat UI may include a general subject field and unstructured text field for a customer to enter their question and initiate a chat with an agent.”; Par 198 – “In one implementation, the solve module 215 is designed to support handling long-form email tickets by detecting intent implementing a complete workflow to resolve a customer's query/concern.”);
training [a machine learning model] using the labeled training dataset (GHOCHE Par 138 – “FIG. 10 is a high level flow chart of a method of training a ML classifier model to identify a category/subcategory of a customer question to perform routing of tickets to agents. In block 1005, historic ticket is ingested, which may include manually labelled category/subcategory routing information, as well as a priority level. In block 1010, a ML model is trained to identify category/subcategory of a customer question for routing purposes. …”), wherein the machine learning model is associated with a customer service email management system (GHOCHE Par 156 – “Customer support tickets may include emails, chats, and other information that is unstructured text generated asynchronously. For example, a customer support chat UI may include a general subject field and unstructured text field for a customer to enter their question and initiate a chat with an agent.”; Par 198 – “In one implementation, the solve module 215 is designed to support handling long-form email tickets by detecting intent implementing a complete workflow to resolve a customer's query/concern.”); and
deploying the machine learning model in the customer service email management system to predict customer service actions for electronic communications at the customer service email management system (GHOCHE Par 138 – “FIG. 10 is a high level flow chart of a method of training a ML classifier model to identify a category/subcategory of a customer question to perform routing of tickets to agents. In block 1005, historic ticket is ingested, which may include manually labelled category/subcategory routing information, as well as a priority level. In block 1010, a ML model is trained to identify category/subcategory of a customer question for routing purposes. This may include, for example, identifying a category/subcategory for identifying an escalation category/subcategory. For example, a customer may be complaining about a repeat problem, or that they want a refund, or that customer service is no good, etc. A ticket corresponding to an escalation risk may be routed to a human agent with training or experience in handling escalation risks. In some implementations, an escalation risk is predicted based in part on other data, such as customer survey data as previously discussed. More generally, escalation risk can be predicted using a model that prioritizes tickets based on past escalations and ticket priority, with customer survey data being still yet another source of data used to train the model.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method/system of PAULI to include training a model for customer service actions prediction, as taught by GHOCHE.
One of ordinary skill would have been motivated to include training a model for customer service actions prediction, in order to accurately identify intent of customer emails and to generate workflows to resolve tickets (GHOCHE Par 213).
REGARDING CLAIM 16, PAULI in view of GHOCHE discloses the computer-implemented method of claim 15.
PAULI further discloses wherein the dataset is few-shot learning data identified using an example selection machine learning model (PAULI Par 49 – “At operation 440, the AI assistant 110 trains a teacher model 132 that is configured to identify samples from the dataset 104 that, if annotated (either through soft-labeling by the LLM 120 or manual labeling by the user 102), are likely to improve the student model 130.”), wherein the example selection machine learning model selects the few-shot learning data (PAULI Par 47 – “Once the initial manual sample annotation is complete, the AI assistant enters a training loop. This training loop begins with training of the student model 130 at operation 420. The AI assistant 110 identifies a set of training samples to use in this current iteration of training of the student model 130. The student model 130 is exclusively trained on soft-labeled samples (soft-labeled by the LLM engine 130). Ground truth labels are only used for evaluating the student model 130. Evaluation involves exclusively ground truth labels.”) based on both an active learning technique (PAULI Par 61 – “Example solutions take advantage of active learning. Active learning approaches aim to identify those data points that are most critical for training a model to understand and categorize a dataset. Here, active learning is used for at least two purposes, namely for selecting those samples that require feedback from the domain expert, and to select samples to be annotated by the LLM 120, to further reduce computing resource usage, training time, and cost. Several sampling strategies are implemented, and the strategy is dynamically selected which is most likely to be successful, given characteristics of the dataset and what has already been learned about it.”) and a clustering technique (PAULI Par 46 – “In some implementations, the assistant 110 performs cluster analysis of the embeddings 124 and, for each cluster, may sample a few points to show the user 102. This approach of clustering at the early stage, rather than letting the teacher model 132 choose, is because there is not enough data yet to train the teacher model 132. For example, the assistant 110 identifies 25 clusters and, from within each cluster, selects a centered sample, one or more fringe or outlier samples (e.g., samples within the cluster but somewhat distant from the center), and a few random samples within the cluster region. These cluster selections can be shown to the user 102 to create initial annotations (e.g., two samples per category).”), wherein the few-shot learning data is identified to support a [hierarchical] categorization of data items in the few-shot learning data (PAULI Par 38 – “As part of an AI-assistance experience, the AI assistant 110 uses the LLM 120 to generate suggestions to the user 102 about how to categorize a datapoint. The assistant 110 is context-aware, as the assistant 110 creates few-shot learning prompts for LLMs 120 in real time. For example, the assistant 110 dynamically re-engineers the few-shot learning prompt.”).
PAUL does not explicitly teach the [square-bracketed] limitation.
GHOCHE discloses the [square-bracketed] limitation. GHOCHE discloses a method/system for automatically generating a natural language workflow for customer support of emails wherein the training data is identified to support a [hierarchical] categorization of data items (GHOCHE Par 98 – “A ticket covers the entire lifecycle of an issue. A dataset of historic tickets would conventionally be manually labelled for routing to agents. For example, a ticket might include fields for category and subcategory. It may also include fields identifying the queue the ticket was sent to. In some cases the agent who answered the ticket may be included. The priority level associated with the ticket may also be included.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method/system of PAULI to include supporting a hierarchical categorization of data, as taught by GHOCHE.
One of ordinary skill would have been motivated to include supporting a hierarchical categorization of data, in order to accurately categorize incoming emails and text tickets for customer service (Par 99).
REGARDING CLAIM 17, PAULI in view of GHOCHE discloses the computer-implemented method of claim 15, wherein the few-shot learning LLM supports a plurality of few-shot learning prompts comprising hierarchical labeling prompts that are designed to guide the few-shot learning LLM to assign labels to data items (PAULI Par 62 – “At operation 520, the AI assistant generates a prompt that includes a request to categorize the sample 510 to be labeled, as well as a list of the current categories, the text of the sample 510 to be labeled, and the text for each of the nearby sampled labels 514.”; Par 70 – “In operation 1006, assistant 110 generates few-shot learning prompts for the LLM 120, where the learning prompts include labeled samples that a student model determines to be similar to a current training example.”) based on a predefined [hierarchical] structure (PAULI Par 31 – “The user 102 is prompted for label inputs 136 for a subset of samples, thus identifying an initial set of ground truth labels 138 for some of the samples. These ground truth labels 138 also identify a set of categories of interest to the user 102 which form the foundation of training for the student model 130.”), wherein the plurality of few-shot learning prompts are associated with prompt templates that include the following (PAULI Par 62 – “At operation 520, the AI assistant generates a prompt that includes a request to categorize the sample 510 to be labeled, as well as a list of the current categories, the text of the sample 510 to be labeled, and the text for each of the nearby sampled labels 514.”; Par 70 – “generates few-shot learning prompts for the LLM 120, where the learning prompts include labeled samples that a student model determines to be similar to a current training example.”): an objective (PAULI Fig. 5 – “Please categorize the following samples”), a labeling structure (PAULI Fig. 5 – “into either of (labeled sample categories)”), an example format, few-shot examples (PAULI Fig. 5 – “list of samples w/ labels”), and a prompt for new data (PAULI Fig. 5 – “samples w/o label”).
PAUL does not explicitly teach the [square-bracketed] limitation.
GHOCHE discloses the [square-bracketed] limitation. GHOCHE discloses a method/system for automatically generating a natural language workflow for customer support of emails comprising hierarchical training data such that a model is trained to assign labels to data items based on a predefined [hierarchical] structure (GHOCHE Par 98 – “A ticket covers the entire lifecycle of an issue. A dataset of historic tickets would conventionally be manually labelled for routing to agents. For example, a ticket might include fields for category and subcategory. It may also include fields identifying the queue the ticket was sent to. In some cases the agent who answered the ticket may be included. The priority level associated with the ticket may also be included.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method/system of PAULI to include supporting a hierarchical categorization of data, as taught by GHOCHE.
One of ordinary skill would have been motivated to include supporting a hierarchical categorization of data, in order to accurately categorize incoming emails and text tickets for customer service (Par 99).
REGARDING CLAIM 18, PAULI in view of GHOCHE discloses the computer-implemented method of claim 15, wherein the updated few-shot learning prompts are accessed via a prompt engineering LLM that supports using a previous error analysis output to automatically update few-shot learning prompts for prompt-based learning using the updated few-shot learning prompts (PAULI Par 76 – “dynamically alter a few-shot learning prompt for the LLM, including labeled samples that a student model determines to be similar to a current training sample;”; Par 38 – “As part of an AI-assistance experience, the AI assistant 110 uses the LLM 120 to generate suggestions to the user 102 about how to categorize a datapoint. The assistant 110 is context-aware, as the assistant 110 creates few-shot learning prompts for LLMs 120 in real time. For example, the assistant 110 dynamically re-engineers the few-shot learning prompt. Each time a new sample is sent to the LLM 120 for generating a soft label 126 or label suggestion 122 for the user 102, the assistant 110 includes reference sentences that the student model 130 identifies as similar (e.g., based on cosine similarity between category probabilities). These prompts thus contextualize what the assistant 110 has already learned about the dataset 104 and the intent of the user 102.”; Par 62 – “At operation 520, the AI assistant generates a prompt that includes a request to categorize the sample 510 to be labeled, as well as a list of the current categories, the text of the sample 510 to be labeled, and the text for each of the nearby sampled labels 514. At decision 530, if the current iteration is an automatic iteration (e.g., at operations 456-458), then this prompt 522 may be submitted to the LLM 120 to generate a soft label 126 for this sample 510 at operation 532.”).
REGARDING CLAIM 19, PAULI in view of GHOCHE discloses the computer-implemented method of claim 15, wherein the few-shot learning prompts are updated few-shot learning prompts that are accessed via a prompt engineering LLM that supports using a previous error analysis output to automatically update few-shot learning prompts for prompt-based learning using the updated few-shot learning prompts (PAULI Par 76 – “dynamically alter a few-shot learning prompt for the LLM, including labeled samples that a student model determines to be similar to a current training sample;”; Par 38 – “As part of an AI-assistance experience, the AI assistant 110 uses the LLM 120 to generate suggestions to the user 102 about how to categorize a datapoint. The assistant 110 is context-aware, as the assistant 110 creates few-shot learning prompts for LLMs 120 in real time. For example, the assistant 110 dynamically re-engineers the few-shot learning prompt. Each time a new sample is sent to the LLM 120 for generating a soft label 126 or label suggestion 122 for the user 102, the assistant 110 includes reference sentences that the student model 130 identifies as similar (e.g., based on cosine similarity between category probabilities). These prompts thus contextualize what the assistant 110 has already learned about the dataset 104 and the intent of the user 102.”; Par 62 – “At operation 520, the AI assistant generates a prompt that includes a request to categorize the sample 510 to be labeled, as well as a list of the current categories, the text of the sample 510 to be labeled, and the text for each of the nearby sampled labels 514. At decision 530, if the current iteration is an automatic iteration (e.g., at operations 456-458), then this prompt 522 may be submitted to the LLM 120 to generate a soft label 126 for this sample 510 at operation 532.”).
REGARDING CLAIM 20, PAULI in view of GHOCHE discloses the computer-implemented method of claim 15, wherein the updated few-shot learning prompts are accessed via a prompt engineering LLM that supports using a previous error analysis output to automatically update few-shot learning prompts for prompt-based learning using the updated few-shot learning prompts (PAULI Par 76 – “dynamically alter a few-shot learning prompt for the LLM, including labeled samples that a student model determines to be similar to a current training sample;”; Par 38 – “As part of an AI-assistance experience, the AI assistant 110 uses the LLM 120 to generate suggestions to the user 102 about how to categorize a datapoint. The assistant 110 is context-aware, as the assistant 110 creates few-shot learning prompts for LLMs 120 in real time. For example, the assistant 110 dynamically re-engineers the few-shot learning prompt. Each time a new sample is sent to the LLM 120 for generating a soft label 126 or label suggestion 122 for the user 102, the assistant 110 includes reference sentences that the student model 130 identifies as similar (e.g., based on cosine similarity between category probabilities). These prompts thus contextualize what the assistant 110 has already learned about the dataset 104 and the intent of the user 102.”; Par 62 – “At operation 520, the AI assistant generates a prompt that includes a request to categorize the sample 510 to be labeled, as well as a list of the current categories, the text of the sample 510 to be labeled, and the text for each of the nearby sampled labels 514. At decision 530, if the current iteration is an automatic iteration (e.g., at operations 456-458), then this prompt 522 may be submitted to the LLM 120 to generate a soft label 126 for this sample 510 at operation 532.”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONATHAN C KIM whose telephone number is (571)272-3327. The examiner can normally be reached Monday to Friday 8:00 AM thru 4:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew C Flanders can be reached at 571-272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JONATHAN C KIM/Primary Examiner, Art Unit 2655
Prosecution Insights