DETAILED ACTION
This action is responsive to communications filed on December 29, 2023. This action is made Non-Final.
Claims 1-20 are pending in the case.
Claims 1, 9, and 17 are independent claims.
Claims 1-5, 8-13, and 16-20 are rejected.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of papers submitted under 35 U.S.C. 119(a)-(d), which papers have been placed of record in the file.
Information Disclosure Statement
The information disclosure statement (IDS(s)) submitted on 12/29/2023 is/are in compliance with the provisions of 37 C.F.R. 1.97. Accordingly, the IDS(s) is/are being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 4, 5, 8-10, 12, 13, 16-18, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pushkin et al., US Patent 11,734,937 (“Pushkin”), in view of Wu, US Publication 2023/0135659 (“Wu”), and further in view of Tsou et al., US Publication 2024/0143643 (“Tsou”).
Claim 1:
Pushkin teaches or suggests a processor implemented method to classify ... into categories using an ensemble of machine learning models, the method comprising:
training, via one or more hardware processor a first machine learning model from a set of machine learning models using a training dataset to learn text representations and classify the learned representations into corresponding category from a set of categories (see Fig. 1-8; col. 5, lines 20-26 - into one of the following categories or classes: account questions, ticket refunds, and flight complaints. Given an example text snippet (e.g., from a chat or phone transcript) of"Hi, I wanted to report a problem on my last flight", the custom classifier model 112 may identify that this text has a label of "flight complaint"; col., 15, lines 53-67 - uses higher-level feature extraction to produce input features to train a model 226 for classification. utilizes a semantic (e.g., feature extraction) 60 model 670 to produce enhanced dataset semantic relations present in the input documents as additional features to be used for model training; col. 12, line 33 – trained models 568-1 to 568-N may be used at inference.);
finetuning via the one or more hardware processors using the training dataset to train a second machine learning model to classify at least one unlabeled ... of unknown category ..., causing the second trained machine learning model to be agnostic to the first trained machine learning model (see Fig. 1-8; col. 1, lines 63-66 - the finetuned language ML model may be trained on labeled documents of the user, for prediction objectives for unlabeled data; col. 8, lines 6-67 - to train one or more ML models, for example, to train a ML model as a text classifier. evaluate the performance of that model (e.g., at generating a label). utilization of this unlabeled (e.g. , original raw) data to boost accuracy of the generated model ... without investing into at times costly annotation process; col. 9, lines 7-10 - pretrain the language machine learning model on prediction objectives for unlabeled data, and outputs this pretrained model as a custom language model 232; col. 16, lines 31-43 - receiving an inference request at the endpoint for an unlabeled document of the user 710; generating, by the language machine learning model, an inference based on the inference request for the unlabeled document of the user 712.); and
generating via the one or more hardware processors an ensemble of machine learning models by using the first machine learning model and the second machine learning model to classify a set of test ... received as input request to corresponding category (see Fig. 1-8; col. 2, lines 13-16 – selecting an ensemble of text classifier language ML models to cumulatively perform an inference; col. 5, lines 20-26 - into one of the following categories or classes: account questions, ticket refunds, and flight complaints. Given an example text snippet (e.g., from a chat or phone transcript) of"Hi, I wanted to report a problem on my last flight", the custom classifier model 112 may identify that this text has a label of "flight complaint"; col. 7, lines 29-62 - receive inference requests from client applications 140A and/or 140B at circle (7), provide the inference requests 160A to the associated hosted model(s) 136, and provide inference results 160B (e.g., predicted classes, predicted entities, etc.) back to applications. selecting an ensemble of text classifier language ML models to cumulatively perform an inference; col. 8, lines 6-46 – to train one or more ML models, for example, to train a ML model as a text classifier; col. 12, lines 33-41 - At inference time, each of the trained models 568-1 to 568-N may be used at inference. Trains multiple different models (e.g., algorithms), it is beneficial to select a subset of best representatives of those (e.g., competing for the highest accuracy) for the inclusion into the ensemble of models used at inference time. Having multiple models available at inference time reduces variance of predictions and generalization error in certain embodiments.).
Pushkin does not explicitly disclose news snippets; based on a premise-hypothesis pair.
Wu teaches or suggests news snippets (see para. 0061 - infer from text 112 ( e.g., input data in text form, textual data, text of news articles, text of financial news articles, text of economic statistics, text from corporate quarterly reports, transcriptions of quarterly earnings calls, text from Wikipedia® entries, text of questions, text of answers), information about entities (e.g., objects, events, situations, concepts, people, places). Time inclusive training data 104 is a set of data (e.g., a training database for machine learning, a body of text from financial news articles, text from publications, historical stock price data) as well as optional labels or classifications in order to provide a set of examples; para. 0062 - time-inclusive training data 104 comprises a set of text ( e.g., words, sentences, paragraphs); para. 0076 - training architecture 200 accepts as input data different types of segments (e.g., individual words, individual sentences, a collection of sentences, partial paragraphs, whole paragraphs, descriptions of events, personal names, location names, dates, times); para. 0080 - word-level pre-training tasks 218 comprise phrase level pre training tasks 218, which may be referred to as training tasks; para. 0082 - segments 202, 204 containing time and date data are recognized from text ( e.g., a time, a date, a sentence, sentences, a paragraph, paragraphs, a collection of news articles, a body of text); para. 0083 - pre-trains neural networks to infer that a capitalized phrase.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Pushkin, to include news snippets for the purpose of efficiently training models to make event related inferences by using portions of news articles, improving model performance, as taught by Wu (0061 and 0076).
Tsou teaches or suggests based on a premise-hypothesis pair (see para. 0060 - goal of zero-shot topic classification is to make use of a pre-trained model within any additional fine-tuning being needed on a task-specific corpus. In some embodiments, natural language inference ("NLI") is employed to accomplish this, using a zero-shot framework. NLI, sometimes called "text entailment", involves the task of determining whether a "hypothesis" is true (i.e., entailment), false (i.e., contradiction), or undetermined (i.e., neutral), given a "premise". The (premise, hypothesis) pairs may be fed into a cross-encoder configured to learn how to predict one of the three labels. In some embodiments, a simple method for performing this zero-shot text classification is to feed in the sequence to be classified as a "premise", along with the "hypothesis" that the text is about the topic. For example, if the utterance is, "how much would it be for both of those products?" and we want to know if this utterance is relating to the topic "pricing", then the system can feed in the premise-hypothesis pair:; para. 0061 - premise="How much would it be for both of those products?; para. 0062 - hypothesis="This text is about pricing."; para. 0063 – he hypothesis used by the model may be, e.g., "they are talking about $TOPIC', where $TOPIC is a key phrase obtained during the preprocessing step described above; para. 0064 - determines whether the text sequence is about the topic.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Pushkin, to include based on a premise-hypothesis pair for the purpose of efficiently performing text entailment based on key phrases, improving model performance for determining the nature of text sequences, as taught by Tsou (0060-0064).
Claim(s) 9 and 17:
Claim(s) 9 and 17 correspond to Claim 1, and thus, Pushkin, Wu, and Tsou teach or suggest the limitations of claim(s) 9 and 17 as well.
Claim 2:
Wu further teaches or suggests wherein the training dataset includes a set of training news snippets, and each of the training news snippets corresponds to a category from the set of categories (see para. 0061 - infer from text 112 ( e.g., input data in text form, textual data, text of news articles, text of financial news articles, text of economic statistics, text from corporate quarterly reports, transcriptions of quarterly earnings calls, text from Wikipedia® entries, text of questions, text of answers), information about entities (e.g., objects, events, situations, concepts, people, places). Time inclusive training data 104 is a set of data (e.g., a training database for machine learning, a body of text from financial news articles, text from publications, historical stock price data) as well as optional labels or classifications in order to provide a set of examples; para. 0062 - time-inclusive training data 104 comprises a set of text ( e.g., words, sentences, paragraphs); para. 0076 - training architecture 200 accepts as input data different types of segments (e.g., individual words, individual sentences, a collection of sentences, partial paragraphs, whole paragraphs, descriptions of events, personal names, location names, dates, times); para. 0080 - word-level pre-training tasks 218 comprise phrase level pre training tasks 218, which may be referred to as training tasks; para. 0082 - segments 202, 204 containing time and date data are recognized from text ( e.g., a time, a date, a sentence, sentences, a paragraph, paragraphs, a collection of news articles, a body of text); para. 0083 - pre-trains neural networks to infer that a capitalized phrase.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Pushkin, to include wherein the training dataset includes a set of training news snippets, and each of the training news snippets corresponds to a category from the set of categories for the purpose of efficiently training models to make event related inferences by using portions of news articles, improving model performance, as taught by Wu (0061 and 0076).
Claim(s) 10 and 18:
Claim(s) 10 and 18 correspond to Claim 2, and thus, Pushkin, Wu, and Tsou teach or suggest the limitations of claim(s) 10 and 18 as well.
Claim 4:
Tsou further teaches or suggests wherein the second machine learning model includes a pre-trained language model (PLM) based zero-shot natural language inference (NLI) (see para. 0060 - goal of zero-shot topic classification is to make use of a pre-trained model within any additional fine-tuning being needed on a task-specific corpus. In some embodiments, natural language inference ("NLI") is employed to accomplish this, using a zero-shot framework. NLI, sometimes called "text entailment", involves the task of determining whether a "hypothesis" is true (i.e., entailment), false (i.e., contradiction), or undetermined (i.e., neutral), given a "premise". The (premise, hypothesis) pairs may be fed into a cross-encoder configured to learn how to predict one of the three labels. In some embodiments, a simple method for performing this zero-shot text classification is to feed in the sequence to be classified as a "premise", along with the "hypothesis" that the text is about the topic. For example, if the utterance is, "how much would it be for both of those products?" and we want to know if this utterance is relating to the topic "pricing", then the system can feed in the premise-hypothesis pair:; para. 0061 - premise="How much would it be for both of those products?; para. 0062 - hypothesis="This text is about pricing."; para. 0063 – he hypothesis used by the model may be, e.g., "they are talking about $TOPIC', where $TOPIC is a key phrase obtained during the preprocessing step described above; para. 0064 - determines whether the text sequence is about the topic. applies a zero-shot topic classifier to every (sliding window text, topic) pair.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Pushkin, to include wherein the second machine learning model includes a pre-trained language model (PLM) based zero-shot natural language inference (NLI) for the purpose of efficiently performing text entailment based on key phrases, improving model performance for determining the nature of text sequences, as taught by Tsou (0060-0064).
Claim(s) 12 and 20:
Claim(s) 12 and 20 correspond to Claim 2, and thus, Pushkin, Wu, and Tsou teach or suggest the limitations of claim(s) 12 and 20 as well.
Claim 5:
Wu further teaches or suggests of news snippet (see para. 0061 - infer from text 112 ( e.g., input data in text form, textual data, text of news articles, text of financial news articles, text of economic statistics, text from corporate quarterly reports, transcriptions of quarterly earnings calls, text from Wikipedia® entries, text of questions, text of answers), information about entities (e.g., objects, events, situations, concepts, people, places). Time inclusive training data 104 is a set of data (e.g., a training database for machine learning, a body of text from financial news articles, text from publications, historical stock price data) as well as optional labels or classifications in order to provide a set of examples; para. 0062 - time-inclusive training data 104 comprises a set of text ( e.g., words, sentences, paragraphs); para. 0076 - training architecture 200 accepts as input data different types of segments (e.g., individual words, individual sentences, a collection of sentences, partial paragraphs, whole paragraphs, descriptions of events, personal names, location names, dates, times); para. 0080 - word-level pre-training tasks 218 comprise phrase level pre training tasks 218, which may be referred to as training tasks; para. 0082 - segments 202, 204 containing time and date data are recognized from text ( e.g., a time, a date, a sentence, sentences, a paragraph, paragraphs, a collection of news articles, a body of text); para. 0083 - pre-trains neural networks to infer that a capitalized phrase.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Pushkin, to include of news snippet for the purpose of efficiently training models to make event related inferences by using portions of news articles, improving model performance, as taught by Wu (0061 and 0076).
Tsou further teaches or suggests wherein the premise-hypothesis pair consists of a hypothesis template of the zero shot natural language inference (NLI) and one or more phrases ... indicative of category (see para. 0060 - goal of zero-shot topic classification is to make use of a pre-trained model within any additional fine-tuning being needed on a task-specific corpus. In some embodiments, natural language inference ("NLI") is employed to accomplish this, using a zero-shot framework. NLI, sometimes called "text entailment", involves the task of determining whether a "hypothesis" is true (i.e., entailment), false (i.e., contradiction), or undetermined (i.e., neutral), given a "premise". The (premise, hypothesis) pairs may be fed into a cross-encoder configured to learn how to predict one of the three labels. In some embodiments, a simple method for performing this zero-shot text classification is to feed in the sequence to be classified as a "premise", along with the "hypothesis" that the text is about the topic. For example, if the utterance is, "how much would it be for both of those products?" and we want to know if this utterance is relating to the topic "pricing", then the system can feed in the premise-hypothesis pair:; para. 0061 - premise="How much would it be for both of those products?; para. 0062 - hypothesis="This text is about pricing."; para. 0063 – he hypothesis used by the model may be, e.g., "they are talking about $TOPIC', where $TOPIC is a key phrase obtained during the preprocessing step described above; para. 0064 - determines whether the text sequence is about the topic. applies a zero-shot topic classifier to every (sliding window text, topic) pair.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Pushkin, to include wherein the premise-hypothesis pair consists of a hypothesis template of the zero shot natural language inference (NLI) and one or more phrases ... indicative of category for the purpose of efficiently performing text entailment based on key phrases, improving model performance for determining the nature of text sequences, as taught by Tsou (0060-0064).
Claim(s) 13:
Claim(s) 13 correspond to Claim 5, and thus, Pushkin, Wu, and Tsou teach or suggest the limitations of claim(s) 13 as well.
Claim 8:
Pushkin further teaches or suggests providing an input request comprising a set of test ... to be classified into corresponding category to the ensemble of machine learning models that includes the first machine learning model and the second machine learning model; and receiving from the ensemble of machine learning models, an ensemble result that classifies the set of test ... corresponding to the category from the set of categories (see Fig. 1-8; col. 2, lines 13-16 – selecting an ensemble of text classifier language ML models to cumulatively perform an inference; col. 5, lines 20-26 - into one of the following categories or classes: account questions, ticket refunds, and flight complaints. Given an example text snippet (e.g., from a chat or phone transcript) of"Hi, I wanted to report a problem on my last flight", the custom classifier model 112 may identify that this text has a label of "flight complaint"; col. 7, lines 29-62 - receive inference requests from client applications 140A and/or 140B at circle (7), provide the inference requests 160A to the associated hosted model(s) 136, and provide inference results 160B (e.g., predicted classes, predicted entities, etc.) back to applications. selecting an ensemble of text classifier language ML models to cumulatively perform an inference; col. 8, lines 6-46 – to train one or more ML models, for example, to train a ML model as a text classifier. Evaluate the performance of that model (e.g., at generating a label). split generator 202 splits the plurality of labeled documents 122 into a first proper subset as a training dataset 204, a second proper subset as an evaluation dataset 206, and a third proper subset as a testing dataset 208. One example of a split is 80%, 10%, and 10% for training dataset 204, evaluation dataset 206, and testing 45 dataset 208, respectively, although any other split may be utilized; col. 12, lines 33-41 - At inference time, each of the trained models 568-1 to 568-N may be used at inference. Trains multiple different models (e.g., algorithms), it is beneficial to select a subset of best representatives of those (e.g., competing for the highest accuracy) for the inclusion into the ensemble of models used at inference time. Having multiple models available at inference time reduces variance of predictions and generalization error in certain embodiments.).
Wu further teaches or suggests news snippets; news snippets based on one or more phrases (see para. 0061 - infer from text 112 ( e.g., input data in text form, textual data, text of news articles, text of financial news articles, text of economic statistics, text from corporate quarterly reports, transcriptions of quarterly earnings calls, text from Wikipedia® entries, text of questions, text of answers), information about entities (e.g., objects, events, situations, concepts, people, places). Time inclusive training data 104 is a set of data (e.g., a training database for machine learning, a body of text from financial news articles, text from publications, historical stock price data) as well as optional labels or classifications in order to provide a set of examples; para. 0062 - time-inclusive training data 104 comprises a set of text ( e.g., words, sentences, paragraphs); para. 0076 - training architecture 200 accepts as input data different types of segments (e.g., individual words, individual sentences, a collection of sentences, partial paragraphs, whole paragraphs, descriptions of events, personal names, location names, dates, times); para. 0080 - word-level pre-training tasks 218 comprise phrase level pre training tasks 218, which may be referred to as training tasks; para. 0082 - segments 202, 204 containing time and date data are recognized from text ( e.g., a time, a date, a sentence, sentences, a paragraph, paragraphs, a collection of news articles, a body of text); para. 0083 - pre-trains neural networks to infer that a capitalized phrase.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Pushkin, to include news snippets; news snippets based on one or more phrases for the purpose of efficiently training models to make event related inferences by using portions of news articles, improving model performance, as taught by Wu (0061 and 0076).
Claim(s) 16:
Claim(s) 16 correspond to Claim 8, and thus, Pushkin, Wu, and Tsou teach or suggest the limitations of claim(s) 16 as well.
Claim(s) 3, 11, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pushkin, in view of Wu, in view of Tsou, and further in view of Ghosh et al., US Publication 2022/0067076 (“Ghosh”).
Claim 3:
Ghosh further teaches or suggests wherein the first machine learning model includes bidirectional long short memory (BiLSTM) based supervised classification (see Abstract – obtains an embedded representation for an event structure of a user query and testimony sentences identified from prior court cases using a trained Bi-LSTM classifier; para. 0004 - methods do not address interpretability of results and the performance and precision of retrieving prior court cases for these methods are less; para. 0005 – technological improvements as solutions to one or more of the above-mentioned technical problems; para. 0032 - provided to a trained Bi-directional long short term memory (Bi-LSTM) based sentence classifier to identify the one or more testimony sentences; para. 0043 – precision of supervised Bi-LSTM classifier is verified manually by using 200 random sentences out of these 10000 and the precision turned out to be 75%; para. 0056 - stressing the importance of additional testimony sentences identified by the Bi-LSTM classifier. In the disclosed method, semantic roles are used that capture an event expressed in a query. For example, in the query ql (in Table 3 and Table 4), the predicate-arguments are: Predicate: set, AO: husband, Al: wife which semantically captures an event and matches it with a prior court case where a similar event has occurred baseline methods are unable to capture such nuanced semantic representations of the underlying events in a query.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Pushkin, to include wherein the first machine learning model includes bidirectional long short memory (BiLSTM) based supervised classification for the purpose of efficiently classifying text segments using semantic feature, improving model precision , as taught by Ghosh (0004, 0005, 0043, and 0056).
Claim(s) 11 and 19:
Claim(s) 11 and 19 correspond to Claim 3, and thus, Pushkin, Wu, and Tsou teach or suggest the limitations of claim(s) 11 and 19 as well.
Allowable Subject Matter
Claims 6, 7, 14, and 15 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Andrew T McIntosh whose telephone number is (571)270-7790. The examiner can normally be reached M-Th 8:00am-5:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tamara Kyle can be reached at 571-272-4241. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW T MCINTOSH/Primary Examiner, Art Unit 2144
Prosecution Insights