Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This office action is in response to correspondence 03/12/26 regarding application 18/636,297, in which Applicant amended claims 1, 2, 5, 7, and 11, cancelled claims 3, 4, 8-10, and 12-20, and added new claims 21-29. Claims 1, 2, 5-7, 11, and 21-29 are pending and have been considered.
Response to Arguments
The examiner agrees with Applicant on page 6 that no new matter was added by the amendments to claims 1, 2, 5, 7, and 11 and addition of new claims 21-29.
Applicant’s arguments on pages 6-8 regarding the 35 U.S.C. 103 rejections based on Wang, Zhang, Peng, Wu, and Verma, as well as new claims 21-29 have been considered but are moot in view of the new grounds for rejection based in part on the newly discovered reference to Li et al. (“Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations”. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10443–10461), which describes generating a synthetic dataset of textual samples using an LLM which is split in to a training set and a test set, which are respectively used to train and evaluate the performance of a various text classifier models. The new grounds for rejection based in part on Li are necessitated by Applicant’s amendments.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 5, 11, and 21-26 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (US 20230196105) in view of Li et al. (“Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations”. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10443–10461).
Consider claim 1, Wang discloses a computerized method (method performed on a computer, Abstract, [0098]) comprising:
(a) obtaining a dataset of non-labeled text-items (unlabeled text inputs 140, Fig. 1, [0020], [0022]);
(b) defining a text classification task (text classification task can be e.g. topic classification, sentiment classification, etc., [0019]);
(c) defining a prompt that commands a Large Language Model (LLM) to generate output that fulfills said text classification task (providing a prompt to the pretrained language model, [0008], which is a large language model, [0034], as an input sequence prompt to neural network 110, [0031], which commands it to generate a labeled example as output, [0059], [0061]-[0064]);
(d) automatically feeding into said LLM said prompt and text-items from the dataset of non-labeled text-items, and automatically generating by said LLM a dataset of LLM-labeled text-items (selecting one or more unlabeled text inputs as context for the task, [0057], and generating a prompt as an input sequence to the pre-trained language, which generates a set of auto-labeled training examples, [0057-0064], [0031], Fig. 1);
(e) automatically training a Machine Learning (ML) model on said dataset of LLM-labeled text-items, and generating a trained ML model (auto-labeled training examples are used to train Task Neural Network 170, [0038]).
Wang does not specifically mention wherein the dataset of LLM-labeled text-items is divided into a training set and a test set; wherein the training set is used to train the ML model and the test set is used to evaluate performance of the trained ML model; (f) deploying the trained ML model in an application that performs said text classification task on newly-received non-labeled text-items.
Li discloses the dataset of LLM-labeled text-items is divided into a training set and a test set (3000 synthetic data points were generated for each candidate label under zero shot settings, and we randomly divided the dataset into training (70%) and test (25%) sets, Section 4.3, page 10446); wherein the training set is used to train the ML model and the test set is used to evaluate performance of the trained ML model (the classification model itself was fine-tuned, and performance was evaluated by comparing the model’s predictions with the gold labels provided in the test sets, Section 4.3, page 10446); deploying the trained ML model in an application that performs said text classification task on newly-received non-labeled text-items (we trained a BERT model using the zero-shot synthetic data and computed its accuracy on the subset of task instances in the evaluation dataset whose instance-level annotation agreement exceeds a threshold, Section 5.2, page 10449; these are “newly received” and provided to the classifier model without labels).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wang such that the dataset of LLM-labeled text-items is divided into a training set and a test set; wherein the training set is used to train the ML model and the test set is used to evaluate performance of the trained ML model; and by deploying the trained ML model in an application that performs said text classification task on newly-received non-labeled text-items in order to skip the costly, time-consuming, and complex process for training data collection and curation for new task domains and categories, as suggested by Li (Section 1, page 10443), with predictably applications in detecting biased or toxic language on online platforms and filtering spam emails, as suggested by Li (Section 1, page 10443). The references cited are analogous art in the same field of natural language processing.
Consider claim 11, Wang in view of Li discloses a system comprising:
one or more hardware processors that are configured to execute code (microprocessors that execute computer program, Wang, [0093]), and that are operably associated with one or more memory units that are configured to execute code (random access memory, Wang, [0093]); wherein the one or more hardware processors are configured to perform the method of claim 1 (see claim 1 above).
Consider claim 21, Wang in view of Li discloses a non-transitory machine-readable medium containing machine-executable instructions designed to cause at least one processor to implement the method (random access memory contains computer program executed by microprocessors, Wang, [0093]) according to claim 1 (see claim 1 above).
Consider claim 2, Wang discloses the dataset of non-labeled text-items comprises only textual data-items (unlabeled text, e.g. from the Internet, [0036]); wherein the ML model is trained on a training dataset labeled text-items generated by the LLM based on the dataset based on the dataset of non-labeled text-items (auto-labeled training examples generated using the non-labeled text input as context are used to train Task Neural Network 170, [0038]).
Wang does not specifically mention a dataset of only non-synthetic textual data-items.
Li discloses a dataset of only non-synthetic textual data-items (real world training data provided by original dataset, Section 4.3, 4.1, page 10445-10446).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wang such that the dataset of non-labeled text-items comprises only textual data-items for reasons similar to those for claim 1.
Consider claim 5, Wang discloses: fine-tuning the LLM to particularly specialize in performing said text classification task, to improve accuracy of the training dataset that the LLM generates for training the first ML model (the system uses the auto-labeled training examples to fine-tune the task neural network to perform the text classification task, [0053], increasing accuracy of the training dataset by removing noisy examples, [0080]-[0081]).
Consider claim 22, Wang discloses the prompt is identical for all of the non-labeled text-items (the last context restaurant review in the input sequence 410 is followed by a “Content” tag that prompts the neural network 110 to start generating a new restaurant review, [0070]).
Consider claim 23, Wang discloses performing post-processing of classification results of the first trained ML model based on one or more predefined rules or threshold values to adjust one or more classifications (the system determines whether a highest probability in the probability distribution exceeds a threshold, [0079], which is a confidence level, [0084], and if so, the task neural network trains using the example, which adjusts accuracy of future classifications by making them more accurate, [0084]).
Consider claim 24, Wang discloses periodically updating the first ML model using further non-labeled text-items processed by the LLM to generate a further dataset of labeled text-items (periodically updating parameters of the task neural through batches of further training examples, which are auto-labeled samples generated using the non-labeled text-items as context, [0077]).
Consider claim 25, Wang discloses periodically assigning a respective confidence score to a respective text-item classified by the first ML model, wherein the respective confidence score represents a probability of the classification being correct (a highest probability in the distribution for each sample, which is a confidence level, i.e. probability the prediction accurately matches the label, [0084]).
Consider claim 26, Wang discloses training a second ML model to perform a different classification task from a classification task performed by the first ML model (e.g. a textual entailment task, [0073], a task neural network used for each task, [0048]).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (US 20230196105) in view of Li et al. (“Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations”. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10443–10461) in further view of Wu et al. (“A Model Ensemble Approach with LLM for Chinese Text Classification”. In: Xu, H., et al. Health Information Processing. Evaluation Track Papers. CHIP 2023. Communications in Computer and Information Science, vol 2080. Springer, Singapore. https://doi.org/10.1007/978-981-97-1717-0_20. Published 20 March 2024).
Consider claim 6, Wang and Li do not, but Wu discloses the LLM comprises a plurality of independent LLMs (in the model ensemble component, Qwen-7b-Chat, ChatGLM2-6b and MacBERT are utilized, pages 218-219, Section 3.1); wherein, for each of the non-labeled text-items: each LLM independently receives said prompt and independently generates a labeling output (Fig 1, page 219, see prompt example in Fig. 2, page 220), and an LLMs Arbitration Unit selects one of a plurality of the labeling outputs based on a pre-defined arbitration scheme (majority vote, Fig 1, page 219).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wang and Li such that the LLM comprises a plurality of independent LLMs, wherein, for each of the non-labeled text-items: each LLM independently receives said prompt and independently generates a labeling output, and an LLMs Arbitration Unit selects one of a plurality of the labeling outputs based on a pre-defined arbitration scheme in order to improve classification of difficult examples, as suggested by Wu (pages 215-216, Section 1), predictably helping to alleviate the problem of uneven samples in text classification tasks, as suggested by Wu (pages 215, Section 1). The references cited are analogous art in the same field of natural language processing.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (US 20230196105) in view of Li et al. (“Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations”. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10443–10461), in further view of Verma et al. (US 20150067833).
Consider claim 7, Wang discloses automatically generating the trained ML model particularly for a text classification task (auto-labeled training examples are used to train Task Neural Network 170 for a particular text classification task, [0038]).
Wang and Li do not specifically mention a task selected from the group consisting of: classifying incoming messages as fraudulent or legitimate, classifying incoming messages as spam or non-spam, or classifying incoming messages as urgent or non-urgent.
Verma discloses a task selected from the group consisting of: classifying incoming messages as fraudulent or legitimate, classifying incoming messages as spam or non-spam, or classifying incoming messages as urgent or non-urgent (analyzing an email to detect fraudulent links, or classifying it as legitimate, [0097], [0105], classifying emails as spam or legitimate, [0004], classifying emails containing urgent language, e.g. “now”, “immediately” as phishing, as those without urgent words as not, [0056], [0057], [0060]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wang and Li by including a task selected from the group consisting of: classifying incoming messages as fraudulent or legitimate, classifying incoming messages as spam or non-spam, or classifying incoming messages as urgent or non-urgent in order to protect the sensitive information of email users, as suggested by Verma ([0005]). The references cited are analogous art in the same field of natural language processing.
Claims 27-29 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (US 20230196105) in view Li et al. (“Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations”. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10443–10461), in further view of Swain et al. (US 20150341300).
Consider claim 27, Wang and Li do not, but Swain discloses deploying includes integrating the first trained ML model into a system enabled to perform one or more actions based on classification results generated by the first trained ML model (classifying emails and sorting them into “Messages”, “FYI”, “Tasks”, etc., Fig 3, [0033], [0034], [0038], [0062], [0074]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wang and Li such that deploying includes integrating the first trained ML model into a system enabled to perform one or more actions based on classification results generated by the first trained ML model in order to improve email presentation, as suggested by Swain ([0022]), predictably reducing the amount of total time that users spend processing email, as suggested by Swain ([0020]). The references cited are analogous art in the same field of text processing.
Consider claim 28, Wang and Li do not, but Swain discloses the one or more actions include at least one action selected from the group of actions consisting of sorting, generating a user alert, flagging content for review, and quarantining content (sorting emails into “Messages”, “FYI”, “Tasks”, etc., buckets, Fig 3, [0033], [0034], [0038], [0062], [0074], noting the claim language “one or more actions”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wang and Li such that the one or more actions include at least one action selected from the group of actions consisting of sorting, generating a user alert, flagging content for review, and quarantining content for reasons similar to those for claim 27.
Consider claim 29, Wang and Li do not, but Swain discloses incorporating user feedback to correct or verify classifications and using the user feedback to refine the first trained ML model via continuous learning (correct classification of emails established by direct feedback from the users by manually classifying emails, and resulting labeled corpus is used to train for classifying new emails, [0067]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wang and Li by incorporating user feedback to correct or verify classifications and using the user feedback to refine the first trained ML model via continuous learning for reasons similar to those for claim 27.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jesse Pullias whose telephone number is 571/270-5135. The examiner can normally be reached on M-F 8:00 AM - 4:30 PM. The examiner’s fax number is 571/270-6135.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Andrew Flanders can be reached on 571/272-7516.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Jesse S Pullias/
Primary Examiner, Art Unit 2655 04/30/26
Prosecution Insights