Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7, 9, 11, 13, 14, 16, 18, 19 are rejected under 35 U.S.C. 103 as being unpatentable over Shi (US 20250181928 A1) in view of Sawtelle (US 20240202329 A1).
Regarding claim 1, Shi discloses “generating, by a knowledge transfer system, a knowledge transfer dataset comprising a set of synthesized data samples adapted for a … classification task, wherein a language machine learning model is guided by a teacher machine learning model in generating the set of synthesized data samples; and” (See Abstract, [0002-0005], [0024]; synthetic samples are produced based on the teacher ML model)
“training, by the knowledge transfer system using the teacher model, a student machine learning model by using the knowledge transfer dataset” (See [0033]; the student model learns from the teacher model using synthetic samples made by teacher, and the synthetic samples are the knowledge transfer dataset)
Shi fails to explicitly disclose that the data samples are adapted for a text classification task.
Sawtelle teaches “data samples adapted for a text classification task” (See [0017], [0018]; Sawtelle discloses using a set of text words in a text message as a training dataset for a message classification engine for classifying text messages that may have a character swap attack).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Shi and Sawtelle before them to modify Shi to obtain data samples that are intended for use in text classification. One would be motivated to do so in order to use the dataset for training another machine learning model on text classification, see e.g., [0018], where Sawtelle uses the text data samples to train a classification engine on classifying text messages that may contain a character swap attack.
Regarding claim 2, Shi discloses “wherein the teacher model was pre-trained for text classification using an original training dataset” (See [0019], [0023]; The teacher model is pre-trained using an original training dataset)
Regarding claim 3, Shi discloses “wherein the original training dataset for the teacher model is inaccessible by the knowledge transfer system” (See [0045]; The original training dataset used for training the teacher model is not available for the knowledge transfer system (the database that the student learns from))
Regarding claim 4, Shi discloses “wherein the teacher model and student model are implemented by different artificial neural network architectures” (See [0047]; teacher models and student models are described as using different convolutional neural network architectures for the best version of the configuration)
Regarding claim 7, Shi discloses “generating, by the knowledge transfer system using the set of synthesized data samples, a set of diversified data samples, wherein the set of diversified data samples is added to the knowledge transfer dataset” (See [0020]; A generator produces diversified samples for teaching student models using teacher assistant models. Each teacher assistant model is a dedicated model to a pair of a teacher model and a generator, and are used to teach student models using the knowledge transfer data set and the diversified data samples)
Regarding claim 9, Shi discloses “wherein the set of diversified data samples are generated by augmenting the set of synthesized data samples using an adversarial strategy” (See [0020]; Using an adversarial loss function helps with generating diversified data samples from the set of synthesized image data samples)
Regarding claims 11 and 16, these claims are similar in scope to claim 1.
Regarding claims 13 and 18, these claims are similar in scope to claim 7.
Regarding claims 14 and 19, these claims are similar in scope to claim 9.
Claim Rejections - 35 USC § 103
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Shi (US 20250181928 A1) in view of Sawtelle (US 20240202329 A1), and further in view of Sanh (DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter).
Regarding claim 5, Shi fails to explicitly disclose, “wherein the student model is differentiated from the teacher model by having at least one of fewer layers and fewer parameters than the teacher model”.
Sanh teaches “wherein the student model is differentiated from the teacher model by having at least one of fewer layers and fewer parameters than the teacher model” (See [Page 2, Section 3, Paragraph 1-2]; Sanh discloses creating a student model by selecting layers and parameters to reduce from the teacher model for the purpose of creating a reduced size version of the teacher model).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Shi and Sanh before them to modify Shi to reduce the number of layers and parameters in the student model so it has fewer layers and parameters than the teacher model. One would be motivated to do so in order to train smaller student models with nearly the same functionality as the teacher model, see e.g., [Page 2, Section 2, Paragraph 1], where Sanh describes that knowledge distillation is a compression technique used to reproduce the behavior of a larger model (teacher) onto a smaller model (student).
Claim Rejections - 35 USC § 103
Claims 6, 12, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Shi (US 20250181928 A1) in view of Sawtelle (US 20240202329 A1), and further in view of Brownlee (A Gentle Introduction to Multiple-Model Machine Learning).
Regarding claim 6, Shi discloses “providing, by the teacher model to the language model” (See [0038]; Shi discloses providing labels generated from a teacher model to a student model through Knowledge Distillation (KD))
Shi fails to explicitly disclose, “wherein the generating the set of synthesized data samples adapted for the text classification task includes: providing… weighted decoding parameters based on the text classification task”.
Brownlee teaches “wherein the generating the set of synthesized data samples adapted for the text classification task includes: providing… weighted decoding parameters based on the text classification task” (See [Page 2, Multiple-Model Techniques], [Page 5, Hybrids Constructed From Multiple Models]; Weighted decoding parameters are described as a combination of two probabilities from two different ML models, and Brownlee discloses using Ensemble Learning to combine predictions from two different ML models).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Shi and Brownlee before them to modify Shi to provide weighted decoding parameters from the teacher to a language model. One would be motivated to do so in order to make use of multiple models and combine their results to obtain improved predictions, see e.g., [Page 2, Multiple-Model Techniques], where Brownlee explains that the goal of an ensemble model is to improve predictions over contributing models.
Regarding claims 12 and 17, these claims are similar in scope to claim 6.
Claim Rejections - 35 USC § 103
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Shi (US 20250181928 A1) in view of Sawtelle (US 20240202329 A1), and further in view of Clement (US 20220308848 A1).
Regarding claim 8, Shi fails to explicitly disclose, “wherein the set of diversified data samples are generated by performing a back-translation of the set of synthesized data samples”.
Clement teaches “wherein the set of diversified data samples are generated by performing a back-translation of the set of synthesized data samples” (See [0108]; Clement discloses generating back-translated data samples from a translated data sample. Source code written in one language that was translated into a second language is processed by a neural transformer model to translate it back into the first language to create back-translated source code).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Shi and Clement before them to modify Shi to perform back-translation on the synthesized data samples. One would be motivated to do so in order to fine-tune the model on the translations generated from performing back-translation, see e.g., [0108], [0110], where Clement discloses performing back-translation on the model until there are no more sequences that can be generated from performing back-translation.
Claim Rejections - 35 USC § 103
Claims 10, 15, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Shi (US 20250181928 A1) in view of Sawtelle (US 20240202329 A1), and further in view of Glass (US 20240111969 A1).
Regarding claim 10, Shi fails to explicitly disclose, “wherein the training the student model using the knowledge transfer dataset includes: updating the student model according to a weighted loss function based on logits output by the teacher model and logits output by the student model”.
Glass teaches “wherein the training the student model using the knowledge transfer dataset includes: updating the student model according to a weighted loss function based on logits output by the teacher model and logits output by the student model” (See [0026], [0027]; Glass discloses training a student model by generating logits based on the teacher model and student model using a weighted loss function, and uses this for a re-ranker that trains the student model).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Shi and Glass before them to modify Shi to update the student model using a weighted loss function that uses logits. One would be motivated to do so in order to describe a process for knowledge distillation for training the student model, see e.g., [0026], [0027], where Glass describes producing logits from the teacher model and student model, and a loss function to weigh the logits, for the purpose of describing a process of knowledge distillation.
Regarding claims 15 and 20, these claims are similar in scope to claim 10.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID KIM whose telephone number is (571)272-4331. The examiner can normally be reached 7:30 AM - 4:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Ell can be reached at (571) 270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/D.K./Examiner, Art Unit 2141
/MATTHEW ELL/Supervisory Patent Examiner, Art Unit 2141
Prosecution Insights