Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3, 8, 10, 16, 18-19 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Hsieh (NPL: Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (from applicant IDS)).
Regarding claim 1, Hsieh teaches:
A method for training a machine-trained model, comprising: (Abstract “Our method extracts LLM rationales as additional supervision for training small models within a multi-task framework”)
in a training example-generating operation, generating a plurality of training examples, each training example being produced by: (Page 3 Section 3 “First, given an LLM and an unlabeled dataset, we prompt the LLM to generate output labels along with rationales to justify the labels.”)
receiving a system instruction that requests a teacher language model to formulate responses to queries that describe final results and processes of producing the final results; (Page 4 section 3.1 “we first curate a prompt template p that articulates how the task should be solved. Each prompt is a triplet (x^p,r^p,y^p), where x^p is an example input, y^p is its corresponding label and r^p is a user-provided rationale that explains why x^p can be categorized as y^p.”)
receiving a client instruction that specifies a query; (Page 4 section 3.1 “As illustrated in Figure 3, given an unlabeled dataset x_i ∈ D,”)
producing a combined prompt that combines the system instruction and the client instruction; (Page 4 section 3.1 “We append each input x_i to p and use it as an input to prompt the LLM to generate rationales and labels for each x_i ∈ D.”)
submitting the combined prompt to the teacher language model, the teacher language model transforming the combined prompt into a teacher-model response, the teacher-model response describing a final result and a process of producing the final result; (Page 4 Section 3.1 “We append each input x_i to p and use it as an input to prompt the LLM to generate rationales and labels for each x_i ∈ D. With the demonstrations seen in p, the LLM is able to mimic the triplet demonstration to generate the rationale ˆr_i and output ˆy_i for x_i.”)
storing a training example in a data store that includes the combined prompt and the teacher-model response, the data store storing the plurality of training examples; and (Section 3.2 page 4 “We first describe the current framework for learning task-specific models. With this framework in place, we extend it to incorporate rationales into the training process. Formally, we denote a dataset as D ={(xi,yi)}N i=1 where each x_i represents an input and y_i is the corresponding desired output label.” Training examples stored in dataset D.)
in a training operation, training parameters of a student language model based on the training examples. (Page 4 section 3.2 “For both scenarios, the smaller model f is trained to minimize the label prediction loss”).
Regarding claim 2, Hsieh teaches claim 1 as outlined above. Hsieh further teaches:
the system instruction instructs the teacher language model to provide a description by directly or indirectly requesting the teacher language model to specify at least one intermediary result that leads to the final result, and the teacher language model satisfies the system instruction by providing said at least one intermediary result and the final result. (Page 2 left column “For example, when asked “Jesse’s room is 11 feet long and 15 feet wide. If she already has 16 square feet of carpet. How much more carpet does she need to cover the whole floor?”, an LLM can be prompted by chain-of-thought (CoT) technique (Wei et al., 2022) to provide intermediate rationales “Area = length ×width. Jesse’s room has 11 × 15 square feet.” that better connects the input to the final answer “(11 × 15) − 16”.”)
Regarding claim 3, Hsieh teaches claim 1 as outlined above. Hsieh further teaches:
the teacher language model is a different model than the student language model, the teacher language model having greater capabilities compared to the student language model, and/or the teacher language model consuming more resources compared to the student language model, and/or the teacher language model having a larger size than the student language model. (Page 3 Section 3. “We propose a new paradigm, Distilling step-by step, that leverages the ability of LLMs to reason about their predictions to train smaller models in a data-efficient way” Also see figure 1)
Regarding claim 8, Hsieh teaches claim 1 as outlined above. Hsieh further teaches:
the client instruction is a multi-modal client instruction that provides a text-based question and an item that includes content other than text, the text-based question being directed to the item. (Page 4 section 3.2 “While our framework supports inputs and outputs of any modality, our experiments limits x and y to be natural language.”)
Regarding claim 10, Hsieh teaches claim 1 as outlined above. Hsieh further teaches:
submitting a student-model prompt to the student language model, and, in response, receiving a student-model response, the student-model response describing a student-model final result and a process for producing the student-model final result; (Page 5 top of left column “We prepend “task prefixes” ([label], [rationale]) to the input examples and train the smaller model to output ˆy_i when [label] is provided and to produce ˆr_i with [rationale]”)
generating a measure of loss that depends on a difference between the teacher-model response and the student-model response; and (equations (1), (2) and (4)).
updating parameters of the student language model based on the loss. (Page 4 equation 1 “For both scenarios, the smaller model f is trained to minimize the label prediction loss”)
Regarding claim 16, Hsieh teaches claim 1 as outlined above. Hsieh further teaches:
providing the student language model to a local system, the local system using the student language model to provide responses to newly-submitted queries. (Page 2 left column “Second, our models outperform LLMs with much smaller model sizes (up to 2000× smaller), drastically reducing the computation cost required for model deployment.” They are deploying their models to local systems.)
Regarding claim 18, Hsieh teaches:
A computer-readable storage medium for storing computer-readable instructions, a processing system executing the computer-readable instructions to perform a training operation, the training operation comprising: (Page 13 appendix A gives more experiment details where they describe their hardware. Thus there must have been some sort of computer-readable instructions).
submitting a student-model prompt to a student language model, and, in response, receiving a student-model response, (Page 5 top of left column “We prepend “task prefixes” ([label], [rationale]) to the input examples and train the smaller model to output ˆy_i when [label] is provided and to produce ˆr_i with [rationale]”)
the student-model prompt expressing a combination of a student-model system instruction and a student-model client instruction, (Page 4 section 3.1 “We append each input x_i to p and use it as an input to prompt the LLM to generate rationales and labels for each x_i ∈ D.”)
the student-model system instruction requesting the student language model to formulate responses to queries that describe student-model final results and processes of producing the student-model final results, (Page 5 top of left column “We prepend “task prefixes” ([label], [rationale]) to the input examples and train the smaller model to output ˆy_i when [label] is provided and to produce ˆr_i with [rationale]”)
the student-model client instruction expressing a query, (Page 4 section 3.1 “As illustrated in Figure 3, given an unlabeled dataset x_i ∈ D,”)
the student-model response describing a student-model final result and a process of producing the student-model final result; (Page 5 top of left column “We prepend “task prefixes” ([label], [rationale]) to the input examples and train the smaller model to output ˆy_i when [label] is provided and to produce ˆr_i with [rationale]”)
receiving a teacher-model response, the teacher-model response being produced by a teacher language model based on a teacher-model prompt, the teacher-model prompt including a teacher-model system instruction that requests the teacher language model to formulate responses to queries that describe teacher-model final results and processes of producing the teacher-model final results, (Page 4 Section 3.1 “We append each input x_i to p and use it as an input to prompt the LLM to generate rationales and labels for each x_i ∈ D. With the demonstrations seen in p, the LLM is able to mimic the triplet demonstration to generate the rationale ˆr_i and output ˆy_i for x_i.”)
the teacher-model response describing a teacher-model final result and a process of producing the teacher-model final result; (Page 4 Section 3.1 “We append each input x_i to p and use it as an input to prompt the LLM to generate rationales and labels for each x_i ∈ D. With the demonstrations seen in p, the LLM is able to mimic the triplet demonstration to generate the rationale ˆr_i and output ˆy_i for x_i.”)
generating a measure of loss that depends on a difference between the teacher-model response and the student-model response; (equations (1), (2) and (4)).
updating parameters of the student language model based on the loss; and (Page 4 equation 1 “For both scenarios, the smaller model f is trained to minimize the label prediction loss”)
repeating the submitting, receiving, generating, and updating for other prompts. (Section 3.2 Describes training which implies multiple iterations.)
Regarding claim 19, Hsieh teaches:
A system for using a transformer-based client language model, comprising: a data store for storing computer-readable instructions; a processing system for executing the computer-readable instructions in the data store, to perform operations including: (Page 13 appendix A gives more experiment details where they describe their hardware. Thus there must have been some sort of processing system).
receiving a client-model system instruction that requests the transformer-based client language model to formulate responses to queries that describe client-model final results and processes of producing the client-model final results; (Page 4 section 3.1 “we first curate a prompt template p that articulates how the task should be solved. Each prompt is a triplet (x^p,r^p,y^p), where x^p is an example input, y^p is its corresponding label and r^p is a user-provided rationale that explains why x^p can be categorized as y^p.”)
receiving a client-model client instruction that specifies a query; (Page 4 section 3.1 “As illustrated in Figure 3, given an unlabeled dataset x_i ∈ D,”)
producing a client-model prompt that includes a combination of the client-model system instruction and the client-model client instruction; and (Page 4 section 3.1 “We append each input x_i to p and use it as an input to prompt the LLM to generate rationales and labels for each x_i ∈ D.”)
submitting the client-model prompt to the transformer-based client language model, the transformer-based client language model transforming the client-model prompt into a client-model response, the client-model response describing a client-model final result and a process of producing the client-model final result via intermediary results, (Page 4 Section 3.1 “We append each input x_i to p and use it as an input to prompt the LLM to generate rationales and labels for each x_i ∈ D. With the demonstrations seen in p, the LLM is able to mimic the triplet demonstration to generate the rationale ˆr_i and output ˆy_i for x_i.”)
the transformer-based client language model producing the client-model response using parameters that are trained based on teacher-model responses produced by a transformer-based teacher language model in response to teacher-model prompts, each teacher-model prompt expressing a combination of a teacher-model system instruction and a teacher-model client instruction, and (Page 5 top of left column “We prepend “task prefixes” ([label], [rationale]) to the input examples and train the smaller model to output ˆy_i when [label] is provided and to produce ˆr_i with [rationale]”)
each teacher-model system instruction requesting the transformer-based teacher language model to formulate teacher-model responses to queries that describe teacher-model final results and processes of producing the teacher-model final results. (Page 4 Section 3.1 “We append each input x_i to p and use it as an input to prompt the LLM to generate rationales and labels for each x_i ∈ D. With the demonstrations seen in p, the LLM is able to mimic the triplet demonstration to generate the rationale ˆr_i and output ˆy_i for x_i.”)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 4-7 are rejected under 35 U.S.C. 103 as being unpatentable by Hsieh in view of Zelikman (NPL ‘STaR: Self-Taught Reasoner Bootstrapping Reasoning With Reasoning’ (2022)).
Regarding claim 4, Hsieh teaches claim 1 as outlined above. Hsieh does not teach the limitations of claim 4. However, Zelikman does:
the teacher language model is a same model as the student language model, acting in a context of a teacher. (Star Bottom of page 2 “We propose what is, to our knowledge, the first technique to allow a pre-trained large language model to iteratively use its language modeling capacity to improve itself.”)
Hsieh and Zelikman are considered analogous art to the claimed invention because they are in the same field of endeavor being model improvement and compression. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the overall teacher student knowledge distillation of Hsieh with the self-model improvement of Zelikman. One would want to do this to improve the models accuracy (Zelikman abstract)
Regarding claim 5, Hsieh teaches claim 1 as outlined above. Hsieh does not teach the limitations of claim 5. However, Zelikman does:
the combined prompt that is provided to the teacher language model also specifies the final result, which serves as a ground-truth answer, and wherein the system instruction asks the teacher language model to describe how the final result is produced. (Star 3rd Paragraph page 2 “in each iteration, first construct a finetuning dataset by attempting to solve the dataset using the current model’s rationale generation ability; then, augment this dataset using rationalization, justifying ground-truth answers to problems the model failed to solve; finally, finetune the large language model on the combined dataset.”)
Hsieh and Zelikman are considered analogous art to the claimed invention because they are in the same field of endeavor being model improvement and compression. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the overall teacher student knowledge distillation of Hsieh with the ground truth of Zelikman. One would want to do this to improve the models accuracy (Zelikman abstract)
Regarding claim 6, Hsieh teaches claim 1 as outlined above. Hsieh does not teach the limitations of claim 6. However, Zelikman does:
using the teacher language model to improve the teacher-model response in one or more improvement operations. (Star Page 3 Section 2 “One initial work on the impact of rationales on language model performance was [3], showing that training a language model on a dataset with explicit rationales preceding the answer could improve a model’s ability to generate the final answer. However, this required many thousands of training examples to be manually annotated with human reasoning. Recently, [5] demonstrated that step-by-step “scratchpads” improve fine-tuned LLM performance and generalization on tasks such as arithmetic, polynomial evaluation, and program evaluation. Similarly, [6] used a single few-shot “chain-of-thought” reasoning prompt to improve model performance on tasks without fine-tuning.”)
Hsieh and Zelikman are considered analogous art to the claimed invention because they are in the same field of endeavor being model improvement and compression. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the overall teacher student knowledge distillation of Hsieh with the teacher model improving itself of Zelikman. One would want to do this to improve the models accuracy (Zelikman abstract)
Regarding claim 7, Hsieh teaches claim 1 as outlined above. Hsieh does not teach the limitations of claim 7. However, Zelikman does:
the teacher language model is invoked in response to a determination that a student-model response fails a prescribed quality test. (Page 4 Section 3.2 “This is fundamentally due to the fact that the algorithm cannot obtain any training signal from failed examples. Inspired by [3], we propose a technique we call “rationalization”. Specifically, we provide the answer as a hint to the model and ask it to generate rationales in the same style as in the previous rationale generation step. Given the answer, the model is able to reason backwards, and hence more easily generate a rationale leading to the correct answer.”)
Hsieh and Zelikman are considered analogous art to the claimed invention because they are in the same field of endeavor being model improvement and compression. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the overall teacher student knowledge distillation of Hsieh with failsafe of Zelikman. One would want to do this to improve the models accuracy (Zelikman abstract)
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable by Hsieh in view of Longpre (NPL: “The Flan Collection: Designing Data and Methods for Effective Instruction Tuning” (From applicant IDS)).
Regarding claim 9, Hsieh teaches claim 1 as outlined above. Hsieh does not teach the limitations of claim 9. However, Longpre does:
wherein the training example-generating operation further comprises extracting a set of queries from a larger collection of queries, the query being one query in the set of queries, wherein the larger collection of queries include plural sub-collections of queries pertaining to different respective categories, and wherein the extracting comprises, for each category-of-interest, selecting a prescribed category-specific amount of queries from a sub-collection pertaining to the category-of-interest. (FLAN Section 3.3 Page 6 “The most recent and concurrent publicly available instruction tuning efforts, like Flan 2022, train on thousands of tasks (Wang et al., 2022c; Iyer et al., 2022), but operate on different task compositions and underlying training methods. To measure the impact of scaling model sizes and tasks for the Flan 2022 collection, we finetune T5-LM adapted models (Small, Base, Large, XL, XXL) on randomly selected task subsets (8, 25, 50, 100, 200, 400, 800, all 1873). Every finetuning run is guaranteed to include the Held-In tasks, so we can estimate how task scaling impacts the model capacity to maintain performance on a given task its already seen.”)
Hsieh and Longpre are considered analogous art to the claimed invention because they are in the same field of endeavor being large language models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the overall teacher student knowledge distillation of Hsieh with the dataset of Longpre. One would want to do this as using this dataset improve model accuracy (Longpre abstract).
Claims 11-15 are rejected under 35 U.S.C. 103 as being unpatentable by Hsieh in view of Liu (NPL: “Adaptive multi-teacher multi-level knowledge distillation” (2020)).
Regarding claim 11, Hsieh teaches claim 1 as outlined above. Hsieh does not teach the limitations of claim 11. However, Liu does:
wherein the set of training examples includes a first set of training examples and a second set of training examples, and (End of section 3.2 page 4 “the soft-target generated by the t-th teacher for the i-th image”)
wherein the training operation performs training using the first set of training examples, and then performs training using the second set of training examples. (Section 3.3 on page 4 describes how the student model learns for the teacher models specifically loss equation (8)).
Hsieh and Liu are considered analogous art to the claimed invention because they are in the same field of endeavor being knowledge distillation. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the overall teacher student knowledge distillation of Hsieh with the multiple teachers of Liu. One would want to do this to improve the accuracy of the student models (Longpre section 5.1.3).
Regarding claim 12, Hsieh in view of Liu teaches claim 11 as outlined above. Liu further teaches:
wherein the teacher language model is one of a first teacher language model or a second teacher language model in a teacher system that includes the first and second teacher language models, the second teacher language model being more capable than the first teacher language model, (Page 3 left column “it is intuitive to allocate a larger weight to the first teacher for this image and smaller weights to the other two teachers to get the integrated soft-target value.”)
wherein the first teacher language model is used to produce the first set of training examples, and (End of section 3.2 page 4 “the soft-target generated by the t-th teacher for the i-th image”)
wherein the second teacher language model is used to produce the second set of training examples. (End of section 3.2 page 4 “the soft-target generated by the t-th teacher for the i-th image”)
Regarding claim 13, Hsieh in view of Liu teaches claim 12 as outlined above. Liu further teaches:
the second teacher language model has a throughput that is higher than a throughput of the first teacher language model, and wherein interaction with the second teacher language model incurs a latency that is higher than a latency of the first teacher language model. (Page 5 section 5.1.1 “With respect to the two CIFAR datasets, we adopt three teacher neural networks, i.e.,ResNet110,VGG-19, andDenseNet121.” In Table 1 they show information about the teacher models which includes number of parameters and compression rate. Showing that some of the teacher models are bigger than others.)
Regarding claim 14, Hsieh in view of Liu teaches claim 11 as outlined above. Liu further teaches:
wherein the first set of training examples are generated for a first set of queries having a first complexity level, and wherein the second set of training examples are generated for a second set of queries having a second complexity level. (Page 3 right column “Larger y_t,i denotes the teacher is more important with regard to the image.”)
Regarding claim 15, Hsieh in view of Liu teaches claim 11 as outlined above. Liu further teaches:
there are more training examples in the first set of training examples compared to the second set of training examples. (Page 3 left Column “the adapter is responsible for adaptively learning instance-level teacher importance weight for integrating soft-targets, which are further utilized to derive the standard knowledge distillation loss and angle based loss.”)
Claims 17 and 20 are rejected under 35 U.S.C. 103 as being unpatentable by Hsieh in view of Baeuml (US 2023/0074406 A1)
Regarding claim 17, Hsieh teaches claim 16 as outlined above. Hsieh does not teach the limitations of claim 17. However, Baeuml does:
the student language model is capable of generating responses to the newly-submitted queries in an offline mode, independent of any network-accessible resources. ([0043] “In some versions of those implementations (e.g., and as described with respect to FIG. 3), the automated assistant 115 can cause one or more of the LLM outputs to be generated in an offline manner (e.g., not responsive to a spoken utterance being received during a dialog session), “)
Hsieh and Baeuml are considered analogous art to the claimed invention because they are in the same field of endeavor being large language models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the overall teacher student knowledge distillation of Hsieh with the offline capabilities of the large language model of Baeuml. One would want to do this to allow for deployment of the student model to offline devices.
Regarding claim 20, Hsieh teaches claim 19 as outlined above. Hsieh does not teach the limitations of claim 20. However, Baeuml does:
wherein the system is a local system that locally implements the transformer-based client language model, and wherein the transformer-based client language model is capable of operating in an offline mode, independent of interaction with network-accessible resources. ([0043] “In some versions of those implementations (e.g., and as described with respect to FIG. 3), the automated assistant 115 can cause one or more of the LLM outputs to be generated in an offline manner (e.g., not responsive to a spoken utterance being received during a dialog session), “)
Hsieh and Baeuml are considered analogous art to the claimed invention because they are in the same field of endeavor being large language models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the overall teacher student knowledge distillation of Hsieh with the offline capabilities of the large language model of Baeuml. One would want to do this to allow for deployment of the student model to offline devices.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL P GRUSZKA whose telephone number is (571)272-5259. The examiner can normally be reached M-F 9:00 AM - 6:00 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li Zhen can be reached at (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL GRUSZKA/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights