Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 2/26/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Status of Claims
The present application is being examined under the claims filed on 1/29/2024.
Claims 1-20 are rejected.
Claims 1-20 are pending.
Specification
The specification filed on 1/29/2024 is acceptable for examination purposes.
Drawings
The drawings filed on 1/29/2024 are acceptable for examination purposes.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1,
Step 1: Claim 1 is a method claim. Therefore, Claims 1-7 are directed to a process.
Step 2A Prong 1:
dividing the holdout set into a plurality of batches, each batch including a respective subset of training examples (mental process – dividing the holdout set into a plurality of batches, each batch including a respective subset of training examples may be performed manually by a user with the aid of pen and paper by observing/analyzing the holdout set and using a judgement to divide the holdout set into a plurality of batches. See MPEP 2106.04(a)(2)(III)(C).)
evaluating the trained surrogate model on the evaluation set by: (mental process – evaluating the trained surrogate model on the evaluation set may be performed manually by a user with the aid of pen and paper by observing/analyzing the trained surrogate model on the evaluation set and using a judgement to evaluate the trained surrogate model. See MPEP 2106.04(a)(2)(III)(C).)
generating a set of predictions [by applying the trained surrogate model to the evaluation set], the set of predictions including a prediction for a training example in the evaluation set (mental process – generating a set of predictions, the set of predictions including a prediction for a training example in the evaluation set may be performed manually by a user with the aid of pen and paper by observing/analyzing a training example in the evaluation set and using a judgement to generate a set of predictions. See MPEP 2106.04(a)(2)(III)(C).)
determining, for the training example in the evaluation set, a loss based on the set of predictions for the training example (mathematical concept – determining, for the training example in the evaluation set, a loss based on the set of predictions for the training example may be performed by mathematical process, calculating the difference between the predictions and the labels for the training examples. Spec. Paragraph [0053] discloses “a loss for each of the training examples, the loss indicating the difference between the predictions and the labels for the training examples.” See MPEP 2106.04(a)(2)(I)(C).)
generating a training set for the first machine-learned language model by selecting, from the evaluation set, a set of training examples with the loss within a predetermined range (mental process – generating a training set for the first machine-learned language model by selecting, from the evaluation set, a set of training examples with the loss within a predetermined range may be performed manually by a user with the aid of pen and paper by selecting a set of training examples with the loss within a predetermined range from the evaluation set and creating a training set based on the selection. See MPEP 2106.04(a)(2)(III)(C).)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
obtaining a holdout set and an evaluation set, the holdout set including a first set of training examples and the evaluation set including a second set of training examples (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
training a surrogate model using the holdout set by iterating through the plurality of batches, wherein the surrogate model is a second machine-learned language model (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
by applying the trained surrogate model to the evaluation set (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
training parameters of the first machine-learned language model for a first epoch using the selected training set (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The additional elements recite generic computer elements and programs at a high-level of generality to perform the judicial exception as well as recitation of generic computer functionality such as a surrogate model, a first machine-learned language model and a second machine-learned language model.
Additional Elements:
obtaining a holdout set and an evaluation set, the holdout set including a first set of training examples and the evaluation set including a second set of training examples (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
training a surrogate model using the holdout set by iterating through the plurality of batches, wherein the surrogate model is a second machine-learned language model (The additional elements are directed to training a surrogate model using the holdout set by iterating through the plurality of batches, wherein the surrogate model is a second machine-learned language model at a high level of generality. Liu et al. (“Accelerate Mini-batch Machine Learning Training With Dynamic Batch Size Fitting”), hereinafter Liu, discloses in Section I, “The currently most widely used training algorithm for large-scale ML tasks is called Minibatch Stochastic Gradient Descent (MGD), which improves upon the simplest but powerful Stochastic Gradient Descent (SGD) [4] method. The basic idea of MGD is to train only a random slice of the dataset in each iteration, which is proved to have similar converge rates as GD on large datasets but is much more efficient [5]. The actual size of the random data slice that is trained in an iteration is determined by a hyperparameter called batch size.” Raschka (“Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning”) discloses in Section 3.5, “Given how the holdout method is most commonly used […]” Liu has recognized training a model by iterating through the plurality of batches (i.e., Minibatch Stochastic Gradient Descent (MGD)) is a well-understood, routine, and conventional activity previously known in the industry. Raschka has recognized using the holdout set is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
by applying the trained surrogate model to the evaluation set (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
training parameters of the first machine-learned language model for a first epoch using the selected training set (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
For the reasons above, Claim 1 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 2-7. The additional limitations of the dependent claims are addressed below.
Regarding Claim 2,
Step 2A Prong 1:
See the rejection of Claim 1 above, which Claim 2 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein the second machine-learned language model has a smaller number of parameters than a number of parameters of the first machine-learned language model (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein the second machine-learned language model has a smaller number of parameters than a number of parameters of the first machine-learned language model (The additional elements are directed to wherein the second machine-learned language model has a smaller number of parameters than a number of parameters of the first machine-learned language model at a high level of generality. Xu et al. (“Multiple-Stage Knowledge Distillation”), hereinafter Xu, discloses in Section I, “In the conventional KD method, the teacher network has a deeper layer and more parameters than those of the student network.” Xu has recognized wherein the second machine-learned model has a smaller number of parameters than a number of parameters of the first machine-learned model is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
Regarding Claim 3,
Step 2A Prong 1:
See the rejection of Claim 1 above, which Claim 3 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
for at least a second epoch, training the parameters of the first machine-learned language model using the selected training set (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
for at least a second epoch, training the parameters of the first machine-learned language model using the selected training set (The additional elements are directed to for at least a second epoch, training the parameters of the first machine-learned language model using the selected training set at a high level of generality. Raschka (“Practical Tips for Finetuning LLMs Using LoRA (Low-Rank Adaptation)”) discloses in Section 5, “In conventional deep learning, we often iterate over a training set multiple times -- iteration over the training set is called an epoch. It's common to run hundreds of training epochs when training convolutional neural networks […]” Raschka has recognized for at least a second epoch, training the parameters of the first machine-learned language model using the selected training set is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
Regarding Claim 4,
Step 2A Prong 1:
See the rejection of Claim 1 above, which Claim 4 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein training the surrogate model comprises repeatedly iterating between:
performing a forward pass step to generate a set of estimated predictions for a batch (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
computing a loss function for the batch based on the set of estimated predictions (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
performing a backpropagation step to update parameters of the surrogate model to reduce the loss function (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein training the surrogate model comprises repeatedly iterating between:
performing a forward pass step to generate a set of estimated predictions for a batch (The additional elements are directed to performing a forward pass step to generate a set of estimated predictions for a batch at a high level of generality. Khan (“From Neurons to Knowledge: Understanding Forward Pass and Backpropagation Step by Step”) discloses in Pages 1 and 4, “[…] there are numerous articles and tutorials available on this topic […] we’ll dive into the core concepts of neural networks, specifically focusing on the step-by-step processes of the forward pass and backpropagation […] The forward pass in a neural network is the process of taking input data, multiplying it by weights, applying activation functions, and passing it through the network’s layers to generate predictions or outputs.” Khan has recognized performing a forward pass step to generate a set of estimated predictions for a batch is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
computing a loss function for the batch based on the set of estimated predictions (The additional elements are directed to computing a loss function for the batch based on the set of estimated predictions at a high level of generality. AI Wiki (“Accuracy and Loss”) discloses in Page 1, “Accuracy and Loss are the two most well-known and discussed metrics in machine learning. […] A loss function, also known as a cost function, takes into account the probabilities or uncertainty of a prediction based on how much the prediction varies from the true value. This gives us a more nuanced view into how well the model is performing.” AI Wiki has recognized computing a loss function for the batch based on the set of estimated predictions is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
performing a backpropagation step to update parameters of the surrogate model to reduce the loss function (The additional elements are directed to performing a backpropagation step to update parameters of the surrogate model to reduce the loss function at a high level of generality. Khan (“From Neurons to Knowledge: Understanding Forward Pass and Backpropagation Step by Step”) discloses in Pages 1 and 8, “[…] there are numerous articles and tutorials available on this topic […] we’ll dive into the core concepts of neural networks, specifically focusing on the step-by-step processes of the forward pass and backpropagation […] Our goal with backpropagation is to update the weights in the network so that they cause the actual output to be closer the target output, there by minimizing the error for each output.” Khan has recognized performing a backpropagation step to update parameters of the surrogate model to reduce the loss function is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
Regarding Claim 5,
Step 2A Prong 1:
See the rejection of Claim 1 above, which Claim 5 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein a training example is a sequence of tokens (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein a training example is a sequence of tokens (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
Regarding Claim 6,
Step 2A Prong 1:
dividing the selected training set into another plurality of batches (mental process – dividing the selected training set into another plurality of batches may be performed manually by a user with the aid of pen and paper by observing/analyzing the selected training set and using a judgement to divide the selected training set into another plurality of batches. See MPEP 2106.04(a)(2)(III)(C).)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein training the first machine-learned language model comprises repeatedly iterating between:
performing a forward pass step to generate another set of estimated predictions for a batch in the another plurality of batches (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
computing a loss function for the batch based on the another set of estimated predictions (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
performing a backpropagation step to update the parameters of the first machine-learned language model to reduce the loss function (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein training the first machine-learned language model comprises repeatedly iterating between:
performing a forward pass step to generate another set of estimated predictions for a batch in the another plurality of batches (The additional elements are directed to performing a forward pass step to generate another set of estimated predictions for a batch in the another plurality of batches at a high level of generality. Khan (“From Neurons to Knowledge: Understanding Forward Pass and Backpropagation Step by Step”) discloses in Pages 1 and 4, “[…] there are numerous articles and tutorials available on this topic […] we’ll dive into the core concepts of neural networks, specifically focusing on the step-by-step processes of the forward pass and backpropagation […] The forward pass in a neural network is the process of taking input data, multiplying it by weights, applying activation functions, and passing it through the network’s layers to generate predictions or outputs.” Khan has recognized performing a forward pass step to generate another set of estimated predictions for a batch in the another plurality of batches is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
computing a loss function for the batch based on the another set of estimated predictions (The additional elements are directed to computing a loss function for the batch based on the another set of estimated predictions at a high level of generality. AI Wiki (“Accuracy and Loss”) discloses in Page 1, “Accuracy and Loss are the two most well-known and discussed metrics in machine learning. […] A loss function, also known as a cost function, takes into account the probabilities or uncertainty of a prediction based on how much the prediction varies from the true value. This gives us a more nuanced view into how well the model is performing.” AI Wiki has recognized computing a loss function for the batch based on the another set of estimated predictions is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
performing a backpropagation step to update the parameters of the first machine-learned language model to reduce the loss function (The additional elements are directed to performing a backpropagation step to update parameters of the first machine-learned language model to reduce the loss function at a high level of generality. Khan (“From Neurons to Knowledge: Understanding Forward Pass and Backpropagation Step by Step”) discloses in Pages 1 and 8, “[…] there are numerous articles and tutorials available on this topic […] we’ll dive into the core concepts of neural networks, specifically focusing on the step-by-step processes of the forward pass and backpropagation […] Our goal with backpropagation is to update the weights in the network so that they cause the actual output to be closer the target output, there by minimizing the error for each output.” Khan has recognized performing a backpropagation step to update parameters of the first machine-learned language model to reduce the loss function is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
Regarding Claim 7,
Step 2A Prong 1:
See the rejection of Claim 1 above, which Claim 7 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
obtaining a total budget indicating a total number of training examples for training the parameters of the first machine-learned language model (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
wherein a number of training examples in the selected training set is based on the total budget (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
obtaining a total budget indicating a total number of training examples for training the parameters of the first machine-learned language model (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
wherein a number of training examples in the selected training set is based on the total budget (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
Regarding Claim 8,
Step 1: Claim 8 is a non-transitory computer readable storage medium claim. Therefore, Claims 8-14 are directed to a manufacture.
Step 2A Prong 1:
divide the holdout set into a plurality of batches, each batch including a respective subset of training examples (mental process – dividing the holdout set into a plurality of batches, each batch including a respective subset of training examples may be performed manually by a user with the aid of pen and paper by observing/analyzing the holdout set and using a judgement to divide the holdout set into a plurality of batches. See MPEP 2106.04(a)(2)(III)(C).)
evaluate the trained surrogate model on the evaluation set by: (mental process – evaluating the trained surrogate model on the evaluation set may be performed manually by a user with the aid of pen and paper by observing/analyzing the trained surrogate model on the evaluation set and using a judgement to evaluate the trained surrogate model. See MPEP 2106.04(a)(2)(III)(C).)
generate a set of predictions [by applying the trained surrogate model to the evaluation set], the set of predictions including a prediction for a training example in the evaluation set (mental process – generating a set of predictions, the set of predictions including a prediction for a training example in the evaluation set may be performed manually by a user with the aid of pen and paper by observing/analyzing a training example in the evaluation set and using a judgement to generate a set of predictions. See MPEP 2106.04(a)(2)(III)(C).)
determine, for the training example in the evaluation set, a loss based on the set of predictions for the training example (mathematical concept – determining, for the training example in the evaluation set, a loss based on the set of predictions for the training example may be performed by mathematical process, calculating the difference between the predictions and the labels for the training examples. Spec. Paragraph [0053] discloses “a loss for each of the training examples, the loss indicating the difference between the predictions and the labels for the training examples.” See MPEP 2106.04(a)(2)(I)(C).)
generate a training set for the first machine-learned language model by selecting, from the evaluation set, a set of training examples with the loss within a predetermined range (mental process – generating a training set for the first machine-learned language model by selecting, from the evaluation set, a set of training examples with the loss within a predetermined range may be performed manually by a user with the aid of pen and paper by selecting a set of training examples with the loss within a predetermined range from the evaluation set and creating a training set based on the selection. See MPEP 2106.04(a)(2)(III)(C).)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
obtain a holdout set and an evaluation set, the holdout set including a first set of training examples and the evaluation set including a second set of training examples (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
train a surrogate model using the holdout set by iterating through the plurality of batches, wherein the surrogate model is a second machine-learned language model (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
by applying the trained surrogate model to the evaluation set (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
train parameters of the first machine-learned language model for a first epoch using the selected training set (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The additional elements recite generic computer elements and programs at a high-level of generality to perform the judicial exception as well as recitation of generic computer functionality such as a surrogate model, a first machine-learned language model and a second machine-learned language model.
Additional Elements:
obtain a holdout set and an evaluation set, the holdout set including a first set of training examples and the evaluation set including a second set of training examples (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
train a surrogate model using the holdout set by iterating through the plurality of batches, wherein the surrogate model is a second machine-learned language model (The additional elements are directed to training a surrogate model using the holdout set by iterating through the plurality of batches, wherein the surrogate model is a second machine-learned language model at a high level of generality. Liu et al. (“Accelerate Mini-batch Machine Learning Training With Dynamic Batch Size Fitting”), hereinafter Liu, discloses in Section I, “The currently most widely used training algorithm for large-scale ML tasks is called Minibatch Stochastic Gradient Descent (MGD), which improves upon the simplest but powerful Stochastic Gradient Descent (SGD) [4] method. The basic idea of MGD is to train only a random slice of the dataset in each iteration, which is proved to have similar converge rates as GD on large datasets but is much more efficient [5]. The actual size of the random data slice that is trained in an iteration is determined by a hyperparameter called batch size.” Raschka (“Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning”) discloses in Section 3.5, “Given how the holdout method is most commonly used […]” Liu has recognized training a model by iterating through the plurality of batches (i.e., Minibatch Stochastic Gradient Descent (MGD)) is a well-understood, routine, and conventional activity previously known in the industry. Raschka has recognized using the holdout set is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
by applying the trained surrogate model to the evaluation set (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
train parameters of the first machine-learned language model for a first epoch using the selected training set (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
For the reasons above, Claim 8 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 9-14. The additional limitations of the dependent claims are addressed below.
Regarding Claim 9,
Step 2A Prong 1:
See the rejection of Claim 8 above, which Claim 9 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein the second machine-learned language model has a smaller number of parameters than a number of parameters of the first machine-learned language model (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein the second machine-learned language model has a smaller number of parameters than a number of parameters of the first machine-learned language model (The additional elements are directed to wherein the second machine-learned language model has a smaller number of parameters than a number of parameters of the first machine-learned language model at a high level of generality. Xu et al. (“Multiple-Stage Knowledge Distillation”), hereinafter Xu, discloses in Section I, “In the conventional KD method, the teacher network has a deeper layer and more parameters than those of the student network.” Xu has recognized wherein the second machine-learned model has a smaller number of parameters than a number of parameters of the first machine-learned model is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
Regarding Claim 10,
Step 2A Prong 1:
See the rejection of Claim 8 above, which Claim 10 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
for at least a second epoch, train the parameters of the first machine-learned language model using the selected training set (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
for at least a second epoch, train the parameters of the first machine-learned language model using the selected training set (The additional elements are directed to for at least a second epoch, training the parameters of the first machine-learned language model using the selected training set at a high level of generality. Raschka (“Practical Tips for Finetuning LLMs Using LoRA (Low-Rank Adaptation)”) discloses in Section 5, “In conventional deep learning, we often iterate over a training set multiple times -- iteration over the training set is called an epoch. It's common to run hundreds of training epochs when training convolutional neural networks […]” Raschka has recognized for at least a second epoch, training the parameters of the first machine-learned language model using the selected training set is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
Regarding Claim 11,
Step 2A Prong 1:
See the rejection of Claim 8 above, which Claim 11 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein the instructions to train the surrogate model comprise instructions to repeatedly iterate between:
performing a forward pass step to generate a set of estimated predictions for a batch (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
computing a loss function for the batch based on the set of estimated predictions (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
performing a backpropagation step to update parameters of the surrogate model to reduce the loss function (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein the instructions to train the surrogate model comprise instructions to repeatedly iterate between:
performing a forward pass step to generate a set of estimated predictions for a batch (The additional elements are directed to performing a forward pass step to generate a set of estimated predictions for a batch at a high level of generality. Khan (“From Neurons to Knowledge: Understanding Forward Pass and Backpropagation Step by Step”) discloses in Pages 1 and 4, “[…] there are numerous articles and tutorials available on this topic […] we’ll dive into the core concepts of neural networks, specifically focusing on the step-by-step processes of the forward pass and backpropagation […] The forward pass in a neural network is the process of taking input data, multiplying it by weights, applying activation functions, and passing it through the network’s layers to generate predictions or outputs.” Khan has recognized performing a forward pass step to generate a set of estimated predictions for a batch is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
computing a loss function for the batch based on the set of estimated predictions (The additional elements are directed to computing a loss function for the batch based on the set of estimated predictions at a high level of generality. AI Wiki (“Accuracy and Loss”) discloses in Page 1, “Accuracy and Loss are the two most well-known and discussed metrics in machine learning. […] A loss function, also known as a cost function, takes into account the probabilities or uncertainty of a prediction based on how much the prediction varies from the true value. This gives us a more nuanced view into how well the model is performing.” AI Wiki has recognized computing a loss function for the batch based on the set of estimated predictions is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
performing a backpropagation step to update parameters of the surrogate model to reduce the loss function (The additional elements are directed to performing a backpropagation step to update parameters of the surrogate model to reduce the loss function at a high level of generality. Khan (“From Neurons to Knowledge: Understanding Forward Pass and Backpropagation Step by Step”) discloses in Pages 1 and 8, “[…] there are numerous articles and tutorials available on this topic […] we’ll dive into the core concepts of neural networks, specifically focusing on the step-by-step processes of the forward pass and backpropagation […] Our goal with backpropagation is to update the weights in the network so that they cause the actual output to be closer the target output, there by minimizing the error for each output.” Khan has recognized performing a backpropagation step to update parameters of the surrogate model to reduce the loss function is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
Regarding Claim 12,
Step 2A Prong 1:
See the rejection of Claim 8 above, which Claim 12 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein a training example is a sequence of tokens (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein a training example is a sequence of tokens (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
Regarding Claim 13,
Step 2A Prong 1:
divide the selected training set into another plurality of batches (mental process – dividing the selected training set into another plurality of batches may be performed manually by a user with the aid of pen and paper by observing/analyzing the selected training set and using a judgement to divide the selected training set into another plurality of batches. See MPEP 2106.04(a)(2)(III)(C).)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein the instructions to train the first machine-learned language model comprise instructions to repeatedly iterate between:
performing a forward pass step to generate another set of estimated predictions for a batch in the another plurality of batches (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
computing a loss function for the batch based on the another set of estimated predictions (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
performing a backpropagation step to update the parameters of the first machine-learned language model to reduce the loss function (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein the instructions to train the first machine-learned language model comprise instructions to repeatedly iterate between:
performing a forward pass step to generate another set of estimated predictions for a batch in the another plurality of batches (The additional elements are directed to performing a forward pass step to generate another set of estimated predictions for a batch in the another plurality of batches at a high level of generality. Khan (“From Neurons to Knowledge: Understanding Forward Pass and Backpropagation Step by Step”) discloses in Pages 1 and 4, “[…] there are numerous articles and tutorials available on this topic […] we’ll dive into the core concepts of neural networks, specifically focusing on the step-by-step processes of the forward pass and backpropagation […] The forward pass in a neural network is the process of taking input data, multiplying it by weights, applying activation functions, and passing it through the network’s layers to generate predictions or outputs.” Khan has recognized performing a forward pass step to generate another set of estimated predictions for a batch in the another plurality of batches is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
computing a loss function for the batch based on the another set of estimated predictions (The additional elements are directed to computing a loss function for the batch based on the another set of estimated predictions at a high level of generality. AI Wiki (“Accuracy and Loss”) discloses in Page 1, “Accuracy and Loss are the two most well-known and discussed metrics in machine learning. […] A loss function, also known as a cost function, takes into account the probabilities or uncertainty of a prediction based on how much the prediction varies from the true value. This gives us a more nuanced view into how well the model is performing.” AI Wiki has recognized computing a loss function for the batch based on the another set of estimated predictions is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
performing a backpropagation step to update the parameters of the first machine-learned language model to reduce the loss function (The additional elements are directed to performing a backpropagation step to update parameters of the first machine-learned language model to reduce the loss function at a high level of generality. Khan (“From Neurons to Knowledge: Understanding Forward Pass and Backpropagation Step by Step”) discloses in Pages 1 and 8, “[…] there are numerous articles and tutorials available on this topic […] we’ll dive into the core concepts of neural networks, specifically focusing on the step-by-step processes of the forward pass and backpropagation […] Our goal with backpropagation is to update the weights in the network so that they cause the actual output to be closer the target output, there by minimizing the error for each output.” Khan has recognized performing a backpropagation step to update parameters of the first machine-learned language model to reduce the loss function is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
Regarding Claim 14,
Step 2A Prong 1:
See the rejection of Claim 8 above, which Claim 14 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
obtain a total budget indicating a total number of training examples for training the parameters of the first machine-learned language model (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
wherein a number of training examples in the selected training set is based on the total budget (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
obtain a total budget indicating a total number of training examples for training the parameters of the first machine-learned language model (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
wherein a number of training examples in the selected training set is based on the total budget (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
Regarding Claim 15,
Step 1: Claim 15 is a system claim. Therefore, Claims 15-20 are directed to a machine.
Step 2A Prong 1:
divide the holdout set into a plurality of batches, each batch including a respective subset of training examples (mental process – dividing the holdout set into a plurality of batches, each batch including a respective subset of training examples may be performed manually by a user with the aid of pen and paper by observing/analyzing the holdout set and using a judgement to divide the holdout set into a plurality of batches. See MPEP 2106.04(a)(2)(III)(C).)
evaluate the trained surrogate model on the evaluation set by: (mental process – evaluating the trained surrogate model on the evaluation set may be performed manually by a user with the aid of pen and paper by observing/analyzing the trained surrogate model on the evaluation set and using a judgement to evaluate the trained surrogate model. See MPEP 2106.04(a)(2)(III)(C).)
generate a set of predictions [by applying the trained surrogate model to the evaluation set], the set of predictions including a prediction for a training example in the evaluation set (mental process – generating a set of predictions, the set of predictions including a prediction for a training example in the evaluation set may be performed manually by a user with the aid of pen and paper by observing/analyzing a training example in the evaluation set and using a judgement to generate a set of predictions. See MPEP 2106.04(a)(2)(III)(C).)
determine, for the training example in the evaluation set, a loss based on the set of predictions for the training example (mathematical concept – determining, for the training example in the evaluation set, a loss based on the set of predictions for the training example may be performed by mathematical process, calculating the difference between the predictions and the labels for the training examples. Spec. Paragraph [0053] discloses “a loss for each of the training examples, the loss indicating the difference between the predictions and the labels for the training examples.” See MPEP 2106.04(a)(2)(I)(C).)
generate a training set for the first machine-learned language model by selecting, from the evaluation set, a set of training examples with the loss within a predetermined range (mental process – generating a training set for the first machine-learned language model by selecting, from the evaluation set, a set of training examples with the loss within a predetermined range may be performed manually by a user with the aid of pen and paper by selecting a set of training examples with the loss within a predetermined range from the evaluation set and creating a training set based on the selection. See MPEP 2106.04(a)(2)(III)(C).)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
a computer processor (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
a non-transitory computer readable storage medium comprising stored instructions that when executed by the computer processor, cause the computer system to: (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
obtain a holdout set and an evaluation set, the holdout set including a first set of training examples and the evaluation set including a second set of training examples (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
train a surrogate model using the holdout set by iterating through the plurality of batches, wherein the surrogate model is a second machine-learned language model (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
by applying the trained surrogate model to the evaluation set (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
train parameters of the first machine-learned language model for a first epoch using the selected training set (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The additional elements recite generic computer elements and programs at a high-level of generality to perform the judicial exception as well as recitation of generic computer functionality such as a computer processor, a non-transitory computer readable storage medium, a surrogate model, a first machine-learned language model and a second machine-learned language model.
Additional Elements:
a computer processor (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
a non-transitory computer readable storage medium comprising stored instructions that when executed by the computer processor, cause the computer system to: (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
obtain a holdout set and an evaluation set, the holdout set including a first set of training examples and the evaluation set including a second set of training examples (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
train a surrogate model using the holdout set by iterating through the plurality of batches, wherein the surrogate model is a second machine-learned language model (The additional elements are directed to training a surrogate model using the holdout set by iterating through the plurality of batches, wherein the surrogate model is a second machine-learned language model at a high level of generality. Liu et al. (“Accelerate Mini-batch Machine Learning Training With Dynamic Batch Size Fitting”), hereinafter Liu, discloses in Section I, “The currently most widely used training algorithm for large-scale ML tasks is called Minibatch Stochastic Gradient Descent (MGD), which improves upon the simplest but powerful Stochastic Gradient Descent (SGD) [4] method. The basic idea of MGD is to train only a random slice of the dataset in each iteration, which is proved to have similar converge rates as GD on large datasets but is much more efficient [5]. The actual size of the random data slice that is trained in an iteration is determined by a hyperparameter called batch size.” Raschka (“Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning”) discloses in Section 3.5, “Given how the holdout method is most commonly used […]” Liu has recognized training a model by iterating through the plurality of batches (i.e., Minibatch Stochastic Gradient Descent (MGD)) is a well-understood, routine, and conventional activity previously known in the industry. Raschka has recognized using the holdout set is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
by applying the trained surrogate model to the evaluation set (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
train parameters of the first machine-learned language model for a first epoch using the selected training set (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
For the reasons above, Claim 15 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 16-20. The additional limitations of the dependent claims are addressed below.
Regarding Claim 16,
Step 2A Prong 1:
See the rejection of Claim 15 above, which Claim 16 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein the second machine-learned language model has a smaller number of parameters than a number of parameters of the first machine-learned language model (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein the second machine-learned language model has a smaller number of parameters than a number of parameters of the first machine-learned language model (The additional elements are directed to wherein the second machine-learned language model has a smaller number of parameters than a number of parameters of the first machine-learned language model at a high level of generality. Xu et al. (“Multiple-Stage Knowledge Distillation”), hereinafter Xu, discloses in Section I, “In the conventional KD method, the teacher network has a deeper layer and more parameters than those of the student network.” Xu has recognized wherein the second machine-learned model has a smaller number of parameters than a number of parameters of the first machine-learned model is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
Regarding Claim 17,
Step 2A Prong 1:
See the rejection of Claim 15 above, which Claim 17 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
for at least a second epoch, train the parameters of the first machine-learned language model using the selected training set (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
for at least a second epoch, train the parameters of the first machine-learned language model using the selected training set (The additional elements are directed to for at least a second epoch, training the parameters of the first machine-learned language model using the selected training set at a high level of generality. Raschka (“Practical Tips for Finetuning LLMs Using LoRA (Low-Rank Adaptation)”) discloses in Section 5, “In conventional deep learning, we often iterate over a training set multiple times -- iteration over the training set is called an epoch. It's common to run hundreds of training epochs when training convolutional neural networks […]” Raschka has recognized for at least a second epoch, training the parameters of the first machine-learned language model using the selected training set is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
Regarding Claim 18,
Step 2A Prong 1:
See the rejection of Claim 15 above, which Claim 18 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein the instructions to train the surrogate model comprise instructions to repeatedly iterate between:
performing a forward pass step to generate a set of estimated predictions for a batch (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
computing a loss function for the batch based on the set of estimated predictions (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
performing a backpropagation step to update parameters of the surrogate model to reduce the loss function (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein the instructions to train the surrogate model comprise instructions to repeatedly iterate between:
performing a forward pass step to generate a set of estimated predictions for a batch (The additional elements are directed to performing a forward pass step to generate a set of estimated predictions for a batch at a high level of generality. Khan (“From Neurons to Knowledge: Understanding Forward Pass and Backpropagation Step by Step”) discloses in Pages 1 and 4, “[…] there are numerous articles and tutorials available on this topic […] we’ll dive into the core concepts of neural networks, specifically focusing on the step-by-step processes of the forward pass and backpropagation […] The forward pass in a neural network is the process of taking input data, multiplying it by weights, applying activation functions, and passing it through the network’s layers to generate predictions or outputs.” Khan has recognized performing a forward pass step to generate a set of estimated predictions for a batch is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
computing a loss function for the batch based on the set of estimated predictions (The additional elements are directed to computing a loss function for the batch based on the set of estimated predictions at a high level of generality. AI Wiki (“Accuracy and Loss”) discloses in Page 1, “Accuracy and Loss are the two most well-known and discussed metrics in machine learning. […] A loss function, also known as a cost function, takes into account the probabilities or uncertainty of a prediction based on how much the prediction varies from the true value. This gives us a more nuanced view into how well the model is performing.” AI Wiki has recognized computing a loss function for the batch based on the set of estimated predictions is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
performing a backpropagation step to update parameters of the surrogate model to reduce the loss function (The additional elements are directed to performing a backpropagation step to update parameters of the surrogate model to reduce the loss function at a high level of generality. Khan (“From Neurons to Knowledge: Understanding Forward Pass and Backpropagation Step by Step”) discloses in Pages 1 and 8, “[…] there are numerous articles and tutorials available on this topic […] we’ll dive into the core concepts of neural networks, specifically focusing on the step-by-step processes of the forward pass and backpropagation […] Our goal with backpropagation is to update the weights in the network so that they cause the actual output to be closer the target output, there by minimizing the error for each output.” Khan has recognized performing a backpropagation step to update parameters of the surrogate model to reduce the loss function is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
Regarding Claim 19,
Step 2A Prong 1:
See the rejection of Claim 15 above, which Claim 19 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein a training example is a sequence of tokens (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein a training example is a sequence of tokens (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
Regarding Claim 20,
Step 2A Prong 1:
divide the selected training set into another plurality of batches (mental process – dividing the selected training set into another plurality of batches may be performed manually by a user with the aid of pen and paper by observing/analyzing the selected training set and using a judgement to divide the selected training set into another plurality of batches. See MPEP 2106.04(a)(2)(III)(C).)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein the instructions to train the first machine-learned language model comprise instructions to repeatedly iterate between:
performing a forward pass step to generate another set of estimated predictions for a batch in the another plurality of batches (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
computing a loss function for the batch based on the another set of estimated predictions (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
performing a backpropagation step to update the parameters of the first machine-learned language model to reduce the loss function (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein the instructions to train the first machine-learned language model comprise instructions to repeatedly iterate between:
performing a forward pass step to generate another set of estimated predictions for a batch in the another plurality of batches (The additional elements are directed to performing a forward pass step to generate another set of estimated predictions for a batch in the another plurality of batches at a high level of generality. Khan (“From Neurons to Knowledge: Understanding Forward Pass and Backpropagation Step by Step”) discloses in Pages 1 and 4, “[…] there are numerous articles and tutorials available on this topic […] we’ll dive into the core concepts of neural networks, specifically focusing on the step-by-step processes of the forward pass and backpropagation […] The forward pass in a neural network is the process of taking input data, multiplying it by weights, applying activation functions, and passing it through the network’s layers to generate predictions or outputs.” Khan has recognized performing a forward pass step to generate another set of estimated predictions for a batch in the another plurality of batches is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
computing a loss function for the batch based on the another set of estimated predictions (The additional elements are directed to computing a loss function for the batch based on the another set of estimated predictions at a high level of generality. AI Wiki (“Accuracy and Loss”) discloses in Page 1, “Accuracy and Loss are the two most well-known and discussed metrics in machine learning. […] A loss function, also known as a cost function, takes into account the probabilities or uncertainty of a prediction based on how much the prediction varies from the true value. This gives us a more nuanced view into how well the model is performing.” AI Wiki has recognized computing a loss function for the batch based on the another set of estimated predictions is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
performing a backpropagation step to update the parameters of the first machine-learned language model to reduce the loss function (The additional elements are directed to performing a backpropagation step to update parameters of the first machine-learned language model to reduce the loss function at a high level of generality. Khan (“From Neurons to Knowledge: Understanding Forward Pass and Backpropagation Step by Step”) discloses in Pages 1 and 8, “[…] there are numerous articles and tutorials available on this topic […] we’ll dive into the core concepts of neural networks, specifically focusing on the step-by-step processes of the forward pass and backpropagation […] Our goal with backpropagation is to update the weights in the network so that they cause the actual output to be closer the target output, there by minimizing the error for each output.” Khan has recognized performing a backpropagation step to update parameters of the first machine-learned language model to reduce the loss function is a well-understood, routine, and conventional activity previously known in the industry. See MPEP 2106.05(d).)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7 are rejected under 35 U.S.C. 103 as being unpatentable over Mindermann et al. (“Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt”) (hereinafter Mindermann), in view of Ouyang et al. (“Efficient NLP Model Finetuning via Multistage Data Filtering”) (hereinafter Ouyang), and further in view of Xie et al. (“DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining”) (hereinafter Xie).
Regarding Claim 1,
Mindermann teaches:
“obtaining a holdout set and an evaluation set, the holdout set including a first set of training examples and the evaluation set including a second set of training examples” (Mindermann, Section 4, “we train on half of the training set and use the other half as a holdout to train the
irreducible loss (IL) model.”; Examiner’s note: obtaining a holdout set (i.e., a holdout to train the irreducible loss (IL) model) and an evaluation set (i.e., training set being used to train), the holdout set including a first set of training examples (i.e., the other half of the training set as a holdout) and the evaluation set including a second set of training examples (i.e., half of the training set) is taught.)
“dividing the holdout set into a plurality of batches, each batch including a respective subset of training examples” (Mindermann, Section 2, “we uniformly pre-sample a larger batch Bt of size nB > nb. Then, we construct a smaller batch bt that consists of the top-ranking nb points in Bt ranked by a label-aware selection function S(xi; yi). We perform a gradient step to minimize a mini-batch loss L(yi; p(yi|xi; Ɵ)) summed over i є bt.”; Examiner’s note: dividing the holdout set into a plurality of batches (i.e., dividing the training set into larger batch Bt and smaller batch bt), each batch including a respective subset of training examples (i.e., the top-ranking nb points in a larger batch and performing a gradient step to minimize a mini-batch loss) is taught. With the training set divided into batches and the known batch-based training as taught in Mindermann, dividing the holdout set into batches is a predictable implementation.)
training a surrogate model using the holdout set by iterating through the plurality of batches (Mindermann, Section 3, “Our method still requires us to train a model on a holdout set, but a final approximation greatly reduces that cost. We can efficiently compute the IL with an “irreducible loss model" (IL model) that is smaller than the target model and has low accuracy (Approximation 3).”; Mindermann, Appendix B, “The irreducible loss models are trained on holdout sets (not test sets, see dataset description in main text). For each dataset, we select the irreducible loss model checkpoint from the epoch with lowest holdout loss on D (as opposed to highest accuracy)”; Mindermann, Algorithm 1,
PNG
media_image1.png
290
323
media_image1.png
Greyscale
; Examiner’s note: training a surrogate model using the holdout set (i.e., the irreducible loss models being trained on holdout sets) by iterating through the plurality of batches (i.e., iterating through a large batch Bt of size nB) is taught.)
“evaluating the trained surrogate model on the evaluation set by:” (Mindermann, Section 3, “In summary, selecting a point that minimizes the holdout loss in Eq. (1), for a model trained on Dt, can be approximated with the following easy-to-compute objective:
PNG
media_image2.png
95
321
media_image2.png
Greyscale
”; Examiner’s note: evaluating the trained surrogate model on the evaluation set by (i.e., evaluating the irreducible loss model with training loss and irreducible holdout loss for each candidate point) is taught.)
“generating a set of predictions by applying the trained surrogate model to the evaluation set, the set of predictions including a prediction for a training example in the evaluation set” (Mindermann, Section 3, “we use the language of Bayesian probability theory. We treat model parameters as a random variable with prior p(Ɵ) and infer a posterior p(Ɵ|Dt) using the already-seen training data Dt.”; Examiner’s note: generating a set of predictions (i.e., using the language of Bayesian probability theory) by applying the trained surrogate model to the evaluation set (i.e., applying irreducible loss model to each training set), the set of predictions including a prediction for a training example in the evaluation set (i.e., inferring a posterior p(Ɵ|Dt) using the already-seen training data Dt) is taught.)
“determining, for the training example in the evaluation set, a loss based on the set of predictions for the training example” (Mindermann, Section 3, “we use the language of Bayesian probability theory. We treat model parameters as a random variable with prior p(Ɵ) and infer a posterior p(Ɵ|Dt) using the already-seen training data Dt. The model has a predictive distribution p(y|x,Dt) = ʃƟp(y|x,Ɵ) p(Ɵ|Dt)dƟ.”; Mindermann, Algorithm 1,
PNG
media_image1.png
290
323
media_image1.png
Greyscale
; Examiner’s note: determining, for the training example in the evaluation set, a loss based on the set of predictions for the training example (i.e. IrreducibleLoss[i] ← L[yi|xi;Dho] for (xi,yi) in training set) is taught.)
Mindermann does not explicitly teach:
“A method for training a first machine-learned language model, comprising:” (preamble)
wherein the surrogate model is a second machine-learned language model
“generating a training set for the first machine-learned language model by selecting, from the evaluation set, a set of training examples with the loss within a predetermined range”
“training parameters of the first machine-learned language model for a first epoch using the selected training set”
Ouyang teaches:
“A method for training a first machine-learned language model, comprising:” (preamble)
“generating a training set for the first machine-learned language model by selecting, from the evaluation set, a set of training examples with the loss within a predetermined range” (Ouyang, Section 1 and Section 6, “[…] large NLP datasets are redundant for model finetuning, and many training examples can be filtered with a low impact on accuracy. We proposed a simple, automatic mechanism for filtering training data: using an automatic loss threshold for skipping backward passes and a lightweight meta predictor for further skipping forward passes […] We automatically maintain a loss threshold from model losses, then train and query a simple predictor to skip both forward and backward passes. So that unnecessary data instances will be filtered out and we achieve great accuracy-efficiency tradeoff.”; Examiner’s note: generating a training set (i.e., filtered training set with an automatic loss threshold) for the first machine-learned language model (i.e., NLP model) by selecting, from the evaluation set, a set of training examples (i.e., filtering out unnecessary data instances from the dataset) with the loss within a predetermined range (i.e., an automatic loss threshold) is taught.)
“training parameters of the first machine-learned language model for a first epoch using the selected training set” (Ouyang, Section 1, “Given a model and training data, the algorithm automatically derives a proper loss threshold Llow […] The algorithm runs training as a multistage process: each stage receives supervision from earlier stages but is more efficient than the former. After training starts: The first stage derives Llow that adapts to the model and the training data. This stage runs both forward and backward passes on training examples. The second stage uses the derived Llow to filter backward passes. With the filtering decisions, it trains a meta predictor using simple linguistic features. This meta predictor will later decide if the given examples are worth training. The third stage queries the meta predictor to filter both forward and backward passes.”; Examiner’s note: training parameters of the first machine-learned language model for a first epoch using the selected training set (i.e., Given training data, the algorithm runs training as a multistage process including loss threshold being derived, forward and backward passes on training examples, using the loss threshold to filter backward passes, training a meta predictor and querying the meta predictor to filter forward and backward passes) is taught.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the prioritized training on points that are learnable, worth learning, and not yet learnt in Mindermann, and the efficient NLP model finetuning via multistage data filtering as taught in Ouyang. Mindermann teaches obtaining a holdout set and an evaluation set, dividing the holdout set into a plurality of batches, training a surrogate model using the holdout set by iterating through the plurality of batches, evaluating the trained surrogate model on the evaluation set, generating a set of predictions by applying the trained surrogate model to the evaluation set, and determining a loss based on the set of predictions for the training example. Ouyang teaches generating a training set for the first machine-learned language model by selecting a set of training examples with the loss within a predetermined range and training parameters of the first machine-learned language model using the selected training set. One of ordinary skill would have motivation to combine Mindermann and Ouyang to “deliver[] both high accuracy and high efficiency (i.e. low training time)” (Ouyang, Section 5.2).
Xie teaches:
wherein the surrogate model is a second machine-learned language model (Xie, Section 2, “[…] we define DoReMi, an algorithm for using a small proxy model to optimize the domain weights of a language modeling dataset, which then improves the training of a large model.”; Examiner’s note: wherein the surrogate model (i.e., a small proxy model) is a second machine-learned language model (i.e., a small language model) is taught.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the prioritized training on points that are learnable, worth learning, and not yet learnt in Mindermann, the efficient NLP model finetuning via multistage data filtering in Ouyang, and the optimizing data mixtures speeds up language model pretraining as taught in Xie. Mindermann teaches obtaining a holdout set and an evaluation set, dividing the holdout set into a plurality of batches, training a surrogate model using the holdout set by iterating through the plurality of batches, evaluating the trained surrogate model on the evaluation set, generating a set of predictions by applying the trained surrogate model to the evaluation set, and determining a loss based on the set of predictions for the training example. Ouyang teaches generating a training set for the first machine-learned language model by selecting a set of training examples with the loss within a predetermined range and training parameters of the first machine-learned language model using the selected training set. Xie teaches that the surrogate model is a second machine-learned language model. One of ordinary skill would have motivation to combine Mindermann, Ouyang and Xie to “consistently improve[] LM training when varying the sizes of the proxy model and the main model trained with optimized domain weights” (Xie, Section 1).
Regarding Claim 2,
The combination of Mindermann, Ouyang and Xie teaches:
“The method of claim 1,” (preamble)
“wherein the second machine-learned language model has a smaller number of parameters than a number of parameters of the first machine-learned language model” (Xie, Section 3, “[…] we use DoReMi domain weights optimized with a 280M-parameter proxy model to train a 8B-parameter main model (30x larger).”; Examiner’s note: wherein the second machine-learned language model has a smaller number of parameters (i.e., a 280M-parameter proxy model) than a number of parameters of the first machine-learned language model (i.e., a 8B-parameter main model) is taught.)
The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein.
Regarding Claim 3,
The combination of Mindermann, Ouyang and Xie teaches:
“The method of claim 1, further comprising” (preamble)
“for at least a second epoch, training the parameters of the first machine-learned language model using the selected training set” (Mindermann, Section 4, “RHO-LOSS reaches target accuracies in fewer epochs than uniform selection on all datasets (Table 2) […] RHO-LOSS reaches the highest accuracy that uniform selection achieves during 50 epochs in just 2 epochs and improves final accuracy (72% vs 70%). Notably, this was possible even though the IL model we used has low accuracy (62.2%) and was trained on ca.”; Mindermann, Table 2,
PNG
media_image3.png
384
662
media_image3.png
Greyscale
; Examiner’s note: for at least a second epoch (i.e., more than or equal to 2 epochs), training the parameters of the first machine-learned language model using the selected training set (i.e., training IL model using the selected datasets such as Clothing-1M, CIFAR10*, CIFAR100*, etc.) is taught.)
The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein.
Regarding Claim 4,
The combination of Mindermann, Ouyang and Xie teaches:
“The method of claim 1,” (preamble)
“wherein training the surrogate model comprises repeatedly iterating between:” (Xie, Section 2, “[…] we define DoReMi, an algorithm for using a small proxy model to optimize the domain weights of a language modeling dataset, which then improves the training of a large model […] We extend DoReMi by running it for multiple rounds, setting the reference domain weights αref for the next round to be ᾶ from the previous round. We call this iterated DoReMi.”; Examiner’s note: wherein the surrogate model (i.e., a small proxy model) comprises repeatedly iterating between (i.e., iterated DoReMi which includes multiple rounds of iterations) is taught.)
“performing a forward pass step to generate a set of estimated predictions for a batch” (Xie, Section 3.1, “All models use a batch size of 512 and maximum token length of 1024. The proxy and reference models have 280M parameters […] We train two 280M models (the reference and proxy models) to optimize the domain weights […] All FLOPs come from standard forward and backward passes.”; Examiner’s note: performing a forward step (i.e., standard forward pass) to generate a set of estimated predictions for a batch (i.e., training the reference and proxy models using a batch size of 512) is taught.)
“computing a loss function for the batch based on the set of estimated predictions” (Xie, Equation 1,
PNG
media_image4.png
52
514
media_image4.png
Greyscale
and Section 2, “where the losses ℓθ(x) = −log pθ(x) and ℓref(x) = −log pref(x) are the negative log-likelihoods of the proxy and reference models respectively in this paper, and |x| is the number of tokens in an example x.”; Examiner’s note: computing a loss function (i.e., equation (1) teaches computing a loss function) for the batch on the set of estimated predictions (i.e., for the batch using the proxy and reference models) is taught.)
“performing a backpropagation step to update parameters of the surrogate model to reduce the loss function” (Xie, Section 2, “[…] we update the proxy model for the objective L(θt−1, αt) using a standard optimizer such as Adam (Kingma and Ba, 2015) or Adafactor (Shazeer and Stern, 2018).”; Xie, Section 3.1, “All models use a batch size of 512 and maximum token length of 1024. The proxy and reference models have 280M parameters […] We train two 280M models (the reference and proxy models) to optimize the domain weights […] All FLOPs come from standard forward and backward passes.”; Examiner’s note: performing a backpropagation step (i.e., standard backward pass) to update parameters of the surrogate model to reduce the loss function (i.e., updating the proxy model using a standard optimizer such as Adam or Adafactor to reduce the loss) is taught.)
The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein.
Regarding Claim 5,
The combination of Mindermann, Ouyang and Xie teaches:
“The method of claim 1,” (preamble)
“wherein a training example is a sequence of tokens” (Xie, Section 3.1, “We train Transformer (Vaswani et al., 2017) decoder-only LMs with the standard next-token language modeling loss […] All models use a batch size of 512 and maximum token length of 1024.”; Examiner’s note: wherein a training example is a sequence of tokens (i.e., training Transformer decoder-only LMs with the standard next-token language modeling loss and all models using maximum token length of 1024) is taught.)
The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein.
Regarding Claim 6,
The combination of Mindermann, Ouyang and Xie teaches:
“The method of claim 1,” (preamble)
“dividing the selected training set into another plurality of batches” (Mindermann, Section 2, “At each training step t, we load a batch bt of size nb from D […] we uniformly pre-sample a larger batch Bt of size nB > nb. Then, we construct a smaller batch bt that consists of the top-ranking nb points in Bt ranked by a label-aware selection function S(xi; yi). We perform a gradient step to minimize a mini-batch loss L(yi; p(yi|xi; Ɵ)) summed over i є bt.”; Examiner’s note: dividing the selected training set into another plurality of batches (i.e., dividing the selected training set into larger batch Bt and smaller batch bt) is taught.)
“wherein training the first machine-learned language model comprises repeatedly iterating between:” (Ouyang, Section 1, “Given a model and training data, the algorithm automatically derives a proper loss threshold Llow […] The algorithm runs training as a multistage process: each stage receives supervision from earlier stages but is more efficient than the former. After training starts: The first stage derives Llow that adapts to the model and the training data. This stage runs both forward and backward passes on training examples. The second stage uses the derived Llow to filter backward passes. With the filtering decisions, it trains a meta predictor using simple linguistic features. This meta predictor will later decide if the given examples are worth training. The third stage queries the meta predictor to filter both forward and backward passes.”; Examiner’s note: wherein training the first machine-learned language model (i.e., Given training data, the algorithm runs training as a multistage process including loss threshold being derived, forward and backward passes on training examples, using the loss threshold to filter backward passes, training a meta predictor and querying the meta predictor to filter forward and backward passes) comprises repeatedly iterating between (i.e., repeated stages of training) is taught.)
“performing a forward pass step to generate another set of estimated predictions for a batch in the another plurality of batches” (Ouyang, Section 1, “Given a model and training data, the algorithm automatically derives a proper loss threshold Llow […] The algorithm runs training as a multistage process: each stage receives supervision from earlier stages but is more efficient than the former. After training starts: The first stage derives Llow that adapts to the model and the training data. This stage runs both forward and backward passes on training examples. The second stage uses the derived Llow to filter backward passes. With the filtering decisions, it trains a meta predictor using simple linguistic features. This meta predictor will later decide if the given examples are worth training. The third stage queries the meta predictor to filter both forward and backward passes.”; Examiner’s note: performing a forward pass step (i.e., forward pass on training examples) to generate another set of estimated predictions (i.e., training a meta predictor using simple linguistic features) for a batch in the another plurality of batches (i.e., training examples) is taught.)
“computing a loss function for the batch based on the another set of estimated predictions” (Ouyang, Section 1, “Given a model and training data, the algorithm automatically derives a proper loss threshold Llow […] The algorithm runs training as a multistage process: each stage receives supervision from earlier stages but is more efficient than the former. After training starts: The first stage derives Llow that adapts to the model and the training data. This stage runs both forward and backward passes on training examples. The second stage uses the derived Llow to filter backward passes. With the filtering decisions, it trains a meta predictor using simple linguistic features. This meta predictor will later decide if the given examples are worth training. The third stage queries the meta predictor to filter both forward and backward passes.”; Examiner’s note: computing a loss function for the batch (i.e., deriving a proper loss threshold that adapts to the model and the training data) on the another set of estimated predictions (i.e., training a meta predictor using simple linguistic features) is taught.)
“performing a backpropagation step to update the parameters of the first machine-learned language model to reduce the loss function” (Ouyang, Section 1, “Given a model and training data, the algorithm automatically derives a proper loss threshold Llow […] The algorithm runs training as a multistage process: each stage receives supervision from earlier stages but is more efficient than the former. After training starts: The first stage derives Llow that adapts to the model and the training data. This stage runs both forward and backward passes on training examples. The second stage uses the derived Llow to filter backward passes. With the filtering decisions, it trains a meta predictor using simple linguistic features. This meta predictor will later decide if the given examples are worth training. The third stage queries the meta predictor to filter both forward and backward passes.”; Examiner’s note: performing a backpropagation step (i.e., backward pass on training examples) to update the parameters of the first machine-learned language model (i.e., training the meta predictor with the filtering decisions) to reduce the loss function (i.e., using the derived loss threshold to filter backward passes) is taught.)
The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein.
Regarding Claim 7,
The combination of Mindermann, Ouyang and Xie teaches:
“The method of claim 1, further comprising” (preamble)
“obtaining a total budget indicating a total number of training examples for training the parameters of the first machine-learned language model, and wherein a number of training examples in the selected training set is based on the total budget” (Mindermann, Section 1, “We propose a selection function grounded in probabilistic modelling that quantifies by how much each point would reduce the generalization loss if we were to train on it […] To approximate optimal selection, we derive an efficient and easy to implement selection function: the reducible holdout loss.”; Mindermann, Section 3, “[…] we show how to (approximately) find the points that would most reduce the holdout loss if we were to train the current model on them […]”; Mindermann, Section 4.2, “RHO-LOSS requires training an IL model on a holdout set, which poses additional costs. Here, we show how to minimize these costs and amortize them across many training runs of target models. The same experiments also show the robustness of RHO-LOSS across architectures and hyperparameter settings. To fit our computational budget, we perform these experiments on moderate-sized clean benchmark datasets although RHO-LOSS produces greater speedups on noisy or redundant web-scraped data”; Examiner’s note: obtaining a total budget indicating a total number of training examples (i.e., RHO-LOSS computational budget for the total selected examples) for training the parameters of the first machine-learned language model (i.e., training the target models), and wherein a number of training examples in the selected training set is based on the total budget (i.e., finding the points that would most reduce the holdout loss and selection function, the reducible holdout loss, to reduce the generalization loss) is taught.)
The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein.
Claims 8-20 are rejected under 35 U.S.C. 103 as being unpatentable over Mindermann et al. (“Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt”) (hereinafter Mindermann), in view of Ouyang et al. (“Efficient NLP Model Finetuning via Multistage Data Filtering”) (hereinafter Ouyang), and further in view of Xie et al. (“DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining”) (hereinafter Xie), and further in view of Bhide et al. (US 20220083899 A1) (hereinafter Bhide).
Regarding Claim 8,
Mindermann teaches:
“obtain a holdout set and an evaluation set, the holdout set including a first set of training examples and the evaluation set including a second set of training examples” (Mindermann, Section 4, “we train on half of the training set and use the other half as a holdout to train the
irreducible loss (IL) model.”; Examiner’s note: obtaining a holdout set (i.e., a holdout to train the irreducible loss (IL) model) and an evaluation set (i.e., training set being used to train), the holdout set including a first set of training examples (i.e., the other half of the training set as a holdout) and the evaluation set including a second set of training examples (i.e., half of the training set) is taught.)
“divide the holdout set into a plurality of batches, each batch including a respective subset of training examples” (Mindermann, Section 2, “we uniformly pre-sample a larger batch Bt of size nB > nb. Then, we construct a smaller batch bt that consists of the top-ranking nb points in Bt ranked by a label-aware selection function S(xi; yi). We perform a gradient step to minimize a mini-batch loss L(yi; p(yi|xi; Ɵ)) summed over i є bt.”; Examiner’s note: dividing the holdout set into a plurality of batches (i.e., dividing the training set into larger batch Bt and smaller batch bt), each batch including a respective subset of training examples (i.e., the top-ranking nb points in a larger batch and performing a gradient step to minimize a mini-batch loss) is taught. With the training set divided into batches and the known batch-based training as taught in Mindermann, dividing the holdout set into batches is a predictable implementation.)
train a surrogate model using the holdout set by iterating through the plurality of batches (Mindermann, Section 3, “Our method still requires us to train a model on a holdout set, but a final approximation greatly reduces that cost. We can efficiently compute the IL with an “irreducible loss model" (IL model) that is smaller than the target model and has low accuracy (Approximation 3).”; Mindermann, Appendix B, “The irreducible loss models are trained on holdout sets (not test sets, see dataset description in main text). For each dataset, we select the irreducible loss model checkpoint from the epoch with lowest holdout loss on D (as opposed to highest accuracy)”; Mindermann, Algorithm 1,
PNG
media_image1.png
290
323
media_image1.png
Greyscale
; Examiner’s note: training a surrogate model using the holdout set (i.e., the irreducible loss models being trained on holdout sets) by iterating through the plurality of batches (i.e., iterating through a large batch Bt of size nB) is taught.)
“evaluate the trained surrogate model on the evaluation set by:” (Mindermann, Section 3, “In summary, selecting a point that minimizes the holdout loss in Eq. (1), for a model trained on Dt, can be approximated with the following easy-to-compute objective:
PNG
media_image2.png
95
321
media_image2.png
Greyscale
”; Examiner’s note: evaluating the trained surrogate model on the evaluation set by (i.e., evaluating the irreducible loss model with training loss and irreducible holdout loss for each candidate point) is taught.)
“generate a set of predictions by applying the trained surrogate model to the evaluation set, the set of predictions including a prediction for a training example in the evaluation set” (Mindermann, Section 3, “we use the language of Bayesian probability theory. We treat model parameters as a random variable with prior p(Ɵ) and infer a posterior p(Ɵ|Dt) using the already-seen training data Dt.”; Examiner’s note: generating a set of predictions (i.e., using the language of Bayesian probability theory) by applying the trained surrogate model to the evaluation set (i.e., applying irreducible loss model to each training set), the set of predictions including a prediction for a training example in the evaluation set (i.e., inferring a posterior p(Ɵ|Dt) using the already-seen training data Dt) is taught.)
“determine, for the training example in the evaluation set, a loss based on the set of predictions for the training example” (Mindermann, Section 3, “we use the language of Bayesian probability theory. We treat model parameters as a random variable with prior p(Ɵ) and infer a posterior p(Ɵ|Dt) using the already-seen training data Dt. The model has a predictive distribution p(y|x,Dt) = ʃƟp(y|x,Ɵ) p(Ɵ|Dt)dƟ.”; Mindermann, Algorithm 1,
PNG
media_image1.png
290
323
media_image1.png
Greyscale
; Examiner’s note: determining, for the training example in the evaluation set, a loss based on the set of predictions for the training example (i.e. IrreducibleLoss[i] ← L[yi|xi;Dho] for (xi,yi) in training set) is taught.)
Mindermann does not explicitly teach:
“A non-transitory computer readable storage medium comprising stored program code, the program code comprising instructions, the instructions when executed causes a processor system to:” (preamble)
wherein the surrogate model is a second machine-learned language model
“generate a training set for the first machine-learned language model by selecting, from the evaluation set, a set of training examples with the loss within a predetermined range”
“train parameters of the first machine-learned language model for a first epoch using the selected training set”
Ouyang teaches:
“generate a training set for the first machine-learned language model by selecting, from the evaluation set, a set of training examples with the loss within a predetermined range” (Ouyang, Section 1 and Section 6, “[…] large NLP datasets are redundant for model finetuning, and many training examples can be filtered with a low impact on accuracy. We proposed a simple, automatic mechanism for filtering training data: using an automatic loss threshold for skipping backward passes and a lightweight meta predictor for further skipping forward passes […] We automatically maintain a loss threshold from model losses, then train and query a simple predictor to skip both forward and backward passes. So that unnecessary data instances will be filtered out and we achieve great accuracy-efficiency tradeoff.”; Examiner’s note: generating a training set (i.e., filtered training set with an automatic loss threshold) for the first machine-learned language model (i.e., NLP model) by selecting, from the evaluation set, a set of training examples (i.e., filtering out unnecessary data instances from the dataset) with the loss within a predetermined range (i.e., an automatic loss threshold) is taught.)
“train parameters of the first machine-learned language model for a first epoch using the selected training set” (Ouyang, Section 1, “Given a model and training data, the algorithm automatically derives a proper loss threshold Llow […] The algorithm runs training as a multistage process: each stage receives supervision from earlier stages but is more efficient than the former. After training starts: The first stage derives Llow that adapts to the model and the training data. This stage runs both forward and backward passes on training examples. The second stage uses the derived Llow to filter backward passes. With the filtering decisions, it trains a meta predictor using simple linguistic features. This meta predictor will later decide if the given examples are worth training. The third stage queries the meta predictor to filter both forward and backward passes.”; Examiner’s note: training parameters of the first machine-learned language model for a first epoch using the selected training set (i.e., Given training data, the algorithm runs training as a multistage process including loss threshold being derived, forward and backward passes on training examples, using the loss threshold to filter backward passes, training a meta predictor and querying the meta predictor to filter forward and backward passes) is taught.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the prioritized training on points that are learnable, worth learning, and not yet learnt in Mindermann, and the efficient NLP model finetuning via multistage data filtering as taught in Ouyang. Mindermann teaches obtaining a holdout set and an evaluation set, dividing the holdout set into a plurality of batches, training a surrogate model using the holdout set by iterating through the plurality of batches, evaluating the trained surrogate model on the evaluation set, generating a set of predictions by applying the trained surrogate model to the evaluation set, and determining a loss based on the set of predictions for the training example. Ouyang teaches generating a training set for the first machine-learned language model by selecting a set of training examples with the loss within a predetermined range and training parameters of the first machine-learned language model using the selected training set. One of ordinary skill would have motivation to combine Mindermann and Ouyang to “deliver[] both high accuracy and high efficiency (i.e. low training time)” (Ouyang, Section 5.2).
Xie teaches:
wherein the surrogate model is a second machine-learned language model (Xie, Section 2, “[…] we define DoReMi, an algorithm for using a small proxy model to optimize the domain weights of a language modeling dataset, which then improves the training of a large model.”; Examiner’s note: wherein the surrogate model (i.e., a small proxy model) is a second machine-learned language model (i.e., a small language model) is taught.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the prioritized training on points that are learnable, worth learning, and not yet learnt in Mindermann, the efficient NLP model finetuning via multistage data filtering in Ouyang, and the optimizing data mixtures speeds up language model pretraining as taught in Xie. Mindermann teaches obtaining a holdout set and an evaluation set, dividing the holdout set into a plurality of batches, training a surrogate model using the holdout set by iterating through the plurality of batches, evaluating the trained surrogate model on the evaluation set, generating a set of predictions by applying the trained surrogate model to the evaluation set, and determining a loss based on the set of predictions for the training example. Ouyang teaches generating a training set for the first machine-learned language model by selecting a set of training examples with the loss within a predetermined range and training parameters of the first machine-learned language model using the selected training set. Xie teaches that the surrogate model is a second machine-learned language model. One of ordinary skill would have motivation to combine Mindermann, Ouyang and Xie to “consistently improve[] LM training when varying the sizes of the proxy model and the main model trained with optimized domain weights” (Xie, Section 1).
Bhide teaches:
““A non-transitory computer readable storage medium comprising stored program code, the program code comprising instructions, the instructions when executed causes a processor system to:” (Bhide, Paragraph [0122], “The computer program product may include a computer readable storage medium with program instructions which are executable by a processor to cause a processor to perform a certain function.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the prioritized training on points that are learnable, worth learning, and not yet learnt in Mindermann, the efficient NLP model finetuning via multistage data filtering in Ouyang, the optimizing data mixtures speeds up language model pretraining in Xie, and the validation of AI models using holdout sets as taught in Bhide. Mindermann teaches obtaining a holdout set and an evaluation set, dividing the holdout set into a plurality of batches, training a surrogate model using the holdout set by iterating through the plurality of batches, evaluating the trained surrogate model on the evaluation set, generating a set of predictions by applying the trained surrogate model to the evaluation set, and determining a loss based on the set of predictions for the training example. Ouyang teaches generating a training set for the first machine-learned language model by selecting a set of training examples with the loss within a predetermined range and training parameters of the first machine-learned language model using the selected training set. Xie teaches that the surrogate model is a second machine-learned language model. Bhide teaches a non-transitory computer readable storage medium comprising stored program code, the program code comprising instructions, executed by a processor. One of ordinary skill would have motivation to combine Mindermann, Ouyang, Xie and Bhide to “automate[] part or all of an AI model validation” (Bhide, Paragraph [0125]).
Regarding Claim 9,
The combination of Mindermann, Ouyang, Xie and Bhide teaches:
“The non-transitory computer readable storage medium of claim 8,” (preamble)
“wherein the second machine-learned language model has a smaller number of parameters than a number of parameters of the first machine-learned language model” (Xie, Section 3, “[…] we use DoReMi domain weights optimized with a 280M-parameter proxy model to train a 8B-parameter main model (30x larger).”; Examiner’s note: wherein the second machine-learned language model has a smaller number of parameters (i.e., a 280M-parameter proxy model) than a number of parameters of the first machine-learned language model (i.e., a 8B-parameter main model) is taught.)
The reasons of obviousness have been noted in the rejection of Claim 8 above and applicable herein.
Regarding Claim 10,
The combination of Mindermann, Ouyang, Xie and Bhide teaches:
“The non-transitory computer readable storage medium of claim 8, wherein the instructions further comprise instructions to,” (preamble)
“for at least a second epoch, train the parameters of the first machine-learned language model using the selected training set” (Mindermann, Section 4, “RHO-LOSS reaches target accuracies in fewer epochs than uniform selection on all datasets (Table 2) […] RHO-LOSS reaches the highest accuracy that uniform selection achieves during 50 epochs in just 2 epochs and improves final accuracy (72% vs 70%). Notably, this was possible even though the IL model we used has low accuracy (62.2%) and was trained on ca.”; Mindermann, Table 2,
PNG
media_image3.png
384
662
media_image3.png
Greyscale
; Examiner’s note: for at least a second epoch (i.e., more than or equal to 2 epochs), training the parameters of the first machine-learned language model using the selected training set (i.e., training IL model using the selected datasets such as Clothing-1M, CIFAR10*, CIFAR100*, etc.) is taught.)
The reasons of obviousness have been noted in the rejection of Claim 8 above and applicable herein.
Regarding Claim 11,
The combination of Mindermann, Ouyang, Xie and Bhide teaches:
“The non-transitory computer readable storage medium of claim 8,” (preamble)
“wherein the instructions to train the surrogate model comprise instructions to repeatedly iterate between:” (Xie, Section 2, “[…] we define DoReMi, an algorithm for using a small proxy model to optimize the domain weights of a language modeling dataset, which then improves the training of a large model […] We extend DoReMi by running it for multiple rounds, setting the reference domain weights αref for the next round to be ᾶ from the previous round. We call this iterated DoReMi.”; Examiner’s note: wherein the instructions to train the surrogate model (i.e., an algorithm for training a small proxy model) comprises repeatedly iterating between (i.e., iterated DoReMi which includes multiple rounds of iterations) is taught.)
“performing a forward pass step to generate a set of estimated predictions for a batch” (Xie, Section 3.1, “All models use a batch size of 512 and maximum token length of 1024. The proxy and reference models have 280M parameters […] We train two 280M models (the reference and proxy models) to optimize the domain weights […] All FLOPs come from standard forward and backward passes.”; Examiner’s note: performing a forward step (i.e., standard forward pass) to generate a set of estimated predictions for a batch (i.e., training the reference and proxy models using a batch size of 512) is taught.)
“computing a loss function for the batch based on the set of estimated predictions” (Xie, Equation 1,
PNG
media_image4.png
52
514
media_image4.png
Greyscale
and Section 2, “where the losses ℓθ(x) = −log pθ(x) and ℓref(x) = −log pref(x) are the negative log-likelihoods of the proxy and reference models respectively in this paper, and |x| is the number of tokens in an example x.”; Examiner’s note: computing a loss function (i.e., equation (1) teaches computing a loss function) for the batch on the set of estimated predictions (i.e., for the batch using the proxy and reference models) is taught.)
“performing a backpropagation step to update parameters of the surrogate model to reduce the loss function” (Xie, Section 2, “[…] we update the proxy model for the objective L(θt−1, αt) using a standard optimizer such as Adam (Kingma and Ba, 2015) or Adafactor (Shazeer and Stern, 2018).”; Xie, Section 3.1, “All models use a batch size of 512 and maximum token length of 1024. The proxy and reference models have 280M parameters […] We train two 280M models (the reference and proxy models) to optimize the domain weights […] All FLOPs come from standard forward and backward passes.”; Examiner’s note: performing a backpropagation step (i.e., standard backward pass) to update parameters of the surrogate model to reduce the loss function (i.e., updating the proxy model using a standard optimizer such as Adam or Adafactor to reduce the loss) is taught.)
The reasons of obviousness have been noted in the rejection of Claim 8 above and applicable herein.
Regarding Claim 12,
The combination of Mindermann, Ouyang, Xie and Bhide teaches:
“The non-transitory computer readable storage medium of claim 8,” (preamble)
“wherein a training example is a sequence of tokens” (Xie, Section 3.1, “We train Transformer (Vaswani et al., 2017) decoder-only LMs with the standard next-token language modeling loss […] All models use a batch size of 512 and maximum token length of 1024.”; Examiner’s note: wherein a training example is a sequence of tokens (i.e., training Transformer decoder-only LMs with the standard next-token language modeling loss and all models using maximum token length of 1024) is taught.)
The reasons of obviousness have been noted in the rejection of Claim 8 above and applicable herein.
Regarding Claim 13,
The combination of Mindermann, Ouyang, Xie and Bhide teaches:
“The non-transitory computer readable storage medium of claim 8, wherein the instructions further comprise instructions to:” (preamble)
“divide the selected training set into another plurality of batches” (Mindermann, Section 2, “At each training step t, we load a batch bt of size nb from D […] we uniformly pre-sample a larger batch Bt of size nB > nb. Then, we construct a smaller batch bt that consists of the top-ranking nb points in Bt ranked by a label-aware selection function S(xi; yi). We perform a gradient step to minimize a mini-batch loss L(yi; p(yi|xi; Ɵ)) summed over i є bt.”; Examiner’s note: dividing the selected training set into another plurality of batches (i.e., dividing the selected training set into larger batch Bt and smaller batch bt) is taught.)
“wherein the instructions to train the first machine-learned language model comprise instructions to repeatedly iterate between:” (Ouyang, Section 1, “Given a model and training data, the algorithm automatically derives a proper loss threshold Llow […] The algorithm runs training as a multistage process: each stage receives supervision from earlier stages but is more efficient than the former. After training starts: The first stage derives Llow that adapts to the model and the training data. This stage runs both forward and backward passes on training examples. The second stage uses the derived Llow to filter backward passes. With the filtering decisions, it trains a meta predictor using simple linguistic features. This meta predictor will later decide if the given examples are worth training. The third stage queries the meta predictor to filter both forward and backward passes.”; Examiner’s note: wherein the instructions to train the first machine-learned language model (i.e., Given training data, the algorithm runs training as a multistage process including loss threshold being derived, forward and backward passes on training examples, using the loss threshold to filter backward passes, training a meta predictor and querying the meta predictor to filter forward and backward passes) comprise instructions to repeatedly iterate between (i.e., repeated stages of training) is taught.)
“performing a forward pass step to generate another set of estimated predictions for a batch in the another plurality of batches” (Ouyang, Section 1, “Given a model and training data, the algorithm automatically derives a proper loss threshold Llow […] The algorithm runs training as a multistage process: each stage receives supervision from earlier stages but is more efficient than the former. After training starts: The first stage derives Llow that adapts to the model and the training data. This stage runs both forward and backward passes on training examples. The second stage uses the derived Llow to filter backward passes. With the filtering decisions, it trains a meta predictor using simple linguistic features. This meta predictor will later decide if the given examples are worth training. The third stage queries the meta predictor to filter both forward and backward passes.”; Examiner’s note: performing a forward pass step (i.e., forward pass on training examples) to generate another set of estimated predictions (i.e., training a meta predictor using simple linguistic features) for a batch in the another plurality of batches (i.e., training examples) is taught.)
“computing a loss function for the batch based on the another set of estimated predictions” (Ouyang, Section 1, “Given a model and training data, the algorithm automatically derives a proper loss threshold Llow […] The algorithm runs training as a multistage process: each stage receives supervision from earlier stages but is more efficient than the former. After training starts: The first stage derives Llow that adapts to the model and the training data. This stage runs both forward and backward passes on training examples. The second stage uses the derived Llow to filter backward passes. With the filtering decisions, it trains a meta predictor using simple linguistic features. This meta predictor will later decide if the given examples are worth training. The third stage queries the meta predictor to filter both forward and backward passes.”; Examiner’s note: computing a loss function for the batch (i.e., deriving a proper loss threshold that adapts to the model and the training data) on the another set of estimated predictions (i.e., training a meta predictor using simple linguistic features) is taught.)
“performing a backpropagation step to update the parameters of the first machine-learned language model to reduce the loss function” (Ouyang, Section 1, “Given a model and training data, the algorithm automatically derives a proper loss threshold Llow […] The algorithm runs training as a multistage process: each stage receives supervision from earlier stages but is more efficient than the former. After training starts: The first stage derives Llow that adapts to the model and the training data. This stage runs both forward and backward passes on training examples. The second stage uses the derived Llow to filter backward passes. With the filtering decisions, it trains a meta predictor using simple linguistic features. This meta predictor will later decide if the given examples are worth training. The third stage queries the meta predictor to filter both forward and backward passes.”; Examiner’s note: performing a backpropagation step (i.e., backward pass on training examples) to update the parameters of the first machine-learned language model (i.e., training the meta predictor with the filtering decisions) to reduce the loss function (i.e., using the derived loss threshold to filter backward passes) is taught.)
The reasons of obviousness have been noted in the rejection of Claim 8 above and applicable herein.
Regarding Claim 14,
The combination of Mindermann, Ouyang, Xie and Bhide teaches:
“The non-transitory computer readable storage medium of claim 8, wherein the instructions further comprise instructions to” (preamble)
“obtain a total budget indicating a total number of training examples for training the parameters of the first machine-learned language model, and wherein a number of training examples in the selected training set is based on the total budget” (Mindermann, Section 1, “We propose a selection function grounded in probabilistic modelling that quantifies by how much each point would reduce the generalization loss if we were to train on it […] To approximate optimal selection, we derive an efficient and easy to implement selection function: the reducible holdout loss.”; Mindermann, Section 3, “[…] we show how to (approximately) find the points that would most reduce the holdout loss if we were to train the current model on them […]”; Mindermann, Section 4.2, “RHO-LOSS requires training an IL model on a holdout set, which poses additional costs. Here, we show how to minimize these costs and amortize them across many training runs of target models. The same experiments also show the robustness of RHO-LOSS across architectures and hyperparameter settings. To fit our computational budget, we perform these experiments on moderate-sized clean benchmark datasets although RHO-LOSS produces greater speedups on noisy or redundant web-scraped data”; Examiner’s note: obtaining a total budget indicating a total number of training examples (i.e., RHO-LOSS computational budget for the total selected examples) for training the parameters of the first machine-learned language model (i.e., training the target models), and wherein a number of training examples in the selected training set is based on the total budget (i.e., finding the points that would most reduce the holdout loss and selection function, the reducible holdout loss, to reduce the generalization loss) is taught.)
The reasons of obviousness have been noted in the rejection of Claim 8 above and applicable herein.
Regarding Claim 15,
Claim 15 recites substantially the same limitations as Claim 8, in the form of a system, therefore
it is rejected under the same rationale.
Regarding Claim 16,
Claim 16 recites substantially the same limitations as Claim 9, in the form of a system, therefore
it is rejected under the same rationale.
Regarding Claim 17,
Claim 17 recites substantially the same limitations as Claim 10, in the form of a system, therefore
it is rejected under the same rationale.
Regarding Claim 18,
Claim 18 recites substantially the same limitations as Claim 11, in the form of a system, therefore
it is rejected under the same rationale.
Regarding Claim 19,
Claim 19 recites substantially the same limitations as Claim 12, in the form of a system, therefore
it is rejected under the same rationale.
Regarding Claim 20,
Claim 20 recites substantially the same limitations as Claim 13, in the form of a system, therefore
it is rejected under the same rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure. Fan et al. teaches the irreducible curriculum for language model pretraining. Kaddour et al. teaches the efficient training algorithms for transformer-based language models.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YONG D RHO whose telephone number is (571)270-0194. The examiner can normally be reached 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at 5712705871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YONG DOO RHO/Examiner, Art Unit 2147
/VIKER A LAMARDO/Supervisory Patent Examiner, Art Unit 2147
