DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is responsive to the Application filed on 4/19/2024. Claims 1-20 are pending in the case. Claims 1 and 19-20 are independent claims.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory obviousness-type double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 U.S.P.Q.2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 U.S.P.Q.2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 U.S.P.Q. 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 U.S.P.Q. 761 (C.C.P.A. 1982); In re Vogel, 422 F.2d 438, 164 U.S.P.Q. 619 (C.C.P.A. 1970); and In re Thorington, 418 F.2d 528, 163 U.S.P.Q. 644 (C.C.P.A. 1969).
A timely filed terminal disclaimer in compliance with 37 C.F.R. §§ 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on a nonstatutory double patenting ground provided the conflicting application or patent either is shown to be commonly owned with this application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement.
Effective January 1, 1994, a registered attorney or agent of record may sign a terminal disclaimer. A terminal disclaimer signed by the assignee must fully comply with 37 C.F.R. § 3.73(b).
Claims 1-20 are rejected on the ground of nonstatutory double patenting over the claims of US 12,282,854 B2.
Instant Application
US 12,282,854 B2
1. A method performed by one or more computers, the method comprising:
training a machine learning model over a sequence of training iterations, comprising, at each of a plurality of training iterations in the sequence of training iterations:
selecting a current batch of model inputs for training the machine learning model at the training iteration, wherein the current batch of model inputs comprises a plurality of model inputs, wherein selecting the current batch of model inputs comprises:
generating a set of candidate batches of model inputs;
generating, for each candidate batch of model inputs, a respective score for the candidate batch of model inputs that characterizes:
(i) an uncertainty of the machine learning model in generating predicted labels for the model inputs in the candidate batch of model inputs, and
(ii) a diversity of the model inputs in the candidate batch of model inputs; and
selecting the current batch of model inputs from the set of candidate batches of model inputs based on the scores;
obtaining a respective target label for each model input in the current batch of model inputs, wherein a target label for a model input defines a model output that should be generated by the machine learning model by processing the model input; and
training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs; and
outputting the trained machine learning model.
1. A method performed by one or more computers, the method comprising:
training a machine learning model over a sequence of training iterations, comprising, at each of a plurality of training iterations in the sequence of training iterations:
selecting a current batch of model inputs for training the machine learning model at the training iteration, wherein the current batch of model inputs comprises a plurality of model inputs, wherein selecting the current batch of model inputs comprises:
generating a set of candidate batches of model inputs;
generating, for each candidate batch of model inputs, a respective score for the candidate batch of model inputs that characterizes:
(i) an uncertainty of the machine learning model in generating predicted labels for the model inputs in the candidate batch of model inputs, and
(ii) a diversity of the model inputs in the candidate batch of model inputs;
selecting the current batch of model inputs from the set of candidate batches of model inputs based on the scores;
obtaining a respective target label for each model input in the current batch of model inputs, wherein a target label for a model input defines a model output that should be generated by the machine learning model by processing the model input; and
training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs; and
outputting the trained machine learning model.
2. The method of claim 1,
wherein for each candidate batch of model inputs, generating the score for the candidate batch of model inputs comprises:
determining, for each pair of model inputs in the candidate batch of model inputs, a respective covariance between:
(i) a predicted label for a first model input in the pair of model inputs, and (ii) a predicted label for a second model input in the pair of model inputs; and
generating the score for the candidate batch of model inputs based on the respective covariance for each pair of model inputs in the candidate batch of model inputs.
1. A method performed by one or more computers, the method comprising:
wherein for each candidate batch of model inputs, generating the score for the candidate batch of model inputs comprises:
determining, for each pair of model inputs in the candidate batch of model inputs, a respective covariance between:
(i) a predicted label for a first model input in the pair of model inputs, and (ii) a predicted label for a second model input in the pair of model inputs; and
generating the score for the candidate batch of model inputs based on the respective covariance for each pair of model inputs in the candidate batch of model inputs; and
3. The method of claim 2, wherein generating the score for the candidate batch of model inputs based on the respective covariance for each pair of model inputs in the candidate batch of model inputs comprises:
generating a determinant of a covariance matrix that comprises the respective covariance for each pair of model inputs in the candidate batch of model inputs; and
determining the score for the candidate batch of model inputs based on the determinant of the covariance matrix.
2. The method of claim 1, wherein generating the score for the candidate batch of model inputs based on the respective covariance for each pair of model inputs in the candidate batch of model inputs comprises:
generating a determinant of a covariance matrix that comprises the respective covariance for each pair of model inputs in the candidate batch of model inputs; and
determining the score for the candidate batch of model inputs based on the determinant of the covariance matrix.
4. The method of claim 3, wherein determining the score for the candidate batch of model inputs based on the determinant of the covariance matrix comprises:
applying a logarithm to the determinant of the covariance matrix.
3. The method of claim 2, wherein determining the score for the candidate batch of model inputs based on the determinant of the covariance matrix comprises:
applying a logarithm to the determinant of the covariance matrix.
5. The method of claim 2, wherein for each pair of model inputs in the candidate batch of model inputs, determining the covariance for the pair of model inputs comprises:
generating a plurality of predicted labels for the first model input in the pair of model inputs using an ensemble of machine learning models;
generating a plurality of predicted labels for the second model input in the pair of model inputs using the ensemble of machine learning models; and
determining the covariance for the pair of model inputs based on:
(i) the plurality of predicted labels for the first model input, and (ii) the plurality of predicted labels for the second model input.
4. The method of claim 1, wherein for each pair of model inputs in the candidate batch of model inputs, determining the covariance for the pair of model inputs comprises:
generating a plurality of predicted labels for the first model input in the pair of model inputs using an ensemble of machine learning models;
generating a plurality of predicted labels for the second model input in the pair of model inputs using the ensemble of machine learning models; and
determining the covariance for the pair of model inputs based on:
(i) the plurality of predicted labels for the first model input, and (ii) the plurality of predicted labels for the second model input.
6. The method of claim 5, wherein the machine learning model is a neural network and the ensemble of machine learning models comprises a plurality of modified neural networks, wherein each modified neural network in the ensemble of machine learning models is a modified version of the neural network.
5. The method of claim 4, wherein the machine learning model is a neural network and the ensemble of machine learning models comprises a plurality of modified neural networks, wherein each modified neural network in the ensemble of machine learning models is a modified version of the neural network.
7. The method of claim 6, wherein each modified neural network in the ensemble of machine learning models is determined by dropping a respective set of parameters from the neural network.
6. The method of claim 5, wherein each modified neural network in the ensemble of machine learning models is determined by dropping a respective set of parameters from the neural network.
8. The method of claim 2, wherein generating the respective covariance for each pair of model inputs in the candidate batch of model inputs comprises:
determining, for each of a plurality of pairs of model parameters of the machine learning model, a respective covariance between:
(i) a first model parameter of the pair of model parameters, and (ii) a second model parameter of the pair of model parameters; and
generating the covariances for the pairs of model inputs based on the covariances for the pairs of model parameters of the machine learning model.
7. The method of claim 1, wherein generating the respective covariance for each pair of model inputs in the candidate batch of model inputs comprises:
determining, for each of a plurality of pairs of model parameters of the machine learning model, a respective covariance between:
(i) a first model parameter of the pair of model parameters, and (ii) a second model parameter of the pair of model parameters; and
generating the covariances for the pairs of model inputs based on the covariances for the pairs of model parameters of the machine learning model.
9. The method of claim 8, wherein the machine learning model is a neural network that comprises an:
(i) an embedding subnetwork that is configured to process a model input to generate an embedding of the model input, and (ii) output layer that is configured to process the embedding of the model input to generate a predicted label for the model input.
8. The method of claim 7, wherein the machine learning model is a neural network that comprises an:
(i) an embedding subnetwork that is configured to process a model input to generate an embedding of the model input, and (ii) output layer that is configured to process the embedding of the model input to generate a predicted label for the model input.
10. The method of claim 9, wherein generating the covariances for the pairs of model inputs based on the covariances for the pairs of model parameters of the machine learning model comprises, for each pair of model inputs comprising a first model input and a second model input:
generating an embedding of the first model input using the embedding subnetwork;
generating an embedding of the second model input using the embedding subnetwork; and
generating the covariance for the pair of model inputs based on the embedding of the first model input, the embedding of the second model input, and covariances for pairs of model parameters included in the output layer of the machine learning model.
9. The method of claim 8, wherein generating the covariances for the pairs of model inputs based on the covariances for the pairs of model parameters of the machine learning model comprises, for each pair of model inputs comprising a first model input and a second model input:
generating an embedding of the first model input using the embedding subnetwork;
generating an embedding of the second model input using the embedding subnetwork; and
generating the covariance for the pair of model inputs based on the embedding of the first model input, the embedding of the second model input, and covariances for pairs of model parameters included in the output layer of the machine learning model.
11. The method of claim 10, wherein for each pair of model inputs comprising a first model input and a second model input, generating the covariance for the pair of model inputs comprises:
computing a matrix product between:
(i) the embedding of the first model input, (ii) a covariance matrix that comprises the covariances for pairs of model parameters included in the output layer of the machine learning model, and (iii) the embedding of the second model input.
10. The method of claim 9, wherein for each pair of model inputs comprising a first model input and a second model input, generating the covariance for the pair of model inputs comprises:
computing a matrix product between:
(i) the embedding of the first model input, (ii) a covariance matrix that comprises the covariances for pairs of model parameters included in the output layer of the machine learning model, and (iii) the embedding of the second model input.
12. The method of claim 8, wherein determining, for each of the plurality of pairs of model parameters of the machine learning model, the respective covariance between:
(i) the first model parameter of the pair of model parameters, and (ii) the second model parameter of the pair of model parameters, comprises:
determining, for each of the plurality of pairs of model parameters of the machine learning model, a respective second derivative of an objective function with respect to the pair of model parameters, wherein the machine learning model has been trained to optimize the objective function; and
processing the second derivatives of the objective function with respect to the pairs of model parameters to generate the covariances of the pairs of model parameters.
11. The method of claim 7, wherein determining, for each of the plurality of pairs of model parameters of the machine learning model, the respective covariance between:
(i) the first model parameter of the pair of model parameters, and (ii) the second model parameter of the pair of model parameters, comprises:
determining, for each of the plurality of pairs of model parameters of the machine learning model, a respective second derivative of an objective function with respect to the pair of model parameters, wherein the machine learning model has been trained to optimize the objective function; and
processing the second derivatives of the objective function with respect to the pairs of model parameters to generate the covariances of the pairs of model parameters.
13. The method of claim 2, wherein for each pair of model inputs in the candidate batch of model inputs, determining the covariance for the pair of model inputs comprises:
determining a quality measure of a first model input in the pair of model inputs based on a value of a predicted label for the first model input relative to values of predicted labels for each other model input in the candidate batch of model inputs;
determining a quality measure of a second model input in the pair of model inputs based on a value of a predicted label for the second model input relative to values of predicted labels for each other model input in the candidate batch of model inputs; and
modifying the covariance for the pair of model inputs based on:
(i) the quality measure of the first model input, and (ii) the quality measure of the second model input.
12. The method of claim 1, wherein for each pair of model inputs in the candidate batch of model inputs, determining the covariance for the pair of model inputs comprises:
determining a quality measure of a first model input in the pair of model inputs based on a value of a predicted label for the first model input relative to values of predicted labels for each other model input in the candidate batch of model inputs;
determining a quality measure of a second model input in the pair of model inputs based on a value of a predicted label for the second model input relative to values of predicted labels for each other model input in the candidate batch of model inputs; and
modifying the covariance for the pair of model inputs based on:
(i) the quality measure of the first model input, and (ii) the quality measure of the second model input.
14. The method of claim 13, wherein the quality measure of the first model input is based on a quantile of the value of the predicted label for the first model input in a set of values that comprises a respective value of a predicted label for each model input in the candidate batch of model inputs.
13. The method of claim 12, wherein the quality measure of the first model input is based on a quantile of the value of the predicted label for the first model input in a set of values that comprises a respective value of a predicted label for each model input in the candidate batch of model inputs.
15. The method of claim 13, wherein the quality measure of the second model input is based on a quantile of the value of the predicted label for the second model input in a set of values that comprises a respective value of a predicted label for each model input in the candidate batch of model inputs.
14. The method of claim 12, wherein the quality measure of the second model input is based on a quantile of the value of the predicted label for the second model input in a set of values that comprises a respective value of a predicted label for each model input in the candidate batch of model inputs.
16. The method of claim 13, wherein modifying the covariance for the pair of model inputs comprises scaling the covariance for the pair of model inputs by the quality measure of the first model input and the quality measure of the second model input.
15. The method of claim 12, wherein modifying the covariance for the pair of model inputs comprises scaling the covariance for the pair of model inputs by the quality measure of the first model input and the quality measure of the second model input.
17. The method of claim 1,
wherein for each candidate batch of model inputs, generating the score for the candidate batch of model inputs comprises:
obtaining, for each model input in the candidate batch of model inputs, a set of classifications of the model input that includes a respective classification generated for the model input by each machine learning model in an ensemble of machine learning models; and
processing the sets of classifications of the model inputs in the candidate batch of model inputs to generate the score for the candidate batch of model inputs as an approximation of an entropy of the candidate batch of model inputs.
24. A method performed by one or more computers, the method comprising:
wherein for each candidate batch of model inputs, generating the score for the candidate batch of model inputs comprises:
obtaining, for each model input in the candidate batch of model inputs, a set of classifications of the model input that includes a respective classification generated for the model input by each machine learning model in an ensemble of machine learning models; and
processing the sets of classifications of the model inputs in the candidate batch of model inputs to generate the score for the candidate batch of model inputs as an approximation of an entropy of the candidate batch of model inputs; and
18. The method of claim 17, wherein processing the sets of classifications of the model inputs in the candidate batch of model inputs to generate the score for the candidate batch of model inputs as an approximation of an entropy of the candidate batch of model inputs comprises:
processing the sets of classifications of the model inputs in the candidate batch of model inputs to generate, for each n-tuple of model inputs from the candidate batch of model inputs, a respective probability distribution over a space of possible joint classifications of model inputs in the n-tuple of model inputs; and
processing the probability distributions for the n-tuples of model inputs in the candidate batch of model inputs to generate the score for the candidate batch of model inputs.
25. The method of claim 24, wherein processing the sets of classifications of the model inputs in the candidate batch of model inputs to generate the score for the candidate batch of model inputs as an approximation of an entropy of the candidate batch of model inputs comprises:
processing the sets of classifications of the model inputs in the candidate batch of model inputs to generate, for each n-tuple of model inputs from the candidate batch of model inputs, a respective probability distribution over a space of possible joint classifications of model inputs in the n-tuple of model inputs; and
processing the probability distributions for the n-tuples of model inputs in the candidate batch of model inputs to generate the score for the candidate batch of model inputs.
19. A system comprising:
one or more computers; and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
training a machine learning model over a sequence of training iterations, comprising, at each of a plurality of training iterations in the sequence of training iterations:
selecting a current batch of model inputs for training the machine learning model at the training iteration, wherein the current batch of model inputs comprises a plurality of model inputs, wherein selecting the current batch of model inputs comprises:
generating a set of candidate batches of model inputs;
generating, for each candidate batch of model inputs, a respective score for the candidate batch of model inputs that characterizes:
(i) an uncertainty of the machine learning model in generating predicted labels for the model inputs in the candidate batch of model inputs, and
(ii) a diversity of the model inputs in the candidate batch of model inputs; and
selecting the current batch of model inputs from the set of candidate batches of model inputs based on the scores;
obtaining a respective target label for each model input in the current batch of model inputs, wherein a target label for a model input defines a model output that should be generated by the machine learning model by processing the model input; and
training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs; and
outputting the trained machine learning model.
22. A system comprising:
one or more computers; and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
training a machine learning model over a sequence of training iterations, comprising, at each of a plurality of training iterations in the sequence of training iterations:
selecting a current batch of model inputs for training the machine learning model at the training iteration, wherein the current batch of model inputs comprises a plurality of model inputs, wherein selecting the current batch of model inputs comprises:
generating a set of candidate batches of model inputs;
generating, for each candidate batch of model inputs, a respective score for the candidate batch of model inputs that characterizes:
(i) an uncertainty of the machine learning model in generating predicted labels for the model inputs in the candidate batch of model inputs, and
(ii) a diversity of the model inputs in the candidate batch of model inputs;
selecting the current batch of model inputs from the set of candidate batches of model inputs based on the scores;
obtaining a respective target label for each model input in the current batch of model inputs, wherein a target label for a model input defines a model output that should be generated by the machine learning model by processing the model input; and
training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs; and
outputting the trained machine learning model.
20. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
training a machine learning model over a sequence of training iterations, comprising, at each of a plurality of training iterations in the sequence of training iterations:
selecting a current batch of model inputs for training the machine learning model at the training iteration, wherein the current batch of model inputs comprises a plurality of model inputs, wherein selecting the current batch of model inputs comprises:
generating a set of candidate batches of model inputs;
generating, for each candidate batch of model inputs, a respective score for the candidate batch of model inputs that characterizes:
(i) an uncertainty of the machine learning model in generating predicted labels for the model inputs in the candidate batch of model inputs, and
(ii) a diversity of the model inputs in the candidate batch of model inputs; and
selecting the current batch of model inputs from the set of candidate batches of model inputs based on the scores;
obtaining a respective target label for each model input in the current batch of model inputs, wherein a target label for a model input defines a model output that should be generated by the machine learning model by processing the model input; and
training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs; and
outputting the trained machine learning model.
23. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
training a machine learning model over a sequence of training iterations, comprising, at each of a plurality of training iterations in the sequence of training iterations:
selecting a current batch of model inputs for training the machine learning model at the training iteration, wherein the current batch of model inputs comprises a plurality of model inputs, wherein selecting the current batch of model inputs comprises:
generating a set of candidate batches of model inputs;
generating, for each candidate batch of model inputs, a respective score for the candidate batch of model inputs that characterizes:
(i) an uncertainty of the machine learning model in generating predicted labels for the model inputs in the candidate batch of model inputs, and
(ii) a diversity of the model inputs in the candidate batch of model inputs;
selecting the current batch of model inputs from the set of candidate batches of model inputs based on the scores;
obtaining a respective target label for each model input in the current batch of model inputs, wherein a target label for a model input defines a model output that should be generated by the machine learning model by processing the model input; and
training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs; and
outputting the trained machine learning model.
Claim Rejections - 35 U.S.C. § 112
The following is a quotation of 35 U.S.C. § 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 2-18 are rejected under 35 U.S.C. § 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention. Claim 2 is incomplete for omitting essential steps, such omission amounting to a gap between the steps. See MPEP § 2172.01. The omitted steps are: determining pairs of model inputs. Claim 2 determines a covariance between a first model input and a second model input but does not first generate the pairs. It is not clear what data is being paired and whether there is any relationship between the paired data. For the purposes of prior art and subject matter eligibility analyses Examiner assumes any one pair is determined at random. Dependent claims inherit the same issue from parent claims and do not resolve it.
Claim Rejections - 35 U.S.C. § 101
35 U.S.C. § 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. § 101 because the claimed invention is directed to an abstract idea without significantly more.
As to claim 1:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, the claim is to a process.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the limitation “at each of a plurality of training iterations in the sequence of training iterations: selecting a current batch of model inputs for training the machine learning model at the training iteration, wherein the current batch of model inputs comprises a plurality of model inputs” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Yes, the limitation “generating a set of candidate batches of model inputs” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Yes, the limitation “generating, for each candidate batch of model inputs, a respective score for the candidate batch of model inputs that characterizes: (i) an uncertainty of the machine learning model in generating predicted labels for the model inputs in the candidate batch of model inputs, and (ii) a diversity of the model inputs in the candidate batch of model inputs” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Yes, the limitation “selecting the current batch of model inputs from the set of candidate batches of model inputs based on the scores” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “training a machine learning model over a sequence of training iterations” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
No, the limitation “training a machine learning model over a sequence of training iterations” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
No, the limitation “obtaining a respective target label for each model input in the current batch of model inputs” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “wherein a target label for a model input defines a model output that should be generated by the machine learning model by processing the model input” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §§ 2106.04(d), 2106.05(h).
No, the limitation “training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
No, the limitation “training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
No, the limitation “outputting the trained machine learning model” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
The additional elements, taken alone or in combination, fail to integrate the judicial exception into a practical application.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “training a machine learning model over a sequence of training iterations” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1).
No, the limitation “training a machine learning model over a sequence of training iterations” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP § 2106.05(f)(2).
No, the limitation “obtaining a respective target label for each model input in the current batch of model inputs” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). Furthermore the additional element is directed to receiving or transmitting data over a network, which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP § 2106.05(d)(II).
No, the limitation “wherein a target label for a model input defines a model output that should be generated by the machine learning model by processing the model input” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP § 2106.05(h).
No, the limitation “training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1).
No, the limitation “training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP § 2106.05(f)(2).
No, the limitation “outputting the trained machine learning model” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). Furthermore the additional element is directed to receiving or transmitting data over a network, which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP § 2106.05(d)(II).
The additional elements, taken alone or in combination, fail to amount to significantly more than the judicial exception.
As to claim 2:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, the claim is to a process.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the limitation “wherein for each candidate batch of model inputs, generating the score for the candidate batch of model inputs comprises: determining, for each pair of model inputs in the candidate batch of model inputs, a respective covariance between: (i) a predicted label for a first model input in the pair of model inputs, and (ii) a predicted label for a second model input in the pair of model inputs” is the abstract idea of a mathematical calculation. See MPEP § 2106.04(a)(2)(I)(C).
Yes, the limitation “generating the score for the candidate batch of model inputs based on the respective covariance for each pair of model inputs in the candidate batch of model inputs” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
The analysis of the parent claim is incorporated.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
The analysis of the parent claim is incorporated.
As to claim 3:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, the claim is to a process.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the limitation “wherein generating the score for the candidate batch of model inputs based on the respective covariance for each pair of model inputs in the candidate batch of model inputs comprises: generating a determinant of a covariance matrix that comprises the respective covariance for each pair of model inputs in the candidate batch of model inputs” is the abstract idea of a mathematical calculation. See MPEP § 2106.04(a)(2)(I)(C).
Yes, the limitation “determining the score for the candidate batch of model inputs based on the determinant of the covariance matrix” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
The analysis of the parent claim is incorporated.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
The analysis of the parent claim is incorporated.
As to claim 4:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, the claim is to a process.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the limitation “wherein determining the score for the candidate batch of model inputs based on the determinant of the covariance matrix comprises: applying a logarithm to the determinant of the covariance matrix” is the abstract idea of a mathematical calculation. See MPEP § 2106.04(a)(2)(I)(C).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
The analysis of the parent claim is incorporated.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
The analysis of the parent claim is incorporated.
As to claim 5:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, the claim is to a process.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the limitation “determining the covariance for the pair of model inputs based on: (i) the plurality of predicted labels for the first model input, and (ii) the plurality of predicted labels for the second model input” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “generating a plurality of predicted labels for the first model input in the pair of model inputs using an ensemble of machine learning models” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
No, the limitation “generating a plurality of predicted labels for the first model input in the pair of model inputs using an ensemble of machine learning models” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
No, the limitation “generating a plurality of predicted labels for the second model input in the pair of model inputs using the ensemble of machine learning models” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
No, the limitation “generating a plurality of predicted labels for the second model input in the pair of model inputs using the ensemble of machine learning models” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
The additional elements, taken alone or in combination, fail to integrate the judicial exception into a practical application.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “generating a plurality of predicted labels for the first model input in the pair of model inputs using an ensemble of machine learning models” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1).
No, the limitation “generating a plurality of predicted labels for the first model input in the pair of model inputs using an ensemble of machine learning models” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP § 2106.05(f)(2).
No, the limitation “generating a plurality of predicted labels for the second model input in the pair of model inputs using the ensemble of machine learning models” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1).
No, the limitation “generating a plurality of predicted labels for the second model input in the pair of model inputs using the ensemble of machine learning models” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP § 2106.05(f)(2).
The additional elements, taken alone or in combination, fail to amount to significantly more than the judicial exception.
As to claim 6:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, the claim is to a process.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
The analysis of the parent claim is incorporated.
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “wherein the machine learning model is a neural network and the ensemble of machine learning models comprises a plurality of modified neural networks, wherein each modified neural network in the ensemble of machine learning models is a modified version of the neural network” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §§ 2106.04(d), 2106.05(h).
The additional elements, taken alone or in combination, fail to integrate the judicial exception into a practical application.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “wherein the machine learning model is a neural network and the ensemble of machine learning models comprises a plurality of modified neural networks, wherein each modified neural network in the ensemble of machine learning models is a modified version of the neural network” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP § 2106.05(h).
The additional elements, taken alone or in combination, fail to amount to significantly more than the judicial exception.
As to claim 7:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, the claim is to a process.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
The analysis of the parent claim is incorporated.
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “wherein each modified neural network in the ensemble of machine learning models is determined by dropping a respective set of parameters from the neural network” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
No, the limitation “wherein each modified neural network in the ensemble of machine learning models is determined by dropping a respective set of parameters from the neural network” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
The additional elements, taken alone or in combination, fail to integrate the judicial exception into a practical application.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “wherein each modified neural network in the ensemble of machine learning models is determined by dropping a respective set of parameters from the neural network” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1).
No, the limitation “wherein each modified neural network in the ensemble of machine learning models is determined by dropping a respective set of parameters from the neural network” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP § 2106.05(f)(2).
The additional elements, taken alone or in combination, fail to amount to significantly more than the judicial exception.
As to claim 8:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, the claim is to a process.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the limitation “wherein generating the respective covariance for each pair of model inputs in the candidate batch of model inputs comprises: determining, for each of a plurality of pairs of model parameters of the machine learning model, a respective covariance between: (i) a first model parameter of the pair of model parameters, and (ii) a second model parameter of the pair of model parameters” is the abstract idea of a mathematical calculation. See MPEP § 2106.04(a)(2)(I)(C).
Yes, the limitation “generating the covariances for the pairs of model inputs based on the covariances for the pairs of model parameters of the machine learning model” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
The analysis of the parent claim is incorporated.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
The analysis of the parent claim is incorporated.
As to claim 9:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, the claim is to a process.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
The analysis of the parent claim is incorporated.
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “wherein the machine learning model is a neural network that comprises an: (i) an embedding subnetwork that is configured to process a model input to generate an embedding of the model input, and (ii) output layer that is configured to process the embedding of the model input to generate a predicted label for the model input” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
No, the limitation “wherein the machine learning model is a neural network that comprises an: (i) an embedding subnetwork that is configured to process a model input to generate an embedding of the model input, and (ii) output layer that is configured to process the embedding of the model input to generate a predicted label for the model input” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
No, the limitation “wherein the machine learning model is a neural network that comprises an: (i) an embedding subnetwork that is configured to process a model input to generate an embedding of the model input, and (ii) output layer that is configured to process the embedding of the model input to generate a predicted label for the model input” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §§ 2106.04(d), 2106.05(h).
The additional elements, taken alone or in combination, fail to integrate the judicial exception into a practical application.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “wherein the machine learning model is a neural network that comprises an: (i) an embedding subnetwork that is configured to process a model input to generate an embedding of the model input, and (ii) output layer that is configured to process the embedding of the model input to generate a predicted label for the model input” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1).
No, the limitation “wherein the machine learning model is a neural network that comprises an: (i) an embedding subnetwork that is configured to process a model input to generate an embedding of the model input, and (ii) output layer that is configured to process the embedding of the model input to generate a predicted label for the model input” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP § 2106.05(f)(2).
No, the limitation “wherein the machine learning model is a neural network that comprises an: (i) an embedding subnetwork that is configured to process a model input to generate an embedding of the model input, and (ii) output layer that is configured to process the embedding of the model input to generate a predicted label for the model input” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP § 2106.05(h).
The additional elements, taken alone or in combination, fail to amount to significantly more than the judicial exception.
As to claim 10:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, the claim is to a process.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the limitation “generating the covariance for the pair of model inputs based on the embedding of the first model input, the embedding of the second model input, and covariances for pairs of model parameters included in the output layer of the machine learning model” is the abstract idea of a mathematical calculation. See MPEP § 2106.04(a)(2)(I)(C).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “generating an embedding of the first model input using the embedding subnetwork” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
No, the limitation “generating an embedding of the first model input using the embedding subnetwork” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
No, the limitation “generating an embedding of the second model input using the embedding subnetwork” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
No, the limitation “generating an embedding of the second model input using the embedding subnetwork” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
The additional elements, taken alone or in combination, fail to integrate the judicial exception into a practical application.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “generating an embedding of the first model input using the embedding subnetwork” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1).
No, the limitation “generating an embedding of the first model input using the embedding subnetwork” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP § 2106.05(f)(2).
No, the limitation “generating an embedding of the second model input using the embedding subnetwork” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1).
No, the limitation “generating an embedding of the second model input using the embedding subnetwork” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP § 2106.05(f)(2).
The additional elements, taken alone or in combination, fail to amount to significantly more than the judicial exception.
As to claim 11:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, the claim is to a process.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the limitation “wherein for each pair of model inputs comprising a first model input and a second model input, generating the covariance for the pair of model inputs comprises: computing a matrix product between: (i) the embedding of the first model input, (ii) a covariance matrix that comprises the covariances for pairs of model parameters included in the output layer of the machine learning model, and (iii) the embedding of the second model input” is the abstract idea of a mathematical calculation. See MPEP § 2106.04(a)(2)(I)(C).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
The analysis of the parent claim is incorporated.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
The analysis of the parent claim is incorporated.
As to claim 12:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, the claim is to a process.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the limitation “determining, for each of the plurality of pairs of model parameters of the machine learning model, a respective second derivative of an objective function with respect to the pair of model parameters” is the abstract idea of a mathematical calculation. See MPEP § 2106.04(a)(2)(I)(C).
Yes, the limitation “determining, for each of the plurality of pairs of model parameters of the machine learning model, a respective second derivative of an objective function with respect to the pair of model parameters” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Yes, the limitation “processing the second derivatives of the objective function with respect to the pairs of model parameters to generate the covariances of the pairs of model parameters” is the abstract idea of a mathematical calculation. See MPEP § 2106.04(a)(2)(I)(C).
Yes, the limitation “processing the second derivatives of the objective function with respect to the pairs of model parameters to generate the covariances of the pairs of model parameters” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “wherein the machine learning model has been trained to optimize the objective function” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
No, the limitation “wherein the machine learning model has been trained to optimize the objective function” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
No, the limitation “wherein the machine learning model has been trained to optimize the objective function” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §§ 2106.04(d), 2106.05(h).
The additional elements, taken alone or in combination, fail to integrate the judicial exception into a practical application.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “wherein the machine learning model has been trained to optimize the objective function” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1).
No, the limitation “wherein the machine learning model has been trained to optimize the objective function” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP § 2106.05(f)(2).
No, the limitation “wherein the machine learning model has been trained to optimize the objective function” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP § 2106.05(h).
The additional elements, taken alone or in combination, fail to amount to significantly more than the judicial exception.
As to claim 13:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, the claim is to a process.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the limitation “determining a quality measure of a first model input in the pair of model inputs based on a value of a predicted label for the first model input relative to values of predicted labels for each other model input in the candidate batch of model inputs” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Yes, the limitation “determining a quality measure of a second model input in the pair of model inputs based on a value of a predicted label for the second model input relative to values of predicted labels for each other model input in the candidate batch of model inputs” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Yes, the limitation “modifying the covariance for the pair of model inputs based on: (i) the quality measure of the first model input, and (ii) the quality measure of the second model input” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
The analysis of the parent claim is incorporated.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
The analysis of the parent claim is incorporated.
As to claim 14:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, the claim is to a process.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the limitation “wherein the quality measure of the first model input is based on a quantile of the value of the predicted label for the first model input in a set of values that comprises a respective value of a predicted label for each model input in the candidate batch of model inputs” is the abstract idea of a mathematical calculation. See MPEP § 2106.04(a)(2)(I)(C).
Yes, the limitation “wherein the quality measure of the first model input is based on a quantile of the value of the predicted label for the first model input in a set of values that comprises a respective value of a predicted label for each model input in the candidate batch of model inputs” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
The analysis of the parent claim is incorporated.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
The analysis of the parent claim is incorporated.
As to claim 15:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, the claim is to a process.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the limitation “wherein the quality measure of the second model input is based on a quantile of the value of the predicted label for the second model input in a set of values that comprises a respective value of a predicted label for each model input in the candidate batch of model inputs” is the abstract idea of a mathematical calculation. See MPEP § 2106.04(a)(2)(I)(C).
Yes, the limitation “wherein the quality measure of the second model input is based on a quantile of the value of the predicted label for the second model input in a set of values that comprises a respective value of a predicted label for each model input in the candidate batch of model inputs” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
The analysis of the parent claim is incorporated.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
The analysis of the parent claim is incorporated.
As to claim 16:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, the claim is to a process.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the limitation “wherein modifying the covariance for the pair of model inputs comprises scaling the covariance for the pair of model inputs by the quality measure of the first model input and the quality measure of the second model input” is the abstract idea of a mathematical calculation. See MPEP § 2106.04(a)(2)(I)(C).
Yes, the limitation “wherein modifying the covariance for the pair of model inputs comprises scaling the covariance for the pair of model inputs by the quality measure of the first model input and the quality measure of the second model input” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
The analysis of the parent claim is incorporated.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
The analysis of the parent claim is incorporated.
As to claim 17:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, the claim is to a process.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the limitation “processing the sets of classifications of the model inputs in the candidate batch of model inputs to generate the score for the candidate batch of model inputs as an approximation of an entropy of the candidate batch of model inputs” is the abstract idea of a mathematical calculation. See MPEP § 2106.04(a)(2)(I)(C).
Yes, the limitation “processing the sets of classifications of the model inputs in the candidate batch of model inputs to generate the score for the candidate batch of model inputs as an approximation of an entropy of the candidate batch of model inputs” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “obtaining, for each model input in the candidate batch of model inputs, a set of classifications of the model input that includes a respective classification generated for the model input by each machine learning model in an ensemble of machine learning models” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
The additional elements, taken alone or in combination, fail to integrate the judicial exception into a practical application.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “obtaining, for each model input in the candidate batch of model inputs, a set of classifications of the model input that includes a respective classification generated for the model input by each machine learning model in an ensemble of machine learning models” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). Furthermore the additional element is directed to receiving or transmitting data over a network, which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP § 2106.05(d)(II).
The additional elements, taken alone or in combination, fail to amount to significantly more than the judicial exception.
As to claim 18:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, the claim is to a process.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the limitation “processing the sets of classifications of the model inputs in the candidate batch of model inputs to generate, for each n-tuple of model inputs from the candidate batch of model inputs, a respective probability distribution over a space of possible joint classifications of model inputs in the n-tuple of model inputs” is the abstract idea of a mathematical calculation. See MPEP § 2106.04(a)(2)(I)(C).
Yes, the limitation “processing the probability distributions for the n-tuples of model inputs in the candidate batch of model inputs to generate the score for the candidate batch of model inputs” is the abstract idea of a mathematical calculation. See MPEP § 2106.04(a)(2)(I)(C).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
The analysis of the parent claim is incorporated.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
The analysis of the parent claim is incorporated.
As to claim 19:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, the claim is to a machine.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the limitation “at each of a plurality of training iterations in the sequence of training iterations: selecting a current batch of model inputs for training the machine learning model at the training iteration, wherein the current batch of model inputs comprises a plurality of model inputs” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Yes, the limitation “generating a set of candidate batches of model inputs” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Yes, the limitation “generating, for each candidate batch of model inputs, a respective score for the candidate batch of model inputs that characterizes: (i) an uncertainty of the machine learning model in generating predicted labels for the model inputs in the candidate batch of model inputs, and (ii) a diversity of the model inputs in the candidate batch of model inputs” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Yes, the limitation “selecting the current batch of model inputs from the set of candidate batches of model inputs based on the scores” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “one or more computers” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §§ 2106.04(d), 2106.05(h).
No, the limitation “one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §§ 2106.04(d), 2106.05(h).
No, the limitation “training a machine learning model over a sequence of training iterations” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
No, the limitation “training a machine learning model over a sequence of training iterations” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
No, the limitation “obtaining a respective target label for each model input in the current batch of model inputs” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “wherein a target label for a model input defines a model output that should be generated by the machine learning model by processing the model input” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §§ 2106.04(d), 2106.05(h).
No, the limitation “training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
No, the limitation “training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
No, the limitation “outputting the trained machine learning model” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
The additional elements, taken alone or in combination, fail to integrate the judicial exception into a practical application.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “one or more computers” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP § 2106.05(h).
No, the limitation “one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP § 2106.05(h).
No, the limitation “training a machine learning model over a sequence of training iterations” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1).
No, the limitation “training a machine learning model over a sequence of training iterations” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP § 2106.05(f)(2).
No, the limitation “obtaining a respective target label for each model input in the current batch of model inputs” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). Furthermore the additional element is directed to receiving or transmitting data over a network, which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP § 2106.05(d)(II).
No, the limitation “wherein a target label for a model input defines a model output that should be generated by the machine learning model by processing the model input” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP § 2106.05(h).
No, the limitation “training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1).
No, the limitation “training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP § 2106.05(f)(2).
No, the limitation “outputting the trained machine learning model” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). Furthermore the additional element is directed to receiving or transmitting data over a network, which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP § 2106.05(d)(II).
The additional elements, taken alone or in combination, fail to amount to significantly more than the judicial exception.
As to claim 20:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, the claim is to a manufacture.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the limitation “at each of a plurality of training iterations in the sequence of training iterations: selecting a current batch of model inputs for training the machine learning model at the training iteration, wherein the current batch of model inputs comprises a plurality of model inputs” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Yes, the limitation “generating a set of candidate batches of model inputs” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Yes, the limitation “generating, for each candidate batch of model inputs, a respective score for the candidate batch of model inputs that characterizes: (i) an uncertainty of the machine learning model in generating predicted labels for the model inputs in the candidate batch of model inputs, and (ii) a diversity of the model inputs in the candidate batch of model inputs” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Yes, the limitation “selecting the current batch of model inputs from the set of candidate batches of model inputs based on the scores” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §§ 2106.04(d), 2106.05(h).
No, the limitation “training a machine learning model over a sequence of training iterations” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
No, the limitation “training a machine learning model over a sequence of training iterations” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
No, the limitation “obtaining a respective target label for each model input in the current batch of model inputs” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “wherein a target label for a model input defines a model output that should be generated by the machine learning model by processing the model input” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §§ 2106.04(d), 2106.05(h).
No, the limitation “training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
No, the limitation “training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
No, the limitation “outputting the trained machine learning model” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
The additional elements, taken alone or in combination, fail to integrate the judicial exception into a practical application.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP § 2106.05(h).
No, the limitation “training a machine learning model over a sequence of training iterations” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1).
No, the limitation “training a machine learning model over a sequence of training iterations” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP § 2106.05(f)(2).
No, the limitation “obtaining a respective target label for each model input in the current batch of model inputs” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). Furthermore the additional element is directed to receiving or transmitting data over a network, which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP § 2106.05(d)(II).
No, the limitation “wherein a target label for a model input defines a model output that should be generated by the machine learning model by processing the model input” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP § 2106.05(h).
No, the limitation “training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1).
No, the limitation “training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP § 2106.05(f)(2).
No, the limitation “outputting the trained machine learning model” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). Furthermore the additional element is directed to receiving or transmitting data over a network, which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP § 2106.05(d)(II).
The additional elements, taken alone or in combination, fail to amount to significantly more than the judicial exception.
Claim Rejections - 35 U.S.C. § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. §§ 102 and 103 (or as subject to pre-AIA 35 U.S.C. §§ 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. § 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1 and 19-20 are rejected under 35 U.S.C. § 102(a)(1) as being anticipated by Erenrich et al. (US 2018/0330280 A1, hereinafter Erenrich).
As to independent claim 1, Erenrich discloses a method performed by one or more computers (“In an implementation, a method for training a machine learning model is provided. The method may be performed on a computer system having one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the computer system to perform the method,” paragraph 0007 lines 1-7), the method comprising:
training a machine learning model over a sequence of training iterations, comprising, at each of a plurality of training iterations in the sequence of training iterations (“The method may include obtaining, by the computer system, a machine learning model and a training dataset, the training dataset including a plurality of training examples; determining, by the computer system, uncertainty scores for the plurality of training examples according to the machine learning model; selecting, by the computer system, a first example batch from the plurality of training examples according to uncertainty scores of the plurality of training examples; updating, by the computer system, the machine learning model according to at least one labeled training example of the example batch; determining, by the computer system, updated uncertainty scores for the plurality of training examples according to the updated machine learning model; and selecting, by the computer system, a second example batch from the plurality of training examples according to the updated uncertainty scores of the plurality of training examples,” paragraph 0007 lines 7-23):
selecting a current batch of model inputs for training the machine learning model at the training iteration, wherein the current batch of model inputs comprises a plurality of model inputs (“The method may include obtaining, by the computer system, a machine learning model and a training dataset, the training dataset including a plurality of training examples; determining, by the computer system, uncertainty scores for the plurality of training examples according to the machine learning model; selecting, by the computer system, a first example batch from the plurality of training examples according to uncertainty scores of the plurality of training examples; updating, by the computer system, the machine learning model according to at least one labeled training example of the example batch; determining, by the computer system, updated uncertainty scores for the plurality of training examples according to the updated machine learning model; and selecting, by the computer system, a second example batch from the plurality of training examples according to the updated uncertainty scores of the plurality of training examples,” paragraph 0007 lines 7-23), wherein selecting the current batch of model inputs comprises:
generating a set of candidate batches of model inputs (“Batch evaluator 704 may receive a batch of data examples from sample evaluator 702. The batch of data examples may be a subset of the training dataset of data examples evaluated by sample evaluator 702, and may be selected by sample evaluator 702 and/or by batch evaluator 704. The batch of data examples may be selected according to their uncertainty scores. The selected batch of data examples may be those having the highest uncertainty scores from the training dataset. For example, the batch of data examples may include the most uncertain 10 percent of data examples from the training data set. The size of the batch of data examples may be predetermined as an absolute number (e.g., 1000 examples) and/or may be predetermined as a percentage of the training dataset. In some implementations, the size of the batch of data examples may be predetermined according to system characteristics and/or may be dynamically determined during system processing,” paragraph 0099 lines 1-17);
generating, for each candidate batch of model inputs, a respective score for the candidate batch of model inputs (“Batch evaluator 704 may obtain determination results and uncertainty scores for all or some of the batch of data examples. In some implementations, batch evaluator 704 may apply the evaluation model 710 to all data examples of the batch of data examples to obtain the determination results and uncertainty scores. In some implementations, where evaluation model 710 has not been updated after determination results and uncertainty scores were determined by sample evaluator 702, batch evaluator 704 may obtain the previously generated results and uncertainty scores,” paragraph 0101 lines 1-11; “After evaluation model 710 updating, batch evaluator 704 may obtain updated determination results and updated uncertainty scores of the remaining data examples of the batch of data examples. Batch evaluator 704 may apply updated evaluation model 710 to the remaining data examples to obtain the updated determination results and updated uncertainty scores. Example selector 706 may select a second set of one or more specific data examples from the remaining data examples of the batch of data examples for user labeling by trainer 708. Trainer 708 may communicate with user interface 524 to obtain user labels of the second set of specific data examples and user these results to again update the evaluation model. This loop may continue until a batch loop end point is reached,” paragraph 0104 lines 1-14) that characterizes:
(i) an uncertainty of the machine learning model in generating predicted labels for the model inputs in the candidate batch of model inputs (“Batch evaluator 704 may receive a batch of data examples from sample evaluator 702. The batch of data examples may be a subset of the training dataset of data examples evaluated by sample evaluator 702, and may be selected by sample evaluator 702 and/or by batch evaluator 704. The batch of data examples may be selected according to their uncertainty scores. The selected batch of data examples may be those having the highest uncertainty scores from the training dataset. For example, the batch of data examples may include the most uncertain 10 percent of data examples from the training data set. The size of the batch of data examples may be predetermined as an absolute number (e.g., 1000 examples) and/or may be predetermined as a percentage of the training dataset. In some implementations, the size of the batch of data examples may be predetermined according to system characteristics and/or may be dynamically determined during system processing,” paragraph 0099 lines 1-17; “An uncertainty score of a data example may indicate a level of uncertainty in a model's evaluation of the data example. In some implementations, a data example may receive both a label and an uncertainty score when evaluated by a machine learning model. That is, the label may represent a determination result of the model and the uncertainty score may represent a confidence level in the determination result. In some implementations, the uncertainty score of the data example may be derived directly from the determination result,” paragraph 0097 lines 1-10), and
(ii) a diversity of the model inputs in the candidate batch of model inputs (“the batch of data examples may further be determined according to a diversity measurement between data examples of the batch. If a batch of data examples is selected purely according to uncertainty scores, some examples of the batch of data examples may be very similar in one or more aspects. For example, a series of duplicate records in a database could lead to a series of identical data examples. Identical data examples would, of course, have an identical uncertainty score. Requiring a user to label more than one of a series of identical data examples would be a waste of time and resources. Two similar, but not identical data examples, may introduce the same difficulty. In some embodiments, once the first of the data examples is labeled, the resultant model updates may be sufficient to significantly reduce the uncertainty score of the second example, rendering it inefficient as a labeling example for the user. In some embodiments, rather than updating the results shown while a user is observing results, results can be optimized for diversity and presented to the user up-front. Accordingly, the system may be configured to determine the data examples of the batch according not just to the uncertainty scores, but also to a diversity score of each data example as compared to the other data examples of the batch. A diversity score of a data examples may be a measurement of the how different each data example of the batch is from each other data example of batch. In selecting the batch of data examples, the system may apply a weight to the uncertainty scores and to the diversity scores of the batch of data examples,” paragraph 0100 lines 1-29); and
selecting the current batch of model inputs from the set of candidate batches of model inputs based on the scores (“In an operation 806, process 800 may include selecting a first batch of data examples from the training dataset. The first batch of data examples may be selected according to uncertainty scores. In some implementations, the first batch of data examples may be selected as the data examples having the highest uncertainty scores in the training dataset,” paragraph 0115 lines 1-7; “In an operation 812, process 800 may include selecting a second batch of data examples from the training dataset. The second batch of data examples may be selected according to the updated uncertainty scores of the training dataset. The second batch of data examples may be used by the system to provide additional data examples to a user for labeling. In turn, the evaluation model may be updated according to the labeling performed on the second batch of data examples,” paragraph 0119 lines 1-9);
obtaining a respective target label for each model input in the current batch of model inputs, wherein a target label for a model input defines a model output that should be generated by the machine learning model by processing the model input (“A claimed solution rooted in computer technology overcomes problems specifically arising in the realm of computer technology. In various implementations, a computing system is configured to provide methods that accelerate machine learning processes. A machine learning model may be trained with human labeled data examples and may be updated as the examples are labeled. A subset or batch of data examples may be selected from a complete data set according to their uncertainty levels. As the batch of data examples are labeled, a machine learning model may be updated and applied to the remaining batch data examples to update their uncertainty levels. The system may select the most uncertain data example from the batch for labeling. While the user continues to label examples of the batch dataset, the system may rescore the complete dataset to select the next batch of examples to be provided to the user as the first batch is completed, thus providing a lag-free and efficient machine learning model training system,” paragraph 0005 lines 1-18; “An uncertainty score of a data example may indicate a level of uncertainty in a model's evaluation of the data example. In some implementations, a data example may receive both a label and an uncertainty score when evaluated by a machine learning model. That is, the label may represent a determination result of the model and the uncertainty score may represent a confidence level in the determination result. In some implementations, the uncertainty score of the data example may be derived directly from the determination result,” paragraph 0097 lines 1-10); and
training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs (“A claimed solution rooted in computer technology overcomes problems specifically arising in the realm of computer technology. In various implementations, a computing system is configured to provide methods that accelerate machine learning processes. A machine learning model may be trained with human labeled data examples and may be updated as the examples are labeled. A subset or batch of data examples may be selected from a complete data set according to their uncertainty levels. As the batch of data examples are labeled, a machine learning model may be updated and applied to the remaining batch data examples to update their uncertainty levels. The system may select the most uncertain data example from the batch for labeling. While the user continues to label examples of the batch dataset, the system may rescore the complete dataset to select the next batch of examples to be provided to the user as the first batch is completed, thus providing a lag-free and efficient machine learning model training system,” paragraph 0005 lines 1-18); and
outputting the trained machine learning model (“In some implementations, batch size may be determined according to a user labeling time and an estimated system processing speed. The batch size may be selected such that the entirety of the batch of data examples can be reevaluated by application of an updated model while the user is labeling a data example. Thus, the user may be provided with the most uncertain example for labeling. The user may label the example, causing the model to update. While the user is labeling the next most uncertain example, the updated model may be applied to the remainder of the batch to update the uncertainty scores. When the user finishes the labeling task, the system may have finished the updating task, and a new example may be selected from the batch based on the updated batch uncertainty scores,” paragraph 0110 lines 1-14).
As to independent claim 19, Erenrich discloses a system comprising:
one or more computers (“A claimed solution rooted in computer technology overcomes problems specifically arising in the realm of computer technology. In various implementations, a computing system is configured to provide methods that accelerate machine learning processes. A machine learning model may be trained with human labeled data examples and may be updated as the examples are labeled. A subset or batch of data examples may be selected from a complete data set according to their uncertainty levels. As the batch of data examples are labeled, a machine learning model may be updated and applied to the remaining batch data examples to update their uncertainty levels. The system may select the most uncertain data example from the batch for labeling. While the user continues to label examples of the batch dataset, the system may rescore the complete dataset to select the next batch of examples to be provided to the user as the first batch is completed, thus providing a lag-free and efficient machine learning model training system,” paragraph 0005 lines 1-18); and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations (“A claimed solution rooted in computer technology overcomes problems specifically arising in the realm of computer technology. In various implementations, a computing system is configured to provide methods that accelerate machine learning processes. A machine learning model may be trained with human labeled data examples and may be updated as the examples are labeled. A subset or batch of data examples may be selected from a complete data set according to their uncertainty levels. As the batch of data examples are labeled, a machine learning model may be updated and applied to the remaining batch data examples to update their uncertainty levels. The system may select the most uncertain data example from the batch for labeling. While the user continues to label examples of the batch dataset, the system may rescore the complete dataset to select the next batch of examples to be provided to the user as the first batch is completed, thus providing a lag-free and efficient machine learning model training system,” paragraph 0005 lines 1-18) comprising:
training a machine learning model over a sequence of training iterations, comprising, at each of a plurality of training iterations in the sequence of training iterations (“The method may include obtaining, by the computer system, a machine learning model and a training dataset, the training dataset including a plurality of training examples; determining, by the computer system, uncertainty scores for the plurality of training examples according to the machine learning model; selecting, by the computer system, a first example batch from the plurality of training examples according to uncertainty scores of the plurality of training examples; updating, by the computer system, the machine learning model according to at least one labeled training example of the example batch; determining, by the computer system, updated uncertainty scores for the plurality of training examples according to the updated machine learning model; and selecting, by the computer system, a second example batch from the plurality of training examples according to the updated uncertainty scores of the plurality of training examples,” paragraph 0007 lines 7-23):
selecting a current batch of model inputs for training the machine learning model at the training iteration, wherein the current batch of model inputs comprises a plurality of model inputs (“The method may include obtaining, by the computer system, a machine learning model and a training dataset, the training dataset including a plurality of training examples; determining, by the computer system, uncertainty scores for the plurality of training examples according to the machine learning model; selecting, by the computer system, a first example batch from the plurality of training examples according to uncertainty scores of the plurality of training examples; updating, by the computer system, the machine learning model according to at least one labeled training example of the example batch; determining, by the computer system, updated uncertainty scores for the plurality of training examples according to the updated machine learning model; and selecting, by the computer system, a second example batch from the plurality of training examples according to the updated uncertainty scores of the plurality of training examples,” paragraph 0007 lines 7-23), wherein selecting the current batch of model inputs comprises:
generating a set of candidate batches of model inputs (“Batch evaluator 704 may receive a batch of data examples from sample evaluator 702. The batch of data examples may be a subset of the training dataset of data examples evaluated by sample evaluator 702, and may be selected by sample evaluator 702 and/or by batch evaluator 704. The batch of data examples may be selected according to their uncertainty scores. The selected batch of data examples may be those having the highest uncertainty scores from the training dataset. For example, the batch of data examples may include the most uncertain 10 percent of data examples from the training data set. The size of the batch of data examples may be predetermined as an absolute number (e.g., 1000 examples) and/or may be predetermined as a percentage of the training dataset. In some implementations, the size of the batch of data examples may be predetermined according to system characteristics and/or may be dynamically determined during system processing,” paragraph 0099 lines 1-17);
generating, for each candidate batch of model inputs, a respective score for the candidate batch of model inputs (“Batch evaluator 704 may obtain determination results and uncertainty scores for all or some of the batch of data examples. In some implementations, batch evaluator 704 may apply the evaluation model 710 to all data examples of the batch of data examples to obtain the determination results and uncertainty scores. In some implementations, where evaluation model 710 has not been updated after determination results and uncertainty scores were determined by sample evaluator 702, batch evaluator 704 may obtain the previously generated results and uncertainty scores,” paragraph 0101 lines 1-11; “After evaluation model 710 updating, batch evaluator 704 may obtain updated determination results and updated uncertainty scores of the remaining data examples of the batch of data examples. Batch evaluator 704 may apply updated evaluation model 710 to the remaining data examples to obtain the updated determination results and updated uncertainty scores. Example selector 706 may select a second set of one or more specific data examples from the remaining data examples of the batch of data examples for user labeling by trainer 708. Trainer 708 may communicate with user interface 524 to obtain user labels of the second set of specific data examples and user these results to again update the evaluation model. This loop may continue until a batch loop end point is reached,” paragraph 0104 lines 1-14) that characterizes:
(i) an uncertainty of the machine learning model in generating predicted labels for the model inputs in the candidate batch of model inputs (“Batch evaluator 704 may receive a batch of data examples from sample evaluator 702. The batch of data examples may be a subset of the training dataset of data examples evaluated by sample evaluator 702, and may be selected by sample evaluator 702 and/or by batch evaluator 704. The batch of data examples may be selected according to their uncertainty scores. The selected batch of data examples may be those having the highest uncertainty scores from the training dataset. For example, the batch of data examples may include the most uncertain 10 percent of data examples from the training data set. The size of the batch of data examples may be predetermined as an absolute number (e.g., 1000 examples) and/or may be predetermined as a percentage of the training dataset. In some implementations, the size of the batch of data examples may be predetermined according to system characteristics and/or may be dynamically determined during system processing,” paragraph 0099 lines 1-17; “An uncertainty score of a data example may indicate a level of uncertainty in a model's evaluation of the data example. In some implementations, a data example may receive both a label and an uncertainty score when evaluated by a machine learning model. That is, the label may represent a determination result of the model and the uncertainty score may represent a confidence level in the determination result. In some implementations, the uncertainty score of the data example may be derived directly from the determination result,” paragraph 0097 lines 1-10), and
(ii) a diversity of the model inputs in the candidate batch of model inputs (“the batch of data examples may further be determined according to a diversity measurement between data examples of the batch. If a batch of data examples is selected purely according to uncertainty scores, some examples of the batch of data examples may be very similar in one or more aspects. For example, a series of duplicate records in a database could lead to a series of identical data examples. Identical data examples would, of course, have an identical uncertainty score. Requiring a user to label more than one of a series of identical data examples would be a waste of time and resources. Two similar, but not identical data examples, may introduce the same difficulty. In some embodiments, once the first of the data examples is labeled, the resultant model updates may be sufficient to significantly reduce the uncertainty score of the second example, rendering it inefficient as a labeling example for the user. In some embodiments, rather than updating the results shown while a user is observing results, results can be optimized for diversity and presented to the user up-front. Accordingly, the system may be configured to determine the data examples of the batch according not just to the uncertainty scores, but also to a diversity score of each data example as compared to the other data examples of the batch. A diversity score of a data examples may be a measurement of the how different each data example of the batch is from each other data example of batch. In selecting the batch of data examples, the system may apply a weight to the uncertainty scores and to the diversity scores of the batch of data examples,” paragraph 0100 lines 1-29); and
selecting the current batch of model inputs from the set of candidate batches of model inputs based on the scores (“In an operation 806, process 800 may include selecting a first batch of data examples from the training dataset. The first batch of data examples may be selected according to uncertainty scores. In some implementations, the first batch of data examples may be selected as the data examples having the highest uncertainty scores in the training dataset,” paragraph 0115 lines 1-7; “In an operation 812, process 800 may include selecting a second batch of data examples from the training dataset. The second batch of data examples may be selected according to the updated uncertainty scores of the training dataset. The second batch of data examples may be used by the system to provide additional data examples to a user for labeling. In turn, the evaluation model may be updated according to the labeling performed on the second batch of data examples,” paragraph 0119 lines 1-9);
obtaining a respective target label for each model input in the current batch of model inputs, wherein a target label for a model input defines a model output that should be generated by the machine learning model by processing the model input (“A claimed solution rooted in computer technology overcomes problems specifically arising in the realm of computer technology. In various implementations, a computing system is configured to provide methods that accelerate machine learning processes. A machine learning model may be trained with human labeled data examples and may be updated as the examples are labeled. A subset or batch of data examples may be selected from a complete data set according to their uncertainty levels. As the batch of data examples are labeled, a machine learning model may be updated and applied to the remaining batch data examples to update their uncertainty levels. The system may select the most uncertain data example from the batch for labeling. While the user continues to label examples of the batch dataset, the system may rescore the complete dataset to select the next batch of examples to be provided to the user as the first batch is completed, thus providing a lag-free and efficient machine learning model training system,” paragraph 0005 lines 1-18; “An uncertainty score of a data example may indicate a level of uncertainty in a model's evaluation of the data example. In some implementations, a data example may receive both a label and an uncertainty score when evaluated by a machine learning model. That is, the label may represent a determination result of the model and the uncertainty score may represent a confidence level in the determination result. In some implementations, the uncertainty score of the data example may be derived directly from the determination result,” paragraph 0097 lines 1-10); and
training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs (“A claimed solution rooted in computer technology overcomes problems specifically arising in the realm of computer technology. In various implementations, a computing system is configured to provide methods that accelerate machine learning processes. A machine learning model may be trained with human labeled data examples and may be updated as the examples are labeled. A subset or batch of data examples may be selected from a complete data set according to their uncertainty levels. As the batch of data examples are labeled, a machine learning model may be updated and applied to the remaining batch data examples to update their uncertainty levels. The system may select the most uncertain data example from the batch for labeling. While the user continues to label examples of the batch dataset, the system may rescore the complete dataset to select the next batch of examples to be provided to the user as the first batch is completed, thus providing a lag-free and efficient machine learning model training system,” paragraph 0005 lines 1-18); and
outputting the trained machine learning model (“In some implementations, batch size may be determined according to a user labeling time and an estimated system processing speed. The batch size may be selected such that the entirety of the batch of data examples can be reevaluated by application of an updated model while the user is labeling a data example. Thus, the user may be provided with the most uncertain example for labeling. The user may label the example, causing the model to update. While the user is labeling the next most uncertain example, the updated model may be applied to the remainder of the batch to update the uncertainty scores. When the user finishes the labeling task, the system may have finished the updating task, and a new example may be selected from the batch based on the updated batch uncertainty scores,” paragraph 0110 lines 1-14).
As to independent claim 20, Erenrich discloses one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations (“In an implementation, a method for training a machine learning model is provided. The method may be performed on a computer system having one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the computer system to perform the method,” paragraph 0007 lines 1-7) comprising:
training a machine learning model over a sequence of training iterations, comprising, at each of a plurality of training iterations in the sequence of training iterations (“The method may include obtaining, by the computer system, a machine learning model and a training dataset, the training dataset including a plurality of training examples; determining, by the computer system, uncertainty scores for the plurality of training examples according to the machine learning model; selecting, by the computer system, a first example batch from the plurality of training examples according to uncertainty scores of the plurality of training examples; updating, by the computer system, the machine learning model according to at least one labeled training example of the example batch; determining, by the computer system, updated uncertainty scores for the plurality of training examples according to the updated machine learning model; and selecting, by the computer system, a second example batch from the plurality of training examples according to the updated uncertainty scores of the plurality of training examples,” paragraph 0007 lines 7-23):
selecting a current batch of model inputs for training the machine learning model at the training iteration, wherein the current batch of model inputs comprises a plurality of model inputs (“The method may include obtaining, by the computer system, a machine learning model and a training dataset, the training dataset including a plurality of training examples; determining, by the computer system, uncertainty scores for the plurality of training examples according to the machine learning model; selecting, by the computer system, a first example batch from the plurality of training examples according to uncertainty scores of the plurality of training examples; updating, by the computer system, the machine learning model according to at least one labeled training example of the example batch; determining, by the computer system, updated uncertainty scores for the plurality of training examples according to the updated machine learning model; and selecting, by the computer system, a second example batch from the plurality of training examples according to the updated uncertainty scores of the plurality of training examples,” paragraph 0007 lines 7-23), wherein selecting the current batch of model inputs comprises:
generating a set of candidate batches of model inputs (“Batch evaluator 704 may receive a batch of data examples from sample evaluator 702. The batch of data examples may be a subset of the training dataset of data examples evaluated by sample evaluator 702, and may be selected by sample evaluator 702 and/or by batch evaluator 704. The batch of data examples may be selected according to their uncertainty scores. The selected batch of data examples may be those having the highest uncertainty scores from the training dataset. For example, the batch of data examples may include the most uncertain 10 percent of data examples from the training data set. The size of the batch of data examples may be predetermined as an absolute number (e.g., 1000 examples) and/or may be predetermined as a percentage of the training dataset. In some implementations, the size of the batch of data examples may be predetermined according to system characteristics and/or may be dynamically determined during system processing,” paragraph 0099 lines 1-17);
generating, for each candidate batch of model inputs, a respective score for the candidate batch of model inputs (“Batch evaluator 704 may obtain determination results and uncertainty scores for all or some of the batch of data examples. In some implementations, batch evaluator 704 may apply the evaluation model 710 to all data examples of the batch of data examples to obtain the determination results and uncertainty scores. In some implementations, where evaluation model 710 has not been updated after determination results and uncertainty scores were determined by sample evaluator 702, batch evaluator 704 may obtain the previously generated results and uncertainty scores,” paragraph 0101 lines 1-11; “After evaluation model 710 updating, batch evaluator 704 may obtain updated determination results and updated uncertainty scores of the remaining data examples of the batch of data examples. Batch evaluator 704 may apply updated evaluation model 710 to the remaining data examples to obtain the updated determination results and updated uncertainty scores. Example selector 706 may select a second set of one or more specific data examples from the remaining data examples of the batch of data examples for user labeling by trainer 708. Trainer 708 may communicate with user interface 524 to obtain user labels of the second set of specific data examples and user these results to again update the evaluation model. This loop may continue until a batch loop end point is reached,” paragraph 0104 lines 1-14) that characterizes:
(i) an uncertainty of the machine learning model in generating predicted labels for the model inputs in the candidate batch of model inputs (“Batch evaluator 704 may receive a batch of data examples from sample evaluator 702. The batch of data examples may be a subset of the training dataset of data examples evaluated by sample evaluator 702, and may be selected by sample evaluator 702 and/or by batch evaluator 704. The batch of data examples may be selected according to their uncertainty scores. The selected batch of data examples may be those having the highest uncertainty scores from the training dataset. For example, the batch of data examples may include the most uncertain 10 percent of data examples from the training data set. The size of the batch of data examples may be predetermined as an absolute number (e.g., 1000 examples) and/or may be predetermined as a percentage of the training dataset. In some implementations, the size of the batch of data examples may be predetermined according to system characteristics and/or may be dynamically determined during system processing,” paragraph 0099 lines 1-17; “An uncertainty score of a data example may indicate a level of uncertainty in a model's evaluation of the data example. In some implementations, a data example may receive both a label and an uncertainty score when evaluated by a machine learning model. That is, the label may represent a determination result of the model and the uncertainty score may represent a confidence level in the determination result. In some implementations, the uncertainty score of the data example may be derived directly from the determination result,” paragraph 0097 lines 1-10), and
(ii) a diversity of the model inputs in the candidate batch of model inputs (“the batch of data examples may further be determined according to a diversity measurement between data examples of the batch. If a batch of data examples is selected purely according to uncertainty scores, some examples of the batch of data examples may be very similar in one or more aspects. For example, a series of duplicate records in a database could lead to a series of identical data examples. Identical data examples would, of course, have an identical uncertainty score. Requiring a user to label more than one of a series of identical data examples would be a waste of time and resources. Two similar, but not identical data examples, may introduce the same difficulty. In some embodiments, once the first of the data examples is labeled, the resultant model updates may be sufficient to significantly reduce the uncertainty score of the second example, rendering it inefficient as a labeling example for the user. In some embodiments, rather than updating the results shown while a user is observing results, results can be optimized for diversity and presented to the user up-front. Accordingly, the system may be configured to determine the data examples of the batch according not just to the uncertainty scores, but also to a diversity score of each data example as compared to the other data examples of the batch. A diversity score of a data examples may be a measurement of the how different each data example of the batch is from each other data example of batch. In selecting the batch of data examples, the system may apply a weight to the uncertainty scores and to the diversity scores of the batch of data examples,” paragraph 0100 lines 1-29); and
selecting the current batch of model inputs from the set of candidate batches of model inputs based on the scores (“In an operation 806, process 800 may include selecting a first batch of data examples from the training dataset. The first batch of data examples may be selected according to uncertainty scores. In some implementations, the first batch of data examples may be selected as the data examples having the highest uncertainty scores in the training dataset,” paragraph 0115 lines 1-7; “In an operation 812, process 800 may include selecting a second batch of data examples from the training dataset. The second batch of data examples may be selected according to the updated uncertainty scores of the training dataset. The second batch of data examples may be used by the system to provide additional data examples to a user for labeling. In turn, the evaluation model may be updated according to the labeling performed on the second batch of data examples,” paragraph 0119 lines 1-9);
obtaining a respective target label for each model input in the current batch of model inputs, wherein a target label for a model input defines a model output that should be generated by the machine learning model by processing the model input (“A claimed solution rooted in computer technology overcomes problems specifically arising in the realm of computer technology. In various implementations, a computing system is configured to provide methods that accelerate machine learning processes. A machine learning model may be trained with human labeled data examples and may be updated as the examples are labeled. A subset or batch of data examples may be selected from a complete data set according to their uncertainty levels. As the batch of data examples are labeled, a machine learning model may be updated and applied to the remaining batch data examples to update their uncertainty levels. The system may select the most uncertain data example from the batch for labeling. While the user continues to label examples of the batch dataset, the system may rescore the complete dataset to select the next batch of examples to be provided to the user as the first batch is completed, thus providing a lag-free and efficient machine learning model training system,” paragraph 0005 lines 1-18; “An uncertainty score of a data example may indicate a level of uncertainty in a model's evaluation of the data example. In some implementations, a data example may receive both a label and an uncertainty score when evaluated by a machine learning model. That is, the label may represent a determination result of the model and the uncertainty score may represent a confidence level in the determination result. In some implementations, the uncertainty score of the data example may be derived directly from the determination result,” paragraph 0097 lines 1-10); and
training the machine learning model on at least the current batch of model inputs using the target labels for the current batch of model inputs (“A claimed solution rooted in computer technology overcomes problems specifically arising in the realm of computer technology. In various implementations, a computing system is configured to provide methods that accelerate machine learning processes. A machine learning model may be trained with human labeled data examples and may be updated as the examples are labeled. A subset or batch of data examples may be selected from a complete data set according to their uncertainty levels. As the batch of data examples are labeled, a machine learning model may be updated and applied to the remaining batch data examples to update their uncertainty levels. The system may select the most uncertain data example from the batch for labeling. While the user continues to label examples of the batch dataset, the system may rescore the complete dataset to select the next batch of examples to be provided to the user as the first batch is completed, thus providing a lag-free and efficient machine learning model training system,” paragraph 0005 lines 1-18); and
outputting the trained machine learning model (“In some implementations, batch size may be determined according to a user labeling time and an estimated system processing speed. The batch size may be selected such that the entirety of the batch of data examples can be reevaluated by application of an updated model while the user is labeling a data example. Thus, the user may be provided with the most uncertain example for labeling. The user may label the example, causing the model to update. While the user is labeling the next most uncertain example, the updated model may be applied to the remainder of the batch to update the uncertainty scores. When the user finishes the labeling task, the system may have finished the updating task, and a new example may be selected from the batch based on the updated batch uncertainty scores,” paragraph 0110 lines 1-14).
Conclusion
The prior art made of record and not relied upon is considered pertinent to Applicant’s disclosure:
US 2024/0119366 A1 disclosing ground truth variance of the model input batch
Applicant is required under 37 C.F.R. § 1.111(c) to consider these references fully when responding to this action.
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
In the interests of compact prosecution, Applicant is invited to contact the examiner via electronic media pursuant to USPTO policy outlined MPEP § 502.03. All electronic communication must be authorized in writing. Applicant may wish to file an Internet Communications Authorization Form PTO/SB/439. Applicant may wish to request an interview using the Interview Practice website: http://www.uspto.gov/patent/laws-and-regulations/interview-practice.
Applicant is reminded Internet e-mail may not be used for communication for matters under 35 U.S.C. § 132 or which otherwise require a signature. A reply to an Office action may NOT be communicated by Applicant to the USPTO via Internet e-mail. If such a reply is submitted by Applicant via Internet e-mail, a paper copy will be placed in the appropriate patent application file with an indication that the reply is NOT ENTERED. See MPEP § 502.03(II).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ryan Barrett whose telephone number is 571 270 3311. The examiner can normally be reached 9:00am to 5:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, Applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Michelle Bechtold can be reached at 571 431 0762. The fax phone number for the organization where this application or proceeding is assigned is 571 273 8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ryan Barrett/
Primary Examiner, Art Unit 2148
Prosecution Insights