DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner remark:
Claims 15-20 recite one or more computer readable storage media, which are not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire (See e.g. Par [0110] of the specification).
Information Disclosure Statement
The information disclosure statements (IDS) (2) submitted on February 22, 2024 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Claim Objections
Claims 1-23 are objected to because of the following informalities:
Claims 1, 8, and 15: "learning curves that predicts" should read "learning curves that predict"
Claims 7, 14, 20, and 21: “one or more processing operation” should read “one or more processing operations”
Claim 15: "program instruction comprising" should read "program instructions comprising"
Claim 21: “terminating, prior to completion, the training on the selected data those of the plurality of machine learning pipelines having a score greater than a selected threshold” should read “terminating, prior to completion, the training on the selected data for those of the plurality of machine learning pipelines having a score less than a selected threshold”
Claims 2-6, 9-13, 16-19, and 22-23 are additionally objected to due to dependency on an objected-to base claim.
Appropriate correction is required.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The disclosure is objected to because of the following informalities:
[0004]: "a one or more" should read "one or more"; "ranking of each of the score" should read "ranking of each of the scores"
[0013]: "in a computing environment in a computing environment by a processor in a computing environment by a processor" should read "in a computing environment by a processor"
[0019]: "training machine learning pipeline" should read "training machine learning pipelines"
[0020]: "that predicted to have" should read "that are predicted to have"
[0021]: "each trained machine learning pipelines" should read "each trained machine learning pipeline"; "curves that predicts" should read "curves that predict"
[0022]: "curve that have" should read "curve that has"; "the various learning curve may can" should read "the various learning curves can"
[0025]: "computationally efficiency" should read "computational efficiency"
[0030]: "may refer to and/or defined as" should read "may refer to and/or be defined as"
[0031]: "in alternate suggestion" should read "in an alternate suggestion"
[0069]: "and a optimization component" should read "and an optimization component 470"
[0071]: "one or more processing operation" should read "one or more processing operations"
[0073]: "each machine learning pipelines" should read "each machine learning pipeline"
[0076]: "each machine learning pipelines" should read "each machine learning pipeline"
[0077]: "based on the application, during the training, one or more" should read "based on the application, during the training, of one or more"; "curves that predicts" should read "curves that predict"
[0078]: "naive bays" should read "naive bayes"
[0080]: "exemplary an system" should read "exemplary system"
[0081]: "functional blocks' of system 500" should read "functional blocks of system 500"; "module blocks' of system 500" should read "module blocks of system 500"
[0091]: "a scores" should read "scores"
[0092]: "may informs" should read "may inform"
[0094]: missing a period at the end of the paragraph
[00104]: "one blocks" should read "one of the blocks"
[00108]: "one or more processing operation" should read "one or more processing operations"
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-25 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The analysis of the claims will follow the 2019 Revised Patent
Subject Matter Eligibility Guidance (“2019 PEG”).
Claim 1
Step 1: The claim recites a method, and therefore is directed to the statutory category of processes.
Step 2A Prong 1: The claim recites, inter alia:
“selecting a machine learning pipeline trained on a dataset according to a ranking of a plurality of machine learning pipelines each permitted to complete training on the dataset in response to applying, during the training, one or more learning curves that predicts a machine learning pipeline performance level”; This limitation encompasses mentally selecting a machine learning pipeline according to a ranking of a plurality of machine learning pipelines each permitted to complete training on the dataset (such as by mentally selecting the highest ranked machine learning pipeline) in response to mentally applying, during the training, one or more learning curves that predicts a machine learning pipeline performance level (such as by mentally predicting a machine learning pipeline performance level during training using one or more learning curves).
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites that the method is “for accelerating machine learning in a computing environment,” however this limitation merely generally links the use of the judicial exception to the field of use/technological environment of machine learning acceleration in a computing environment (MPEP 2106.05(h)). The claim further recites that the method is performed “by one or more processors,” however this limitation amounts to mere instructions to apply a judicial exception using a generic computer (MPEP 2106.05(f)).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step mirrors that of Step 2A Prong 2 above. As an ordered whole, the claim is directed to a mentally performable process of selecting a machine learning pipeline according to a ranking of a plurality of machine learning pipelines in response to applying, during training, one or more learning curves that predicts a machine learning pipeline performance level. Nothing in the claim provides significantly more than this. As such, the claim is not patent eligible.
Claim 2
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
“generating… the one or more learning curves for each of the plurality of machine learning pipelines while training on the dataset”; This limitation encompasses mentally generating the one or more learning curves for each of the plurality of machine learning pipelines while training on the dataset, such as by mentally mapping a number of training iterations spent to a validation loss.
“learning from the one or more learning curves to apply and predict a machine learning pipeline performance level on each training operation”; This limitation encompasses mentally learning from the one or more learning curves to apply and predict a machine learning performance level on each training operation, such as by mentally learning that a particular learning curve corresponds to a particular machine learning pipeline performance level.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “storing the one or more learning curves for each of the plurality of machine learning pipelines while training on the dataset,” however this limitation amounts to the insignificant extra-solution activity of mere data gathering (MPEP 2106.05(g)).
Step 2B: The claim does not contain significantly more than the judicial exception. The “storing the one or more learning curves…” limitation, in addition to reciting insignificant extra solution activity, is also directed to the well-understood, routine, and conventional activity of storing and retrieving information in memory (MPEP 2106.06(d)(II)(iv) Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015)).
Claim 3
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
“scoring each of the plurality of machine learning pipelines according to the one or more learning curves while training on the dataset”; This limitation encompasses mentally scoring each of the plurality of machine learning pipelines according to the one or more learning curves while training on the dataset.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. No further additional elements are recited, see analysis of claim 1.
Step 2B: The claim does not contain significantly more than the judicial exception. No further additional elements are recited, see analysis of claim 1.
Claim 4
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the same judicial exception as claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “permitting the training of those of the plurality of machine learning pipelines having a score, assigned in response to applying the one or more learning curves, greater than a defined threshold,” however, this limitation amounts to mere instructions to apply the above-mentioned judicial exception (MPEP 2106.05(f)) of applying one or more learning curves by permitting a generically recited training operation based on the one or more learning curves.
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step mirrors that of Step 2A Prong 2 above.
Claim 5
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the same judicial exception as claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “terminating the training of those of the plurality of machine learning pipelines having a score, assigned in response to applying the one or more learning curves, less than a defined threshold,” however, this limitation amounts to mere instructions to apply the above-mentioned judicial exception (MPEP 2106.05(f)) of applying one or more learning curves by terminating a generically recited training operation of the plurality of machine learning pipelines based on the one or more learning curves.
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step mirrors that of Step 2A Prong 2 above.
Claim 6
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
“ranking each of the plurality of machine learning pipelines according to scoring each of the plurality of machine learning pipelines according to the one or more learning curves while training on the dataset”; This limitation encompasses mentally ranking each of the plurality of machine learning pipelines according to scoring each of the plurality of machine learning pipelines according to the one or more learning curves while training on the dataset.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. No further additional elements are recited, see analysis of claim 1.
Step 2B: The claim does not contain significantly more than the judicial exception. No further additional elements are recited, see analysis of claim 1.
Claim 7
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the same judicial exception as claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “receiving the dataset for training a machine learning pipeline, wherein a machine learning pipeline includes one or more machine learning models, a plurality of various data curations, one or more processing operation, or a combination thereof,” however this limitation amounts to the insignificant extra-solution activity of mere data gathering (MPEP 2106.05(g)).
Step 2B: The claim does not contain significantly more than the judicial exception. The “receiving the dataset…” limitation, in addition to reciting insignificant extra-solution activity, is also directed to the well-understood, routine, and conventional activity of receiving or transmitting data over a network (MPEP 2106.05(d)(II)(i) OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)).
Claims 8-14
Step 1: The claims recite a system and therefore are directed to the statutory category of machines.
Step 2A Prong 1: The claims recite the same judicial exception as claims 1-7, respectively (Examiner notes that claim 9 additionally recites that the predicted performance level on each training operation is “for a subsequent machine learning pipeline on the dataset,” however the step of “learning from the one or more learning curves to apply and predict a machine learning pipeline performance level” is still mentally performable given this further limitation, as one could mentally learn to apply and predict a machine learning pipeline performance level for a subsequent pipeline from the one or more learning curves).
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The analysis at this step mirrors that of claims 1-7, respectively, except insofar as claims 8-14 are directed to a system rather than a method. The limitation of “A system… comprising: one or more computers with executable instructions that when executed cause the system to: [perform the method]” amounts to mere instructions to apply a judicial exception using a generic computer (MPEP 2106.05(f)).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step mirrors that of claims 1-7, respectively, except insofar as claims 8-14 are directed to a system rather than a method. The limitation of “A system… comprising: one or more computers with executable instructions that when executed cause the system to: [perform the method]” amounts to mere instructions to apply a judicial exception using a generic computer (MPEP 2106.05(f)).
Claims 15-17, 19-20
Step 1: The claims recite a computer program product comprising one or more computer readable storage media (excluding transitory signals, see instant specification [00110]), and therefore are directed to the statutory category of articles of manufacture.
Step 2A Prong 1: The claims recite the same judicial exception as claims 1-3 and 6-7, respectively. (Examiner notes that claim 16 additionally recites that the predicted performance level on each training operation is “for a subsequent machine learning pipeline on the dataset,” however the step of “learning from the one or more learning curves to apply and predict a machine learning pipeline performance level” is still mentally performable given this further limitation, as one could mentally learn to apply and predict a machine learning pipeline performance level for a subsequent pipeline from the one or more learning curves).
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The analysis at this step mirrors that of claims 1-3 and 6-7, respectively, except insofar as claims 15-17 and 19-20 are directed to a computer program product rather than a method. The limitation of “A computer program product… comprising: one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instruction comprising: program instructions to… [perform the method]” amounts to mere instructions to apply a judicial exception using a generic computer (MPEP 2106.05(f)).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step mirrors that of claims 1-3 and 6-7, respectively, except insofar as claims 15-17 and 19-20 are directed to a computer program product rather than a method. The limitation of “A computer program product… comprising: one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instruction comprising: program instructions to… [perform the method]” amounts to mere instructions to apply a judicial exception using a generic computer (MPEP 2106.05(f)).
Claim 18
Step 1: An article of manufacture, as above.
Step 2A Prong 1: The claim recites the same judicial exception as claim 15.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “permit the training of those of the plurality of machine learning pipelines having a score, assigned in response to applying the one or more learning curves, greater than a defined threshold; or terminate the training of those of the plurality of machine learning pipelines having a score, assigned in response to applying the one or more learning curves, less than the defined threshold,” however, this limitation amounts to mere instructions to apply the above-mentioned judicial exception (MPEP 2106.05(f)) of applying one or more learning curves by permitting or terminating a generically recited training operation of the plurality of machine learning pipelines based on the one or more learning curves.
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step mirrors that of Step 2A Prong 2 above.
Claim 21
Step 1: The claim recites a method, and therefore is directed to the statutory category of processes.
Step 2A Prong 1: The claim recites, inter alia:
“scoring each of the plurality of machine learning pipelines according to one or more learning curves while training on selected data”; This limitation encompasses mentally scoring each of the plurality of machine learning pipelines according to one or more learning curves while training on selected data.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “receiving selected data for training a plurality of machine learning pipelines, wherein a machine learning pipeline includes one or more machine learning models, a plurality of various data curations, one or more processing operation, or a combination thereof,” however, this limitation amounts to the insignificant extra-solution activity of mere data gathering (MPEP 2106.05(g)). The claim further recites “permitting completion of the training on the selected data for those of the plurality of machine learning pipelines having a score greater than a selected threshold; and terminating, prior to completion, the training on the selected data those of the plurality of machine learning pipelines having a score greater than a selected threshold,” however these limitations amount to mere instructions to apply the above-mentioned judicial exception (MPEP 2106.05(f)) of scoring each of the plurality of machine learning pipelines by permitting or terminating a generically recited training operation based on the score. The claim further recites that the method is “for accelerating machine learning in a computing environment,” however this limitation merely generally links the use of the judicial exception to the field of use/technological environment of machine learning acceleration in a computing environment (MPEP 2106.05(h)). The claim further recites that the method is performed “by one or more processors,” however this limitation amounts to mere instructions to apply a judicial exception using a generic computer (MPEP 2106.05(f)).
Step 2B: The claim does not contain significantly more than the judicial exception. The “receiving selected data…” limitation, in addition to reciting insignificant extra-solution activity, is also directed to the well-understood, routine, and conventional activity of receiving or transmitting data over a network (MPEP 2106.05(d)(II)(i) OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)). Otherwise, the analysis at this step mirrors that of Step 2A Prong 2 above. As an ordered whole, the claim is directed to a mentally performable process of scoring each of a plurality of machine learning pipelines according to one or more learning curves while training on selected data in order to determine whether to permit completion of the training or terminate the training. Nothing in the claim provides significantly more than this. As such, the claim is not patent eligible.
Claim 22
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
“generating… the one or more learning curves for each of the plurality of machine learning pipelines while training on the selected data”; This limitation encompasses mentally generating the one or more learning curves for each of the plurality of machine learning pipelines while training on the selected data, such as by mentally mapping a number of training iterations spent to a validation loss.
“learning from the one or more learning curves to apply and predict a machine learning pipeline performance level for a subsequent machine learning pipeline on the selected data”; This limitation encompasses mentally learning from the one or more learning curves to apply and predict a machine learning performance level for a subsequent machine learning pipeline on the selected data, such as by mentally learning that a particular learning curve corresponds to a particular machine learning pipeline performance level.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “storing the one or more learning curves for each of the plurality of machine learning pipelines while training on the selected data,” however this limitation amounts to the insignificant extra-solution activity of mere data gathering (MPEP 2106.05(g)).
Step 2B: The claim does not contain significantly more than the judicial exception. The “storing the one or more learning curves…” limitation, in addition to reciting insignificant extra solution activity, is also directed to the well-understood, routine, and conventional activity of storing and retrieving information in memory (MPEP 2106.06(d)(II)(iv) Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015)).
Claim 23
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
“identifying at least one of the plurality of machine learning pipelines as a preferred machine learning pipeline in response to completion of the training on the selected data”; This limitation encompasses mentally identifying at least one of the plurality of machine learning pipelines as a preferred machine learning pipeline in response to completion of the training on the selected data.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. No further additional elements are recited, see analysis of claim 21.
Step 2B: The claim does not contain significantly more than the judicial exception. No further additional elements are recited, see analysis of claim 21.
Claim 24
Step 1: The claim recites a method, and therefore is directed to the statutory category of processes.
Step 2A Prong 1: The claim recites, inter alia:
“assigning a learning curve score, using one or more learning curves, to the one or more machine learning pipelines during the training”; This limitation encompasses mentally assigning a learning curve score, using one or more learning curves, to the one or more machine learning pipelines during the training, such as by mentally determining a score for each pipeline based on a corresponding learning curve.
“identifying a trained machine learning pipeline from those of the one or more machine learning pipelines having completed the training based on a ranking of each of the scores”; This limitation encompasses mentally identifying a trained machine learning pipeline from those of the one or more machine learning pipelines having completed the training based on a ranking of each of the scores, such as by mentally identifying the highest ranked pipeline.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “training one or more machine learning pipelines using selected data,” however this limitation merely generally links the above-mentioned judicial exceptions to the field of use of model training (MPEP 2106.05(h)). The claim additionally recites “allowing the training of those of the one or more machine learning pipelines having the learning curve score greater than a selected threshold while terminating the training of those of the one or more machine learning pipelines having the learning curve score less than a selected threshold,” however this limitation amounts to mere instructions to apply the above-mentioned judicial exception (MPEP 2106.05(f)) of assigning a learning curve score to the one or more machine learning pipelines by permitting or terminating a generically recited training operation based on the learning curve score. The claim further recites that the method is “for accelerating machine learning in a computing environment,” however this limitation merely generally links the use of the judicial exception to the field of use/technological environment of machine learning acceleration in a computing environment (MPEP 2106.05(h)). The claim further recites that the method is performed “by one or more processors,” however this limitation amounts to mere instructions to apply a judicial exception using a generic computer (MPEP 2106.05(f)).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step mirrors that of Step 2A Prong 2 above. As an ordered whole, the claim is directed to a mentally performable process of assigning a learning curve score to one or more machine learning pipelines during training in order to determine whether to permit completion of the training or terminate the training, and then identifying a trained machine learning pipeline based on a ranking of the scores. Nothing in the claim provides significantly more than this. As such, the claim is not patent eligible.
Claim 25
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
“learning at least a partial learning curve for the one or more machine learning pipelines during the training”; This limitation encompasses mentally learning at least a partial learning curve for the one or more machine learning pipelines during the training.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “storing the at least a partial learning curve with a plurality of historically learned learning curves; and storing configurations of each of the one or more machine learning pipelines while being trained on the selected data,” however these limitations amount to the insignificant extra-solution activity of mere data gathering (MPEP 2106.05(g)).
Step 2B: The claim does not contain significantly more than the judicial exception. The storing limitations, in addition to reciting insignificant extra-solution activity, are also directed to the well-understood, routine, and conventional activity of storing and retrieving information in memory (MPEP 2106.05(d)(II)(iv) Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015)).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 3, 6-8, 10, 13-15, 17, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Buratti et al. (Ordalia: Deep Learning Hyperparameter Search via Generalization Error Bounds Extrapolation) (hereinafter “Buratti”) in view of Hapke et al. (Building Machine Learning Pipelines) (hereinafter “Hapke”).
Regarding claim 1, Buratti discloses “A method for accelerating machine learning in a computing environment by one or more processors comprising:
selecting a machine learning… [model] trained on a dataset according to a ranking of a plurality of machine learning… [models] each permitted to complete training on the dataset in response to applying, during the training, one or more learning curves that predicts a machine learning… [model] performance level” (Buratti, IV. B.: “Ordalia takes as input (1) n neural networks sampled at random from the search space H, (2) a training and (3) a validation set… Each time a neural network is trained using (mi, ei) resources, its validation error stored and used to extrapolate the dominant parameters
a
^
,
b
^
of its validation learning curve
l
^
η
(
∙
)
=
a
^
η
-
b
^
, where η is the product of the training sample size and number of epochs η = me3. Once the dominant parameters
a
^
and
b
^
are obtained, Ordalia computes the neural network’s performance projections and uses them as a metric to evaluate their learning progresses and decide which configuration to early-terminate” and Algorithm 1: “Ordalia-Termination(OT). Ordalia splits the training resources into exponentially larger sets and samples n configurations at random from the search space. Then at each round the algorithm computes the Ordalias’s projection for each of the surviving configuration up to that round and then ranks them accordingly: at round i just the top-
η
i
-
1
η
i
are retained for the successive rounds. During the last round all the resources η are used and the top performing configuration returned”; Examiner notes that returning the top performing configuration corresponds to “selecting a machine learning model trained on a dataset,” the ranking of the surviving configurations during the last round corresponds to “a ranking of a plurality of machine learning models each permitted to complete training on the dataset,” and using the validation learning curve for each configuration at each training round to compute a performance projection for ranking the models corresponds to “applying, during the training, one or more learning curves that predicts a machine learning model performance level”).
Buratti does not appear to explicitly disclose the further limitations of the claim.
However, Hapke discloses “machine learning pipeline” (Hapke, Overview of the Steps in a Machine Learning Pipeline, paragraph 1: “A machine learning pipeline starts with the ingestion of new training data and ends with receiving some kind of feedback on how your newly trained model is performing. This feedback can be a production performance metric or feedback from users of your product. The pipeline includes a variety of steps, including data preprocessing, model training, and model analysis, as well as the deployment of the model”).
Hapke and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified Buratti with the teachings of Hapke such that machine learning model(s) is substituted with machine learning pipeline(s), and one would have been motivated to do so for the purpose of automating model life cycle steps, which in turn would prevent bugs and standardize machine learning workflows to improve efficiency and reduce set up time (see Hapke, “Why Machine Learning Pipelines?” section).
Regarding claim 3, the rejection of claim 1 is incorporated. Buratti as modified by Hapke further discloses “scoring each of the plurality of machine learning pipelines according to the one or more learning curves while training on the dataset” (Buratti, IV. B.: “Each time a neural network is trained using (mi, ei) resources, its validation error stored and used to extrapolate the dominant parameters
a
^
,
b
^
of its validation learning curve
l
^
η
(
∙
)
=
a
^
η
-
b
^
, where η is the product of the training sample size and number of epochs η = me3. Once the dominant parameters
a
^
and
b
^
are obtained, Ordalia computes the neural network’s performance projections and uses them as a metric to evaluate their learning progresses and decide which configuration to early-terminate”; Examiner notes that using each neural network’s performance projection (computed from the learning curve) as a metric for evaluation corresponds to “scoring each of the plurality of machine learning pipelines according to the one or more learning curves while training on the dataset”).
Regarding claim 6, the rejection of claim 1 is incorporated. Buratti as modified by Hapke further discloses “ranking each of the plurality of machine learning pipelines according to scoring each of the plurality of machine learning pipelines according to the one or more learning curves while training on the dataset” (Buratti, IV. B.: “Each time a neural network is trained using (mi, ei) resources, its validation error stored and used to extrapolate the dominant parameters
a
^
,
b
^
of its validation learning curve
l
^
η
(
∙
)
=
a
^
η
-
b
^
, where η is the product of the training sample size and number of epochs η = me3. Once the dominant parameters
a
^
and
b
^
are obtained, Ordalia computes the neural network’s performance projections and uses them as a metric to evaluate their learning progresses and decide which configuration to early-terminate” and Algorithm 1: “Then at each round the algorithm computes the Ordalias’s projection for each of the surviving configuration up to that round and then ranks them accordingly”; Examiner notes that using each neural network’s performance projection (computed from the learning curve) as a metric for evaluation corresponds to “scoring each of the plurality of machine learning pipelines according to the one or more learning curves while training on the dataset,” and ranking the networks according to the projections corresponds to “ranking each of the plurality of machine learning pipelines according to scoring”).
Regarding claim 7, the rejection of claim 1 is incorporated. Buratti further discloses “receiving the dataset for training a machine learning… [model]” (Buratti, IV. B.: “Ordalia takes as input (1) n neural networks sampled at random from the search space H, (2) a training and (3) a validation set).
Buratti does not appear to explicitly disclose the further limitations of the claim.
However, Hapke discloses “…a machine learning pipeline, wherein a machine learning pipeline includes one or more machine learning models, a plurality of various data curations, one or more processing operation, or a combination thereof” (Hapke, Overview of the Steps in a Machine Learning Pipeline, paragraph 1: “A machine learning pipeline starts with the ingestion of new training data and ends with receiving some kind of feedback on how your newly trained model is performing… The pipeline includes a variety of steps, including data preprocessing, model training, and model analysis, as well as the deployment of the model”).
Hapke and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified Buratti with the teachings of Hapke such that the received dataset is for training a machine learning pipeline, wherein a machine learning pipeline includes one or more machine learning models, a plurality of various data curations, one or more processing operation, or a combination thereof, and one would have been motivated to do so for the purpose of automating model life cycle steps, which in turn would prevent bugs and standardize machine learning workflows to improve efficiency and reduce set up time. (see Hapke, “Why Machine Learning Pipelines?” section).
Regarding claim 8, Buratti discloses, “A system for accelerating machine learning in a computing environment, comprising: one or more computers with executable instructions (Buratti, V: “Hardware Environment: the experiments have been executed on Google Cloud Platform (GCP) with N1-highmem machines, with 4-core Intel Haswell vCPUs with 26 GB memory, running Debian 4.9 and Python 3.6.3. The neural networks training has been performed on a NVIDIA Tesla V100 GPUs.”) that when executed cause the system to:
select a machine learning… [model] trained on a dataset according to a ranking of a plurality of machine learning… [models] each permitted to complete training on the dataset in response to applying, during the training, one or more learning curves that predicts a machine learning… [model] performance level” (Buratti, IV. B.: “Ordalia takes as input (1) n neural networks sampled at random from the search space H, (2) a training and (3) a validation set…Each time a neural network is trained using (mi, ei) resources, its validation error stored and used to extrapolate the dominant parameters
a
^
,
b
^
of its validation learning curve
l
^
η
(
∙
)
=
a
^
η
-
b
^
, where η is the product of the training sample size and number of epochs η = me3. Once the dominant parameters
a
^
and
b
^
are obtained, Ordalia computes the neural network’s performance projections and uses them as a metric to evaluate their learning progresses and decide which configuration to early-terminate” and Algorithm 1: “Ordalia-Termination(OT). Ordalia splits the training resources into exponentially larger sets and samples n configurations at random from the search space. Then at each round the algorithm computes the Ordalias’s projection for each of the surviving configuration up to that round and then ranks them accordingly: at round i just the top-
η
i
-
1
η
i
are retained for the successive rounds. During the last round all the resources η are used and the top performing configuration returned”; Examiner notes that returning the top performing configuration corresponds to “selecting a machine learning model trained on a dataset,” the ranking of the surviving configurations during the last round corresponds to “a ranking of a plurality of machine learning models each permitted to complete training on the dataset,” and using the validation learning curve for each configuration at each training round to compute a performance projection for ranking the models corresponds to “applying, during the training, one or more learning curves that predicts a machine learning model performance level”).
Buratti does not appear to explicitly disclose the further limitations of the claim.
However, Hapke discloses “machine learning pipeline” (Hapke, Overview of the Steps in a Machine Learning Pipeline, paragraph 1: “A machine learning pipeline starts with the ingestion of new training data and ends with receiving some kind of feedback on how your newly trained model is performing. This feedback can be a production performance metric or feedback from users of your product. The pipeline includes a variety of steps, including data preprocessing, model training, and model analysis, as well as the deployment of the model”).
Hapke and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified Buratti with the teachings of Hapke such that machine learning model(s) is substituted with machine learning pipeline(s), and one would have been motivated to do so for the purpose of automating model life cycle steps, which in turn would prevent bugs and standardize machine learning workflows to improve efficiency and reduce set up time. (see Hapke, “Why Machine Learning Pipelines?” section).
Regarding claim 10, the rejection of claim 8 is incorporated. Claim 10 is a system claim corresponding to method claim 3, and the remainder of the rejection follows the same rationale given for the rejection of claim 3 above.
Regarding claim 13, the rejection of claim 8 is incorporated. Claim 13 is a system claim corresponding to method claim 6, and the remainder of the rejection follows the same rationale given for the rejection of claim 6 above.
Regarding claim 14, the rejection of claim 8 is incorporated. Claim 14 is a system claim corresponding to method claim 7, and the remainder of the rejection follows the same rationale given for the rejection of claim 7 above.
Regarding claim 15, Buratti discloses “A computer program product for accelerating machine learning in a computing environment, the computer program product comprising: one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media (Buratti, V: “Hardware Environment: the experiments have been executed on Google Cloud Platform (GCP) with N1-highmem machines, with 4-core Intel Haswell vCPUs with 26 GB memory, running Debian 4.9 and Python 3.6.3. The neural networks training has been performed on a NVIDIA Tesla V100 GPUs.”), the program instruction comprising:
program instructions to select a machine learning… [model] trained on a dataset according to a ranking of a plurality of machine learning… [models] each permitted to complete training on the dataset in response to applying, during the training, one or more learning curves that predicts a machine learning… [model] performance level” (Buratti, IV. B.: “Ordalia takes as input (1) n neural networks sampled at random from the search space H, (2) a training and (3) a validation set…Each time a neural network is trained using (mi, ei) resources, its validation error stored and used to extrapolate the dominant parameters
a
^
,
b
^
of its validation learning curve
l
^
η
(
∙
)
=
a
^
η
-
b
^
, where η is the product of the training sample size and number of epochs η = me3. Once the dominant parameters
a
^
and
b
^
are obtained, Ordalia computes the neural network’s performance projections and uses them as a metric to evaluate their learning progresses and decide which configuration to early-terminate” and Algorithm 1: “Ordalia-Termination(OT). Ordalia splits the training resources into exponentially larger sets and samples n configurations at random from the search space. Then at each round the algorithm computes the Ordalias’s projection for each of the surviving configuration up to that round and then ranks them accordingly: at round i just the top-
η
i
-
1
η
i
are retained for the successive rounds. During the last round all the resources η are used and the top performing configuration returned”; Examiner notes that returning the top performing configuration corresponds to “selecting a machine learning model trained on a dataset,” the ranking of the surviving configurations during the last round corresponds to “a ranking of a plurality of machine learning models each permitted to complete training on the dataset,” and using the validation learning curve for each configuration at each training round to compute a performance projection for ranking the models corresponds to “applying, during the training, one or more learning curves that predicts a machine learning model performance level”).
Buratti does not appear to explicitly disclose the further limitations of the claim.
However, Hapke discloses “machine learning pipeline” (Hapke, Overview of the Steps in a Machine Learning Pipeline, paragraph 1: “A machine learning pipeline starts with the ingestion of new training data and ends with receiving some kind of feedback on how your newly trained model is performing. This feedback can be a production performance metric or feedback from users of your product. The pipeline includes a variety of steps, including data preprocessing, model training, and model analysis, as well as the deployment of the model”).
Hapke and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified Buratti with the teachings of Hapke such that machine learning model(s) is substituted with machine learning pipeline(s), and one would have been motivated to do so for the purpose of automating model life cycle steps, which in turn would prevent bugs and standardize machine learning workflows to improve efficiency and reduce set up time. (see Hapke, “Why Machine Learning Pipelines?” section).
Regarding claim 17, the rejection of claim 15 is incorporated. Claim 17 is a computer program product claim corresponding to method claim 3, and the remainder of the rejection follows the same rationale given for the rejection of claim 3 above.
Regarding claim 19, the rejection of claim 15 is incorporated. Claim 19 is a computer program product claim corresponding to method claim 6, and the remainder of the rejection follows the same rationale given for the rejection of claim 6 above.
Regarding claim 20, the rejection of claim 15 is incorporated. Claim 20 is a computer program product claim corresponding to method claim 7, and the remainder of the rejection follows the same rationale given for the rejection of claim 7 above.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Buratti in view of Hapke, and further in view of Kitamura et al. (US20200134453) (hereinafter “Kitamura”).
Regarding claim 2, the rejection of claim 1 is incorporated. Buratti as modified by Hapke further discloses “generating… the one or more learning curves for each of the plurality of machine learning pipelines while training on the dataset (Buratti, IV. B.: “Each time a neural network is trained using (mi, ei) resources, its validation error stored and used to extrapolate the dominant parameters
a
^
,
b
^
of its validation learning curve
l
^
η
(
∙
)
=
a
^
η
-
b
^
, where η is the product of the training sample size and number of epochs η = me3); and
learning from the one or more learning curves to apply and predict a machine learning pipeline performance level on each training operation (Buratti, IV. B.: “Once the dominant parameters
a
^
and
b
^
are obtained, Ordalia computes the neural network’s performance projections” and Algorithm 2: “Ordalia-Projection (OP). For a given neural network configuration h Ordalia evaluates its overall learning process by extrapolating l(h)’s dominant parameters
a
^
and
b
^
in the log-log space, and then computing the projection
a
^
η
B
-
b
^
”).
Neither Buratti nor Hapke appear to explicitly disclose the further limitations of the claim.
However, Kitamura discloses “storing… one or more learning curves” (Kitamura, Fig. 1: “Past Learning Curve” in Storage Device 11 and [0036]: “The storage device 11 stores data necessary for the processing of the hyperparameter search. Examples of the necessary data include: training data used when the learning in the parameter model or the neural networks is advanced; and learning curves which correspond to hyperparameters tried so far and are to be used in the learning curve prediction”).
Kitamura and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the combination of Buratti/Hapke with the teachings of Kitamura to include “storing the one or more learning curves for each of the plurality of machine learning pipelines while training on the dataset,” and one would have been motivated to do so for the purpose of being able to access previous learning curves to use in future learning curve predictions (see Kitamura, [0036]).
Claims 4-5, 11-12, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Buratti in view of Hapke and Domhan et al. (Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves) (hereinafter “Domhan”).
Regarding claim 4, the rejection of claim 1 is incorporated. Neither Buratti nor Hapke appear to explicitly disclose the further limitations of the claim.
However, Domhan discloses “permitting… training of those of… [a] plurality of machine learning… [models] having a score, assigned in response to applying… one or more learning curves, greater than a defined threshold” (Domhan, 3.1: “In this section, we describe how we probabilistically extrapolate from a short initial portion of a learning curve to a later point. When running SGD on DNNs we measure validation performance in regular intervals. Let y1:n denote the observed performance values for the first n intervals. In our problem setup, we observe y1:n and aim to predict performance ym after a large number of intervals m >> n. We solve this problem using a probabilistic model. Our basic approach is to model the partially observed learning curve y1:n by a set of parametric model families” and 3.2: “Each time the optimizer queries the performance l(λ) of a hyperparameter setting λ we train a DNN using λ as usual, except that we terminate this run early if our extrapolation model predicts the network to eventually yield worse performance than
y
^
. More precisely, at regular intervals i during SGD training we measure and save validation set performance yi. There are emax epochs, k such intervals per epoch, and every p epochs, we gather the performance values y1:n of the n intervals so far and run MCMC to probabilistically extrapolate performance to the final step m = k×emax. We then consider the predicted probability P(ym ≥
y
^
|y1:n) that the network, after training for m intervals, will exceed the performance
y
^
. If this probability is above a threshold δ then training continues as usual for the next p epochs. Otherwise, training is terminated”; Examiner notes that the predicted probability that the network, after training for m intervals, will exceed the performance
y
^
corresponds to a score assigned in response to applying learning curve y1:n, and when the score is above a threshold δ, training is permitted to continue).
Domhan and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the combination of Buratti/Hapke with the teachings of Domhan to include “permitting the training of those of the plurality of machine learning pipelines having a score, assigned in response to applying the one or more learning curves, greater than a defined threshold,” and one would have been motivated to do so for the purpose of speeding up automatic hyperparameter optimization (see Domhan, Abstract).
Regarding claim 5, the rejection of claim 1 is incorporated. Neither Buratti nor Hapke appear to explicitly disclose the further limitations of the claim.
However, Domhan discloses “terminating… training of those of… [a] plurality of machine learning… [models] having a score, assigned in response to applying… one or more learning curves, less than a defined threshold” (Domhan, 3.1: “In this section, we describe how we probabilistically extrapolate from a short initial portion of a learning curve to a later point. When running SGD on DNNs we measure validation performance in regular intervals. Let y1:n denote the observed performance values for the first n intervals. In our problem setup, we observe y1:n and aim to predict performance ym after a large number of intervals m >> n. We solve this problem using a probabilistic model. Our basic approach is to model the partially observed learning curve y1:n by a set of parametric model families” and 3.2: “Each time the optimizer queries the performance l(λ) of a hyperparameter setting λ we train a DNN using λ as usual, except that we terminate this run early if our extrapolation model predicts the network to eventually yield worse performance than
y
^
. More precisely, at regular intervals i during SGD training we measure and save validation set performance yi. There are emax epochs, k such intervals per epoch, and every p epochs, we gather the performance values y1:n of the n intervals so far and run MCMC to probabilistically extrapolate performance to the final step m = k×emax. We then consider the predicted probability P(ym ≥
y
^
|y1:n) that the network, after training for m intervals, will exceed the performance
y
^
. If this probability is above a threshold δ then training continues as usual for the next p epochs. Otherwise, training is terminated”; Examiner notes that the predicted probability that the network, after training for m intervals, will exceed the performance
y
^
corresponds to a score assigned in response to applying learning curve y1:n, and when the score is not above a threshold δ, training is terminated).
Domhan and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the combination of Buratti/Hapke with the teachings of Domhan to include “terminating the training of those of the plurality of machine learning pipelines having a score, assigned in response to applying the one or more learning curves, less than a defined threshold,” and one would have been motivated to do so for the purpose of speeding up automatic hyperparameter optimization (see Domhan, Abstract).
Regarding claim 11, the rejection of claim 8 is incorporated. Claim 11 is a system claim corresponding to method claim 4, and the remainder of the rejection follows the same rationale given for the rejection of claim 4 above.
Regarding claim 12, the rejection of claim 8 is incorporated. Claim 12 is a system claim corresponding to method claim 5, and the remainder of the rejection follows the same rationale given for the rejection of claim 5 above.
Regarding claim 18, the rejection of claim 15 is incorporated. Neither Buratti nor Hapke appear to explicitly disclose the further limitations of the claim.
However, Domhan discloses “permit… training of those of… [a] plurality of machine learning… [models] having a score, assigned in response to applying… one or more learning curves, greater than a defined threshold; or terminate the training of those of the plurality of machine learning… [models] having a score, assigned in response to applying the one or more learning curves, less than the defined threshold” (Domhan, 3.1: “In this section, we describe how we probabilistically extrapolate from a short initial portion of a learning curve to a later point. When running SGD on DNNs we measure validation performance in regular intervals. Let y1:n denote the observed performance values for the first n intervals. In our problem setup, we observe y1:n and aim to predict performance ym after a large number of intervals m >> n. We solve this problem using a probabilistic model. Our basic approach is to model the partially observed learning curve y1:n by a set of parametric model families” and 3.2: “Each time the optimizer queries the performance l(λ) of a hyperparameter setting λ we train a DNN using λ as usual, except that we terminate this run early if our extrapolation model predicts the network to eventually yield worse performance than
y
^
. More precisely, at regular intervals i during SGD training we measure and save validation set performance yi. There are emax epochs, k such intervals per epoch, and every p epochs, we gather the performance values y1:n of the n intervals so far and run MCMC to probabilistically extrapolate performance to the final step m = k×emax. We then consider the predicted probability P(ym ≥
y
^
|y1:n) that the network, after training for m intervals, will exceed the performance
y
^
. If this probability is above a threshold δ then training continues as usual for the next p epochs. Otherwise, training is terminated”; Examiner notes that the predicted probability that the network, after training for m intervals, will exceed the performance
y
^
corresponds to a score assigned in response to applying learning curve y1:n, and when the score is above a threshold δ, training is permitted to continue, otherwise it is terminated).
Domhan and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the combination of Buratti/Hapke with the teachings of Domhan to include “program instructions to: permit the training of those of the plurality of machine learning pipelines having a score, assigned in response to applying the one or more learning curves, greater than a defined threshold; or terminate the training of those of the plurality of machine learning pipelines having a score, assigned in response to applying the one or more learning curves, less than the defined threshold,” and one would have been motivated to do so for the purpose of speeding up automatic hyperparameter optimization (see Domhan, Abstract).
Claims 9 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Buratti in view of Hapke, Kitamura, and Chandrashekaran et al. (Speeding up Hyper-parameter Optimization by Extrapolation of Learning Curves using Previous Builds) (hereinafter “Chandrashekaran”).
Regarding claim 9, the rejection of claim 8 is incorporated. Buratti as modified by Hapke further discloses “generate… the one or more learning curves for each of the plurality of machine learning pipelines while training on the dataset…” (Buratti, IV. B.: “Each time a neural network is trained using (mi, ei) resources, its validation error stored and used to extrapolate the dominant parameters
a
^
,
b
^
of its validation learning curve
l
^
η
(
∙
)
=
a
^
η
-
b
^
, where η is the product of the training sample size and number of epochs η = me3).
Neither Buratti nor Hapke appear to explicitly disclose the further limitations of the claim.
However, Kitamura discloses “store… one or more learning curves” (Kitamura, Fig. 1: “Past Learning Curve” in Storage Device 11 and [0036]: “The storage device 11 stores data necessary for the processing of the hyperparameter search. Examples of the necessary data include: training data used when the learning in the parameter model or the neural networks is advanced; and learning curves which correspond to hyperparameters tried so far and are to be used in the learning curve prediction”).
Kitamura and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the combination of Buratti/Hapke with the teachings of Kitamura to include “storing the one or more learning curves for each of the plurality of machine learning pipelines while training on the dataset,” and one would have been motivated to do so for the purpose of being able to access previous learning curves to use in future learning curve predictions (see Kitamura, [0036]).
Neither Buratti, Hapke, nor Kitamura appear to explicitly disclose the further limitations of the claim.
However, Chandrashekaran discloses “learn from… one or more learning curves to apply and predict a machine learning… [model] performance level on each training operation for a subsequent machine learning… [model] on… [a] dataset” (Chandrashekaran, 3: “We further use lk(λ) = Lk(Aλ,Dtrain,Dvalid) to denote the best validation accuracy till epoch k that Aλ achieves on data Dvalid when trained on Dtrain and 3.1: “In this section, we describe how we probabilistically extrapolate from a short initial portion of the learning curve for a given model, and information from previous builds to predict the validation accuracy at a later point. For simplicity, let yr,k indicates the best validation accuracy observed till epoch k for the rth build, i.e. yr,k = lk(λr). yr,1:n = [yr,1,yr,2,...,yr,n] denotes the observed performance values for the rth build till epoch n. m is the final epoch after which a build will be terminated (m > n). Y1:r−1,1:m = [y1,m,y2,m,...,yr−1,m] is the observed performance values from previous r − 1 builds. Our objective is, given R−1 completed previous builds (Y1:R−1,1:m), and n epochs from the current build (yR,1:n), to predict the performance of the current build at termination: yR,m. We solve this problem using an ensemble of learning curve models as described below”; Examiner notes that the performance values from previous r-1 builds correspond to “one or more learning curves” (see also 3.2, Algorithm 1, “previous learning curves: Y1:r−1,1:m”) which are used to predict the validation accuracy at a later point (corresponding to “machine learning model performance level”) for subsequent model r).
Chandrashekaran and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the combination of Buratti/Hapke/Kitamura with the teachings of Chandrashekaran to include causing the system to “learn from the one or more learning curves to apply and predict a machine learning pipeline performance level on each training operation for a subsequent machine learning pipeline on the dataset,” and one would have been motivated to do so for the purpose of using the trajectories of previous model builds to identify and terminate poorly performing builds quickly (see Chandrashekaran, 1 Introduction, paragraph 4).
Regarding claim 16, the rejection of claim 15 is incorporated. Claim 16 is a computer program product claim corresponding to system claim 9, and the remainder of the rejection follows the same rationale given for the rejection of claim 9 above.
Claims 21 and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Domhan in view of Hapke.
Regarding claim 21, Domhan discloses “A method for accelerating machine learning in a computing environment by one or more processors comprising:
receiving selected data for training a plurality of machine learning…[models] (Domhan, 4.1: “We used three popular datasets concerning object recognition from small-sized images: the image recognition datasets CIFAR-10 and CIFAR-100 [Krizhevsky, 2009] and the well known MNIST dataset [LeCun et al., 1989]… For performing the hyperparameter search on CIFAR-10 and CIFAR-100, we randomly split the training data into training and validation sets containing 40,000 and 10,000 examples, respectively. Likewise, for MNIST, we split the training data into a training set containing 50,000 examples and a validation set containing 10,000 examples. We used the deep learning framework CAFFE [Jia et al., 2014] to train DNNs on a single GPU per run”) …
scoring each of the plurality of machine learning…[models] according to one or more learning curves while training on selected data (Domhan, 3.1: “In this section, we describe how we probabilistically extrapolate from a short initial portion of a learning curve to a later point. When running SGD on DNNs we measure validation performance in regular intervals. Let y1:n denote the observed performance values for the first n intervals. In our problem setup, we observe y1:n and aim to predict performance ym after a large number of intervals m >> n. We solve this problem using a probabilistic model. Our basic approach is to model the partially observed learning curve y1:n by a set of parametric model families” and 3.2: “Each time the optimizer queries the performance l(λ) of a hyperparameter setting λ we train a DNN using λ as usual, except that we terminate this run early if our extrapolation model predicts the network to eventually yield worse performance than
y
^
. More precisely, at regular intervals i during SGD training we measure and save validation set performance yi. There are emax epochs, k such intervals per epoch, and every p epochs, we gather the performance values y1:n of the n intervals so far and run MCMC to probabilistically extrapolate performance to the final step m = k×emax. We then consider the predicted probability P(ym ≥
y
^
|y1:n) that the network, after training for m intervals, will exceed the performance
y
^
”; Examiner notes that the predicted probability that the network, after training for m intervals, will exceed the performance
y
^
corresponds to a score according to learning curve y1:n)
permitting completion of the training on the selected data for those of the plurality of machine learning… [models] having a score greater than a selected threshold (Domhan, 3.2: “If this probability is above a threshold δ then training continues as usual for the next p epochs”); and
terminating, prior to completion, the training on the selected data for those of the plurality of machine learning… [models] having a score less than a selected threshold (Domhan, 3.2: “If this probability is above a threshold δ then training continues as usual for the next p epochs. Otherwise, training is terminated”).
Domhan does not appear to explicitly disclose the further limitations of the claim.
However, Hapke discloses “machine learning pipeline” and “wherein a machine learning pipeline includes one or more machine learning models, a plurality of various data curations, one or more processing operation, or a combination thereof” (Hapke, Overview of the Steps in a Machine Learning Pipeline, paragraph 1: “A machine learning pipeline starts with the ingestion of new training data and ends with receiving some kind of feedback on how your newly trained model is performing… The pipeline includes a variety of steps, including data preprocessing, model training, and model analysis, as well as the deployment of the model”).
Hapke and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified Buratti with the teachings of Hapke such that the plurality of machine learning models is substituted with a plurality of machine learning pipelines, wherein a machine learning pipeline includes one or more machine learning models, a plurality of various data curations, one or more processing operation, or a combination thereof, and one would have been motivated to do so for the purpose of automating model life cycle steps, which in turn would prevent bugs and standardize machine learning workflows to improve efficiency and reduce set up time (see Hapke, “Why Machine Learning Pipelines?” section).
Regarding claim 23, the rejection of claim 21 is incorporated. Domhan as modified by Hapke further discloses “identifying at least one of the plurality of machine learning pipelines as a preferred machine learning pipeline in response to completion of the training on the selected data” (Domhan 4.4, Results for Large CNNs on CIFAR-10: “We trained all networks for a maximum of 800 epochs and evaluated 100 configurations. Table 5 compares the performance of the best network resulting from this experiment to the state-of-the-art for CIFAR-10 without data augmentation”).
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Domhan in view of Hapke, Kitamura, and Chandrashekaran.
Regarding claim 22, the rejection of claim 21 is incorporated. Domhan as modified by Hapke further discloses “generating… the one or more learning curves for each of the plurality of machine learning pipelines while training on the selected data…” (Domhan, 3.1: “In this section, we describe how we probabilistically extrapolate from a short initial portion of a learning curve to a later point. When running SGD on DNNs we measure validation performance in regular intervals. Let y1:n denote the observed performance values for the first n intervals. In our problem setup, we observe y1:n and aim to predict performance ym after a large number of intervals m >> n. We solve this problem using a probabilistic model. Our basic approach is to model the partially observed learning curve y1:n by a set of parametric model families”).
Neither Domhan nor Hapke appear to explicitly disclose the further limitations of the claim.
However, Kitamura discloses “storing… one or more learning curves” (Kitamura, Fig. 1: “Past Learning Curve” in Storage Device 11 and [0036]: “The storage device 11 stores data necessary for the processing of the hyperparameter search. Examples of the necessary data include: training data used when the learning in the parameter model or the neural networks is advanced; and learning curves which correspond to hyperparameters tried so far and are to be used in the learning curve prediction”).
Kitamura and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the combination of Domhan/Hapke with the teachings of Kitamura to include “storing the one or more learning curves for each of the plurality of machine learning pipelines while training on the selected data,” and one would have been motivated to do so for the purpose of being able to access previous learning curves to use in future learning curve predictions (see Kitamura, [0036]).
Neither Domhan, Hapke, nor Kitamura appear to explicitly disclose the further limitations of the claim.
However, Chandrashekaran discloses “learning from… one or more learning curves to apply and predict a machine learning… [model] performance level on each training operation for a subsequent machine learning… [model] on… selected data” (Chandrashekaran, 3: “We further use lk(λ) = Lk(Aλ,Dtrain,Dvalid) to denote the best validation accuracy till epoch k that Aλ achieves on data Dvalid when trained on Dtrain and 3.1: “In this section, we describe how we probabilistically extrapolate from a short initial portion of the learning curve for a given model, and information from previous builds to predict the validation accuracy at a later point. For simplicity, let yr,k indicates the best validation accuracy observed till epoch k for the rth build, i.e. yr,k = lk(λr). yr,1:n = [yr,1,yr,2,...,yr,n] denotes the observed performance values for the rth build till epoch n. m is the final epoch after which a build will be terminated (m > n). Y1:r−1,1:m = [y1,m,y2,m,...,yr−1,m] is the observed performance values from previous r − 1 builds. Our objective is, given R−1 completed previous builds (Y1:R−1,1:m), and n epochs from the current build (yR,1:n), to predict the performance of the current build at termination: yR,m. We solve this problem using an ensemble of learning curve models as described below”; Examiner notes that the performance values from previous r-1 builds correspond to “one or more learning curves” (see also 3.2, Algorithm 1, “previous learning curves: Y1:r−1,1:m”) which are used to predict the validation accuracy at a later point (corresponding to “machine learning model performance level”) for subsequent model r).
Chandrashekaran and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the combination of Domhan/Hapke/Kitamura with the teachings of Chandrashekaran to include “learning from the one or more learning curves to apply and predict a machine learning pipeline performance level on each training operation for a subsequent machine learning pipeline on the selected data,” and one would have been motivated to do so for the purpose of using the trajectories of previous model builds to identify and terminate poorly performing builds quickly (see Chandrashekaran, 1 Introduction, paragraph 4).
Claim 24 is rejected under 35 U.S.C. 103 as being unpatentable over Buratti in view of Domhan and Hapke.
Regarding claim 24, Buratti discloses “A method for accelerating machine learning in a computing environment by one or more processors comprising:
training one or more machine learning… [models] using selected data (Buratti, IV. B.: “Ordalia takes as input (1) n neural networks sampled at random from the search space H, (2) a training and (3) a validation set… Each time a neural network is trained using (mi, ei) resources, its validation error stored and used to extrapolate the dominant parameters
a
^
,
b
^
of its validation learning curve
l
^
η
(
∙
)
=
a
^
η
-
b
^
, where η is the product of the training sample size and number of epochs η = me3”);
assigning a learning curve score, using one or more learning curves, to the one or more machine learning… [models] during the training (Buratti, IV. B.: “Once the dominant parameters
a
^
and
b
^
are obtained, Ordalia computes the neural network’s performance projections and uses them as a metric to evaluate their learning progresses and decide which configuration to early-terminate”; Examiner notes that the performance projection of each neural network computed from the dominant parameters of the learning curve corresponds to a “learning curve score”);
…
identifying a trained machine learning… [model] from those of the one or more machine learning… [models] having completed the training based on a ranking of each of the scores” (Buratti, Algorithm 1: “Ordalia-Termination(OT). Ordalia splits the training resources into exponentially larger sets and samples n configurations at random from the search space. Then at each round the algorithm computes the Ordalias’s projection for each of the surviving configuration up to that round and then ranks them accordingly: at round i just the top-
η
i
-
1
η
i
are retained for the successive rounds. During the last round all the resources η are used and the top performing configuration returned”; Examiner notes that returning the top performing configuration corresponds to “identifying a trained machine learning model,” the surviving configurations at the last round of training correspond to “one or more machine learning models having completed the training,” and the ranking of the surviving configurations based on the Ordalia’s projection (performance projection) corresponds to “a ranking of each of the scores”).
Buratti does not appear to explicitly disclose the further limitations of the claim.
However, Domhan discloses “allowing… training of those of… one or more machine learning… [models] having… [a] learning curve score greater than a selected threshold while terminating the training of those of the one or more machine learning… [models] having the learning curve score less than a selected threshold” (Domhan, 3.1: “In this section, we describe how we probabilistically extrapolate from a short initial portion of a learning curve to a later point. When running SGD on DNNs we measure validation performance in regular intervals. Let y1:n denote the observed performance values for the first n intervals. In our problem setup, we observe y1:n and aim to predict performance ym after a large number of intervals m >> n. We solve this problem using a probabilistic model. Our basic approach is to model the partially observed learning curve y1:n by a set of parametric model families” and 3.2: “Each time the optimizer queries the performance l(λ) of a hyperparameter setting λ we train a DNN using λ as usual, except that we terminate this run early if our extrapolation model predicts the network to eventually yield worse performance than
y
^
. More precisely, at regular intervals i during SGD training we measure and save validation set performance yi. There are emax epochs, k such intervals per epoch, and every p epochs, we gather the performance values y1:n of the n intervals so far and run MCMC to probabilistically extrapolate performance to the final step m = k×emax. We then consider the predicted probability P(ym ≥
y
^
|y1:n) that the network, after training for m intervals, will exceed the performance
y
^
. If this probability is above a threshold δ then training continues as usual for the next p epochs. Otherwise, training is terminated”; Examiner notes that the predicted probability that the network, after training for m intervals, will exceed the performance
y
^
corresponds to a learning curve score using learning curve y1:n, and when the score is above a threshold δ, training is permitted to continue, otherwise it is terminated).
Domhan and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified Buratti with the teachings of Domhan to include allowing the training of those of the one or more machine learning models having the learning curve score greater than a selected threshold while terminating the training of those of the one or more machine learning models having the learning curve score less than a selected threshold, and one would have been motivated to do so for the purpose of speeding up automatic hyperparameter optimization (see Domhan, Abstract).
Neither Buratti nor Domhan appear to explicitly disclose the further limitations of the claim.
However, Hapke discloses “machine learning pipeline” (Hapke, Overview of the Steps in a Machine Learning Pipeline, paragraph 1: “A machine learning pipeline starts with the ingestion of new training data and ends with receiving some kind of feedback on how your newly trained model is performing. This feedback can be a production performance metric or feedback from users of your product. The pipeline includes a variety of steps, including data preprocessing, model training, and model analysis, as well as the deployment of the model”).
Hapke and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the combination of Buratti/Domhan with the teachings of Hapke such that machine learning model(s) is substituted with machine learning pipeline(s), and one would have been motivated to do so for the purpose of automating model life cycle steps, which in turn would prevent bugs and standardize machine learning workflows to improve efficiency and reduce set up time. (see Hapke, “Why Machine Learning Pipelines?” section).
Claim 25 is rejected under 35 U.S.C. 103 as being unpatentable over Buratti in view of Domhan, Hapke, Kitamura, and Elshawi et al. (Automated Machine Learning: State-of-The-Art and Open Challenges) (“Elshawi”).
Regarding claim 25, the rejection of claim 24 is incorporated. Buratti as modified by Domhan and Hapke further discloses “learning at least a partial learning curve for the one or more machine learning pipelines during the training” (Buratti, IV. B.: “Ordalia takes as input (1) n neural networks sampled at random from the search space H, (2) a training and (3) a validation set… Each time a neural network is trained using (mi, ei) resources, its validation error stored and used to extrapolate the dominant parameters
a
^
,
b
^
of its validation learning curve
l
^
η
(
∙
)
=
a
^
η
-
b
^
, where η is the product of the training sample size and number of epochs η = me3. Once the dominant parameters
a
^
and
b
^
are obtained, Ordalia computes the neural network’s performance projections”).
Neither Buratti nor Domhan nor Hapke appear to explicitly disclose the further limitations of the claim.
However, Kitamura discloses “storing… at least a partial learning curve with a plurality of historically learned learning curves” (Kitamura, Fig. 1: “Past Learning Curve” in Storage Device 11 and [0036]: “The storage device 11 stores data necessary for the processing of the hyperparameter search. Examples of the necessary data include: training data used when the learning in the parameter model or the neural networks is advanced; and learning curves which correspond to hyperparameters tried so far and are to be used in the learning curve prediction”).
Kitamura and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the combination of Buratti/Domhan/Hapke with the teachings of Kitamura to include “storing the at least a partial learning curve with a plurality of historically learned learning curves,” and one would have been motivated to do so for the purpose of being able to access previous learning curves to use in future learning curve predictions (see Kitamura, [0036]).
Neither Buratti, Domhan, Hapke, nor Kitamura appear to explicitly disclose the further limitations of the claim.
However, Elshawi discloses “storing configurations of each of… one or more machine learning… [models] while being trained on… selected data” (Elshawi, 5.2, paragraph 4: “Rafiki21 has been introduced as a distributed framework which is based on the idea of using previous models that achieved high performance on the same tasks [126]. In this framework, regarding the data and parameters storage, the data uploaded by user to be trained is stored in a Hadoop Distributed File System (HDFS). During training, there is a database for each model storing the best version of parameters from hyper-parameter tuning process. This database is kept in memory as it is accessed and updated frequently”).
Elshawi and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the combination of Buratti/Domhan/Hapke/Kitamura with the teachings of Elshawi to include “storing configurations of each of the one or more machine learning pipelines while being trained on the selected data,” and one would have been motivated to do so for the purpose of being able to retrieve and use previous models that achieved high performance on the same tasks (see Elshawi, 5.2, paragraph 4).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GWYNEVERE A DETERDING whose telephone number is (571) 272-7657. The examiner can normally be reached Mon-Fri. 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/G.A.D./Examiner, Art Unit 2125
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125
Prosecution Insights