Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendments
The previous claim objections are withdrawn due to Applicant’s amendments.
Response to Arguments
Applicant’s arguments filed 04/06/2026 on pages 9-12 of Remarks regarding the rejection under 35 U.S.C. 101 with respect to claims 1-17 have been fully considered but they are not persuasive.
Beginning on page 10, Applicant asserts that the amended claim 1 recites additional elements that integrate the judicial exception into a practical application under the second prong of Step 2A because the claim comprises an additional feature or a combination of features which demonstrates an improvement to the conventional automated machine learning technology. However, Examiner respectfully disagrees. MPEP 2106.04(d)(1) talks about a claim reciting a judicial exception is not directed to the judicial exception if it also recites additional elements demonstrating that the claim as a whole integrates the exception into a practical application. MPEP 2106.05(a) talks about the judicial exception alone cannot provide the improvement. The improvement can be provided by one or more additional elements. Improving an abstract element is not sufficient to integrate into a practical application. See updated rejection below. See updated rejection below.
Applicant’s arguments on pages 12-16 regarding the rejection under 35 U.S.C. 103 with respect to claims 1-17 regarding limitations “sorting the one or more first parameter sets with respect to the inference speed or architecture corresponding to the learning environment; and choosing a top percentage of the one or more first parameter sets sorted according to the learning condition inputted as the one or more second parameter sets to minimize a search range and a time required for optimization” have been fully considered but are moot. New references Tan and Jamieson have been incorporated below to teach the newly presented limitations.
Applicant’s arguments on pages 12-16 regarding the rejection under 35 U.S.C. 103 with respect to claims 1-17 regarding limitations “wherein the learning condition is related to learning environment, inference speed, or search range” have been fully considered but are not persuasive.
Beginning on page 15, Applicant asserts that in Fishkov, the inference speed and search range are targets explored internally by a Bayesian optimization algorithm, whereas, in the claimed invention, these are selection criteria directly specified by the user as external constraints of the deployment environment. However, the selection criteria directly specified by the user as external constraints of the deployment environment is not claimed in the limitations, therefore Fishkov teaches wherein the learning condition is related to learning environment, inference speed, or search range (Fishkov [0060]: “Bayesian optimization balances exploration and exploitation to search an entire domain (e.g., range, set, etc.) of possible hyperparameters. For example, the service attempts to minimize a validation error with respect to the known data, by executing a plurality of trials for a given machine-learning algorithm using different sets of hyperparameters.”;)
Claim Objections
Claim 17 is objected to because of the following informalities:
In claim 17, line 1, “A a non-transitory” should read “A non-transitory”.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-5, 8-13 and 16-17 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1,
Claim 1 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 1 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“registering one or more first parameter sets including combinations of different set data for one or more parameters having an influence on performance of learning models;”
“choosing one or more second parameter sets from the one or more first parameter sets based on a learning condition inputted;”
“choosing one of the learning models as an application model based on the validation scores,”
“wherein the learning condition is related to learning environment, inference speed, or search range, and”
“wherein choosing the one or more second parameter sets comprises:
“sorting the one or more first parameter sets with respect to the inference speed or architecture corresponding to the learning environment; and”
“choosing a top percentage of the one or more first parameter sets sorted according to the learning condition inputted as the one or more second parameter sets to minimize search range and a time required for optimization.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., registering, choosing, sorting). The above limitations in the context of this claim encompass, inter alia, registering parameter sets, choosing parameter sets, choosing learning models, sorting parameter sets, choosing a top percentage (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“producing learning models corresponding to the one or more second parameter sets by performing learning for network functions based on the one or more second parameter sets and given input datasets and calculating validation scores for the learning models;”
The following additional elements are directed to model selection using hyperparameter search and cross-validation to identify the most accurate classifier at a high level of generality. Mayhew et al. (US 20200303078), hereinafter Mayhew, discloses in [0164] “two different types of CV schemes were initially considered: conventional 5-fold cross-validation and leave-one-study-out (LOSO) cross-validation. For trials of 5-fold CV, standard methodology for randomly partitioning all IMX samples into five non-overlapping subsets of roughly similar sample sizes was used. For trials of LOSO CV, each study was treated as a CV partition. In this way, at each step (“fold”) in LOSO CV, a candidate model is trained on all studies but one, and the trained model is then used to generate predictions for the remaining study.” Mayhew has recognized using cross-validation in model selection as well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The limitations:
“producing learning models corresponding to the one or more second parameter sets by performing learning for network functions- based on the one or more second parameter sets and given input datasets and calculating validation scores for the learning models;”
The following additional elements are directed to model selection using hyperparameter search and cross-validation to identify the most accurate classifier at a high level of generality. Mayhew et al. (US 20200303078), hereinafter Mayhew, discloses in [0164] “two different types of CV schemes were initially considered: conventional 5-fold cross-validation and leave-one-study-out (LOSO) cross-validation. For trials of 5-fold CV, standard methodology for randomly partitioning all IMX samples into five non-overlapping subsets of roughly similar sample sizes was used. For trials of LOSO CV, each study was treated as a CV partition. In this way, at each step (“fold”) in LOSO CV, a candidate model is trained on all studies but one, and the trained model is then used to generate predictions for the remaining study.” Mayhew has recognized using cross-validation in model selection as well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
The claim is not patent eligible.
Regarding Claim 2,
Claim 2 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 2 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“combining the different set data for the one or more parameters to produce a plurality of candidate parameter sets;”
“determining one or more candidate parameter sets as the one or more first parameter sets according to results of the cross validation.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., combining, determining). The above limitations in the context of this claim encompass, inter alia, combining the different set data and determining candidate parameter sets (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“performing cross validation for the plurality of candidate parameter sets by performing the learning for the network functions with respect to the plurality of candidate parameter sets through a first dataset and determining at least one or more candidate parameter sets as the at least one or more first parameter sets according to results of the cross validation.”
The following additional elements are directed to model selection using hyperparameter search and cross-validation to identify the most accurate classifier at a high level of generality. Mayhew et al. (US 20200303078), hereinafter Mayhew, discloses in [0164] “two different types of CV schemes were initially considered: conventional 5-fold cross-validation and leave-one-study-out (LOSO) cross-validation. For trials of 5-fold CV, standard methodology for randomly partitioning all IMX samples into five non-overlapping subsets of roughly similar sample sizes was used. For trials of LOSO CV, each study was treated as a CV partition. In this way, at each step (“fold”) in LOSO CV, a candidate model is trained on all studies but one, and the trained model is then used to generate predictions for the remaining study.” Mayhew has recognized using cross-validation in model selection as well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The limitations:
“performing cross validation for the plurality of candidate parameter sets by performing the learning for the network functions with respect to the plurality of candidate parameter sets through a first dataset and determining at least one or more candidate parameter sets as the at least one or more first parameter sets according to results of the cross validation.”
The following additional elements are directed to model selection using hyperparameter search and cross-validation to identify the most accurate classifier at a high level of generality. Mayhew et al. (US 20200303078), hereinafter Mayhew, discloses in [0164] “two different types of CV schemes were initially considered: conventional 5-fold cross-validation and leave-one-study-out (LOSO) cross-validation. For trials of 5-fold CV, standard methodology for randomly partitioning all IMX samples into five non-overlapping subsets of roughly similar sample sizes was used. For trials of LOSO CV, each study was treated as a CV partition. In this way, at each step (“fold”) in LOSO CV, a candidate model is trained on all studies but one, and the trained model is then used to generate predictions for the remaining study.” Mayhew has recognized using cross-validation in model selection as well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
The claim is not patent eligible.
Regarding Claim 3,
Claim 3 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 3 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: Please see the corresponding analysis of Claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“wherein the performing the cross validation and the determining the one or more candidate parameter sets as the one or more first parameter sets are repeatedly performed, based on a second dataset which is different from the first dataset.”
The following additional elements are directed to model selection using hyperparameter search and cross-validation to identify the most accurate classifier at a high level of generality. Mayhew et al. (US 20200303078), hereinafter Mayhew, discloses in [0164] “two different types of CV schemes were initially considered: conventional 5-fold cross-validation and leave-one-study-out (LOSO) cross-validation. For trials of 5-fold CV, standard methodology for randomly partitioning all IMX samples into five non-overlapping subsets of roughly similar sample sizes was used. For trials of LOSO CV, each study was treated as a CV partition. In this way, at each step (“fold”) in LOSO CV, a candidate model is trained on all studies but one, and the trained model is then used to generate predictions for the remaining study.” Mayhew has recognized using cross-validation in model selection as well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The limitations:
“wherein the performing the cross validation and the determining the one or more candidate parameter sets as the one or more first parameter sets are repeatedly performed, based on a second dataset which is different from the first dataset.”
The following additional elements are directed to model selection using hyperparameter search and cross-validation to identify the most accurate classifier at a high level of generality. Mayhew et al. (US 20200303078), hereinafter Mayhew, discloses in [0164] “two different types of CV schemes were initially considered: conventional 5-fold cross-validation and leave-one-study-out (LOSO) cross-validation. For trials of 5-fold CV, standard methodology for randomly partitioning all IMX samples into five non-overlapping subsets of roughly similar sample sizes was used. For trials of LOSO CV, each study was treated as a CV partition. In this way, at each step (“fold”) in LOSO CV, a candidate model is trained on all studies but one, and the trained model is then used to generate predictions for the remaining study.” Mayhew has recognized using cross-validation in model selection as well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
The claim is not patent eligible.
Regarding Claim 4,
Claim 4 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 4 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein in the determining the one or more candidate parameter sets as the one or more first parameter sets, statistical comparison is performed based on the average and standard deviation of the validation scores, and the one or more candidate parameter sets having performance greater than a given baseline are determined as the one or more first parameter sets.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., determining). The above limitations in the context of this claim encompass, inter alia, determining candidate parameter sets (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“wherein the results of the cross validation comprise an average and standard deviation of the validation scores calculated for the plurality of candidate parameter sets, and”
The following additional elements are directed to model selection using hyperparameter search and cross-validation to identify the most accurate classifier at a high level of generality. Mayhew et al. (US 20200303078), hereinafter Mayhew, discloses in [0164] “two different types of CV schemes were initially considered: conventional 5-fold cross-validation and leave-one-study-out (LOSO) cross-validation. For trials of 5-fold CV, standard methodology for randomly partitioning all IMX samples into five non-overlapping subsets of roughly similar sample sizes was used. For trials of LOSO CV, each study was treated as a CV partition. In this way, at each step (“fold”) in LOSO CV, a candidate model is trained on all studies but one, and the trained model is then used to generate predictions for the remaining study.” Mayhew has recognized using cross-validation in model selection as well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The limitations:
“wherein the results of the cross validation comprise an average and standard deviation of validation scores calculated for the plurality of candidate parameter sets, and”
The following additional elements are directed to model selection using hyperparameter search and cross-validation to identify the most accurate classifier at a high level of generality. Mayhew et al. (US 20200303078), hereinafter Mayhew, discloses in [0164] “two different types of CV schemes were initially considered: conventional 5-fold cross-validation and leave-one-study-out (LOSO) cross-validation. For trials of 5-fold CV, standard methodology for randomly partitioning all IMX samples into five non-overlapping subsets of roughly similar sample sizes was used. For trials of LOSO CV, each study was treated as a CV partition. In this way, at each step (“fold”) in LOSO CV, a candidate model is trained on all studies but one, and the trained model is then used to generate predictions for the remaining study.” Mayhew has recognized using cross-validation in model selection as well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
The claim is not patent eligible.
Regarding Claim 5,
Claim 5 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 5 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein the one or more first parameter sets comprise set data for parameters of types of network functions, an optimizer, a learning rate, or data augmentation.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., registering). The above limitations in the context of this claim encompass, inter alia, registering parameter sets (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 1.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 8,
Claim 8 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 8 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: Please see the corresponding analysis of Claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“wherein the validation scores are calculated based on at least one of recall, precision, accuracy, and a combination thereof.”
The following additional elements are directed to model selection using hyperparameter search and cross-validation to identify the most accurate classifier at a high level of generality. Mayhew et al. (US 20200303078), hereinafter Mayhew, discloses in [0164] “two different types of CV schemes were initially considered: conventional 5-fold cross-validation and leave-one-study-out (LOSO) cross-validation. For trials of 5-fold CV, standard methodology for randomly partitioning all IMX samples into five non-overlapping subsets of roughly similar sample sizes was used. For trials of LOSO CV, each study was treated as a CV partition. In this way, at each step (“fold”) in LOSO CV, a candidate model is trained on all studies but one, and the trained model is then used to generate predictions for the remaining study.” Mayhew has recognized using cross-validation in model selection as well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The limitations:
“wherein the validation scores are calculated based on at least one of recall, precision, accuracy, and a combination thereof.”
The following additional elements are directed to model selection using hyperparameter search and cross-validation to identify the most accurate classifier at a high level of generality. Mayhew et al. (US 20200303078), hereinafter Mayhew, discloses in [0164] “two different types of CV schemes were initially considered: conventional 5-fold cross-validation and leave-one-study-out (LOSO) cross-validation. For trials of 5-fold CV, standard methodology for randomly partitioning all IMX samples into five non-overlapping subsets of roughly similar sample sizes was used. For trials of LOSO CV, each study was treated as a CV partition. In this way, at each step (“fold”) in LOSO CV, a candidate model is trained on all studies but one, and the trained model is then used to generate predictions for the remaining study.” Mayhew has recognized using cross-validation in model selection as well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
The claim is not patent eligible.
Regarding Claim 9,
Claim 9 recites an apparatus for performing steps similar of claim 1 and is rejected with the same rationale, mutatis mutandis, in view of the following additional elements, considered individually and as an ordered combination with the additional elements identified above, failing to integrate the abstract idea into a practical application or amount to significantly more than the abstract idea:
“a memory for storing a program for the automated machine learning; and”
“a processor for executing the program and configured to:”
This is a recitation of generic computing components to be used in performing the abstract idea, which does not integrate the abstract idea into a practical application or amount to significantly more than the abstract idea. See MPEP 2106.05(f).
Regarding Claim 10,
Claim 10 recites an apparatus for performing steps substantially similar to those of claim 2 and is rejected with the same rationale, mutatis mutandis.
Regarding Claim 11,
Claim 11 recites an apparatus for performing steps substantially similar to those of claim 3 and is rejected with the same rationale, mutatis mutandis.
Regarding Claim 12,
Claim 12 recites an apparatus for performing steps substantially similar to those of claim 4 and is rejected with the same rationale, mutatis mutandis.
Regarding Claim 13,
Claim 13 recites an apparatus for performing steps substantially similar to those of claim 5 and is rejected with the same rationale, mutatis mutandis.
Regarding Claim 16,
Claim 16 recites an apparatus for performing steps substantially similar to those of claim 8 and is rejected with the same rationale, mutatis mutandis.
Regarding Claim 17,
Claim 17 recites a non-transitory recording medium for performing steps similar of claim 1 and is rejected with the same rationale, mutatis mutandis, in view of the following additional elements, considered individually and as an ordered combination with the additional elements identified above, failing to integrate the abstract idea into a practical application or amount to significantly more than the abstract idea:
“A computer program stored in a non-transitory recording medium to execute a method for automated machine learning, the method comprising:”
This is a recitation of generic computing components to be used in performing the abstract idea, which does not integrate the abstract idea into a practical application or amount to significantly more than the abstract idea. See MPEP 2106.05(f).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5, 8, 9, 13, 16 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Krishna et al. (US20210295107A1); hereinafter Krishna in view of Andonie et al. (Hyperparameter optimization in learning systems); hereinafter Andonie in view of Fishkov et al. (US 20200184382 A1); hereinafter Fishkov in view of Tan et al. (MnasNet: Platform-Aware Neural Architecture Search for Mobile); hereinafter Tan and in further view of Jamieson et al. (Non-stochastic Best Arm Identification and Hyperparameter Optimization); hereinafter Jamieson
Claim 1 is rejected over Krishna, Andonie, Fishkov, Tan and Jamieson. Regarding claim 1, Krishna teaches a method for automated machine learning comprising: (“The embodiments described herein are directed to automatically determining hyperparameters for machine learning models.”; [0005])
registering one or more first parameter sets including combinations of different set data for one or more parameters having an influence on the performance of learning models; (Krishna [0002]: “The use of particular hyperparameters by a machine learning model may affect the machine learning model's performance. As such, before executing a machine learning model, hyperparameters for a machine learning model are typically first selected and tuned. “; and [0036]: “FIG. 5A illustrates a hyperparameter search space 500 that may be generated by hyperparameter determination computing device 102 and stored, for example, in database 116. Hyperparameter search space 500 includes various hyperparameter types 502, 504, 506, 508, 510, 512. For example, hyperparameter type a.sub.1 502 illustrates exemplary pre-processing hyperparameters that one or more machine learning models may be configured with.”; and [0037]: ”Hyperparameter search space 500 is hierarchically organized, where edges 520 indicate possible selections of a lower-level hyperparameter type given the selection of a higher-level hyperparameter type.”;)
choosing one or more second parameter sets from the one or more first parameter sets based on a learning condition inputted; (Krishna [0113, Figure 6]: “Proceeding to step 910, a plurality of probabilities are generated based on execution of the probability determination model. The probabilities correspond to the first set of hyperparameters. For example, hyperparameter determination computing device 102 may generate the probabilities based on execution of probability determination model 550. At step 912, a second set of hyperparameters from the pool of hyperparameters are determined based on the plurality of probabilities. At step 914, the machine learning model is configured with the second set of hyperparameters”;)
producing learning models corresponding to one or more second parameter sets by performing learning for network functions based on the one or more second parameter sets and given input datasets, and (Krishna [0012]: “The operations may include determining a second set of hyperparameters from the pool of hyperparameters based on the first plurality of values. The operations may also include configuring the machine learning model with the second set of hyperparameters.”;)
Krishna does not teach calculating validation scores for the learning models; and
choosing one of the learning models as an application model based on the validation scores.
However, Andonie teaches calculating validation scores for the learning models; and (Andonie [page 2, column 2]: “Each time we try different hyperparameters, we have to train a model on the training data, make predictions on the validation data, and then calculate the validation metric. This optimization is usually done by re-training multiple models with different combinations of hyperparameter values and evaluating their performance. We call this re-training + evaluation for one set of hyperparameter values a trial.”;)
choosing one of the learning models as an application model based on the validation scores. (Andonie [page 2, column 1]: “we want to find the model hyperparameters that yield the best score on the validation set metric.”; and [page 2, column 2]: “The validation process is more complex than described here. Each combination of hyperparameter values may result in a different model and Eq. (1) evaluates and compares the objective function for different models. The process of finding the best-performing model from a set of models that were produced by different hyperparameter settings is called model selection [41].”;)
It would have been obvious before the effective filing date to combine the selection of hyperparameters of Krishna with the multiple models trained with different combinations of hyperparameters to efficiently optimize hyperparameters (page 3, 2.1 Grid Search). Krishna and Andonie are analogous art because they concern producing the most optimal machine learning model based on finding the optimal set of hyperparameters.
Krishna does not teach wherein the learning condition is related to learning environment, inference speed, or search range, and
However, Fishkov teaches wherein the learning condition is related to learning environment, inference speed, or search range, and (Fishkov [0060]: “Bayesian optimization balances exploration and exploitation to search an entire domain (e.g., range, set, etc.) of possible hyperparameters. For example, the service attempts to minimize a validation error with respect to the known data, by executing a plurality of trials for a given machine-learning algorithm using different sets of hyperparameters.”;)
It would have been obvious before the effective filing date to combine the selection of hyperparameters of Krishna with the hyperparameter search range of Fishkov to improve the selection of hyperparameter sets (Fishkov, [0060]). Krishna and Fishkov are analogous art because they both concern optimizing machine learning through hyperparameter selection.
Krishna does not teach wherein the choosing the one or more second parameter sets comprises:
sorting the one or more first parameter sets with respect to the inference speed or architecture corresponding to the learning environment; and
However, Tan teaches wherein the choosing the one or more second parameter sets comprises:
sorting the one or more first parameter sets with respect to the inference speed or architecture corresponding to the learning environment; and (Tan [Abstract]: “we propose an automated mobile neural architecture search (MNAS) approach, which explicitly incorporate model latency into the main objective so that the search can identify a model that achieves a good trade-off between accuracy and latency … our approach directly measures real-world inference latency by executing the model on mobile phones.”; and [page 6]: “we pick three top-performing MnasNet models, with different latency-accuracy trade-offs from the same search experiment and compare them with existing mobile models.”; Note: The execution of the model on mobile phones is the learning environment.)
It would have been obvious before the effective filing date to combine the selection of hyperparameters of Krishna with the model selection based on latency of Tan to improve resource efficiency (Tan, page 2). Krishna and Tan are analogous art because they both concern parameter optimization.
Krishna does not teach choosing a top percentage of the one or more first parameter sets sorted according to the learning condition inputted as the one or more second parameter sets to minimize a search range and a time required for optimization.
However, Jamieson teaches choosing a top percentage of the one or more first parameter sets sorted according to the learning condition inputted as the one or more second parameter sets to minimize a search range and a time required for optimization. (Jamieson [page 4]: “given an input budget, uniformly allocate the budget to a set of arms for a predefined amount of iterations, evaluate their performance, throw out the worst half, and repeat until just one arm remains.”; [page 1, 1. Introduction]: “terminate these poor-performing hyperparameter settings early in a principled online fashion to speed up hyperparameter optimization”; and [Abstract]: “we typically achieve comparable test accuracies an order of magnitude faster than baseline methods.”)
It would have been obvious before the effective filing date to combine the selection of hyperparameters of Krishna with the termination of poor performing hyperparameter settings of Jamieson to speed up hyperparameter optimization (Jamieson, page 1, 1. Introduction). Krishna and Jamieson are analogous art because they both concern hyperparameter optimization.
Claim 5 is rejected over Krishna, Andonie, Fishkov, Tan and Jamieson with the incorporation of claim 1.
Regarding claim 5, Krishna teaches wherein one or more first parameter sets comprise set data for parameters of types of network functions, an optimizer, a learning rate, or data augmentation. (Krishna [0056]: “FIG. 5B illustrates a probability determination model 550 with conditional probabilities for the hyperparameter types 502, 504, 506, 508, 510, 512 of FIG. 5A. In this example, probability determination model 550 is in the form a neural network with a first layer 580, a second layer 582, a third layer 584, and a fourth layer 586.”; Note: In paragraph [46] of the present invention’s specification, “a network function may be used with the same meaning as a neural network.”)
Claim 8 is rejected over Krishna, Andonie, Fishkov, Tan and Jamieson with the incorporation of claim 1.
Regarding claim 8, Krishna teaches wherein the validation scores are calculated based on at least one of recall, precision, accuracy, and a combination thereof. (Krishna [0045]: “For example, hyperparameter determination computing device 102 may execute the trained machine learning model, which is configured with the first set of hyperparameters, to operate on validation data (e.g., supervised data). Execution of the machine learning model may generate output results (e.g., classified data). Hyperparameter determination computing device 102 may compare the output results to expected results (e.g., correct results, correct data) to generate a score (e.g., value). The score may be based on one or more of an accuracy and precision of the output results, for example.”;)
Claim 9 is rejected over Krishna, Andonie, Fishkov, Tan and Jamieson.
Regarding claim 9, Krishna teaches an apparatus for automated machine learning, comprising: (Krishna [0005]: “The apparatus and methods described herein may be applied to hyperparameter selection for machine learning models used across a variety of applications, such as to machine learning models that determine search results in response to a search request.”;)
a memory for storing a program for the automated machine learning; and (Krishna [0093]: “and hyperparameter probability determination engine 408 may be implemented as an executable program maintained in a tangible, non-transitory memory, such as instruction memory 207 of FIG. 2, that may be executed by one or processors, such as processor 201 of FIG. 2.”;)
a processor for executing the program and, the program, when executed by the processor configured to: (Krishna [0012]: “a non-transitory computer readable medium has instructions stored thereon, where the instructions, when executed by at least one processor, cause a computing device to perform operations that include configuring a machine learning model with a first set of hyperparameters from a pool of hyperparameters.”;)
The remainder of claim 9 is claim 1 in the form of an apparatus and is rejected for the same reasons as claim 1 stated above.
Dependent claim 13 is claim 5 in the form of an apparatus and is rejected for the same reasons as claim 5 stated above. For the rejection of the limitations specifically pertaining to the apparatus of claim 9, see the rejection of claim 9 above.
Dependent claim 16 is claim 8 in the form of an apparatus and is rejected for the same reasons as claim 8 stated above. For the rejection of the limitations specifically pertaining to the apparatus of claim 9, see the rejection of claim 9 above.
Claim 17 is rejected over Krishna, Andonie, Fishkov, Tan and Jamieson.
Regarding claim 17, Krishna teaches a non-transitory computer readable medium comprising a computer program, when executed by a processor, configured to perform a method for automated machine learning, the method comprising: (Krishna [0012]: “a non-transitory computer readable medium has instructions stored thereon, where the instructions, when executed by at least one processor, cause a computing device to perform operations that include configuring a machine learning model with a first set of hyperparameters from a pool of hyperparameters.”;)
The remainder of claim 17 is claim 1 in the form of a non-transitory recording medium and is rejected for the same reasons as claim 1 stated above.
Claims 2, 3, 10 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Krishna, Andonie, Fishkov, Tan and Jamieson in further view of Nguyen et al. (US20190042887A1); hereinafter Nguyen
Claim 2 is rejected over Krishna, Andonie, Fishkov, Tan, Jamieson and Nguyen with the incorporation of claim 1. Regarding claim 2, Krishna teaches combining the different set data for the one or more parameters to produce a plurality of candidate parameter sets; (Krishna [0036]: “Hyperparameter search space 500 includes various hyperparameter types 502, 504, 506, 508, 510, 512. For example, hyperparameter type a.sub.1 502 illustrates exemplary pre-processing hyperparameters that one or more machine learning models may be configured with.”; [0036]; and [0037]: “Hyperparameter search space 500 is hierarchically organized, where edges 520 indicate possible selections of a lower-level hyperparameter type given the selection of a higher-level hyperparameter type.”;)
Krishna does not teach performing cross validation for the plurality of candidate parameter sets by performing the learning for the network functions with respect to the plurality of candidate parameter sets through a first dataset;
and determining one or more candidate parameter sets as the at least one or more first parameter sets according to the results of the cross validation.
However, Nguyen teaches performing cross validation for the plurality of candidate parameter sets by performing the learning for the network functions with respect to the plurality of candidate parameter sets through a first dataset; (Nguyen [0024]: “Training management module 120 can be configured to evaluate the effectiveness of each of the trained predictive models (FIGS. 2, 3G). For example, management module 120 is operable to estimate the effectiveness of each trained predictive model. In some implementations, cross-validation can be used to estimate the effectiveness of each trained predictive model by applying each model to the test set. For example, the training management module 120 can provide to each of the model training systems 160 the test set of data to determine the effectiveness of the model developed by the training system. The model training module 164 can apply the transformation of the data pipeline to the test set, apply the trained machine learning model to the test set to determine a value for the dependent variable for each row in the test set and return the determined values for each row to training management module 120 (e.g., via the caching layer).”; [0073]; and “The modelling system 100 may support multiple machine learning algorithms to train models including, but not limited to, generalized linear regression models (linear, logistic, exponential, and other regression models), decision trees (random forest, gradient boosted trees, xgboost), support vector machines and neural networks”; Note: In paragraph [46] of the present invention’s specification, “a network function may be used with the same meaning as a neural network.”)
and determining one or more candidate parameter sets as the at least one or more first parameter sets according to the results of the cross validation. (Nguyen [0056]: “Multiple different hyper parameter configurations can be applied in training, generating multiple different trained predictive models. The generated models (e.g., candidate generated models with candidate hyper parameter sets) can be evaluated and a particular trained model selected. For example, the evaluation may include an evaluation of the effectiveness of the candidate hyper parameter sets, based on the generated models, and the selection may be based on a measure of predictive performance that may be determined as an optimal measure of performance, for the set of candidate hyper parameter sets.”;)
It would have been obvious before the effective filing date to combine the selection of hyperparameters of Krishna with the cross validation of Nguyen to efficiently measure the effectiveness of a trained predictive model based on candidate hyperparameter sets (Nguyen, [0072]). Krishna and Nguyen are analogous art because they both concern hyperparameter optimization for machine learning model production.
Claim 3 is rejected over Krishna, Andonie and Nguyen with the incorporation of claim 1.
Regarding claim 3, Krishna does not teach performing the cross validation and the determining the one or more candidate parameter sets as the one or more first parameter sets are repeatedly performed, based on a second dataset which is different from the first dataset.
However, Nguyen teaches performing the cross validation and the determining the at least one or more candidate parameter sets as the at least one or more first parameter sets are repeatedly performed, based on a second dataset which is different from the first dataset. (Nguyen [0078]: “The predictive model generated by each training system 160 or effectiveness metric of the predictive model generated by each training system 160 can be returned to training management system 110 and evaluated as discussed above. Rounds of model training can be repeated using new hyper parameters until a model reaches a threshold level of effectiveness or other condition is met. In some embodiments, training rounds can be repeated until the change in effectiveness between two rounds drops below a pre-defined threshold. In any event, whether performed in multiple rounds or a single round the most performant hyper parameter set may be selected (212).”; and [0079]: “Training to select a predictive model may involve training on only a sample of the training data, or not all of the training data at one time. For example, if k-fold cross-validation was used to estimate the effectiveness of the trained models, then the selected model will not have been trained with all of the training data at the time it is selected, but rather only K-1 partitions of the training data.”;)
It would have been obvious before the effective filing date to combine the selection of hyperparameters of Krishna with the cross validation of Nguyen to efficiently measure the effectiveness of a trained predictive model based on candidate hyperparameter sets. Krishna and Nguyen are analogous art because they both concern hyperparameter optimization for machine learning model production.
Dependent claim 10 is claim 2 in the form of an apparatus and is rejected for the same reasons as claim 2 stated above. For the rejection of the limitations specifically pertaining to the apparatus of claim 9, see the rejection of claim 9 above.
Dependent claim 11 is claim 3 in the form of an apparatus and is rejected for the same reasons as claim 3 stated above. For the rejection of the limitations specifically pertaining to the apparatus of claim 9, see the rejection of claim 9 above.
Claims 4 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Krishna, Andonie, Fishkov, Tan and Jamieson and in further view of Hughes et al. (US20200202171A1); hereinafter Hughes
Claim 4 is rejected over Krishna, Andonie, Fishkov, Tan, Jamieson and Hughes with the incorporation of claim 1.
Regarding claim 4, Krishna teaches the one or more candidate parameter sets having performance greater than a given baseline are determined as at least one or more first parameter sets. (Krishna [0063]: “Hyperparameter determination computing device 102 may continue the above process until, for example, a generated score is beyond (e.g., above) a threshold (e.g., a predetermined threshold). In some examples, hyperparameter determination computing device 102 repeats the process for a number of iterations (e.g., a number of predetermined iterations). As training progresses, the hyper-parameter configuration that yields a best validation accuracy score is stored (e.g., cached and updated). At the end of training, the machine learning model is configured with this stored hyper-parameter configuration. In some examples, the machine learning model is configured with the last selected subset of hyperparameters. In some examples, the machine learning model is configured with hyperparameters associated with the highest probabilities. In some examples, the machine learning model is configured with hyperparameters associated with the highest values corresponding to learnable parameters.”; Note: The threshold is the baseline.)
Krishna does not teach wherein the results of the cross validation comprise an average and standard deviation of the validation scores calculated for the plurality of candidate parameter sets, and
wherein in the determining the one or more candidate parameter sets as the one or more first parameter sets, statistical comparison is performed based on the average and standard deviation of the validation scores, and
However, Hughes teaches wherein the results of the cross validation comprise an average and standard deviation of the validation scores calculated for the plurality of candidate parameter sets, and (Hughes [0174]: “The model convergence metric may be a standard deviation of metrics across runs, across cross-validation folds, and/or across cross-validation averages. Other stopping criteria for the annotation process 400 may be used.”;)
wherein in the determining the one or more candidate parameter sets as the one or more first parameter sets, statistical comparison is performed based on the average and standard deviation of the validation scores, and (Hughes [0159]: “As in the baseline selection process, the annotation server 202 uses the model type to select an appropriate search space. A search space consists of a family of algorithms, their associated loss functions, and potential hyperparameters for tuning the algorithm. During a single hyperparameter optimization run, an algorithm and sample hyperparameters are selected, a model is trained and metrics are calculated.”;)
It would have been obvious before the effective filing date to combine the selection of hyperparameters of Krishna with the standard deviation of metrics of Hughes for rapid and focused training of machine learning models (Hughes, [0169]). Krishna and Hughes are analogous art because they both concern hyperparameter optimization for machine learning model production.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID H TRAN whose telephone number is (703)756-1525. The examiner can normally be reached M-F 9:30 am - 5:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at (571) 270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID H TRAN/Examiner, Art Unit 2147
/VIKER A LAMARDO/Supervisory Patent Examiner, Art Unit 2147
