Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
Acknowledgment is made of the Information Disclosure Statement dated 12/28/2022. All of the cited references have been considered.
Drawings
The drawings have been received on 12/28/2022. These drawings are accepted.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Objections
Claim 9 objected to because of the following informalities:
In claim 9, line 1, “A apparatus for automated machine learning” should be “An apparatus for automated machine learning.”
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-17 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1,
Claim 1 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 1 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“registering at least one or more first parameter sets including combinations of different set data for at least one or more parameters having an influence on performance of learning models;”
“choosing at least one or more second parameter sets to be used for production of the learning models from the at least one or more first parameter sets, based on learning conditions inputted;”
“choosing one of the produced learning models as an application model, based on the calculated validation scores.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., registering, choosing). The above limitations in the context of this claim encompass, inter alia, registering parameter sets, choosing parameter sets and choosing produced learning models (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“producing learning models corresponding to the at least one or more second parameter sets by performing learning for network functions- based on the chosen at least one or more second parameter sets and given input datasets and calculating validation scores for the respective learning models produced;”
The following additional elements are directed to model selection using hyperparameter search and cross-validation to identify the most accurate classifier at a high level of generality. Mayhew et al. (US 20200303078), hereinafter Mayhew, discloses in [0164] “two different types of CV schemes were initially considered: conventional 5-fold cross-validation and leave-one-study-out (LOSO) cross-validation. For trials of 5-fold CV, standard methodology for randomly partitioning all IMX samples into five non-overlapping subsets of roughly similar sample sizes was used. For trials of LOSO CV, each study was treated as a CV partition. In this way, at each step (“fold”) in LOSO CV, a candidate model is trained on all studies but one, and the trained model is then used to generate predictions for the remaining study.” Mayhew has recognized using cross-validation in model selection as well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The limitations:
“producing learning models corresponding to the at least one or more second parameter sets by performing learning for network functions- based on the chosen at least one or more second parameter sets and given input datasets and calculating validation scores for the respective learning models produced;”
The following additional elements are directed to model selection using hyperparameter search and cross-validation to identify the most accurate classifier at a high level of generality. Mayhew et al. (US 20200303078), hereinafter Mayhew, discloses in [0164] “two different types of CV schemes were initially considered: conventional 5-fold cross-validation and leave-one-study-out (LOSO) cross-validation. For trials of 5-fold CV, standard methodology for randomly partitioning all IMX samples into five non-overlapping subsets of roughly similar sample sizes was used. For trials of LOSO CV, each study was treated as a CV partition. In this way, at each step (“fold”) in LOSO CV, a candidate model is trained on all studies but one, and the trained model is then used to generate predictions for the remaining study.” Mayhew has recognized using cross-validation in model selection as well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
The claim is not patent eligible.
Regarding Claim 2,
Claim 2 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 2 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“combining the different set data for the at least one or more parameters to produce a plurality of candidate parameter sets;”
“determining at least one or more candidate parameter sets as the at least one or more first parameter sets according to results of the cross validation.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., combining, determining). The above limitations in the context of this claim encompass, inter alia, combining the different set data and determining candidate parameter sets (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“performing cross validation for the plurality of candidate parameter sets by performing the learning for the network functions with respect to the produced respective candidate parameter sets through a first dataset and determining at least one or more candidate parameter sets as the at least one or more first parameter sets according to results of the cross validation.”
The following additional elements are directed to model selection using hyperparameter search and cross-validation to identify the most accurate classifier at a high level of generality. Mayhew et al. (US 20200303078), hereinafter Mayhew, discloses in [0164] “two different types of CV schemes were initially considered: conventional 5-fold cross-validation and leave-one-study-out (LOSO) cross-validation. For trials of 5-fold CV, standard methodology for randomly partitioning all IMX samples into five non-overlapping subsets of roughly similar sample sizes was used. For trials of LOSO CV, each study was treated as a CV partition. In this way, at each step (“fold”) in LOSO CV, a candidate model is trained on all studies but one, and the trained model is then used to generate predictions for the remaining study.” Mayhew has recognized using cross-validation in model selection as well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The limitations:
“performing cross validation for the plurality of candidate parameter sets by performing the learning for the network functions with respect to the produced respective candidate parameter sets through a first dataset and determining at least one or more candidate parameter sets as the at least one or more first parameter sets according to results of the cross validation.”
The following additional elements are directed to model selection using hyperparameter search and cross-validation to identify the most accurate classifier at a high level of generality. Mayhew et al. (US 20200303078), hereinafter Mayhew, discloses in [0164] “two different types of CV schemes were initially considered: conventional 5-fold cross-validation and leave-one-study-out (LOSO) cross-validation. For trials of 5-fold CV, standard methodology for randomly partitioning all IMX samples into five non-overlapping subsets of roughly similar sample sizes was used. For trials of LOSO CV, each study was treated as a CV partition. In this way, at each step (“fold”) in LOSO CV, a candidate model is trained on all studies but one, and the trained model is then used to generate predictions for the remaining study.” Mayhew has recognized using cross-validation in model selection as well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
The claim is not patent eligible.
Regarding Claim 3,
Claim 3 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 3 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: Please see the corresponding analysis of Claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“wherein the performing the cross validation and the determining the at least one or more candidate parameter sets as the at least one or more first parameter sets are repeatedly performed, based on a second dataset which is different from the first dataset.”
The following additional elements are directed to model selection using hyperparameter search and cross-validation to identify the most accurate classifier at a high level of generality. Mayhew et al. (US 20200303078), hereinafter Mayhew, discloses in [0164] “two different types of CV schemes were initially considered: conventional 5-fold cross-validation and leave-one-study-out (LOSO) cross-validation. For trials of 5-fold CV, standard methodology for randomly partitioning all IMX samples into five non-overlapping subsets of roughly similar sample sizes was used. For trials of LOSO CV, each study was treated as a CV partition. In this way, at each step (“fold”) in LOSO CV, a candidate model is trained on all studies but one, and the trained model is then used to generate predictions for the remaining study.” Mayhew has recognized using cross-validation in model selection as well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The limitations:
“wherein the performing the cross validation and the determining the at least one or more candidate parameter sets as the at least one or more first parameter sets are repeatedly performed, based on a second dataset which is different from the first dataset.”
The following additional elements are directed to model selection using hyperparameter search and cross-validation to identify the most accurate classifier at a high level of generality. Mayhew et al. (US 20200303078), hereinafter Mayhew, discloses in [0164] “two different types of CV schemes were initially considered: conventional 5-fold cross-validation and leave-one-study-out (LOSO) cross-validation. For trials of 5-fold CV, standard methodology for randomly partitioning all IMX samples into five non-overlapping subsets of roughly similar sample sizes was used. For trials of LOSO CV, each study was treated as a CV partition. In this way, at each step (“fold”) in LOSO CV, a candidate model is trained on all studies but one, and the trained model is then used to generate predictions for the remaining study.” Mayhew has recognized using cross-validation in model selection as well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
The claim is not patent eligible.
Regarding Claim 4,
Claim 4 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 4 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein in the determining the at least one or more candidate parameter sets as the first parameter sets, statistical comparison is performed based on the average and standard deviation of the validation scores, and the at least one or more candidate parameter sets having performance greater than a given baseline are determined as the at least one or more first parameter sets.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., determining). The above limitations in the context of this claim encompass, inter alia, determining candidate parameter sets (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“wherein the results of the cross validation comprise an average and standard deviation of validation scores calculated for the respective candidate parameter sets, and”
The following additional elements are directed to model selection using hyperparameter search and cross-validation to identify the most accurate classifier at a high level of generality. Mayhew et al. (US 20200303078), hereinafter Mayhew, discloses in [0164] “two different types of CV schemes were initially considered: conventional 5-fold cross-validation and leave-one-study-out (LOSO) cross-validation. For trials of 5-fold CV, standard methodology for randomly partitioning all IMX samples into five non-overlapping subsets of roughly similar sample sizes was used. For trials of LOSO CV, each study was treated as a CV partition. In this way, at each step (“fold”) in LOSO CV, a candidate model is trained on all studies but one, and the trained model is then used to generate predictions for the remaining study.” Mayhew has recognized using cross-validation in model selection as well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The limitations:
“wherein the results of the cross validation comprise an average and standard deviation of validation scores calculated for the respective candidate parameter sets, and”
The following additional elements are directed to model selection using hyperparameter search and cross-validation to identify the most accurate classifier at a high level of generality. Mayhew et al. (US 20200303078), hereinafter Mayhew, discloses in [0164] “two different types of CV schemes were initially considered: conventional 5-fold cross-validation and leave-one-study-out (LOSO) cross-validation. For trials of 5-fold CV, standard methodology for randomly partitioning all IMX samples into five non-overlapping subsets of roughly similar sample sizes was used. For trials of LOSO CV, each study was treated as a CV partition. In this way, at each step (“fold”) in LOSO CV, a candidate model is trained on all studies but one, and the trained model is then used to generate predictions for the remaining study.” Mayhew has recognized using cross-validation in model selection as well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
The claim is not patent eligible.
Regarding Claim 5,
Claim 5 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 5 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein the at least one or more first parameter sets comprise the set data for at least one of parameters of types of network functions, an optimizer, a learning rate, and data augmentation.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., registering,). The above limitations in the context of this claim encompass, inter alia, registering parameter sets (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 1.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 6,
Claim 6 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 6 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein the learning conditions are related to at least one of learning environment, inference speed, and search range.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., choosing). The above limitations in the context of this claim encompass, inter alia, choosing parameter sets (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 1.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 7,
Claim 7 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 7 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“sorting the at least one or more first parameter sets with respect to at least one of architecture and the inference speed; and”
“choosing a given top percentage of the at least one or more first parameter sets sorted according to the learning conditions inputted as the at least one or more second parameter sets.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., sorting, choosing). The above limitations in the context of this claim encompass, inter alia, sorting parameter sets, choosing a given top percentage (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 1.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 8,
Claim 8 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 8 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: Please see the corresponding analysis of Claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“wherein the validation scores are calculated based on at least one of recall, precision, accuracy, and a combination thereof.”
The following additional elements are directed to model selection using hyperparameter search and cross-validation to identify the most accurate classifier at a high level of generality. Mayhew et al. (US 20200303078), hereinafter Mayhew, discloses in [0164] “two different types of CV schemes were initially considered: conventional 5-fold cross-validation and leave-one-study-out (LOSO) cross-validation. For trials of 5-fold CV, standard methodology for randomly partitioning all IMX samples into five non-overlapping subsets of roughly similar sample sizes was used. For trials of LOSO CV, each study was treated as a CV partition. In this way, at each step (“fold”) in LOSO CV, a candidate model is trained on all studies but one, and the trained model is then used to generate predictions for the remaining study.” Mayhew has recognized using cross-validation in model selection as well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The limitations:
“wherein the validation scores are calculated based on at least one of recall, precision, accuracy, and a combination thereof.”
The following additional elements are directed to model selection using hyperparameter search and cross-validation to identify the most accurate classifier at a high level of generality. Mayhew et al. (US 20200303078), hereinafter Mayhew, discloses in [0164] “two different types of CV schemes were initially considered: conventional 5-fold cross-validation and leave-one-study-out (LOSO) cross-validation. For trials of 5-fold CV, standard methodology for randomly partitioning all IMX samples into five non-overlapping subsets of roughly similar sample sizes was used. For trials of LOSO CV, each study was treated as a CV partition. In this way, at each step (“fold”) in LOSO CV, a candidate model is trained on all studies but one, and the trained model is then used to generate predictions for the remaining study.” Mayhew has recognized using cross-validation in model selection as well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
The claim is not patent eligible.
Regarding Claim 9,
Claim 9 recites an apparatus for performing steps similar of claim 1 and is rejected with the same rationale, mutatis mutandis, in view of the following additional elements, considered individually and as an ordered combination with the additional elements identified above, failing to integrate the abstract idea into a practical application or amount to significantly more than the abstract idea:
“a memory for storing a program for the automated machine learning; and”
“a processor for executing the program and configured to:”
This is a recitation of generic computing components to be used in performing the abstract idea, which does not integrate the abstract idea into a practical application or amount to significantly more than the abstract idea. See MPEP 2106.05(f).
Regarding Claim 10,
Claim 10 recites an apparatus for performing steps substantially similar to those of claim 2 and is rejected with the same rationale, mutatis mutandis.
Regarding Claim 11,
Claim 11 recites an apparatus for performing steps substantially similar to those of claim 3 and is rejected with the same rationale, mutatis mutandis.
Regarding Claim 12,
Claim 12 recites an apparatus for performing steps substantially similar to those of claim 4 and is rejected with the same rationale, mutatis mutandis.
Regarding Claim 13,
Claim 13 recites an apparatus for performing steps substantially similar to those of claim 5 and is rejected with the same rationale, mutatis mutandis.
Regarding Claim 14,
Claim 14 recites an apparatus for performing steps substantially similar to those of claim 6 and is rejected with the same rationale, mutatis mutandis.
Regarding Claim 15,
Claim 15 recites an apparatus for performing steps substantially similar to those of claim 7 and is rejected with the same rationale, mutatis mutandis.
Regarding Claim 16,
Claim 16 recites an apparatus for performing steps substantially similar to those of claim 8 and is rejected with the same rationale, mutatis mutandis.
Regarding Claim 17,
Claim 17 recites a non-transitory recording medium for performing steps similar of claim 1 and is rejected with the same rationale, mutatis mutandis, in view of the following additional elements, considered individually and as an ordered combination with the additional elements identified above, failing to integrate the abstract idea into a practical application or amount to significantly more than the abstract idea:
“A computer program stored in a non-transitory recording medium to execute a method for automated machine learning, the method comprising:”
This is a recitation of generic computing components to be used in performing the abstract idea, which does not integrate the abstract idea into a practical application or amount to significantly more than the abstract idea. See MPEP 2106.05(f).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5, 8, 9, 13, 16 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Krishna et al. (US20210295107A1); hereinafter Krishna in view of Andonie et al. (Hyperparameter optimization in learning systems); hereinafter Andonie
Claim 1 is rejected over Krishna and Andonie. Regarding claim 1, Krishna teaches a method for automated machine learning comprising: (“The embodiments described herein are directed to automatically determining hyperparameters for machine learning models.”; [0005])
registering at least one or more first parameter sets including combinations of different set data for at least one or more parameters having an influence on the performance of learning models; (“The use of particular hyperparameters by a machine learning model may affect the machine learning model's performance. As such, before executing a machine learning model, hyperparameters for a machine learning model are typically first selected and tuned. “; [0002]; and “FIG. 5A illustrates a hyperparameter search space 500 that may be generated by hyperparameter determination computing device 102 and stored, for example, in database 116. Hyperparameter search space 500 includes various hyperparameter types 502, 504, 506, 508, 510, 512. For example, hyperparameter type a.sub.1 502 illustrates exemplary pre-processing hyperparameters that one or more machine learning models may be configured with.”; [0036]; and ”Hyperparameter search space 500 is hierarchically organized, where edges 520 indicate possible selections of a lower-level hyperparameter type given the selection of a higher-level hyperparameter type.”; [0037])
choosing at least one or more second parameter sets to be used for production of the learning models from the at least one or more first parameter sets, based on learning conditions inputted; (“Proceeding to step 910, a plurality of probabilities are generated based on execution of the probability determination model. The probabilities correspond to the first set of hyperparameters. For example, hyperparameter determination computing device 102 may generate the probabilities based on execution of probability determination model 550. At step 912, a second set of hyperparameters from the pool of hyperparameters are determined based on the plurality of probabilities. At step 914, the machine learning model is configured with the second set of hyperparameters”; [0113], Figure 6)
producing learning models corresponding to the at least one or more second parameter sets by performing learning for network functions based on the chosen at least one or more second parameter sets and given input datasets, and (“The operations may include determining a second set of hyperparameters from the pool of hyperparameters based on the first plurality of values. The operations may also include configuring the machine learning model with the second set of hyperparameters.”; [0012])
Krishna does not teach calculating validation scores for respective learning models produced; and
choosing one of the produced learning models as an application model, based on the calculated validation scores.
However, Andonie teaches calculating validation scores for respective learning models produced; and (“Each time we try different hyperparameters, we have to train a model on the training data, make predictions on the validation data, and then calculate the validation metric. This optimization is usually done by re-training multiple models with different combinations of hyperparameter values and evaluating their performance. We call this re-training + evaluation for one set of hyperparameter values a trial.”; page 2, column 2)
choosing one of the produced learning models as an application model, based on the calculated validation scores. (“we want to find the model hyperparameters that yield the best score on the validation set metric.”; page 2, column 1; and “The validation process is more complex than described here. Each combination of hyperparameter values may result in a different model and Eq. (1) evaluates and compares the objective function for different models. The process of finding the best-performing model from a set of models that were produced by different hyperparameter settings is called model selection [41].”; page 2, column 2)
It would have been obvious before the effective filing date to combine the selection of hyperparameters of Krishna with the multiple models trained with different combinations of hyperparameters to efficiently optimize hyperparameters (page 3, 2.1 Grid Search). Krishna and Andonie are analogous art because they concern producing the most optimal machine learning model based on finding the optimal set of hyperparameters.
Claim 5 is rejected over Krishna and Andonie with the incorporation of claim 1.
Regarding claim 5, Krishna teaches wherein the at least one or more first parameter sets comprise the set data for at least one of parameters of types of network functions, an optimizer, a learning rate, and data augmentation. (“FIG. 5B illustrates a probability determination model 550 with conditional probabilities for the hyperparameter types 502, 504, 506, 508, 510, 512 of FIG. 5A. In this example, probability determination model 550 is in the form a neural network with a first layer 580, a second layer 582, a third layer 584, and a fourth layer 586.”; [0056]; Note: In paragraph [46] of the present invention’s specification, “a network function may be used with the same meaning as a neural network.”)
Claim 8 is rejected over Krishna and Andonie with the incorporation of claim 1.
Regarding claim 8, Krishna teaches wherein the validation scores are calculated based on at least one of recall, precision, accuracy, and a combination thereof. (“For example, hyperparameter determination computing device 102 may execute the trained machine learning model, which is configured with the first set of hyperparameters, to operate on validation data (e.g., supervised data). Execution of the machine learning model may generate output results (e.g., classified data). Hyperparameter determination computing device 102 may compare the output results to expected results (e.g., correct results, correct data) to generate a score (e.g., value). The score may be based on one or more of an accuracy and precision of the output results, for example.”; [0045])
Claim 9 is rejected over Krishna and Andonie.
Regarding claim 9, Krishna teaches a apparatus for automated machine learning, comprising: (“The apparatus and methods described herein may be applied to hyperparameter selection for machine learning models used across a variety of applications, such as to machine learning models that determine search results in response to a search request.”; [0005])
a memory for storing a program for the automated machine learning; and (“and hyperparameter probability determination engine 408 may be implemented as an executable program maintained in a tangible, non-transitory memory, such as instruction memory 207 of FIG. 2, that may be executed by one or processors, such as processor 201 of FIG. 2.”; [0093])
a processor for executing the program and configured to: (“a non-transitory computer readable medium has instructions stored thereon, where the instructions, when executed by at least one processor, cause a computing device to perform operations that include configuring a machine learning model with a first set of hyperparameters from a pool of hyperparameters.”; [0012])
The remainder of claim 9 is claim 1 in the form of an apparatus and is rejected for the same reasons as claim 1 stated above.
Dependent claim 13 is claim 5 in the form of an apparatus and is rejected for the same reasons as claim 5 stated above. For the rejection of the limitations specifically pertaining to the apparatus of claim 9, see the rejection of claim 9 above.
Dependent claim 16 is claim 8 in the form of an apparatus and is rejected for the same reasons as claim 8 stated above. For the rejection of the limitations specifically pertaining to the apparatus of claim 9, see the rejection of claim 9 above.
Claim 17 is rejected over Krishna and Andonie.
Regarding claim 17, Krishna teaches a computer program stored in a non-transitory recording medium to execute a method for automated machine learning, the method comprising: (“a non-transitory computer readable medium has instructions stored thereon, where the instructions, when executed by at least one processor, cause a computing device to perform operations that include configuring a machine learning model with a first set of hyperparameters from a pool of hyperparameters.”; [0012])
The remainder of claim 17 is claim 1 in the form of a non-transitory recording medium and is rejected for the same reasons as claim 1 stated above.
Claims 2, 3, 10 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Krishna and Andonie in further view of Nguyen et al. (US20190042887A1); hereinafter Nguyen
Claim 2 is rejected over Krishna, Andonie and Nguyen with the incorporation of claim 1. Regarding claim 2, Krishna teaches combining the different set data for the at least one or more parameters to produce a plurality of candidate parameter sets; (“Hyperparameter search space 500 includes various hyperparameter types 502, 504, 506, 508, 510, 512. For example, hyperparameter type a.sub.1 502 illustrates exemplary pre-processing hyperparameters that one or more machine learning models may be configured with.”; [0036]; and ”Hyperparameter search space 500 is hierarchically organized, where edges 520 indicate possible selections of a lower-level hyperparameter type given the selection of a higher-level hyperparameter type.”; [0037])
Krishna does not teach performing cross validation for the plurality of candidate parameter sets by performing the learning for the network functions with respect to the produced respective candidate parameter sets through a first dataset;
and determining at least one or more candidate parameter sets as the at least one or more first parameter sets according to the results of the cross validation.
However, Nguyen teaches performing cross validation for the plurality of candidate parameter sets by performing the learning for the network functions with respect to the produced respective candidate parameter sets through a first dataset; (“Training management module 120 can be configured to evaluate the effectiveness of each of the trained predictive models (FIGS. 2, 3G). For example, management module 120 is operable to estimate the effectiveness of each trained predictive model. In some implementations, cross-validation can be used to estimate the effectiveness of each trained predictive model by applying each model to the test set. For example, the training management module 120 can provide to each of the model training systems 160 the test set of data to determine the effectiveness of the model developed by the training system. The model training module 164 can apply the transformation of the data pipeline to the test set, apply the trained machine learning model to the test set to determine a value for the dependent variable for each row in the test set and return the determined values for each row to training management module 120 (e.g., via the caching layer).”; [0073]; and “The modelling system 100 may support multiple machine learning algorithms to train models including, but not limited to, generalized linear regression models (linear, logistic, exponential, and other regression models), decision trees (random forest, gradient boosted trees, xgboost), support vector machines and neural networks”; [0024]; Note: In paragraph [46] of the present invention’s specification, “a network function may be used with the same meaning as a neural network.”)
and determining at least one or more candidate parameter sets as the at least one or more first parameter sets according to the results of the cross validation. (“Multiple different hyper parameter configurations can be applied in training, generating multiple different trained predictive models. The generated models (e.g., candidate generated models with candidate hyper parameter sets) can be evaluated and a particular trained model selected. For example, the evaluation may include an evaluation of the effectiveness of the candidate hyper parameter sets, based on the generated models, and the selection may be based on a measure of predictive performance that may be determined as an optimal measure of performance, for the set of candidate hyper parameter sets.”; [0056])
It would have been obvious before the effective filing date to combine the selection of hyperparameters of Krishna with the cross validation of Nguyen to efficiently measure the effectiveness of a trained predictive model based on candidate hyperparameter sets. Krishna and Nguyen are analogous art because they both concern hyperparameter optimization for machine learning model production.
Claim 3 is rejected over Krishna, Andonie and Nguyen with the incorporation of claim 1.
Regarding claim 3, Krishna does not teach performing the cross validation and the determining the at least one or more candidate parameter sets as the at least one or more first parameter sets are repeatedly performed, based on a second dataset which is different from the first dataset.
However, Nguyen teaches performing the cross validation and the determining the at least one or more candidate parameter sets as the at least one or more first parameter sets are repeatedly performed, based on a second dataset which is different from the first dataset. (“The predictive model generated by each training system 160 or effectiveness metric of the predictive model generated by each training system 160 can be returned to training management system 110 and evaluated as discussed above. Rounds of model training can be repeated using new hyper parameters until a model reaches a threshold level of effectiveness or other condition is met. In some embodiments, training rounds can be repeated until the change in effectiveness between two rounds drops below a pre-defined threshold. In any event, whether performed in multiple rounds or a single round the most performant hyper parameter set may be selected (212).”; [0078] and “Training to select a predictive model may involve training on only a sample of the training data, or not all of the training data at one time. For example, if k-fold cross-validation was used to estimate the effectiveness of the trained models, then the selected model will not have been trained with all of the training data at the time it is selected, but rather only K-1 partitions of the training data.”; [0079])
It would have been obvious before the effective filing date to combine the selection of hyperparameters of Krishna with the cross validation of Nguyen to efficiently measure the effectiveness of a trained predictive model based on candidate hyperparameter sets. Krishna and Nguyen are analogous art because they both concern hyperparameter optimization for machine learning model production.
Dependent claim 10 is claim 2 in the form of an apparatus and is rejected for the same reasons as claim 2 stated above. For the rejection of the limitations specifically pertaining to the apparatus of claim 9, see the rejection of claim 9 above.
Dependent claim 11 is claim 3 in the form of an apparatus and is rejected for the same reasons as claim 3 stated above. For the rejection of the limitations specifically pertaining to the apparatus of claim 9, see the rejection of claim 9 above.
Claims 4 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Krishna and Andonie and in further view of Hughes et al. (US20200202171A1); hereinafter Hughes
Claim 4 is rejected over Krishna, Andonie and Hughes with the incorporation of claim 1.
Regarding claim 4, Krishna teaches the at least one or more candidate parameter sets having performance greater than a given baseline are determined as at least one or more first parameter sets. (“Hyperparameter determination computing device 102 may continue the above process until, for example, a generated score is beyond (e.g., above) a threshold (e.g., a predetermined threshold). In some examples, hyperparameter determination computing device 102 repeats the process for a number of iterations (e.g., a number of predetermined iterations). As training progresses, the hyper-parameter configuration that yields a best validation accuracy score is stored (e.g., cached and updated). At the end of training, the machine learning model is configured with this stored hyper-parameter configuration. In some examples, the machine learning model is configured with the last selected subset of hyperparameters. In some examples, the machine learning model is configured with hyperparameters associated with the highest probabilities. In some examples, the machine learning model is configured with hyperparameters associated with the highest values corresponding to learnable parameters.”; [0063]; Note: The threshold is the baseline.)
Krishna does not teach wherein the results of the cross validation comprise an average and standard deviation of validation scores calculated for the respective candidate parameter sets, and
wherein in the determining the at least one or more candidate parameter sets as the first parameter sets, statistical comparison is performed based on the average and standard deviation of the validation scores, and
However, Hughes teaches wherein the results of the cross validation comprise an average and standard deviation of validation scores calculated for the respective candidate parameter sets, and (“The model convergence metric may be a standard deviation of metrics across runs, across cross-validation folds, and/or across cross-validation averages. Other stopping criteria for the annotation process 400 may be used.”; [0174])
wherein in the determining the at least one or more candidate parameter sets as the first parameter sets, statistical comparison is performed based on the average and standard deviation of the validation scores, and (“As in the baseline selection process, the annotation server 202 uses the model type to select an appropriate search space. A search space consists of a family of algorithms, their associated loss functions, and potential hyperparameters for tuning the algorithm. During a single hyperparameter optimization run, an algorithm and sample hyperparameters are selected, a model is trained and metrics are calculated.”; [0159])
It would have been obvious before the effective filing date to combine the selection of hyperparameters of Krishna with the standard deviation of metrics of Hughes for rapid and focused training of machine learning models (Hughes, [0169]). Krishna and Hughes are analogous art because they both concern hyperparameter optimization for machine learning model production.
Claims 6 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Krishna and Andonie and in further view of Fishkov et al. (US20200184382A1); hereinafter Fishkov
Claim 6 is rejected over Krishna, Andonie and Fishkov with the incorporation of claim 1.
Regarding claim 6, Krishna does not teach wherein the learning conditions are related to at least one of learning environment, inference speed, and search range.
However, Fishkov teaches wherein the learning conditions are related to at least one of learning environment, inference speed, and search range. (“Bayesian optimization balances exploration and exploitation to search an entire domain (e.g., range, set, etc.) of possible hyperparameters. For example, the service attempts to minimize a validation error with respect to the known data, by executing a plurality of trials for a given machine-learning algorithm using different sets of hyperparameters.”; [0060])
It would have been obvious before the effective filing date to combine the selection of hyperparameters of Krishna with the hyperparameter search range of Fishkov to improve the selection of hyperparameter sets (Fishkov, [0060]). Krishna and Fishkov are analogous art because they both concern optimizing machine learning through hyperparameter selection.
Dependent claim 14 is claim 6 in the form of an apparatus and is rejected for the same reasons as claim 6 stated above. For the rejection of the limitations specifically pertaining to the apparatus of claim 9, see the rejection of claim 9 above.
Claims 7 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Krishna, Andonie and Fishkov in further view of Okada et al. (US20220172115A1); hereinafter Okada
Claim 7 is rejected over Krishna, Andonie, Fishkov and Okada with the incorporation of claim 1.
Regarding claim 7, Krishna does not teach sorting the at least one or more first parameter sets with respect to at least one of architecture and the inference speed; and
choosing a given top percentage of the at least one or more first parameter sets sorted according to the learning conditions inputted as the at least one or more second parameter sets.
However, Okada teaches sorting the at least one or more first parameter sets with respect to at least one of architecture and the inference speed; and (“After the process of the step S108, the parameter combination optimization unit 35 ranks relationships between the not-excluded (in other words, extracted) combination patterns and the accuracy results associated with the corresponding models in the descending order of the accuracy and stores them, for example, in the storage apparatus 14 (a step S109). In the example illustrated in FIG. 4, the ranking is as illustrated in FIG. 4C.”; [0053])
choosing a given top percentage of the at least one or more first parameter sets sorted according to the learning conditions inputted as the at least one or more second parameter sets. (“By performing a series of steps described with reference to the flow chart of FIG. 3, for example, as illustrated in FIG. 4A, it is possible to efficiently narrow down the combination patterns of parameters. Note that the narrowed-down combination patterns may be used when the next analysis is performed (for example, when the learning processing is performed by using the analysis data that differ from the analysis data used for the current learning processing). At this time, the combination patterns are used for the analysis (e.g., tuning of the hyperparameters by the grid search) in descending order from the high ranking (i.e., high rank).”; [0058]; and “The “allowable range” may be preset, for example, by the user of the parameter tuning apparatus 1, or may be automatically set by the parameter tuning apparatus 1. At this time, the “allowable range” may be set by an absolute value of accuracy, or may be set as a relative range (e.g., xx % from a high precision side).”; [0040])
It would have been obvious before the effective filing date to combine the selection of hyperparameters of Krishna with the hyperparameter tuning of Okada to improve the generalization capability and the accuracy of the model after the combinations of parameter values and the range of parameter values are sufficiently narrowed down (Okada, [0062]). Krishna and Okada are analogous art because they both concern hyperparameter optimization.
Dependent claim 15 is claim 7 in the form of an apparatus and is rejected for the same reasons as claim 7 stated above. For the rejection of the limitations specifically pertaining to the apparatus of claim 9, see the rejection of claim 9 above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID H TRAN whose telephone number is (703)756-1525. The examiner can normally be reached M-F 9:30 am - 5:30 pm.
Examiner interviews are a