DETAILED ACTION
This action is responsive to the Application/amendment filed on 03/21/2024. Claims 1-20 are pending in the case. Claims 1, 8, and 15 are independent claims.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55 and Effective Filing Date of 03/28/2023 is acknowledged.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/21/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 112
Claims 1-20 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 2, 5, 7, 9, 12, 14, 16, 19 recites the limitation the scores of the cross-validations used to update the best score. There is insufficient antecedent basis for this limitation in the claim. Examiner will interpret the limitation as a scores of the cross-validations used to update the best score.
Claims 2, 5, 7, 9, 12, 14, 16, 19 recites the limitation the defined order. There is insufficient antecedent basis for this limitation in the claim. Examiner will interpret the limitation as a defined order.
Claims 1, 8 and 15 recites the limitation this hyperparameter combination. There is insufficient antecedent basis for this limitation in the claim. This rejection extends to claims 1-20 as dependent claims inherent deficiencies from parent claims. Examiner will interpret the limitation as a hyperparameter combination.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 as the focus of the claims are directed towards judicial exceptions without significantly more.
Regarding Claim 1:
Subject Matter Eligibility Analysis Step 2A Prong 1:
The claim recites performing several hyperparameter combination tests, each hyperparameter combination test including cross-validation using a validation data set, the cross-validation defining several performance tests on at least one validation data subset which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))). Alternatively the limitation can be considered an abstract idea as, under the broadest reasonable interpretation, it covers performance of the limitation in the mind with physical aid. The limitations encompass using evaluation to choose subsets of a set and perform an evaluation on the subsets performance. See 2106.04.(a)(2).III.C.
The claim recites computing a performance score which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))).
The claim recites comparing the computed performance score with a best score which is an abstract idea (Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A)))) of comparing a score with a value.
The claim recites stopping testing of this hyperparameter combination in response to the computed performance score being lower than the best score which is an abstract idea (Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A)))). Alternatively the limitation can be considered an abstract idea as, under the broadest reasonable interpretation, it covers performance of the limitation in the mind. The limitations encompass using evaluation to choose an action to perform or not perform based on the judgement of the evaluation. See 2106.04.(a)(2).III.C.
The claim recites each cross-validation further comprising updating the best score in response to all of the performance scores computed for this cross-validation being higher than the best score which is an abstract idea (Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A)))) of comparing a score with a value. Alternatively the limitation can be considered an abstract idea as, under the broadest reasonable interpretation, it covers performance of the limitation in the mind. The limitations encompass using evaluation to compare and choose a value. See 2106.04.(a)(2).III.C.
The claim recites each the updated best score then corresponding to a lowest performance score from among the performance scores computed for this cross-validation which is an abstract idea (Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A)))) of comparing a score with a value. Alternatively the limitation can be considered an abstract idea as, under the broadest reasonable interpretation, it covers performance of the limitation in the mind. The limitations encompass using evaluation to compare and choose a value. See 2106.04.(a)(2).III.C.
The claim recites defining the optimal combination of hyperparameters…and, the optimal combination of hyperparameters…corresponding to the optimal combination of hyperparameters that defined a last best score once all of the hyperparameter combination tests have been carried out which is an abstract idea (Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A)))) of comparing a score with a value. Alternatively the limitation can be considered an abstract idea as, under the broadest reasonable interpretation, it covers performance of the limitation in the mind. The limitations encompass using evaluation to compare and choose a value. See 2106.04.(a)(2).III.C.
Subject Matter Eligibility Analysis Step 2A Prong 2:
for the automatic learning model specifies a particular technological environment in which the abstract idea is to take place, i.e. a field of use (see MPEP 2106.05(h))
Subject Matter Eligibility Analysis Step 2B:
Additional elements (a) do not integrate the abstract idea into a practical application nor do the additional limitation provide significantly more than the abstract idea because the limitation merely specifies a field of use in which the abstract idea is to take place, i.e. a field of use (see MPEP 2106.05(h)).
The additional element(s) outlined in Step 2A Prong 2 in the claim do/does not include any additional elements, when considered separately and in combination, that amount to an integration of the judicial exception into a practical application, nor significantly more than the judicial exception for the reasons set forth in step 2A prong 2 analysis above. The claim is not patent eligible.
Regarding Claim 2:
The rejection of claim 1 is incorporated and further claim recites further additional
elements/limitations:
Subject Matter Eligibility Analysis Step 2A Prong 1:
The claim recites carrying out the performance tests defined by the cross-validation in a given order which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))). Alternatively the limitation can be considered an abstract idea as, under the broadest reasonable interpretation, it covers performance of the limitation in the mind with physical aid. The limitations encompass using evaluation to choose subsets of a set and perform an evaluation on the subsets performance. See 2106.04.(a)(2).III.C.
The claim recites defining an order for the performance tests each time the best score is updated and, with the defined order thus corresponding to an ascending order of the scores of the cross-validations used to update the best score which, under the broadest reasonable interpretation, covers performance of the limitation in the mind. The limitations encompass using judgement to determine an order. See 2106.04.(a)(2).III.C.
Subject Matter Eligibility Analysis Step 2A Prong 2:
The claim does not contain elements that would warrant a Step 2A Prong 2 analysis.
Subject Matter Eligibility Analysis Step 2B:
The claim does not include any additional element, when considered separately and in combination, that amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 3:
The rejection of claim 1 is incorporated and further claim recites further additional
elements/limitations:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Subject Matter Eligibility Analysis Step 2A Prong 2:
automatic learning model is selected from among a linear model, a decision tree, or an artificial neural network specifies a particular technological environment in which the abstract idea is to take place, i.e. a field of use (see MPEP 2106.05(h))
Subject Matter Eligibility Analysis Step 2B:
Additional elements (a) do not integrate the abstract idea into a practical application nor do the additional limitation provide significantly more than the abstract idea because the limitation merely specifies a field of use in which the abstract idea is to take place, i.e. a field of use (see MPEP 2106.05(h)).
The additional element(s) outlined in Step 2A Prong 2 in the claim do/does not include any additional elements, when considered separately and in combination, that amount to an integration of the judicial exception into a practical application, nor significantly more than the judicial exception for the reasons set forth in step 2A prong 2 analysis above. The claim is not patent eligible.
Regarding Claim 4:
The rejection of claim 3 is incorporated and further claim recites further additional
elements/limitations:
Subject Matter Eligibility Analysis Step 2A Prong 1:
The claim recites wherein each hyperparameter combination is sought using a grid search, a random search, or a Bayesian search which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))).
Subject Matter Eligibility Analysis Step 2A Prong 2:
The claim does not contain elements that would warrant a Step 2A Prong 2 analysis.
Subject Matter Eligibility Analysis Step 2B:
The claim does not include any additional element, when considered separately and in combination, that amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 5:
The rejection of claim 3 is incorporated and further claim recites further additional
elements/limitations:
Subject Matter Eligibility Analysis Step 2A Prong 1:
The claim recites carrying out the performance tests defined by the cross-validation in a given order which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))). Alternatively the limitation can be considered an abstract idea as, under the broadest reasonable interpretation, it covers performance of the limitation in the mind with physical aid. The limitations encompass using evaluation to choose subsets of a set and perform an evaluation on the subsets performance. See 2106.04.(a)(2).III.C.
The claim recites defining an order for the performance tests each time the best score is updated and, with the defined order thus corresponding to an ascending order of the scores of the cross-validations used to update the best score which, under the broadest reasonable interpretation, covers performance of the limitation in the mind. The limitations encompass using judgement to determine an order. See 2106.04.(a)(2).III.C.
Subject Matter Eligibility Analysis Step 2A Prong 2:
The claim does not contain elements that would warrant a Step 2A Prong 2 analysis.
Subject Matter Eligibility Analysis Step 2B:
The claim does not include any additional element, when considered separately and in combination, that amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 6:
The rejection of claim 1 is incorporated and further claim recites further additional
elements/limitations:
Subject Matter Eligibility Analysis Step 2A Prong 1:
The claim recites wherein each hyperparameter combination is sought using a grid search, a random search, or a Bayesian search. which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))).
Subject Matter Eligibility Analysis Step 2A Prong 2:
The claim does not contain elements that would warrant a Step 2A Prong 2 analysis.
Subject Matter Eligibility Analysis Step 2B:
The claim does not include any additional element, when considered separately and in combination, that amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 7:
The rejection of claim 6 is incorporated and further claim recites further additional
elements/limitations:
Subject Matter Eligibility Analysis Step 2A Prong 1:
The claim recites carrying out the performance tests defined by the cross-validation in a given order which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))). Alternatively the limitation can be considered an abstract idea as, under the broadest reasonable interpretation, it covers performance of the limitation in the mind with physical aid. The limitations encompass using evaluation to choose subsets of a set and perform an evaluation on the subsets performance. See 2106.04.(a)(2).III.C.
The claim recites defining an order for the performance tests each time the best score is updated and, with the defined order thus corresponding to an ascending order of the scores of the cross-validations used to update the best score which, under the broadest reasonable interpretation, covers performance of the limitation in the mind. The limitations encompass using judgement to determine an order. See 2106.04.(a)(2).III.C.
Subject Matter Eligibility Analysis Step 2A Prong 2:
The claim does not contain elements that would warrant a Step 2A Prong 2 analysis.
Subject Matter Eligibility Analysis Step 2B:
The claim does not include any additional element, when considered separately and in combination, that amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception. The claim is not patent eligible.
Regarding claim 8:
Subject Matter Eligibility Analysis Step 2A Prong 1:
The claim recites performing several hyperparameter combination tests, each hyperparameter combination test including cross-validation using a validation data set, the cross-validation defining several performance tests on at least one validation data subset which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))). Alternatively the limitation can be considered an abstract idea as, under the broadest reasonable interpretation, it covers performance of the limitation in the mind with physical aid. The limitations encompass using evaluation to choose subsets of a set and perform an evaluation on the subsets performance. See 2106.04.(a)(2).III.C.
The claim recites computing a performance score which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))).
The claim recites comparing the computed performance score with a best score which is an abstract idea (Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A)))) of comparing a score with a value.
The claim recites stopping testing of this hyperparameter combination in response to the computed performance score being lower than the best score which is an abstract idea (Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A)))). Alternatively the limitation can be considered an abstract idea as, under the broadest reasonable interpretation, it covers performance of the limitation in the mind. The limitations encompass using evaluation to choose an action to perform or not perform based on the judgement of the evaluation. See 2106.04.(a)(2).III.C.
The claim recites each cross-validation further comprising updating the best score in response to all of the performance scores computed for this cross-validation being higher than the best score which is an abstract idea (Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A)))) of comparing a score with a value. Alternatively the limitation can be considered an abstract idea as, under the broadest reasonable interpretation, it covers performance of the limitation in the mind. The limitations encompass using evaluation to compare and choose a value. See 2106.04.(a)(2).III.C.
The claim recites each the updated best score then corresponding to a lowest performance score from among the performance scores computed for this cross-validation which is an abstract idea (Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A)))) of comparing a score with a value. Alternatively the limitation can be considered an abstract idea as, under the broadest reasonable interpretation, it covers performance of the limitation in the mind. The limitations encompass using evaluation to compare and choose a value. See 2106.04.(a)(2).III.C.
The claim recites defining the optimal combination of hyperparameters…and, the optimal combination of hyperparameters…corresponding to the optimal combination of hyperparameters that defined a last best score once all of the hyperparameter combination tests have been carried out which is an abstract idea (Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A)))) of comparing a score with a value. Alternatively the limitation can be considered an abstract idea as, under the broadest reasonable interpretation, it covers performance of the limitation in the mind. The limitations encompass using evaluation to compare and choose a value. See 2106.04.(a)(2).III.C.
Subject Matter Eligibility Analysis Step 2A Prong 2:
non-transitory computer-readable media recites a generic computer on which to perform the abstract idea, e.g. "apply it on a computer" (see MPEP 2106.05(f))
processor recites a generic computer on which to perform the abstract idea, e.g. "apply it on a computer" (see MPEP 2106.05(f))
the automatic learning model specifies a particular technological environment in which the abstract idea is to take place, i.e. a field of use (see MPEP 2106.05(h))
Subject Matter Eligibility Analysis Step 2B:
Additional elements (a) and (b) do not integrate the abstract idea into a practical application nor do the additional limitation provide significantly more than the abstract idea because the limitation amount to no more than mere instructions to apply the exception using a generic computer component. Please see MPEP §2106.05(f).
Additional elements (c) do not integrate the abstract idea into a practical application nor do the additional limitation provide significantly more than the abstract idea because the limitation merely specifies a field of use in which the abstract idea is to take place, i.e. a field of use (see MPEP 2106.05(h)).
The additional element(s) outlined in Step 2A Prong 2 in the claim do/does not include any additional elements, when considered separately and in combination, that amount to an integration of the judicial exception into a practical application, nor significantly more than the judicial exception for the reasons set forth in step 2A prong 2 analysis above. The claim is not patent eligible.
Regarding claim 9:
The rejection of claim 8 is incorporated and, further, recites additional elements/limitations:
Claim 9 is rejected under that same 101 claim analysis due to the substantially similarity of the limitations and additional elements of claim 9 found in claim 2.
Regarding claim 10:
The rejection of claim 8 is incorporated and, further, recites additional elements/limitations:
Claim 10 is rejected under that same 101 claim analysis due to the substantially similarity of the limitations and additional elements of claim 10 found in claim 3.
Regarding claim 11:
The rejection of claim 10 is incorporated and, further, recites additional elements/limitations:
Claim 11 is rejected under that same 101 claim analysis due to the substantially similarity of the limitations and additional elements of claim 11 found in claim 4.
Regarding claim 12:
The rejection of claim 10 is incorporated and, further, recites additional elements/limitations:
Claim 12 is rejected under that same 101 claim analysis due to the substantially similarity of the limitations and additional elements of claim 12 found in claim 5.
Regarding claim 13:
The rejection of claim 8 is incorporated and, further, recites additional elements/limitations:
Claim 13 is rejected under that same 101 claim analysis due to the substantially similarity of the limitations and additional elements of claim 13 found in claim 6.
Regarding claim 14:
The rejection of claim 13 is incorporated and, further, recites additional elements/limitations:
Claim 14 is rejected under that same 101 claim analysis due to the substantially similarity of the limitations and additional elements of claim 14 found in claim 7.
Regarding claim 15:
Subject Matter Eligibility Analysis Step 2A Prong 1:
The claim recites performing several hyperparameter combination tests, each hyperparameter combination test including cross-validation using a validation data set, the cross-validation defining several performance tests on at least one validation data subset which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))). Alternatively the limitation can be considered an abstract idea as, under the broadest reasonable interpretation, it covers performance of the limitation in the mind with physical aid. The limitations encompass using evaluation to choose subsets of a set and perform an evaluation on the subsets performance. See 2106.04.(a)(2).III.C.
The claim recites computing a performance score which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))).
The claim recites comparing the computed performance score with a best score which is an abstract idea (Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A)))) of comparing a score with a value.
The claim recites stopping testing of this hyperparameter combination in response to the computed performance score being lower than the best score which is an abstract idea (Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A)))). Alternatively the limitation can be considered an abstract idea as, under the broadest reasonable interpretation, it covers performance of the limitation in the mind. The limitations encompass using evaluation to choose an action to perform or not perform based on the judgement of the evaluation. See 2106.04.(a)(2).III.C.
The claim recites each cross-validation further comprising updating the best score in response to all of the performance scores computed for this cross-validation being higher than the best score which is an abstract idea (Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A)))) of comparing a score with a value. Alternatively the limitation can be considered an abstract idea as, under the broadest reasonable interpretation, it covers performance of the limitation in the mind. The limitations encompass using evaluation to compare and choose a value. See 2106.04.(a)(2).III.C.
The claim recites each the updated best score then corresponding to a lowest performance score from among the performance scores computed for this cross-validation which is an abstract idea (Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A)))) of comparing a score with a value. Alternatively the limitation can be considered an abstract idea as, under the broadest reasonable interpretation, it covers performance of the limitation in the mind. The limitations encompass using evaluation to compare and choose a value. See 2106.04.(a)(2).III.C.
The claim recites defining the optimal combination of hyperparameters…and, the optimal combination of hyperparameters…corresponding to the optimal combination of hyperparameters that defined a last best score once all of the hyperparameter combination tests have been carried out which is an abstract idea (Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A)))) of comparing a score with a value. Alternatively the limitation can be considered an abstract idea as, under the broadest reasonable interpretation, it covers performance of the limitation in the mind. The limitations encompass using evaluation to compare and choose a value. See 2106.04.(a)(2).III.C.
Subject Matter Eligibility Analysis Step 2A Prong 2:
non-transitory memory recites a generic computer on which to perform the abstract idea, e.g. "apply it on a computer" (see MPEP 2106.05(f))
processor in communication with the non-transitory memory recites a generic computer on which to perform the abstract idea, e.g. "apply it on a computer" (see MPEP 2106.05(f))
the automatic learning model specifies a particular technological environment in which the abstract idea is to take place, i.e. a field of use (see MPEP 2106.05(h))
Subject Matter Eligibility Analysis Step 2B:
Additional elements (a) and (b) do not integrate the abstract idea into a practical application nor do the additional limitation provide significantly more than the abstract idea because the limitation amount to no more than mere instructions to apply the exception using a generic computer component. Please see MPEP §2106.05(f).
Additional elements (c) do not integrate the abstract idea into a practical application nor do the additional limitation provide significantly more than the abstract idea because the limitation merely specifies a field of use in which the abstract idea is to take place, i.e. a field of use (see MPEP 2106.05(h)).
The additional element(s) outlined in Step 2A Prong 2 in the claim do/does not include any additional elements, when considered separately and in combination, that amount to an integration of the judicial exception into a practical application, nor significantly more than the judicial exception for the reasons set forth in step 2A prong 2 analysis above. The claim is not patent eligible.
Regarding claim 16:
The rejection of claim 15 is incorporated and, further, recites additional elements/limitations:
Claim 16 is rejected under that same 101 claim analysis due to the substantially similarity of the limitations and additional elements of claim 16 found in claim 2.
Regarding claim 17:
The rejection of claim 15 is incorporated and, further, recites additional elements/limitations:
Claim 17 is rejected under that same 101 claim analysis due to the substantially similarity of the limitations and additional elements of claim 17 found in claim 3.
Regarding claim 18:
The rejection of claim 17 is incorporated and, further, recites additional elements/limitations:
Claim 18 is rejected under that same 101 claim analysis due to the substantially similarity of the limitations and additional elements of claim 18 found in claim 4.
Regarding claim 19:
The rejection of claim 17 is incorporated and, further, recites additional elements/limitations:
Claim 19 is rejected under that same 101 claim analysis due to the substantially similarity of the limitations and additional elements of claim 19 found in claim 5.
Regarding claim 20:
The rejection of claim 15 is incorporated and, further, recites additional elements/limitations:
Claim 20 is rejected under that same 101 claim analysis due to the substantially similarity of the limitations and additional elements of claim 20 found in claim 6.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 3-4, 6, 8, 10-11, 13, 15, 17-18, 20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Soper et al(“Greed Is Good: Rapid Hyperparameter Optimization and Model Selection Using Greedy k-Fold Cross Validation”, henceforth known as Soper).
Regarding claim 1:
Soper discloses performing several hyperparameter combination tests, each hyperparameter combination test including cross-validation using a validation data set, the cross-validation defining several performance tests on at least one validation data subset(Soper, Page 12, Paragraph 2, “The greedy k-fold cross validation algorithm begins by obtaining a partial performance estimate for each candidate model using just the first fold as a validation set, after having trained the model using the remaining folds.”)
Soper discloses each performance test including: computing a performance score(Soper, Page 12, Paragraph 2, “…The model with the best mean performance at that moment is then identified” See also Soper, Page 13, Paragraph 2, “…Every time a candidate model becomes fully evaluated, that model’s overall performance is compared to the performance of the currently known, fully evaluated, best-performing model. If the newly completed model is found to be inferior to the currently known best model, then an inferior model counter is incremented” where the mean performance corresponds to a performance scores)
Soper discloses comparing the computed performance score with a best score(Soper, Page 14, Algorithm 3, Soper, Page 12, Paragraph 2, “The model with the best mean performance at that moment is then identified, after which its next available fold is used as a validation set and the model’s mean performance is updated” and Soper, Page 13, Paragraph 2, “…If, however, a newly completed candidate model is found to be superior to the currently known best then the newly completely model replaces the previous best-performing model” where comparing the mean performance of models to determine the best performing model corresponds to comparing a computed performance score with a best score)
Soper discloses and stopping testing of this hyperparameter combination in response to the computed performance score being lower than the best score(Soper, Page 14, Algorithm 3 and Soper, Page 13, Paragraph 2, “…the greedy cross validation to decide whether to continue searching or to stop the search process early on the data…an early stopping percentage (ε) as an input parameter. The product of ε and the number of candidate models through a standard ceiling function to yield an early stopping threshold. Every time a candidate model becomes fully evaluated, that model’s overall performance is compared to the performance of the currently known, fully evaluated, best-performing model. If the newly completed model is found to be inferior to the currently known best model, then an inferior model counter is incremented. Whenever the value of the inferior model counter exceeds the early stopping threshold, the search is stopped immediately” where the evaluation of the mean performance being lower causing an early stop condition)
Soper discloses each cross-validation further comprising updating the best score in response to all of the performance scores computed for this cross-validation being higher than the best score(Soper, Page 14, Algorithm 3 and Soper, Page 13, Paragraph 2, “…the greedy cross validation to decide whether to continue searching or to stop the search process early on the data…an early stopping percentage (ε) as an input parameter. The product of ε and the number of candidate models through a standard ceiling function to yield an early stopping threshold. Every time a candidate model becomes fully evaluated, that model’s overall performance is compared to the performance of the currently known, fully evaluated, best-performing model. If the newly completed model is found to be inferior to the currently known best model, then an inferior model counter is incremented. Whenever the value of the inferior model counter exceeds the early stopping threshold” where (See also Soper, Page 12, Paragraph 2, “This process repeats until either the computational budget has been exhausted or an early stopping criterion has been met, at which time the algorithm returns the best, fully evaluated model)), the updated best score then corresponding to a lowest performance score from among the performance scores computed for this cross-validation(Soper, Page 10, Paragraph 1, “evaluating as many models as possible given constraints of the computational budget (or when a stopping criterion is met in of a computational budget), the candidate model with the most desirable performance characteristics is chosen as the final model. A wide variety of performance metrics are feasible for the model selection process (e.g., best classification accuracy, lowest mean squared error, etc)” where the performance metric among the wide variety of performance metrics is the mean squared error and the best-performance model used is the lowest mean squared error )
Soper discloses and defining the optimal combination of hyperparameters for the automatic learning model, the optimal combination of hyperparameters for the automatic learning model corresponding to the optimal combination of hyperparameters that defined a last best score once all of the hyperparameter combination tests have been carried out(“After choosing θopt, one final model f(D;θopt) is created using the optimized settings and the entire training set D”)
Regarding claim 3:
The rejection of claim 1 with prior art is incorporated and further:
Soper discloses wherein the automatic learning model is selected from among a linear model, a decision tree, or an artificial neural network(Soper, Page 16, Paragraph 2,“As noted previously, a variety of ML algorithms and real-world datasets were used in the experiments to compare the performance of the greedy k-fold cross validation method
to the standard (baseline) k-fold cross validation and successive halving methods. These
algorithms included the Bernoulli Naïve Bayes, Decision Tree, and K-Nearest Neighbors
(KNN) classifiers, each of which was chosen due to its distinct approach to performing
the classification task” where one of the experiments being part of a decision tree corresponds to an a decision tree being selected to use as the automatic learning model)
Regarding claim 4:
The rejection of claim 3 with prior art is incorporated and further:
Soper discloses wherein each hyperparameter combination is sought using a grid search, a random search, or a Bayesian search(Soper, Page 16, Paragraph 3, “The candidate models that were evaluated in the experiments varied according to the values of their hyperparameters, with the hyperparameter settings for each model being chosen randomly” where the hyperparameters settings being chosen randomly corresponds to hyperparameter combination being sought using a random search)
Regarding claim 6:
The rejection of claim 1 with prior art is incorporated and further:
Soper discloses wherein each hyperparameter combination is sought using a grid search, a random search, or a Bayesian search(Soper, Page 16, Paragraph 3, “The candidate models that were evaluated in the experiments varied according to the values of their hyperparameters, with the hyperparameter settings for each model being chosen randomly” where the hyperparameters settings being chosen randomly corresponds to hyperparameter combination being sought using a random search)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 2, 5, 7, 9, 12, 14, 16, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Soper et al(“Greed Is Good: Rapid Hyperparameter Optimization and Model Selection Using Greedy k-Fold Cross Validation”, henceforth known as Soper) in view of Birattari et al(“Lazy Learning Meets the Recursive Least Squares Algorithm”, henceforth known as Birattari).
Regarding claim 2:
The rejection of claim 1 with prior art is incorporated and further:
Soper discloses carrying out the performance tests defined by the cross-validation in a given order “As indicated in the while loop, the greedy k-fold cross validation algorithm behaves greedily by always pursuing the most promising available option, with the extent to which an option is promising being determined by the current mean performance of its corresponding model.” where choosing the promising available option by evaluating the next fold for the candidate model that currently has the best-known level of performance corresponds to carrying out the performance tests defined by the cross-validation in a given order)
Soper does not explicitly disclose, however Birattari discloses defining an order for the performance tests each time the best score is updated, with the defined order thus corresponding to an ascending order of the scores of the cross-validations used to update the best score(Birattari, Page 379, Paragraph 2, Equation 10 and “…the final prediction of the value Yq is obtained as a weighted average of the best b models, where b is a parameter of the algorithm. Suppose the predictions yq(k) and the error vectors ecv(k) have been ordered creating a sequence of integers {ki} so that MSE( ki ) ≤ MSE(kj),
∀
i < j” where ordering the cross validation error vectors into an ascending sequence and choosing the best/first b models in that ascending sequence corresponds to an ascending order of scores of cross validation used to update the best score with the defined order thus corresponding to an ascending order of the scores of the cross-validations used to update the best score as the best models will be at the beginning of the ordered sequence and when a new best score is added it will update the best models/top models)
References Soper and Birattari are analogous art because they are from the same problem solving area of seeking to reduce computation burden of evaluating multiple candidate models using cross-validation to decide models.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Soper and Birattari before him or her, to modify the order used to select models of Soper to include the ascending order of Birattari as, in order to select the best items of a list, it is a indexing/ranking convention to order the list from best to worst performance and, as Birattari orders using lowest MSE to highest MSE it is an ascending order.
Regarding claim 5:
The rejection of claim 3 with prior art is incorporated and further:
Soper discloses carrying out the performance tests defined by the cross-validation in a given order “As indicated in the while loop, the greedy k-fold cross validation algorithm behaves greedily by always pursuing the most promising available option, with the extent to which an option is promising being determined by the current mean performance of its corresponding model.” where choosing the promising available option by evaluating the next fold for the candidate model that currently has the best-known level of performance corresponds to carrying out the performance tests defined by the cross-validation in a given order)
Soper does not explicitly disclose, however Birattari discloses defining an order for the performance tests each time the best score is updated, with the defined order thus corresponding to an ascending order of the scores of the cross-validations used to update the best score(Birattari, Page 379, Paragraph 2, Equation 10 and “…the final prediction of the value Yq is obtained as a weighted average of the best b models, where b is a parameter of the algorithm. Suppose the predictions yq(k) and the error vectors ecv(k) have been ordered creating a sequence of integers {ki} so that MSE( ki ) ≤ MSE(kj),
∀
i < j” where ordering the cross validation error vectors into an ascending sequence and choosing the best/first b models in that ascending sequence corresponds to an ascending order of scores of cross validation used to update the best score with the defined order thus corresponding to an ascending order of the scores of the cross-validations used to update the best score as the best models will be at the beginning of the ordered sequence and when a new best score is added it will update the best models/top models)
References Soper and Birattari are analogous art because they are from the same problem solving area of seeking to reduce computation burden of evaluating multiple candidate models using cross-validation to decide models.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Soper and Birattari before him or her, to modify the order used to select models of Soper to include the ascending order of Birattari as, in order to select the best items of a list, it is a indexing/ranking convention to order the list from best to worst performance and, as Birattari orders using lowest MSE to highest MSE it is an ascending order.
Regarding claim 7:
The rejection of claim 6 with prior art is incorporated and further:
Soper discloses carrying out the performance tests defined by the cross-validation in a given order “As indicated in the while loop, the greedy k-fold cross validation algorithm behaves greedily by always pursuing the most promising available option, with the extent to which an option is promising being determined by the current mean performance of its corresponding model.” where choosing the promising available option by evaluating the next fold for the candidate model that currently has the best-known level of performance corresponds to carrying out the performance tests defined by the cross-validation in a given order)
Soper does not explicitly disclose, however Birattari discloses defining an order for the performance tests each time the best score is updated, with the defined order thus corresponding to an ascending order of the scores of the cross-validations used to update the best score(Birattari, Page 379, Paragraph 2, Equation 10 and “…the final prediction of the value Yq is obtained as a weighted average of the best b models, where b is a parameter of the algorithm. Suppose the predictions yq(k) and the error vectors ecv(k) have been ordered creating a sequence of integers {ki} so that MSE( ki ) ≤ MSE(kj),
∀
i < j” where ordering the cross validation error vectors into an ascending sequence and choosing the best/first b models in that ascending sequence corresponds to an ascending order of scores of cross validation used to update the best score with the defined order thus corresponding to an ascending order of the scores of the cross-validations used to update the best score as the best models will be at the beginning of the ordered sequence and when a new best score is added it will update the best models/top models)
References Soper and Birattari are analogous art because they are from the same problem solving area of seeking to reduce computation burden of evaluating multiple candidate models using cross-validation to decide models.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Soper and Birattari before him or her, to modify the order used to select models of Soper to include the ascending order of Birattari as, in order to select the best items of a list, it is a indexing/ranking convention to order the list from best to worst performance and, as Birattari orders using lowest MSE to highest MSE it is an ascending order.
Regarding claim 9:
The rejection of claim 8 is incorporated and, further, claim 9 is rejected under that same claim analysis due to the substantially similarity of the limitations and additional elements of claim 9 found in claim 2.
Regarding claim 12:
The rejection of claim 10 is incorporated and, further, claim 12 is rejected under that same claim analysis due to the substantially similarity of the limitations and additional elements of claim 12 found in claim 5.
Regarding claim 14:
The rejection of claim 13 is incorporated and, further, claim 14 is rejected under that same claim analysis due to the substantially similarity of the limitations and additional elements of claim 14 found in claim 7.
Regarding claim 16:
The rejection of claim 15 is incorporated and, further, claim 16 is rejected under that same claim analysis due to the substantially similarity of the limitations and additional elements of claim 16 found in claim 2.
Regarding claim 19:
The rejection of claim 17 is incorporated and, further, claim 19 is rejected under that same claim analysis due to the substantially similarity of the limitations and additional elements of claim 19 found in claim 5.
Relevant Art:
While not used in the rejection, Examiner found relevant prior outlined below:
“Hyperparameter Optimization Using Successive Halving with Greedy Cross Validation” henceforth known as Soper as it discusses using cross-validation for model selection
“Unsupervised stratification of cross-validation for accuracy estimation” by Diamantidis as a heavily cited prior art that discusses ordered cross-validation according to their similarity to the center of the instance space
“A comparison of approaches to improve worst-case predictive model performance over patient subpopulations” by Pfohl as it discusses minimax worst-case model selection and performance
“Futility Analysis in the Cross–Validation of Machine Learning Models” by Kuhn that discuses cross-validation and tuning
“EFFICIENT, ADAPTIVE CROSS-VALIDATION FOR TUNING AND COMPARING MODELS, WITH APPLICATION TO DRUG DISCOVERY” by Shen that discuses cross-validation and tuning
“Random Search for Hyper-Parameter Optimization” by Bergestra that discusses searching for hyper-parameters and optimization
Scikit versions 1.2 and prior versions(https://scikit-learn.org/1.2/modules/classes.html) as a general framework for cross validator functions used for model selection
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHARLES JEFFREY JONES JR whose telephone number is (703)756-1414. The examiner can normally be reached Monday - Friday 8:00 - 5:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at 571-272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.J.J./Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights