DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 02/24/2026 have been fully considered but they are not persuasive.
Regarding applicant’s remarks directed to the rejection of claims under 35 USC § 103, the arguments are directed to newly amended limitations that were not previously examined by the examiner. Therefore, applicants arguments are rendered moot. The examiner refers to the rejection under 35 USC § 103 in the current office action for more details.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-4, 7-13, 16-18 and 21-26 is/are rejected under 35 U.S.C. 103 as being unpatentable over US Pub No. US20160110657A1 Gibiansky et al. (“Gibiansky”) in view of Yang, Li, and Abdallah Shami. "On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice." arXiv preprint arXiv:2007.15745 (2020). (“Yang”) in further view of Andonie, Răzvan. "Hyperparameter optimization in learning systems." Journal of Membrane Computing 1.4 (2019): 279-291 (“Andonie”) and evidenced by Wikipedia (2021, May 18th). Greedy Algorithm. https://web.archive.org/web/20210518031653/https://en.wikipedia.org/wiki/Greedy_algorithm. (“Wikipedia”).
In regards to claim 1,
Gibiansky teaches A computer-implemented method comprising: identifying, for a given machine learning algorithm, three or more hyperparameters and possible values for each of the three or more hyperparameters,
(Gibiansky, “[0042] Unlike the traditional grid-search-based tuning method, the model-based automatic parameter tuning method described herein is able to explore the entire space formed by different models together with their associated parameters. The model-based automatic parameter tuning method described herein is further able to intelligently and automatically detect effective search directions and refine the tuning region, and hence arrive at the desired result [identifying, for a given machine learning algorithm, three or more hyperparameters and possible values for each of the three or more hyperparameters; ie identifying desired parameters for a model] in an efficient way.”)
Gibiansky teaches and automatically tuning at least the first hyperparameter [of the decision tree-based machine learning algorithm pertaining to the at least one decision tree split quality metric], and the second hyperparameter [of the decision tree-based machine learning algorithm pertaining to the number of data features associated with one or more decision tree splits], based at least in part on the determined first value and the determined at least a second value;
(Gibiansky, “[0042] Unlike the traditional grid-search-based tuning method, the model-based automatic parameter tuning method described herein is able to explore the entire space formed by different models together with their associated parameters. The model-based automatic parameter tuning method [automatically tuning… hyperparameters; wherein the particular parameters (criterion, max_features, min_samples_leaf) are those in Yang wherein the manner of obtaining the particular parameters are through the greedy algorithm of Andonie] described herein is further able to intelligently and automatically detect effective search directions and refine the tuning region, and hence arrive at the desired result in an efficient way.”)
Gibiansky teaches wherein the method is performed by at least one processing device comprising a processor coupled to a memory.
(Gibianksy, “[0010] According to one innovative aspect of the subject matter described in this disclosure, a system comprises: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the system to: receive data; determine a first candidate machine learning method; tune one or more parameters of the first candidate machine learning method; determine that the first candidate machine learning method and a first parameter configuration for the first candidate machine learning method are the best based on a measure of fitness subsequent to satisfaction of a stop condition; and output the first candidate machine learning method and the first parameter configuration for the first candidate machine learning method.”)
However, Gibiansky does not explicitly teach wherein the given machine learning algorithm comprises a decision tree-based machine learning algorithm, wherein the decision tree-based machine learning algorithm comprises a gradient boosting algorithm,
and wherein the three or more hyperparameters comprise: (i) a first hyperparameter of the decision tree-based machine learning algorithm pertaining to at least one decision tree split quality metric,
(ii) a second hyperparameter of the decision tree-based machine learning algorithm pertaining to a number of data features associated with one or more decision tree splits,
and (iii) a third hyperparameter of the decision tree-based machine learning algorithm pertaining to a number of data features associated with one or more leaf nodes;
determining a first value, among the possible values, for the first of the three or more hyperparameters by iterating through each of the possible values for the first hyperparameter, generating a version of the decision tree-based machine learning algorithm corresponding to each iteration, scoring each version of the decision tree-based machine learning algorithm, and identifying the first value based on the scoring of each version, wherein the first value comprises a particular value for the first hyperparameter, and wherein identifying the particular value for the first hyperparameter comprises comparing scores attributed to the scored versions of the decision tree-based machine learning algorithm and selecting the value corresponding to a particular score relative to the other scores;
determining at least a second value, among the possible values, for at least the second of the three or more hyperparameters by iterating through each of the possible values for the at least the second of the three or more hyperparameters, with each iteration being carried out in conjunction with using the identified first value for the first hyperparameter, generating a version of the decision tree-based machine learning algorithm corresponding to each iteration, scoring each version of the decision tree-based machine learning algorithm, and identifying at least the second value for the at least the second of the three or more hyperparameters based on the scoring of each version, wherein the at least a second value comprises at least one particular value for the at least the second of the three or more hyperparameters, and wherein identifying the at least one particular value for the at least the second of the three or more hyperparameters comprises, for each of the at least the second of the three or more hyperparameters, comparing scores attributed to the scored versions of the decision tree-based machine learning algorithm and selecting the value corresponding to a particular score relative to the other scores;
Yang teaches wherein the given machine learning algorithm comprises a decision tree-based machine learning algorithm,
(Yang, Section 3.1.5, “Decision tree (DT) [53] [wherein the given machine learning algorithm comprises a decision tree-based machine learning algorithm] is a common classification method that uses a tree-structure to model decisions and possible consequences by summarizing a set of classification rules from the data. A DT has three main components: a root node representing the entire data; multiple decision nodes indicating decision tests and sub-node splits over each feature; and several leaf nodes representing the result classes [54]. DT algorithms recursively split the training set with better feature values to achieve good decisions on each subset. Pruning, which means removing some of the sub-nodes of decision nodes, is used in DT to avoid over-fitting. Since a deeper tree has more sub-trees to make more accurate decisions, the maximum tree depth, ’max depth’, is an essential hyper-parameter that controls the complexity of DT algorithms [55]. There are many other important HPs to be tuned to build effective DT models [56].”)
Yang teaches wherein the decision tree-based machine learning algorithm comprises a gradient boosting algorithm,
(Yang, Section 3.1.5, “Based on the concept of DT models, many decision-tree-based ensemble algorithms have been proposed to improve model performance by combining multiple decision trees, including random forest (RF), extra trees (ET), and extreme gradient boosting (XGBoost) [wherein the decision tree-based machine learning algorithm comprises a gradient boosting algorithm] models.”)
Yang teaches and wherein the three or more hyperparameters comprise: (i) a first hyperparameter of the decision tree-based machine learning algorithm pertaining to at least one decision tree split quality metric ie criterion,
(Yang, Section 3.1.5, “Firstly, the quality of splits can be measured by setting a measuring function, denoted by ’criterion’ in sklearn. Gini impurity and information gain are the two main types of measuring functions.”)
Yang teaches (ii) a second hyperparameter of the decision tree-based machine learning algorithm pertaining to a number of data features associated with one or more decision tree splits ie max_features,
(Yang, Section 3.1.5, “The number of considered features to generate the best split, ’max_features’, can also be tuned as a feature selection process.”)
Yang teaches and (iii) a third hyperparameter of the decision tree-based machine learning algorithm pertaining to a number of data features associated with one or more leaf nodes ie min_samples_leaf;
(Yang, Section 3.1.5, “Moreover, there are several discrete hyper-parameters related to the splitting process: the minimum number of data points to split a decision node or to obtain a leaf node, denoted by ’min_samples_split’ and ’min_samples_leaf’, respectively; the ’max_leaf_nodes’, indicating the maximum number of leaf nodes, and the ’min weight fraction leaf’ that means the minimum weighted fraction of the total weights, can also be tuned to improve model performance [31] [56].”)
Andonie teaches determining a first value, among the possible values, for the first of the three or more hyperparameters by iterating through each of the possible values for the first hyperparameter, generating a version of the decision tree-based machine learning algorithm corresponding to each iteration, scoring each version of the decision tree-based machine learning algorithm, and identifying the first value based on the scoring of each version, wherein the first value comprises a particular value for the first hyperparameter, and wherein identifying the particular value for the first hyperparameter comprises comparing scores attributed to the scored versions of the decision tree-based machine learning algorithm and selecting the value corresponding to a particular score relative to the other scores;
determining at least a second value, among the possible values, for at least the second of the three or more hyperparameters by iterating through each of the possible values for the at least the second of the three or more hyperparameters, with each iteration being carried out in conjunction with using the identified first value for the first hyperparameter, generating a version of the decision tree-based machine learning algorithm corresponding to each iteration, scoring each version of the decision tree-based machine learning algorithm, and identifying at least the second value for the at least the second of the three or more hyperparameters based on the scoring of each version, wherein the at least a second value comprises at least one particular value for the at least the second of the three or more hyperparameters, and wherein identifying the at least one particular value for the at least the second of the three or more hyperparameters comprises, for each of the at least the second of the three or more hyperparameters, comparing scores attributed to the scored versions of the decision tree-based machine learning algorithm and selecting the value corresponding to a particular score relative to the other scores;
Examiner’s note: The determining limitations first/second/third hyperparameter values is interpreted to be a greedy algorithm wherein a first set of hyperparameters is considered iteratively to determine the ‘best’ hyperparameter for the first hyperparameter and the first hyperparameter is then used in part to iteratively determine the ‘best’ hyperparameter for the second hyperparameter from the second set of hyperparameters and so on.
(Andonie, Section 2, “A simple strategy for hyperparameter optimization is a greedy approach [Examiner notes: One of ordinary skills in the art would be able to recognize a greedy approach would mean making the locally optimal choice at each stage]: investigate the local neighborhood of a decision tree-based hyperparameter configuration: vary one hyperparameter at a time [by iterating through each of the possible values for the first/second/third hyperparameter; ie varying one hyperparameter at a time as in iterating through each element in the respective possible values. In the context of the applied references, it would be iterating through possible values for the criterion hyperparameter, first starting at “gini” and then going to “entropy”] and measure how performance changes [generating a version of the decision tree-based machine learning algorithm corresponding to each iteration wherein performance of the hyperparameter can only be observed from instantiating the model with the hyperparameter and executing it, scoring each version of the decision tree-based machine learning algorithm…comparing scores…and selecting the value corresponding to a particular score relative to the other scores; Andonie discloses measuring performance changes]. The only information obtained with this analysis is how different hyperparameter values perform in the context of a single instantiation of the other hyperparameters.”; Since Andonie teaches varying one hyperparameter at a time in a greedy approach, Examiner interprets the algorithm disclosed to be setting the respective hyperparameter as the locally optimal solution before moving to iterate (vary) subsequent hyperparameters with the previously determined hyperparameter. See as evidenced by Wikipedia)
Gibiansky and Yang are both considered to be analogous to the claimed invention because they are in the same field of hyperparameter tuning. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Gibiansky to incorporate the teachings of Yang in order to provide a common decision tree model with respective important hyperparameters to tune as doing so would build effective DT models (Yang, Section 3.1.5, “Decision tree (DT) [53] is a common classification method that uses a tree-structure to model decisions and possible consequences by summarizing a set of classification rules from the data… There are many other important HPs to be tuned to build effective DT models [56].”)
Andonie is considered to be analogous to the claimed invention because they are in the same field of hyperparameter optimization. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Statnikov and Weerts to incorporate the teachings of Andonie in order to provide a simple greedy algorithm to be utilized in hyperparameter optimization as doing so would yield locally optimal results to approximate a globally optimal solution in a reasonable amount of time. (Wikipedia, “In many problems, a greedy strategy does not usually produce an optimal solution, but nonetheless, a greedy heuristic may yield locally optimal solutions that approximate a globally optimal solution in a reasonable amount of time.”)
In regards to claim 2 and analogous claims 12 and 17,
Gibiansky in view of Yang and Andonie teach The computer-implemented method of claim 1,
Gibiansky teaches further comprising: performing one or more cross-validation techniques on the decision tree-based machine learning algorithm subsequent to the automatic tuning of the three or more hyperparameters.
(Gibiansky, [0093], “For example, the machine learning method unit 230 using analysis (e.g. k-fold cross-validation) trains a GBM machine learning model with the parameters received from the parameter optimization unit 240.”)
In regards to claim 3,
Gibiansky in view of Yang and Andonie teach The computer-implemented method of claim 2,
Gibiansky teaches wherein performing the one or more cross-validation techniques comprises performing one or more k-fold cross-validation techniques.
(Gibiansky, [0093], “For example, the machine learning method unit 230 using analysis (e.g. k-fold cross-validation) trains a GBM machine learning model with the parameters received from the parameter optimization unit 240.”)
In regards to claim 4 and analogous claims 13 and 18,
Gibiansky in view of Yang and Andonie teach The computer-implemented method of claim 3,
Yang teaches wherein performing one or more k-fold cross-validation techniques comprises using a number of folds equivalent to the number of hyperparameters comprised in the three or more hyperparameters of the decision tree-based machine learning algorithm.
(Yang, Section 7.1, “In the next step, the performance metrics and evaluation methods are configured. For each experiment on the selected two datasets, 3-fold cross validation [using a number of folds equivalent to the number of hyperparameters comprised in the three or more hyperparameters; wherein 3 hyperparameters were used and 3 folds were used in the cross-validation] is implemented to evaluate the involved HPO methods.”)
In regards to claim 7 and analogous claims 21 and 24,
Gibiansky in view of Yang and Andonie teach The computer-implemented method of claim 1,
Yang teaches wherein the at least a portion of possible values for each of the three or more hyperparameters comprise one or more of: at least one categorical value and at least one numerical value.
(Yang, Table 3 teaches criterion as a categorical value and max_features as a numerical value
PNG
media_image1.png
203
786
media_image1.png
Greyscale
)
In regards to claim 8 and analogous claims 22 and 25,
Gibiansky in view of Yang and Andonie teach The computer-implemented method of claim 1,
Andonie teaches wherein the at least a portion of possible values for each of the three or more hyperparameters varies in number of values, wherein iterating through each of the at least a portion of the possible values for the first hyperparameter comprises performing a first number of iterations based on a number of possible values across the three or more hyperparameters, and wherein iterating through each of the at least a portion of the possible values for the at least a second of the three or more hyperparameters comprises performing a second number of iterations based on the number of possible values across the three or more hyperparameters.
(Andonie, Section 2, “A simple strategy for hyperparameter optimization is a greedy approach: investigate the local neighborhood of a given hyperparameter configuration: vary one hyperparameter at a time [wherein the at least a portion of possible values for each of the three or more hyperparameters varies in number of values, wherein iterating through each of the at least a portion of the possible values for the first/second/third hyperparameter comprises performing a first/second/third number of iterations based on a number of possible values across the three or more hyperparameters; wherein varying one parameter at a time is iterating through each of the possible values for the first/second/third hyperparameters as it only considers a hyperparameter set at a single time] and measure how performance changes. The only information obtained with this analysis is how different hyperparameter values perform in the context of a single instantiation of the other hyperparameters.”)
In regards to claim 9,
Gibiansky in view of Yang and Andonie teach The computer-implemented method of claim 1,
Yang teaches wherein the decision tree-based machine learning algorithm further comprises one of: a support vector machines (SVM) algorithm, a random forest algorithm, a k-means clustering algorithm, a density-based spatial clustering of applications with noise (DBSCAN) algorithm, an agglomerative clustering algorithm, and a neural network.
(Yang, Section 3.1.5, “Based on the concept of DT models, many decision-tree-based ensemble algorithms have been proposed to improve model performance by combining multiple decision trees, including random forest (RF), extra trees (ET), and extreme gradient boosting (XGBoost) models.”)
In regards to claim 10 and analogous claims 23 and 26,
Gibiansky in view of Yang and Andonie teach The computer-implemented method of claim 1,
Yang teaches wherein scoring each version of the given machine algorithm comprises using at least one set of training data.
(Yang, Section 3, “To boost ML models by HPO, firstly, we need to find out what the key hyper-parameters are that people need to tune to fit the ML models into specific problems or datasets [using at least one set of training data].”)
Claims 11 and 16 are rejected on the same grounds under 35 U.S.C. 103 as claim 1 as they are substantially similar.
Claims 12 and 17 are rejected on the same grounds under 35 U.S.C. 103 as claim 2 as they are substantially similar.
Claims 13 and 18 are rejected on the same grounds under 35 U.S.C. 103 as claim 4 as they are substantially similar.
Claims 14 and 19 are rejected on the same grounds under 35 U.S.C. 103 as claim 5 as they are substantially similar.
Claims 15 and 20 are rejected on the same grounds under 35 U.S.C. 103 as claim 6 as they are substantially similar.
Claims 21 and 24 are rejected on the same grounds under 35 U.S.C. 103 as claim 7 as they are substantially similar.
Claims 22 and 25 are rejected on the same grounds under 35 U.S.C. 103 as claim 8 as they are substantially similar.
Claims 23 and 26 are rejected on the same grounds under 35 U.S.C. 103 as claim 10 as they are substantially similar.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US20190236487A1 Huang teaches Machine learning hyperparameter tuning tool
US20190370684A1 Gunes teaches System for automatic, simultaneous feature selection and hyperparameter tuning for a machine learning model
[0047] For example, the decision tree model type hyperparameters may include a maximum number of decision tree levels, a number of bins to use for numeric variables during calculation of the decision tree, and a split criterion for each tree node. A value for each of these hyperparameters is defined in each hyperparameter configuration for the decision tree model type.
US20200057944A1 Loh et al. teaches Hyperparameter optimization method and apparatus
[0078] Referring again to FIG. 5, in operation S120, the optimization apparatus 100 calculates evaluation scores for a plurality of hyperparameter samples constituting the initial hyperparameter sample set. Specifically, the optimization apparatus 100 may apply a first hyperparameter sample among the hyperparameter samples to the target model and train the target model using the given dataset 211. Then, the optimization apparatus 100 may calculate an evaluation score for the first hyperparameter sample by evaluating (e.g., cross-validating) the performance of the target model. The optimization apparatus 100 may calculate an evaluation score for each of the hyperparameter samples constituting the initial hyperparameter sample set by repeating the above process on the other hyperparameter samples. If the evaluation scores were already calculated in the previous operation S100, operation S120 can be omitted.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASMINE THAI whose telephone number is (703)756-5904. The examiner can normally be reached M-F 8-4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.T.T./Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129
Prosecution Insights