DETAILED ACTION
This action is in response to the claims filed 01/24/2024 for Application number 18/421,416. Claims 1-20 are currently pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/24/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 4-5, 7, 9, 11-12, 14-15, 17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Rashidi et al. ("US 20210150412 A1", hereinafter "Rashidi") in view of Xia et al. ("A boosted decision tree approach using Bayesian hyper-parameter optimization for credit scoring", hereinafter "Xia").
Regarding claim 1, Rashidi teaches A system for performing model selection using hyperparameter optimization in conjunction with feature selection (Abstract, “each associated with a particular combination of features and hyperparameters”), comprising:
one or more processors (¶0057); and
one or more non-transitory, computer-readable media storing instructions that, when executed by the one or more processors (¶0052), cause operations comprising:
receiving training data intended for generating a final machine learning model (“The data may be in any suitable format or type (e.g. database, array, flat file, concatenated file, etc.), and may comprise balanced data (e.g. having an equal proportion of positive and negative outcomes or classifications) which may be used for training and initial validation of the generated ML models” [¶0019]), wherein the final machine learning model, once generated, includes a selection from a plurality of input features based on the training data (“This ensures that the resulting models are not only optimized by hyperparameters, but also feature selection (different feature sets may be used for each model, (would include the final model) as independent subsets of the originally provided feature set within the input data).” [¶0023]) and uses a first hyperparameter configuration (“Furthermore, for each combination of a given feature set and model type, models may be generated and trained with different hyperparameter values (e.g. different gamma and c values for support vector machines, etc.).” [¶0024]);
wherein the hyperparameter search space comprises ranges for a set of hyperparameters for the final machine learning model (“In a further implementation, the method includes generating a plurality of values for each hyperparameter, the plurality of values distributed across a predetermined range;” [¶0045]);
generating a set of hyperparameter configurations from the hyperparameter search space (“generate a plurality of hyperparameter sets; generate a plurality of machine learning models, each machine learning model utilizing one of the plurality of feature sets and one of the plurality of hyperparameter sets;” [¶0047]);
generating, for each hyperparameter configuration in the set of hyperparameter configurations, a feature vector by executing a feature selection method, thereby generating a set of feature vectors corresponding to the set of hyperparameter configurations (“In some implementations, different feature selection processes may be performed in separate pipelines or for use by different models. This ensures that the resulting models are not only optimized by hyperparameters, but also feature selection (different feature sets may be used for each model, as independent subsets of the originally provided feature set within the input data)” [¶0023; note: Feature selection implies the generation of feature vectors.]);
based on the set of feature vectors and the training data, generating a set of candidate models corresponding to the set of hyperparameter configurations, wherein each candidate model in the set of candidate models uses a feature vector in the set of feature vectors as input features and is trained using a hyperparameter configuration corresponding to the feature vector (“generate a plurality of hyperparameter sets; generate a plurality of machine learning models, each machine learning model utilizing one of the plurality of feature sets and one of the plurality of hyperparameter sets; train each of the plurality of machine learning models using a first subset of the input data;” [¶0047]);
ranking the set of candidate models based on a performance metric (“At step 114, the results of the model training and validation may be provided to the user as an ordered or ranked list, with the order corresponding to a selected scoring characteristic (e.g. accuracy, AUC ROC, etc.).” [¶0026]); and
based on the rankings of the set of candidate models, selecting the final machine learning model from the set of candidate models (“At step 320, in some implementations, the system may generate a stand-alone application using a selected model (e.g. feature set, model type, and hyperparameter tuning) from the plurality of models identified in the aggregated scores (e.g. by selecting the corresponding model in the user interface).” [¶0034]).
However fails to explicitly teach training a preliminary machine learning model using the training data to generate learned parameter values of the preliminary machine learning model, wherein the preliminary machine learning model includes an entirety of the plurality of input features, and wherein the learned parameter values comprise feature values and node-level statistics of a gradient-boosted algorithm;
Xia teaches training a preliminary machine learning model using the training data to generate learned parameter values of the preliminary machine learning model, wherein the preliminary machine learning model includes an entirety of the plurality of input features (“Afterward, a model-based feature selection algorithm is applied to filter un- representative features. The feature importance scores are derived from the initial model and sorted. Then, a wrapper feature selection algorithm is utilized according to the calculated feature im- portance scores. This stage finally outputs a feature subset which is expected to remove redundant features.” [pg. 231, §3., ¶1]), and wherein the learned parameter values comprise feature values and node-level statistics of a gradient-boosted algorithm (“Then, a wrapper feature selec- tion algorithm is utilized according to the calculated feature importance scores. This stage finally outputs a feature subset which is expected to remove redundant features. In the final step, TPE is used to tune the hyper-parameters, and the final model is built with selected features and optimized parameters” [pg. 231, §3., ¶1; note: XGboost provides weight, gain, and cover as feature importance scores thus would correspond to node-level statistics of a gradient boosted algorithm.]))
based on the learned parameter values of the preliminary machine learning model, defining a hyperparameter search space (“The application of hyper-parameter optimization in XGBoost follows the framework of SMBO (Fig. 2). The model that initializes random values of the hyper-parameters in the specific searching space are summarized in Table 1.” [pg. 231, left col ¶3]),
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Rashidi’s teachings by training an initial model to generate learned parameter values as taught by Xia. One would have been motivated to make this modification in order to remove redundant features and tune hyperparameters to build a final model. [pg. 231, §3., ¶1, Xia]
Regarding claim 2, it is substantially similar to claim 1 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Regarding claim 4, Rashidi/Xia teaches The method of claim 2, where Xia teaches wherein the machine learning model uses a boosted gradient ensemble algorithm. (“This paper aims to propose a sequential ensemble credit scoring model based on a variant of gradient boosting machine (i.e., extreme gradient boosting (XGBoost)).” [Abstract])
Regarding claim 5, Rashidi/Xia teaches The method of claim 2, Xia teaches wherein the feature selection method for a hyperparameter configuration comprises:
initializing a standard feature set to train a prototype machine learning model with, wherein the prototype machine learning model uses an algorithm as the machine learning model and is trained on the training data, and wherein the prototype machine learning model uses the hyperparameter configuration for its hyperparameter values (“Afterward, a model-based feature selection algorithm is applied to filter un- representative features. The feature importance scores are derived from the initial model and sorted. Then, a wrapper feature selec- tion algorithm is utilized according to the calculated feature im- portance scores. This stage finally outputs a feature subset which is expected to remove redundant features. In the final step, TPE is used to tune the hyper-parameters, and the final model is built with selected features and optimized parameters.” [¶231, 3., ¶1; note: The examiner is interpreting the initial model to be equivalent to a “prototype/preliminary” model. Based off the instant specification, preliminary and prototype can be considered to be the same model.);
processing the prototype machine learning model to extract a feature importance vector, wherein the feature importance vector specifies an importance of each feature in the standard feature set in generating outputs of the prototype machine learning model (“Similar to CART and RF, tree boosting techniques can output feature importance score, which is a metric for measuring the im-portance of features in splitting” [pg. 231, right col, ¶2]); and
eliminating a set number of features from the standard feature set based on the feature importance vector. (“first, feature selection improves the performance of XGBoost empirically. The phenomenon may be attributed to the fact that feature selection removes the redundant features, which may be used for splitting when building model with all the features.” [pg. 231, right col, ¶2])
Same motivation to combine the teachings of Rashidi/Xia as claim 2.
Regarding claim 7, Rashidi/Xia teaches The method of claim 2, wherein ranking the set of candidate models based on the performance metric comprises:
Rashidi teaches partitioning the training data into a sample set and a testing set (“As discussed above, input data 202 is split into training data 204 and testing data 206, e.g. via a randomized or semi-randomized selection process (in any division or ratio, such as 80% training data and 20% testing data).” [¶0027]);
training each candidate model in the set of candidate models on the sample set; for each candidate model in the set of candidate models (“train each of the plurality of machine learning models using a first subset of the input data;” [¶0047]),
Xia teaches performing cross-validation testing of the candidate model using the testing set to obtain a performance metric (“The loss function L in TPE is critical to the optimization performance; thus, we try several metrics and finally decide to use 10-fold cross-validation logistic loss to measure the performances of model under corresponding parameters.” [pg. 232, left col, ¶1]); and
ranking the set of candidate models based on respective performance metrics. (“In Fig. 7 , the bars represent the average rank of the models over five datasets. The three horizontal lines denote the threshold at different significant levels. As shown in Fig. 7 , XGBoost-based models occupy the top four spots among all models.” [pg. 238, right col, ¶1])
Same motivation to combine the teachings of Rashidi/Xia as claim 2.
Regarding claim 9, Rashidi/Xia teaches The method of claim 2, Xia teaches wherein the search technique for generating the set of hyperparameter configurations is an iterative deepening space search algorithm. (“The hyper-parameter setting and the corresponding 10-fold cross-validation logistic loss are stored in H . Then, the model iterates the following steps:.. The iteration stops when the maximum number of iteration is reached.” [pg. 232, left col, ¶2])
Same motivation to combine the teachings of Rashidi/Xia as claim 2.
Regarding claim 11, Rashidi/Xia teaches The method of claim 2, Rashidi teaches wherein the machine learning model uses a stochastic gradient boosting algorithm. (“Different models may be generated for each feature selection identified at step 106, and may utilize different supervised machine learning algorithms (e.g. neural network, logistic regression, naïve Bayes, K-nearest neighbor, support vector machine, gradient boosting machine, and random forest).” [¶0024])
Regarding claim 12, it is substantially similar to claims 1 and 2 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Regarding claims 14-15, 17 and 19, they are substantially similar to claims 4-5, 7 and 9 respectively, and are rejected in the same manner, the same art, and reasoning applying.
Claims 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Rashidi in view of Xia and further in view of Karl et al. ("Multi-Objective Hyperparameter Optimization in Machine Learning—An Overview", hereinafter "Karl").
Regarding claim 3, Rashidi/Xia teaches The method of claim 2, wherein the search technique for generating the set of hyperparameter configurations is a Latin Hypercube sampling algorithm.
Karl teaches wherein the search technique for generating the set of hyperparameter configurations is a Latin Hypercube sampling algorithm. (“BO starts on an archive A of evaluated configurations, typically sampled through Latin hypercube sampling” [pg. 15, §4.4, ¶1])
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Rashidi’s/Xia’s teachings by using Laten Hypercube sampling as taught by Karl. One would have been motivated to make this modification in order to ensure the diversity of solutions when applying Bayesian optimization. [pg. 15, §4.4, ¶1, Karl]
Regarding claim 13, it is substantially similar to claim 3 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Claims 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Rashidi in view of Xia and further in view of Xia et al. ("A Model-Free Feature Selection Technique of Feature Screening and Random Forest-Based Recursive Feature Elimination", hereinafter "Xia2").
Regarding claim 6, Rashidi/Xia teaches The method of claim 2, wherein generating a candidate model in the set of candidate models from a feature vector in the set of feature vectors comprises:
Rashidi teaches training the candidate model using the training data, wherein the candidate model uses the training feature set as input features. (“Accordingly, for a given feature selection, hundreds of models may be generated (e.g. two hundred hyperparameter values for each of seven supervised learning algorithms, or 700 models per feature selection, in some implementations) … At step 108, each model may be trained using the training subset of balanced data (e.g. first portion of balanced data discussed above), and at step 110, each model may be tested using the validation subset of balanced data (e.g. second portion of balanced data discussed above).” [¶0025])
However Rashidi/Xia fails to explicitly teach
performing a random permutation on the feature vector to generate a training feature set;
Xia2 teaches performing a random permutation on the feature vector to generate a training feature set; (“Random forest accesses the relevance of a predictor by the permutation importance measure [58,67], which is used to eliminate features in the wrapper phase. T is measure is based on the idea that a variable Xj is relevant to Y if the prediction error increases when we break the link between Xj and Y, and this link can be broken by random permuting the observations of Xj.” [pg. 4, right col, ¶3])
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Rahsidi’s/Xia’s teachings by randomly permutating feature vectors as taught by Xia2. One would have been motivated to make this modification in order determine the important/relevance of a predictor which can be used to eliminate features. [Xia2, pg. 4, right col, ¶3]
Regarding claim 16, it is substantially similar to claim 6 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Claims 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Rashidi in view of Xia and further in view of Schonlau ("Boosted regression (boosting): An introductorytutorial and a Stata plugin", hereinafter "Schonlau").
Regarding claim 10, Rashidi/Xia teaches The method of claim 2, Xia teaches wherein each hyperparameter configuration in the set of hyperparameter configurations comprises: a maximum tree depth (“Tree complexity is another important parameter. In GB, tree complexity is controlled by two alternative hyper-parameters: the maximum tree depth 1 and the tree size 2” [pg. 228, left top col, ¶1]), a minimum child weight (See Table 1, pg. 230, “minimum child weight”), a maximum tree breadth (“Tree complexity is another important parameter. In GB, tree complexity is controlled by two alternative hyper-parameters: the maximum tree depth 1 and the tree size 2 (corresponds to “tree breadth”)” [pg. 228, left top col, ¶1]), and an amount of improvement in a loss function (“Boosting fits additive base learners to minimize the loss function provided.” [pg. 227, §2.1, ¶1]).
However fails to explicitly teach an average bias of residuals at a node
Schonlau teaches an average bias of residuals at a node
(“The residuals from the model are computed. A regression tree is fitted to the residuals. For each terminal node, the average y-value of the residuals that the node contains is computed. The regression tree is used to predict the residuals. (In the first step, this means that a regression tree is fitted to the difference between the observation and the average y-value. The tree then predicts those differences.) The boosting regression model—consisting of the sum of all previous regression trees—is updated to reflect the current regression tree” [pg. 336, ¶2])
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Rashidi’s/Xia’s teachings in order to use an average bias of residuals at a node as taught by Schonlau. One would have been motivated to make this modification in order to apply boosted regression which is a highly flexible regression method which allows the research to specify the x-variables without specifying the functional relationship to the response. [pg. 330, bottom para – pg. 331, top para, Schonlau]
Regarding claim 20, it is substantially similar to claim 10 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Allowable Subject Matter
Claims 8 and 18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claims 8 and 18:
based on the rankings of the set of candidate models, generating a feature allocation map, wherein the feature allocation map correlates each candidate model in the set of candidate models with an integer number of features;
for each candidate model in the set of candidate models, use a feature ranking technique to extract a number of features equal to that specified by the feature allocation map; and
generating the machine learning model to use the extracted features as input.
None of the prior art cited discloses the specific feature allocation map and specific steps recited in claims 8 and 18.
The closest prior art of record is Rashidi which discloses an automatic model selection method using feature selection and hyperparameter optimization however does not explicitly disclose using a feature allocation map, using a feature ranking technique to extract a number of features equal to that specified by the feature allocation map, and generating the machine learning model to use the extracted features.
Xia, Karl, Xia2, and Schonlau all discloses various features of the claims and all directed to methods of model selection, feature selection, hyperparameter optimization, and boosting algorithms however they do not disclose using a feature allocation map, using a feature ranking technique to extract a number of features equal to that specified by the feature allocation map, and generating the machine learning model to use the extracted features.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL H HOANG whose telephone number is (571)272-8491. The examiner can normally be reached Mon-Fri 8:30AM-4:30PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL H HOANG/PRIMARY EXAMINER, Art Unit 2122
Prosecution Insights