Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner Notes
Examiner cites particular paragraphs, columns and line numbers in the references as applied to the claims below for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the applicant fully consider the references cited in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner.
Specification
The disclosure is objected to because of the following informalities: paragraph [0022], where it states, "the data retrieval and pipeline the initialization module 21". The module is elsewhere named the data acquisition and pipeline initialization module 21. Appropriate correction is required.
The disclosure is objected to because of the following informalities: paragraph [0031], where it states, "can use the feature intersection number (i.e., the above-mentioned feature intersection number and the total feature number)". The parenthetical is circular and does not state the computation that is performed. Appropriate correction is required.
The disclosure is objected to because of the following informalities: paragraph [0040], where it states, "selects the initial pipeline P7". Pipeline P7 is identified elsewhere in the same paragraph as a new pipeline. Appropriate correction is required.
The disclosure is objected to because of the following informalities: paragraph [0024], where it states that a kernel-based method "(such as Gaussian Process)" is used to obtain the prediction result and then states "In other words" that each initial pipeline is trained and tested on the dataset. These are different operations, and it is unclear which one produces the prediction result and the accuracy. Appropriate correction is required.
The disclosure is objected to because of the following informalities: paragraphs [0021] and [0041], where the trade names "XGBoost", "AutoSklearn" and "H2O" are used without generic terminology. Each mark should be capitalized wherever it appears and be accompanied by generic terminology. See MPEP 608.01(v). Appropriate correction is required.
Claim Objections
Claim 1 is objected to because of the following informalities: where it states, "the data acquisition and pipeline initialization module use at least one algorithm". Appropriate correction is required.
Claim 2 is objected to because of the following informalities: where it states, "module receive the algorithm" and "module use the algorithm”. Appropriate correction is required.
Claim 4 is objected to because of the following informalities: where it states, "the initial hyperparameter include". The verb should read "includes". Appropriate correction is required.
Claim 5 is objected to because of the following informalities: where it states, "module receive the dataset". Appropriate correction is required.
Claim 16 is objected to because of the following informalities: where it states, "wherein the method includes following steps". Appropriate correction is required.
Claim Rejections – 35 U.S.C. § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Regarding claims 1 and 16, the claims recite "to determine a recommended new pipeline in the new pipeline". Only "a new pipeline", in the singular, is introduced, so it is unclear how one pipeline is determined "in" a single pipeline. Paragraph [0036] of the specification selects among plural candidate pipelines P7 to P10 by highest sampling score. For purposes of examination, the limitation is understood as selecting a recommended new pipeline from one or more candidate new pipelines.
Regarding claims 1 and 16, the claims recite "from the recommended new pipelines". There is insufficient antecedent basis for this limitation in the claims. The claims introduce a single "recommended new pipeline" and recite no iteration that would produce several. For purposes of examination, the phrase is understood as the set of pipelines recommended over one or more iterations.
Regarding claims 1 and 16, the claims recite a "target recommended new pipeline". In paragraph [0040], the pool from which the selection is made contains initial pipelines P1 to P6, and the example selects initial pipelines P1 and P2. It is therefore unclear whether a "target recommended new pipeline" may be an initial pipeline. For purposes of examination, the term is understood as a pipeline selected for the ensemble from the evaluated pool.
Regarding claims 1 and 16, the claims recite an "inter algorithm diversity value" and an "intra algorithm hyperparameter distance value" while requiring only "at least one algorithm" and "at least one initial pipeline". Where there is only one algorithm, there is nothing between algorithms to compare, and the claims do not state which two things each value relates. For purposes of examination, the terms are understood as set forth in the Claim Interpretation section above.
Regarding claims 3 and 17, the claims recite "a first best accuracy initial pipeline prediction result and a second best accuracy initial pipeline prediction result". "First best" and "second best" may mean the best and second-best pipelines overall, or the best pipeline of a first algorithm and the best pipeline of a second algorithm. Only claim 4 supplies the second meaning, and claim 17 has no counterpart to claim 4. For purposes of examination, either meaning is understood to satisfy the claims.
Regarding claim 4, the claim recites "the first initial pipeline includes a first best accuracy initial pipeline and at least one first other initial pipeline". A single pipeline cannot include several pipelines. The "first initial pipeline" appears to name a group of pipelines, and "the first accuracy" is then used for different members of that group. For purposes of examination, the limitation is understood as a first group of initial pipelines that share the first algorithm, each having its own accuracy.
Regarding claims 11- 13, the claims recite lists joined by "and", for example "the sampling function includes an Expected Improvement (EI), an Upper Confidence Bound (UCB), a Probability of Improvement (POI), and an Entropy Search (ES)". It is unclear whether every listed member is required or whether any one of them suffices. The categories also do not correspond, because a "value" cannot include a contingency table as in claim 11 or a kernel as in claim 13, and claim 12 lists distances as a "correlation value". The specification describes each list with "can include" and uses one member at a time, such as UCB in paragraph [0034] and cosine similarity in paragraph [0030]. For purposes of examination, each list is understood to require at least one of the listed members.
Regarding claim 18, the claim recites "receiving, by the data acquisition and pipeline initialization module, the dataset through the transceiver" as part of the step of obtaining the three values. Claim 16 already uses the dataset in the earlier evaluating step, so placing its receipt inside a later step renders the order of the steps unclear. For purposes of examination, the claim is understood to require that the dataset be received before it is used.
Claims 2-15 and 17-20 are also rejected because they depend from a rejected base claim and do not cure its deficiencies.
Claim Rejections – 35 U.S.C. § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-6 and 9-19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1:
Claim 1 is directed to "A device for recommending pipelines for an ensemble model", a device, and is therefore directed to a machine, which is one of the four statutory categories.
Step 2A, Prong One:
Claim 1 recites the following limitations:
the pipeline sampling score calculation module uses prediction result, the accuracy, and a new pipeline to obtain at least one inter algorithm diversity value, at least one inter feature correlation value, and at least one intra algorithm hyperparameter distance value;
the pipeline sampling score calculation module uses the inter algorithm diversity value, the inter feature correlation value, and the intra algorithm hyperparameter distance value to obtain at least one sampling score corresponding to the new pipeline;
the pipeline recommendation module uses the sampling score to determine a recommended new pipeline in the new pipeline;
the ensemble model recommendation module uses an ensemble learning method to select at least one target recommended new pipeline from the recommended new pipelines.
The first two limitations recite mathematical calculations. The specification computes the three values as a cosine similarity (paragraph [0030]), a feature intersection ratio (paragraph [0031]) and a radial basis function kernel value (paragraph [0032]), and computes the sampling score as the value of an acquisition function such as UCB (paragraph [0034]). The last two limitations can be performed in the human mind through observation, evaluation, judgment, and opinion, with the aid of pen and paper, and therefore recite a mental process. The specification determines the recommended pipeline by picking the candidate with the highest score (paragraph [0036]) and selects the target pipelines with weights computed as selection counts divided by a total (paragraph [0040]). See MPEP 2106.04(a)(2).
Accordingly, claim 1 recites a judicial exception (i.e., an abstract idea).
Step 2A, Prong Two:
Claim 1 recites the following additional elements:
a storage medium, storing a plurality of modules, and a processor, coupled to the storage medium, and accessing and executing the plurality of modules;
the data acquisition and pipeline initialization module use at least one algorithm to generate at least one initial pipeline;
the pipeline performance evaluation module uses a dataset to obtain at least one prediction result corresponding to the initial pipeline, and uses the dataset to obtain at least one accuracy corresponding to the initial pipeline.
The storage medium, the processor and the modules are generic computer components that are used as tools to perform the abstract idea. See MPEP 2106.05(f). Generating an initial pipeline and evaluating it on a dataset are recited at a high level of generality and merely supply the data on which the calculations operate, which constitutes insignificant extra-solution activity. See MPEP 2106.05(g).
The specification states that the disclosure recommends pipelines of similar accuracy with significantly fewer trials (paragraphs [0007] and [0041], Tables 1 and 2). It attributes that result to a Gaussian-process performance model whose hybrid kernel is established from the three values and is evaluated with a sampling function (paragraphs [0029], [0033] and [0034]). Claim 1 does not recite those components. It requires only that the three values be "used" to obtain a score, in any manner. A claim must include the components or steps that provide the improvement described in the specification, and claim 1 does not. Accordingly, the additional elements do not integrate the judicial exception into a practical application.
Step 2B:
The additional elements within this claim fail to amount to significantly more than the abstract idea (MPEP 2106.05). The storage medium, the processor and the modules are generic computer components. Generating pipelines from algorithms and evaluating them on a dataset are well-understood, routine, and conventional activities in automated machine learning, as the specification itself acknowledges in describing how models are conventionally trained and evaluated over combinations of hyperparameters and how ensemble learning is often used to select among them (paragraphs [0002] and [0003]). The elements, both individually and in combination, do no more than apply the abstract idea using generic computing components. Consequently, claim 1 as a whole does not amount to significantly more than the recited judicial exception and the claim is not eligible.
Claim 2 is dependent on claim 1, and therefore inherits the same judicial exception recited in claim 1. Further, claim 2 recites a transceiver coupled to the processor, through which the data acquisition and pipeline initialization module receive the algorithm, at least one hyperparameter range, an initial number, a preset number of target pipeline and a training time, and use them to generate the initial pipeline. The transceiver is a generic computer component, and receiving these inputs amounts to mere data gathering, which is insignificant extra-solution activity. See MPEP 2106.05(g). Receiving data over a network is also well-understood, routine, and conventional. See MPEP 2106.05(d)(II). Accordingly, for the same reasons presented with respect to claim 1, the additional elements are not indicative of integration into a practical application, nor do they amount to significantly more than the recited judicial exception. Thus, claim 2 is not eligible.
Claim 3 is dependent on claim 1, and therefore inherits the same judicial exception recited in claim 1. Further, claim 3 recites that the pipeline sampling score calculation module uses a first best accuracy initial pipeline prediction result and a second best accuracy initial pipeline prediction result to obtain the inter algorithm diversity value, which further specifies the inputs to a mathematical calculation and therefore recites a mathematical concept. Accordingly, for the same reasons presented with respect to claim 1, the claim does not recite additional elements that are indicative of integration into a practical application, nor that amount to significantly more than the recited judicial exception. Thus, claim 3 is not eligible.
Claim 4 is dependent on claim 3, and therefore inherits the same judicial exception recited in claim 3. Further, claim 4 recites that the algorithm includes a first algorithm and a different second algorithm, that the initial pipeline includes a first initial pipeline and a second initial pipeline having corresponding initial hyperparameters, prediction results and accuracies, and that each of the first and second initial pipelines includes a best accuracy initial pipeline whose accuracy is greater than that of the other initial pipelines. These limitations describe the data on which the calculation operates and a comparison of accuracies, which can be performed in the human mind through observation, evaluation, judgment, and opinion, with the aid of pen and paper, and therefore recite a mental process. Accordingly, for the same reasons presented with respect to claim 3, the claim does not recite additional elements that are indicative of integration into a practical application, nor that amount to significantly more than the recited judicial exception. Thus, claim 4 is not eligible.
Claim 5 is dependent on claim 1, and therefore inherits the same judicial exception recited in claim 1. Further, claim 5 recites a transceiver through which the dataset and the new pipeline are received, which amounts to mere data gathering by a generic computer component. See MPEP 2106.05(g). Claim 5 also recites that the pipeline sampling score calculation module uses the initial feature set and the new feature set to obtain the inter feature correlation value, which further specifies the inputs to a mathematical calculation and therefore recites a mathematical concept.
Accordingly, for the same reasons presented with respect to claim 1, the additional elements are not indicative of integration into a practical application, nor do they amount to significantly more than the recited judicial exception. Thus, claim 5 is not eligible.
Claim 6 is dependent on claim 1, and therefore inherits the same judicial exception recited in claim 1. Further, claim 6 recites that the pipeline sampling score calculation module uses the initial hyperparameter and the new hyperparameter to obtain the intra algorithm hyperparameter distance value, which further specifies the inputs to a mathematical calculation and therefore recites a mathematical concept. Accordingly, for the same reasons presented with respect to claim 1, the claim does not recite additional elements that are indicative of integration into a practical application, nor that amount to significantly more than the recited judicial exception. Thus, claim 6 is not eligible.
Claim 9 is dependent on claim 1, and therefore inherits the same judicial exception recited in claim 1. Further, claim 9 recites the following additional element, wherein the algorithm includes a feature selection algorithm and a model algorithm, which amounts to merely indicating a field of use or technological environment in which to apply a judicial exception. See MPEP 2106.05(h). Since the claim does not recite additional elements that either integrate the judicial exception into a practical application or provide significantly more than the judicial exception, claim 9 is not eligible.
Claim 10 is dependent on claim 1, and therefore inherits the same judicial exception recited in claim 1. Further, claim 10 recites that the pipeline performance evaluation module uses a kernel-based method and the dataset to obtain the prediction result and the accuracy. The kernel-based method is recited at a high level of generality and, like the evaluation in claim 1, merely supplies the data on which the calculations operate. See MPEP 2106.05(g). Accordingly, for the same reasons presented with respect to claim 1, the additional elements are not indicative of integration into a practical application, nor do they amount to significantly more than the recited judicial exception. Thus, claim 10 is not eligible.
Claim 11 is dependent on claim 1, and therefore inherits the same judicial exception recited in claim 1. Further, claim 11 recites that the inter algorithm diversity value includes a cosine similarity and a contingency table, which names the mathematical relationship used and therefore recites a mathematical concept. Thus, for the same reasons presented with respect to claim 1, claim 11 is not eligible.
Claim 12 is dependent on claim 1, and therefore inherits the same judicial exception recited in claim 1. Further, claim 12 recites that the inter feature correlation value includes an absolute value of a Pearson correlation coefficient, an absolute value of a Spearman correlation coefficient, a feature intersection number divided by a total feature number, a Euclidean Distance, and a Mahalanobis Distance, which names the mathematical relationship used and therefore recites a mathematical concept. Thus, for the same reasons presented with respect to claim 1, claim 12 is not eligible.
Claim 13 is dependent on claim 1, and therefore inherits the same judicial exception recited in claim 1. Further, claim 13 recites that the intra algorithm hyperparameter distance value includes a Radial Basis Function kernel, a Laplace kernel, a Matern kernel, and a Rational Quadratic kernel, which names the mathematical relationship used and therefore recites a mathematical concept. Thus, for the same reasons presented with respect to claim 1, claim 13 is not eligible.
Claim 14 is dependent on claim 1, and therefore inherits the same judicial exception recited in claim 1. Further, claim 14 recites that the pipeline recommendation module uses a preset execution time and the sampling score to determine the recommended new pipeline. Deciding when to stop evaluating candidates based on elapsed time can be performed in the human mind through observation, evaluation, judgment, and opinion, with the aid of pen and paper, and therefore recites a mental process. Thus, for the same reasons presented with respect to claim 1, claim 14 is not eligible.
Claim 15 is dependent on claim 1, and therefore inherits the same judicial exception recited in claim 1. Further, claim 15 recites that the ensemble model recommendation module uses a preset number and an ensemble learning method to select the target recommended new pipeline. Selecting a preset number of candidates can be performed in the human mind through observation, evaluation, judgment, and opinion, with the aid of pen and paper, and therefore recites a mental process. Thus, for the same reasons presented with respect to claim 1, claim 15 is not eligible.
Claim 16 is directed to "A method for recommending pipelines for an ensemble model", a method, and is therefore directed to a process, which is one of the four statutory categories. Claims 16-19 recite substantially the same limitations as those recited in claims 1, 3, 5 and 6, respectively, applied to the method of claim 16. Thus, for the same reasons presented with respect to claims 1, 3, 5 and 6, claims 16-19 are directed to an abstract idea without significantly more and are not eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6 and 9-19 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (US 2022/0300821 A1, hereinafter Wang) in view of Swersky et al. (US 2018/0349158 A1, hereinafter Swersky), further in view of Gunes et al. (US 10,600,005 B2, hereinafter Gunes), and further in view of Owhadi Kareshk et al. (US 2025/0077876 A1, hereinafter Owhadi Kareshk).
As per Claim 1, Wang discloses,
A device for recommending pipelines for an ensemble model, including:
a storage medium, storing a plurality of modules, wherein the plurality of modules includes a data acquisition and pipeline initialization module, a pipeline performance evaluation module, a pipeline sampling score calculation module, a pipeline recommendation module and an ensemble model recommendation module; and a processor, coupled to the storage medium, and accessing and executing the plurality of modules, (Wang, Figure 8 and paras [0059]-[0062] teach a computer platform 800 having a CPU 804 and an HDD 806 that stores a program made up of modules 842 through 858, including a pipeline generation module 844, a ranking/scoring module 848, a hyperparameter module 850 and an ensemble module 854. Para [0046] teaches an evaluation operation module 330. Para [0060] teaches that the functionalities may be combined into fewer modules than shown. The pipeline generation module is understood to be the data acquisition and pipeline initialization module, the evaluation operation module is understood to be the pipeline performance evaluation module, the ranking/scoring and hyperparameter modules are understood to be the sampling score calculation and pipeline recommendation modules, and the ensemble module is understood to be the ensemble model recommendation module.)
wherein the data acquisition and pipeline initialization module use at least one algorithm to generate at least one initial pipeline; (Wang, Figure 2 and para [0044] teach a Bayesian Optimization model training and selection method 250 that includes model candidates 210, a hyperparameter optimization function 215 and feature selection 220. Para [0051] teaches that the model candidates include a support-vector machine (SVM) algorithm and a gradient boost machine (XGBM). Para [0061] teaches that the pipeline generation module 844 generates multiple pipelines, including a Bayesian Optimized pipeline. This is understood to be the same step described in para [0021]-[0022] of the Specification, where a feature selection algorithm and a model algorithm with hyperparameter values form each initial pipeline.)
the pipeline performance evaluation module uses a dataset to obtain at least one prediction result corresponding to the initial pipeline, and uses the dataset to obtain at least one accuracy corresponding to the initial pipeline; (Wang, Figure 3 shows training data 310 and holdout data supplied to the candidate pipelines, and para [0046] teaches that the evaluation operation module 330 executes a test regarding the accuracy of the pipelines. Testing a pipeline on data produces the outputs of that pipeline for the data, which are understood to be the prediction result. Para [0044] teaches that the candidate pipelines are ranked by a ranking function 260 in terms of accuracy, and Figure 4 shows the accuracy obtained for each pipeline, which is understood to be the accuracy.)
the ensemble model recommendation module uses an ensemble learning method to select at least one target recommended new pipeline from the recommended new pipelines. (Wang, Figure 2 and paras [0007], [0045] and [0062], and claim 6, teach an ensemble function 265 and an ensemble module 854 that generate an ensemble from at least two of the candidate machine learning pipelines to form a single predictive model.)
Wang does not disclose but Swersky discloses:
the pipeline sampling score calculation module uses the prediction result, the accuracy, and a new pipeline to obtain ... at least one intra algorithm hyperparameter distance value, (Swersky paras [0158]-[0159] teach a probabilistic model of an objective function relating the hyper-parameters of a machine learning system to its performance, the model comprising a Gaussian process whose covariance function is specified by a kernel, such as the 5/2 Matern kernel of equation (1), evaluated between points x and x' that "may represent sets of hyper-parameter values of a machine learning system". Para [0160] teaches that the model is conditioned on all previously obtained evaluations of the objective function, which are understood to be the accuracies of the initial pipelines. Para [0111] teaches that, for a categorical variable, a common kernel function "assigns a 1 if two inputs belong to the same category, and 0 otherwise". With the algorithm type as the categorical variable, the kernel value between the hyper-parameters of a previously evaluated configuration and those of a new configuration contributes only where both belong to the same algorithm. This is understood to be the intra algorithm hyperparameter distance value, and is consistent with para [0032] of the Specification, where the value is 0 when the algorithms differ.)
and the pipeline sampling score calculation module uses the ... values to obtain at least one sampling score corresponding to the new pipeline; (Swersky paras [0177]-[0178] teach an acquisition utility function that associates each point at which the objective function may be evaluated with a value representing the utility of evaluating it, the function being specified from the predictive mean and predictive covariance of the probabilistic model. Figures 2A-2D and para [0179] show the values of the acquisition utility function 231. Paras [0183]-[0186] teach the probability of improvement, expected improvement, regret minimization and entropy search acquisition utility functions. The acquisition utility function value of a new point is understood to be the sampling score corresponding to the new pipeline.)
the pipeline recommendation module uses the sampling score to determine a recommended new pipeline in the new pipeline; (Swersky para [0177] teaches that the next point at which to evaluate the objective function is selected as the point that maximizes the acquisition utility function, and para [0179] teaches that the next point is selected as the point at which the acquisition utility function 231 takes on its maximum value 230. Wang para [0004] likewise teaches generating a priority score for each candidate pipeline and using the candidate pipeline having the highest priority score.)
Wang is considered to be analogous art to the claimed invention because it is in the same field of automated machine learning in which candidate pipelines are generated by Bayesian optimization, evaluated, ranked and combined into an ensemble. Swersky is also considered to be analogous art because it is in the same field of selecting the hyper-parameter values of a machine learning system by Bayesian optimization. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the Bayesian optimization pipeline search of Wang to score each new candidate pipeline with the Gaussian-process kernel and acquisition utility function of Swersky, and to recommend the candidate at which the acquisition utility function is greatest. One of ordinary skill would have been motivated to do so because Wang names Bayesian optimization for its model training and selection method (paras [0039] and [0044]) but leaves its implementation open, while Swersky teaches that each evaluation of a machine learning configuration may require a significant amount of time and/or power (para [0141]) and that Bayesian optimization techniques "are designed to reduce the number of such evaluations" (para [0142]). The modification applies a known technique to a known method ready for improvement, with predictable results.
Wang in view of Swersky does not disclose but Gunes discloses:
at least one inter feature correlation value, (Gunes, col. 1:45-49 and col. 17:10-22, teach that each evaluated unit is a pair made up of a feature set, which is a subset of the variables of the input dataset, and a hyperparameter configuration. Col. 19:18-30 teaches that an estimation model is trained on the evaluated pairs and predicts the accuracy of every pair that has not been trained. Col. 16:34-40 teaches the rationale that "feature sets A and B would have a similar test error for a hyperparameter configuration if they both include and exclude some essential information". Col. 25:24-27 teaches that the feature set is treated as a nominal input to the estimation model. Swersky para [0111] teaches that a nominal, i.e. categorical, input is embedded in a continuous space "where similar categories will be closer in terms of Euclidean distance" and is then "concatenated with a continuous input and used in a standard kernel". When combined, the kernel term computed between the feature set of an evaluated pipeline and the feature set of a new pipeline is a value expressing how the two feature sets relate. This is understood to be the inter feature correlation value described in para [0031] of the Specification. Gunes, col. 18:49-60, further teaches that each trained model is executed on a validation dataset 126 to predict a target variable value for each observation vector and that an accuracy value is computed from those predictions.)
Gunes is considered to be analogous art to the claimed invention because it is in the same field of jointly selecting a feature set and hyperparameters for a machine learning model. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the pipeline search of Wang in view of Swersky to evaluate pairs of feature sets and hyperparameter configurations as taught by Gunes, and to include the feature set of each pipeline as an input of the kernel of Swersky. One of ordinary skill would have been motivated to do so because the method of Wang already performs feature selection together with hyperparameter optimization (paras [0010] and [0044]), and Gunes teaches that combining feature selection with hyperparameter tuning in this way improves model automation "at a minimal computational cost" (col. 26:18-21). Gunes states that its estimation model may be "any other supervised modeling algorithm" (col. 15:63-65), so using the Gaussian process of Swersky in that role is a substitution of one known performance model for another with predictable results, and the categorical embedding of Swersky (para [0111]) accommodates the nominal feature set input of Gunes (col. 25:24-27).
Wang in view of Swersky and Gunes does not disclose but Owhadi Kareshk discloses:
at least one inter algorithm diversity value, (Owhadi Kareshk, Figure 1 and para [0020], teach a model selector 140 that includes a diversity score generator 142 and a correctness score generator 144. Paras [0022]-[0023] teach that different ML algorithms, for example a support vector machine algorithm and linear regression, and different hyperparameters are used to generate the multiple ML models. Paras [0027]-[0031] and [0061] teach that a diversity score D(o_i, o_j) is generated for each pair of ML models from the output data that each model of the pair produced for the same input data, the score reflecting how different the respective outputs are. The output data is understood to be the prediction result, and the diversity score between models of different algorithms is understood to be the inter algorithm diversity value. Para [0017] and claim 5 further teach weights that trade the diversity value against the correctness value when the models are selected. Paras [0027], [0051] and [0064] further teach that the model selector 140 selects a subset of the trained models, based on the diversity scores and the correctness scores, for use in an ensemble setup using hard or soft voters.)
Owhadi Kareshk is considered to be analogous art to the claimed invention because it is in the same field of selecting, from automatically generated machine-learned models, the models to be used in an ensemble. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the pipeline search of Wang in view of Swersky and Gunes to compute the diversity score of Owhadi Kareshk between the trained pipelines of different algorithms, and to use that diversity score together with the hyperparameter and feature set kernel values in obtaining the sampling score of each new pipeline. One of ordinary skill would have been motivated to do so because Wang forms an ensemble without stating how its members are chosen (para [0045]), while Owhadi Kareshk teaches that diverse or uncorrelated errors "make ensemble learning more effective" (para [0027]) and that, of two very accurate models whose predictions are similar, only one will be added to the ensemble (para [0044]). One of ordinary skill would have recognized from these teachings that a candidate of an algorithm whose trained models behave like the models already found is unlikely to be chosen for the ensemble, and would have weighted the acquisition utility value of each candidate by the diversity score already computed between the trained models of the candidate's algorithm and those of the other algorithm, in the manner that Owhadi Kareshk already weights diversity against correctness (para [0017], claim 5). Training effort is thereby directed to candidates that are likely to join the ensemble. The resulting sampling score depends on the hyperparameter kernel value, the feature set kernel value and the diversity score, as claimed.
As per claim 2, Wang in view of Swersky, Gunes and Owhadi Kareshk discloses,
further including a transceiver coupled to the processor, wherein the data acquisition and pipeline initialization module receive the algorithm, at least one hyperparameter range, an initial number, a preset number of target pipeline and a training time through the transceiver; the data acquisition and pipeline initialization module use the algorithm, the hyperparameter range, the initial number, the preset number of target pipeline, and the training time to generate the initial pipeline. (Wang, Figure 8 and paras [0059] and [0062], teach a communication interface 816 and a communication interface module 858 used to send and receive data and instructions. Figure 2 and para [0045] teach a user input 280, which supplies the data and a run time, used to control the settings of the operations. Gunes, col. 4:31-45, teaches a communication interface 106 for receiving and transmitting data between devices. Col. 9:61 to col. 10:13 teaches receiving an indicator of the model type together with a maximum time. Col. 12:18-21 teaches receiving a lower bound value and an upper bound value for each hyperparameter, which is understood to be the hyperparameter range. Col. 16:57-63 and col. 17:28-32 teach receiving the percentage of combinations and the number of feature sets to select for the initial model training, which is understood to be the initial number. Owhadi Kareshk para [0046] and claim 8 teach a maximum number of ML models to select for the ensemble, which is understood to be the preset number of target pipelines.)
As per claim 3, Wang in view of Swersky, Gunes and Owhadi Kareshk discloses,
wherein the pipeline sampling score calculation module uses a first best accuracy initial pipeline prediction result and a second best accuracy initial pipeline prediction result to obtain the inter algorithm diversity value. (Owhadi Kareshk para [0029] teaches that a diversity score is generated for each pair of ML models, and para [0035] teaches that a correctness score, such as accuracy, is generated for each ML model. The pairs for which diversity scores are generated therefore include the pair formed by the most accurate model of a first algorithm and the most accurate model of a second algorithm. It would have been obvious to one of ordinary skill in the art to use the most accurate model of each algorithm as the representative of that algorithm, because para [0044] of Owhadi Kareshk teaches that the selection favors the models with the highest correctness.)
As per claim 4, Wang in view of Swersky, Gunes and Owhadi Kareshk discloses,
wherein an initial hyperparameter corresponds to the initial pipeline, wherein the algorithm includes a first algorithm and a second algorithm, wherein the first algorithm is different from the second algorithm; the initial pipeline includes a first initial pipeline and a second initial pipeline; the initial hyperparameter include a first initial hyperparameter and a second initial hyperparameter; the first initial pipeline includes the first initial hyperparameter corresponding to the first algorithm, and the second initial pipeline includes the second initial hyperparameter corresponding to the second algorithm; the prediction result includes a first prediction result and a second prediction result, and the accuracy includes a first accuracy and a second accuracy; the first prediction result corresponds to the first initial pipeline, and the first accuracy corresponds to the first initial pipeline, the second prediction result corresponds to the second initial pipeline, and the second accuracy corresponds to the second initial pipeline; the first initial pipeline includes a first best accuracy initial pipeline and at least one first other initial pipeline, wherein the first accuracy of the first best accuracy initial pipeline is greater than the first accuracy of the first other initial pipeline, wherein the first best accuracy initial pipeline corresponds to the first best accuracy initial pipeline prediction result; the second initial pipeline includes a second best accuracy initial pipeline and at least one second other initial pipeline, wherein the second accuracy of the second best accuracy initial pipeline is greater than the second accuracy of the second other initial pipeline, wherein the second best accuracy initial pipeline corresponds to the second best accuracy initial pipeline prediction result. (Wang para [0051] teaches SVM and XGBM model candidates, each of which is subject to hyperparameter optimization. Owhadi Kareshk para [0023] teaches that a first ML algorithm is used to generate a first number of ML models, and a second ML algorithm is used to generate a second number of ML models, where a different set of hyperparameters is used for each ML model. Para [0030] teaches that each ML model produces its own output data, and para [0035] teaches that a correctness score is generated for each ML model. Within each of the two groups of models, one model has a correctness score greater than that of the others. This is understood to be the first and the second-best accuracy initial pipeline, each with its corresponding prediction result.)
As per claim 5, Wang in view of Swersky, Gunes and Owhadi Kareshk discloses,
further including a transceiver coupled to the processor, wherein the dataset includes a plurality of features, wherein an initial feature set corresponds to the initial pipeline, and the initial feature set includes at least one of the plurality of features, wherein the data acquisition and pipeline initialization module receive the dataset through the transceiver; the pipeline sampling score calculation module receives the new pipeline through the transceiver, wherein a new feature set corresponds to the new pipeline, and the new feature set includes at least one of the plurality of features; the pipeline sampling score calculation module uses the initial feature set and the new feature set to obtain the inter feature correlation value. (Wang, Figure 5 and para [0048], teach uploading the data used for training, and Figure 2 teaches that the user input 280 supplies the data and "forced pipelines / features" by way of the communication interface of paras [0059] and [0062]. Gunes, col. 7:57-62, teaches that the training dataset and validation dataset are accessed using communication interface 106. Col. 1:45-49 teaches that each feature set is a subset of the plurality of variables included in the input dataset, and col. 17:23-28 teaches that the list of unique feature sets may be provided as an input. As discussed with respect to claim 1, the kernel term of Swersky para [0111] applied to the nominal feature set input of Gunes is computed between the feature set of an evaluated pipeline and the feature set of a new pipeline.)
As per claim 6, Wang in view of Swersky, Gunes and Owhadi Kareshk discloses,
wherein an initial hyperparameter corresponds to the initial pipeline, wherein a new hyperparameter corresponds to the new pipeline, wherein the pipeline sampling score calculation module uses the initial hyperparameter and the new hyperparameter to obtain the intra algorithm hyperparameter distance value. (Swersky para [0159] teaches that the kernel is evaluated between points x and x' that represent sets of hyper-parameter values, and para [0160] teaches the cross-covariances between a point x and the set of points at which the objective function was previously evaluated. The previously evaluated points are understood to be the initial hyperparameters and point x is understood to be the new hyperparameter.)
As per claim 9, Wang in view of Swersky, Gunes and Owhadi Kareshk discloses,
wherein the algorithm includes a feature selection algorithm and a model algorithm. (Wang, Figure 2 and paras [0010] and [0044], teach that the Bayesian Optimization method 250 includes model candidates 210 and feature selection 220.)
As per claim 10, Wang in view of Swersky, Gunes and Owhadi Kareshk discloses,
wherein the pipeline performance evaluation module uses a kernel-based method and the dataset to obtain the prediction result, and uses the kernel-based method and the dataset to obtain the accuracy. (Wang para [0051] teaches a support-vector machine algorithm as a model candidate. Gunes, col. 11:59 to col. 12:10, teaches a support vector machine model type that uses linear and low-degree polynomial kernels and that is trained and validated on the datasets to compute a validation error. A support vector machine is understood to be a kernel-based method.)
As per claim 11, Wang in view of Swersky, Gunes and Owhadi Kareshk discloses,
wherein the inter algorithm diversity value includes a cosine similarity and a contingency table. (Owhadi Kareshk para [0027] teaches that "any loss, similarity, or agreement evaluation function can be used for pairwise model comparisons". Para [0031] teaches comparing the respective outputs of the two models to determine the number or percentage that are the same or different, for example that 98 classifications out of one hundred are the same. A count of the paired outputs on which two models agree and disagree is a tabulation of their joint outcomes, which is understood to be a contingency table measure.)
As per claim 12, Wang in view of Swersky, Gunes and Owhadi Kareshk discloses,
wherein the inter feature correlation value includes an absolute value of a Pearson correlation coefficient, an absolute value of a Spearman correlation coefficient, a feature intersection number divided by a total feature number, a Euclidean Distance, and a Mahalanobis Distance. (Swersky para [0111] teaches embedding each category of a categorical variable in a continuous space where similar categories are closer in terms of Euclidean distance. Applied to the nominal feature set input of Gunes, col. 25:24-27, the value relating two feature sets is a Euclidean distance between the embedded feature sets.)
As per claim 13, Wang in view of Swersky, Gunes and Owhadi Kareshk discloses,
wherein the intra algorithm hyperparameter distance value includes a Radial Basis Function kernel (RBF kernel), a Laplace kernel, a Matern kernel, and a Rational Quadratic kernel. (Swersky para [0159] teaches a 5/2 Matern kernel and teaches that other kernels may be used, including an automatic relevance determination squared exponential kernel, which is understood to be a radial basis function kernel, and a rational quadratic kernel.)
As per claim 14, Wang in view of Swersky, Gunes and Owhadi Kareshk discloses,
wherein the pipeline recommendation module uses a preset execution time and the sampling score to determine the recommended new pipeline in the new pipeline. (Wang, Figure 2, teaches a user-supplied run time, and paras [0034] and [0062] teach that the hyperparameters are tuned to achieve a maximum performance within a certain amount of time. Gunes, col. 10:8-13, teaches optimization parameters that indicate when training is complete, such as a maximum number of configuration evaluations or a maximum time. Swersky paras [0143] and [0177] teach that points are selected with the acquisition utility function and evaluated iteratively.)
As per claim 15, Wang in view of Swersky, Gunes and Owhadi Kareshk discloses,
wherein the ensemble model recommendation module uses a preset number and an ensemble learning method to select the target recommended new pipeline from the recommended new pipelines. (Owhadi Kareshk para [0046] and claim 8 teach selecting the subset such that the number of ML models in the subset does not exceed a first number.)
Claims 16-19 recite substantially the same limitations as those recited in claims 1, 3, 5 and 6, respectively, applied to the method of claim 16. Thus, for the same reasons presented with respect to claims 1, 3, 5 and 6, claims 16-19 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Swersky, Gunes and Owhadi Kareshk.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to SAMUEL NWUHA whose telephone number is (571)272-9367. The examiner can
normally be reached Monday-Friday; 7:30 am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a
USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use
the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor,
Kevin Young can be reached at (571) 270-3180. The fax phone number for the organization where this
application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from
Patent Center. Unpublished application information in Patent Center is available to registered users. To
file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit
https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and
https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional
questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like
assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or
571-272-1000.
/SAMUEL OBINNA NNAJI NWUHA/
Examiner, Art Unit 2194
/KEVIN L YOUNG/Supervisory Patent Examiner, Art Unit 2194