Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Detailed Action
The following action is in response to the communication(s) received on 11/12/2025.
As of the claims filed 11/12/2025:
Claims 1, 7, 8, 14, 15, and 21 have been amended.
Claims 2, 9, and 16 have been canceled.
Claims 1, 3, 5, 7, 8, 10, 12, 14, 16, 17, 19, and 21 are now pending.
Claims 1, 8, and 15 are independent claims.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 11/12/2025 has been entered.
Response to Arguments
Applicant’s arguments filed 11/12/2025 have been fully considered, but are not fully persuasive.
The amended limitations have been considered for eligibility and are persuasive. Thus, the 35 USC § 101 rejections have been withdrawn.
Applicant arguments regarding the amended limitations have been considered for novelty and non-obviousness, but are unpersuasive.
Applicant asserts that the prior art does not teach extracting… a second set of features related to a new optimization problem as Pise “simply discloses that it is possible to obtain knowledge about performance of different algorithms of a given dataset” (p.26). Applicant further asserts that the limitation extracting… a second set of features related to a new optimization problem corresponds to requiring the machine learning model to be trained on the first set of features before applying the second set of features. (p.27 ¶1) While the previously applied manner does not teach this limitation, in view of the additional evidence from the art, Pise still remains teaching the limitation (Pise [p.208 2nd col last ¶] … Then based on the problem of classification, or the new problem at hand, mapping is done between the datasets and the benchmark performance of different classifiers. K-similar datasets are returned.) (Note: the new problem at hand corresponds to a second set of features; mapping for the new problem corresponds to a new optimization problem)
Applicant further asserts that the prior art does not teach iterating… through successively-ranked optimization algorithms until one or more conditions are satisfied (p.28 last ¶), and that Pise does not teach actually executing the algorithms. Examiner respectfully submits that the execution of the algorithms at the time of mapping is not explicitly recited in the claims and cannot be read into the broadest reasonable interpretation. Thus, Pise remains teaching this limitation (Pise [p.208 2nd col last ¶] … based on the problem of classification, or the new problem at hand, mapping is done between the datasets and the benchmark performance of different classifiers. K-similar datasets are returned. Then ranking of classification algorithms is performed and based on the highest rank, the classification algorithm is recommended for the problem at hand.) (Note: All algorithms being ranked based on their benchmark performance, thus fully trained and evaluated, corresponds to iterating through the optimization algorithms; selecting the highest rank corresponds to satisfying a condition)
Applicant further asserts that the dependent claims are allowable by virtue of dependency to their arguments regarding the claims above. Examiner respectfully submits that the dependent claims remain rejected at least by virtue of dependency to their respective parent claims for the reasons given above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3, 5, 7-10, 12, 14, 15-17, 19, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Rijn et al., “Algorithm Selection on Data Streams” (hereinafter Rijn), in view of Pise et al., “Algorithm selection for classification problems” (hereinafter Pise).
Regarding Claim 1, Rijn teaches:
A computer-implemented method (Rijn [p.330 2nd ¶] For all classifiers, we have recorded the predictive accuracy, the runtime, and RAM Hours on each data stream.) (Note: the recorded runtime and RAM hours correspond to the performance on a computer that comprises a processor and memory.)
comprising: extracting, by a processor, a first set of features from a plurality of optimization problems; (Rijn [p.327 3rd ¶] The field of meta-learning addresses the question what machine learning algorithms work well on what data. The algorithm selection problem, formalised by Rice in [14], is a natural problem from the field of meta-learning. According to the definition of Rice, the problem space P consists of all machine learning tasks from a certain domain, the feature space F contains measurable characteristics calculated upon this data (called meta-features), the algorithm space A is the set of all considered algorithms that can execute these tasks and the performance space Y represents the mapping of these algorithms to a set of performance measures. The task is for any given x ∈ P, to select the algorithm α ∈ A that maximizes a predefined performance measure y ∈ Y, which is a classification problem. Similar ranking and regression problems are derived from this.
Much effort has been devoted to the development of meta-features that effectively describe the characteristics of the data (called meta-features). Commonly used meta-features are typically categorised as one of the following: simple (number of instances, number of attributes, number of classes), statistical (mean standard deviation of attributes, mean kurtosis of attributes, mean skewness of attributes), information theoretic (class entropy, mean entropy of attributes, noise-signal ratio) … (Note: problem space P corresponds to the plurality of optimization problems; the statistical and information theoretic meta-features corresponds to the first set of features.)
receiving, by the processor, respective characteristics of a plurality of optimization algorithms, the characteristics of each algorithm based on application of the optimization algorithm applied to each optimization problem of the plurality of optimization problems; (Rijn [p.327 4th ¶] Much effort has been devoted to the development of meta-features that effectively describe the characteristics of the data (called meta-features). Commonly used meta-features are typically categorised as one of the following: simple (number of instances, number of attributes, number of classes), statistical (mean standard deviation of attributes, mean kurtosis of attributes, mean skewness of attributes), information theoretic (class entropy, mean entropy of attributes, noise-signal ratio) or landmarkers [12] (performance of a simple classifier on the data). The authors of [18] give an extensive description of many meta-features. Furthermore, they propose a new type of meta-feature, pair-wise meta-rules.) (Note: landmarkers correspond to the characteristics of each algorithm; each classifier corresponds to each optimization algorithm)
training, by the processor, a plurality of machine learning models on a first portion of a dataset, the dataset comprising the first set of features and the respective characteristics; (Rijn [p.332 2nd ¶] In this experiment we want to determine whether meta-knowledge can improve the predictive performance of data stream algorithms in the following setting. Consider an ensemble of algorithms that are all trained on the same data stream. For each window of size w, an abstract meta-algorithm determines which algorithm will be used to predict the next window of instances, based on data characteristics measured in the previous window and the meta-knowledge. Note that the performance of the meta-algorithm depends on the size of this window. Meta-features calculated over a small window size are probably not able to adequately represent the characteristics of the data, whereas calculating meta-features over large windows is computationally expensive. Since our previous experiment obtained good results with a window size of 1,000, we perform our experiments with the same window size.)
selecting a trained machine learning model based on a second portion of the dataset; (Rijn [p.331 1st ¶] The goal is to predict which algorithm performs best, measured over the whole data stream. In order to obtain deeper insight into what kind of targets we can predict, we also defined three sub tasks, i.e., predicting the best instance incremental classifier (A = 5), predicting the best batch incremental classifier (A = 4) and predicting the best ensemble (A = 4). We have selected the “Decision Stump” and “Random Forest” classifiers (as implemented in Weka 3.7.11 [9]) as meta-algorithms. The Random Forest algorithm has proven to be a useful meta-algorithm in prior work [18], while models obtained from a single decision tree or stump are especially easy to interpret. Both classifiers are tree-based, which guards them against modeling irrelevant features. We ran the Random Forest algorithm with 100 trees and 10 attributes. We estimate the performance of the meta-algorithm by doing 10 times 10-fold cross-validation, and compare its performance against predicting the majority class. For each meta-dataset, we have filtered out the instances that contain a unique class value; since they will either only appear in the training or test set, these do not form a reliable source for estimating the accuracy.) (Note: measuring the performance by doing cross-validation corresponds to evaluating the machine learning model on a second portion of the dataset (the test sets used during cross-validation))
Rijn does not teach, but Pise further teaches:
extracting, by the processor, a second set of features related to a new optimization problem; (Pise [p.208 2nd col last ¶] In this paper, algorithm selection is proposed for classification problems in data mining. The characteristics of datasets and the performance of classification algorithms are found out. Then based on the problem of classification, or the new problem at hand, mapping is done between the datasets and the benchmark performance of different classifiers. K-similar datasets are returned. (Note: the new problem at hand corresponds to a second set of features; mapping for the new problem corresponds to a new optimization problem)
obtaining, by the processor, predicted performance characteristics for each optimization algorithm based on application of the selected trained machine learning model on the second set of features. (Pise [p.208 2nd col last ¶] In this paper, algorithm selection is proposed for classification problems in data mining. The characteristics of datasets and the performance of classification algorithms are found out. Then based on the problem of classification, or the new problem at hand, mapping is done between the datasets and the benchmark performance of different classifiers. K-similar datasets are returned. Then ranking of classification algorithms is performed and based on the highest rank, the classification algorithm is recommended for the problem at hand. Hence the user doesn’t need to waste time for working on different data mining algorithms, fine tuning the parameters for different algorithms. The algorithm is directly recommended for his problem.) (Note: the ranking of classification algorithms for the new problem corresponds to obtaining predicted performance characteristics of each optimization algorithm)
ranking, by the processor, each optimization algorithm according to the predicted performance characteristics. (Pise [p.208 2nd col last ¶] Then based on the problem of classification, or the new problem at hand, mapping is done between the datasets and the benchmark performance of different classifiers. K-similar datasets are returned. Then ranking of classification algorithms is performed and based on the highest rank, the classification algorithm is recommended for the problem at hand.
PNG
media_image1.png
459
573
media_image1.png
Greyscale
) (Note: the benchmark performance corresponds to the predicted performance characteristics)
and iterating, by the processor, through successively-ranked optimization algorithms until one or more conditions are satisfied, (Pise [p.208 2nd col last ¶] In this paper, algorithm selection is proposed for classification problems in data mining. The characteristics of datasets and the performance of classification algorithms are found out. Then based on the problem of classification, or the new problem at hand, mapping is done between the datasets and the benchmark performance of different classifiers. K-similar datasets are returned. Then ranking of classification algorithms is performed and based on the highest rank, the classification algorithm is recommended for the problem at hand.) (Note: Ranking of classification algorithms and determining the highest rank corresponds to satisfying; all algorithms being ranked to find the highest ranked algorithm corresponds to satisfying the condition)
wherein the predicted performance characteristics comprise a CPU run-time for executing the optimization algorithm and a performance metric of the successively-ranked optimization algorithms (Pise [p.204 2nd col 3rd ¶] Further knowledgebase represents knowledge about the performance of many different algorithms on specific dataset. This knowledge may involve training time, test time, error rate and some more parameters. But only classifier accuracy is used in the proposed approach. [p.208 2nd col last ¶] Then based on the problem of classification, or the new problem at hand, mapping is done between the datasets and the benchmark performance of different classifiers. K-similar datasets are returned. Then ranking of classification algorithms is performed and based on the highest rank, the classification algorithm is recommended for the problem at hand) (Note: the further knowledgebase corresponds to the performance characteristics; the performance of each algorithm corresponds to executing each optimization algorithm; since training time requires a CPU for execution, training time corresponds to a CPU run-time.)
Pise and Rijn are analogous to the present invention because both are from the same field of endeavor of meta-learning regarding selecting the algorithm to use on a new dataset. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the algorithm ranking method from Pise into Rijn’s algorithm selection method. The motivation would be to “to choose the selection mapping S returning the best algorithm α from algorithm space A” (Pise p.204 1st col 1st ¶).
Regarding Claim 3, Rijn/Pise respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Rijn further teaches:
The computer-implemented method of claim 1, wherein: each of the first set of features and the second set of features is based on tabular data and graph structures generated from the tabular data. (Rijn [p.328 2nd ¶] In order to obtain a reasonable number of experiments on data streams, we propose a new type of data generator that generates data streams based on real world data [16]. It takes a dataset as input, preferably consisting of real world data and a reasonable number of features, and builds a Bayesian Network over it, which is then used to generate instances based on the probability tables. These streams can also be combined together to simulate concept drift, similar to what is commonly done with the Covertype, Pokerhand and Electricity dataset [4].)
Regarding Claim 5, Rijn/Pise respectively teaches and incorporates the claimed limitations and rejections of Claim 4. Rijn does not teach, but Pise further teaches:
The computer-implemented method of claim 4, further comprising: executing, by the processor, a first-ranked optimization algorithm on the new optimization problem. (Pise [p.209 1st col 1st ¶] The proposed algorithm selection approach recommends approximate best classifier using classification accuracy measure. It helps non-experts in deciding algorithm. The experimentation shows predicted accuracies are matching with the actual accuracies for more than 90 % of the benchmark datasets used for experimentation.) (Note: comparing the predicted accuracies with the actual accuracies corresponds to executing the first-ranked optimization algorithm on the new optimization problem (the actual dataset).)
Pise and Rijn are analogous to the present invention because both are from the same field of endeavor of meta-learning regarding selecting the algorithm to use on a new dataset. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the method of executing the selected algorithm into the new dataset from Pise into Rijn’s algorithm selection method. The motivation would be to evaluate the “actual and predicted values of accuracy for features: number of attributes, number of instances, number of class labels.” (Pise p.206 2nd col 2nd ¶).
Regarding Claim 7, Rijn/Pise respectively teaches and incorporates the claimed limitations and rejections of Claim 6. Pise further teaches:
The computer-implemented method of claim 6, wherein the one or more conditions are: an actual run-time of the CPU run-time and an actual performance metric of the performance metric that is acceptable; or attain a run-time limit of the CPU run-time; or expectation of no further improvement on the CPU run-time and the performance metric of the successively-ranked optimization algorithms. (Pise [p.208 2nd col last ¶] In this paper, algorithm selection is proposed for classification problems in data mining. The characteristics of datasets and the performance of classification algorithms are found out. Then based on the problem of classification, or the new problem at hand, mapping is done between the datasets and the benchmark performance of different classifiers. K-similar datasets are returned. Then ranking of classification algorithms is performed and based on the highest rank, the classification algorithm is recommended for the problem at hand.) (Note: All algorithms being ranked, thus fully trained and evaluated, corresponds to no further improvements done on the CPU run-time and the performance metrics of the successively-ranked optimization algorithms.)
Independent Claim 8 recites A system comprising: a processor; and a memory storing instructions that, when executed by the processor, configure the system (Rijn [p.330 2nd ¶] For all classifiers, we have recorded the predictive accuracy, the runtime, and RAM Hours on each data stream.) to perform precisely the methods of Claim 1. Thus, Claim 8 is rejected for reasons set forth in Claim 1. (Note: the recorded runtime and RAM hours correspond to the performance on a computer that comprises a processor and memory.)
Claim(s) 10, 12, and 14, dependent on Claim 8, also recite the system configured to perform precisely the methods of Claims 3, 5, and 7, respectively, and thus are rejected for reasons set forth in these claims.
Independent Claim 15 recites A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to (Rijn [p.330 2nd ¶] For all classifiers, we have recorded the predictive accuracy, the runtime, and RAM Hours on each data stream.) perform precisely the methods of Claim 1. Thus, Claim 15 is rejected for reasons set forth in Claim 1. (Note: the recorded runtime and RAM hours correspond to the performance on a computer that comprises a processor and memory.)
Claim(s) 17, 19, and 21 , dependent on Claim 15, also recite the system configured to perform precisely the methods of Claims 3, 5, and 7, respectively, and thus are rejected for reasons set forth in these claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSEP HAN whose telephone number is (703)756-1346. The examiner can normally be reached Mon-Fri 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached on (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.H./Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122