Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1-2, 11, 16, and 19 were amended. Claims 1 – 20 are pending and examined herein.
Claims 11 – 15 are rejected under 35 U.S.C. 112(b).
Claims 1 – 20 are rejected under 35 U.S.C. 101.
Claims 1 – 20 are rejected under 35 U.S.C. 103.
Response to Amendment
The amendment filed March 12th, 2026 has been entered. Claims 1-2, 11, 16, and 19 were amended. Claims 1-20 are pending and are examined herein. Applicant’s amendments to the claims have overcome each and every objection previously set forth in the Non-Final Rejection Office Action mailed December 12th, 2025. However, new 112(b) issue has been introduced to the amended claim.
Response to Arguments
Applicant's arguments filed March 12th, 2026 regarding the 35 U.S.C. § 101 rejection for being directed to an abstract idea without significantly more have been fully considered but they are not persuasive.
Applicant argues that the amended claims are directed to an improvement in machine learning field because the claims now recite a time series meta learner and a general meta learner. However, the amended limitations do not change the character of the claims as a whole. The claims still recite computing time series meta features, generating predicted performance information or a performance score, comparing predicted performance to ground truth performance, updating model parameters, and selecting a forecasting model based on the generated score or predicted performance. These limitations are directed to mathematical prediction, scoring, comparison, optimization, and model selection.
The recited time series meta learner and general meta learner identify the machine learning components used to perform the claimed prediction and scoring, but the claims do not recite a specific improvement to computer functionality or to the internal operations of a neural network. Rather, the claims functionality state that one learner predicts performance based on relationships across different time windows and the other predicts performance based on a single time window. Any alleged improvement in model selection accuracy results from the mathematical evaluation of time series and model performance.
Accordingly, the amended claims do not integrate the judicial exception into a practical application and do not add significantly more than the judicial exception. The rejection under 35 U.S.C. § 101 is maintained.
Applicant’s arguments, see pages 12-14, with respect to the rejection(s) of claim(s) 1-20 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Agrawal et al. (U.S. Pub. 2019/0095756 A1).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 11 - 15 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 11 recites “a plurality of time series datasets,” but later recites “different windows of time of the time-series dataset” and “a single time window of the time-series dataset.” It is unclear which time-series dataset is intended by “the time-series dataset,” because claim 11 previously recites a plurality of time-series datasets. There is insufficient antecedent basis for this limitation in the claim. For examination purposes, they will refer to “one of the plurality of time-series dataset” or “each of the plurality of time-series datasets.”
Claims 12 – 15 are dependent on claim 11. They do not resolve the issue of indefiniteness and are rejected with the same rationale.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1 - 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
MPEP § 2109(III) sets out steps for evaluating whether a claim is drawn to patent-eligible subject matter. The analysis of claims 1 – 20, in accordance with these steps, follows.
Step 1 Analysis:
Step 1 is to determine whether the claim is directed to a statutory category (process, machine, manufacture, or composition of matter.
Claims 1 – 10 are directed to a method, meaning that it is directed to the statutory category of process. Claims 11 – 15 are directed to a method, which is also the statutory category of process. Claims 16 – 20 are directed to an apparatus for data processing, which can be an article of machine.
Step 2A Prong One, Step 2A Prong Two, and Step 2B Analysis:
Step 2A Prong One asks if the claim recites a judicial exception (abstract idea, law of nature, or natural phenomenon). If the claim recites a judicial exception, analysis proceeds to Step 2A Prong Two, which asks if the claim recites additional elements that integrate the abstract idea into a practical application. If the claim does not integrate the judicial exception, analysis proceeds to Step 2B, which asks if the claim amounts to significantly more than the judicial exception. If the claim does not amount to significantly more than the judicial exception, the claim is not eligible subject matter under 35 U.S.C. 101.
Regarding claim 1, the following claim elements are abstract ideas:
computing a time-series meta-feature vector based on the time-series dataset; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components. In addition, computing feature vector based on the timeseries dataset could fall under a mathematical concept.)
generating a performance score for a forecasting model (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
wherein the time-series meta-learner is trained to predict model performance based on relationships across different windows of time of the time-series dataset, (Predicting performance based on relationships could recite mathematical relationship, which is mathematical concept.)
and wherein the general meta-learner is trained to predict model performance based on a single time window of the time-series dataset; (Predicting performance based on datasets could recite mathematical relationship, which is mathematical concept.)
selecting the forecasting model from a plurality of forecasting models based on the performance score; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
The following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
A method for data processing, comprising: receiving a time-series dataset; (This is mere data gathering, an insignificant extra solution activity, which is a well-understood, routine conventional activity. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(d). Therefore, this does not amount to significantly more than the judicial exception.)
using a meta-learner machine learning model that takes the time-series meta-feature vector as input; (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
wherein the meta-learner machine learning model comprises an artificial neural network (ANN) including a time-series meta-learner and a general meta-learner separate from the time-series meta-learner, (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
and generating predicted time-series data based on the time-series dataset using the selected forecasting model. (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 2, the rejection of claim 1 is incorporated herein. Further, claim 2 recites the following abstract ideas:
dividing the time-series dataset into a plurality of time windows of the time-series dataset; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
and identifying a time window of the plurality of time windows of the time-series dataset. (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
Claim 2 further recites following additional element
wherein the forecasting model is selected based on the identified time window. (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 3, the rejection of claim 1 is incorporated herein. Further, claim 3 recites the following abstract ideas:
computing a plurality of meta-features based on the time-series dataset; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
Claim 3 further recites following additional element
and generating the time-series meta-feature vector based on the plurality of meta-features. (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 4, the rejection of claim 3 is incorporated herein. Further, claim 4 recites the following additional element:
the plurality of meta-features include an aggregate statistic of the time-series dataset. (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 5, the rejection of claim 3 is incorporated herein. Further, claim 5 recites the following abstract idea:
performing a principal component analysis on the plurality of meta-features to obtain the time-series meta-feature vector. (Performing a principal component analysis to obtain feature vector recites a mathematical calculation, which is mathematical concept.)
Claim 5 does not recite additional elements.
Regarding claim 6, the rejection of claim 1 is incorporated herein. Further, claim 6 recites the following abstract ideas:
generating first predicted performance data for each of the plurality of forecasting models (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
wherein the forecasting model is selected based on the first predicted performance data. (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
Claim 6 further recites following additional element
using a time-series meta-learner of the meta-learner machine learning model, (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 7, the rejection of claim 6 is incorporated herein. Further, claim 7 recites the following abstract ideas:
generating second predicted performance data for each of the plurality of forecasting models (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
wherein the forecasting model is selected based on the second predicted performance data. (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
Claim 7 further recites following additional element
using a general meta-learner of the meta-learner machine learning model, (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 8, the rejection of claim 7 is incorporated herein. Further, claim 8 recites the following additional element:
providing the second predicted performance data as an input to the time-series meta-learner. (This is mere transmitting data, which is a well-understood, routine conventional activity. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(d). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 9, the rejection of claim 1 is incorporated herein. Further, claim 9 recites the following abstract ideas:
identifying a plurality of hyperparameters for each of the plurality of forecasting models; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
and selecting a hyperparameter from the plurality of hyperparameters (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
Claim 9 further recites following additional element
using the meta-learner machine learning model, wherein the predicted time-series data is based on the selected hyperparameter. (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 10, the rejection of claim 1 is incorporated herein. Further, claim 10 recites the following additional element:
receiving a time-series training set; (This is mere data gathering, an insignificant extra solution activity, which is a well-understood, routine conventional activity. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(d). Therefore, this does not amount to significantly more than the judicial exception.)
and training the selected forecasting model based on the time-series training set, wherein the predicted time-series data is generated based on the training. (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 11, the following claim elements are abstract ideas:
identifying a training set comprising a plurality of time-series datasets, a plurality of forecasting models, and ground-truth performance data for the plurality of forecasting models applied to each of the plurality of time-series datasets; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
generating predicted performance data for the plurality of forecasting models applied to each of the plurality of time-series datasets (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
comparing the predicted performance data to the ground-truth performance data; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
to train the time-series meta-learner to predict model performance based on relationships across different windows of time of the time-series dataset and to train the general meta-learner to predict model performance based on a single time window of the time-series dataset. (Predicting performance based on datasets could recite mathematical relationship, which is mathematical concept.)
Claim 11 further recites following additional element
using a meta-learner machine learning model, wherein the meta-learner machine learning model comprises an artificial neural network (ANN) including a time-series meta-learner and a general meta-learner separate from the time-series meta-learner; (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
and updating parameters of the meta-learner machine learning model based on the comparison. (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 12, the rejection of claim 11 is incorporated herein. Further, claim 12 recites the following abstract ideas:
computing a loss function based on the predicted performance data and the ground-truth performance data, (Computing a loss function based on the predicted data and the ground-truth data recites a mathematical calculation, which is mathematical concept.)
Claim 12 further recites following additional element
wherein the parameters of the meta-learner machine learning model are based on the loss function. (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 13, the rejection of claim 12 is incorporated herein. Further, claim 13 recites the following abstract ideas:
computing a time-series loss term based on an output of a time-series meta-learner; (Computing a timeseries loss term based on an output recites a mathematical calculation, which is mathematical concept.)
and computing a general loss term based on an output of a general meta-learner, (Computing a general loss term based on an output recites a mathematical calculation, which is mathematical concept.)
Claim 13 further recites following additional element
wherein the loss function comprises the time-series loss term and the general loss term. (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 14, the rejection of claim 11 is incorporated herein. Further, claim 14 recites the following additional element:
applying each of the plurality of forecasting models to each of the plurality of time-series datasets to obtain the ground-truth performance data. (This is mere data gathering and outputting, an insignificant extra solution activity, which does not integrate the judicial exception into a practical application. The broadest reasonable interpretation of this claim is storing information in memory, which is a well-understood, routine conventional activity. See MPEP § 2106.05(d)(II)(iv). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 15, the rejection of claim 14 is incorporated herein. Further, claim 15 recites the following additional element:
training a forecasting model of the plurality of forecasting models on each of the plurality of time-series datasets to obtain a trained forecasting model, wherein the ground-truth performance data is based on the trained forecasting model. (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 16, the following claim elements are abstract ideas:
compute a plurality of meta-features based on a time-series dataset; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
select a forecasting model from a plurality of forecasting models based on the time-series dataset. (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
wherein the time-series meta-learner is trained to predict model performance based on relationships across different windows of time of the time-series dataset, and wherein the general meta-learner is trained to predict model performance based on a single time window of the time-series dataset. (Predicting performance based on datasets could recite mathematical relationship, which is mathematical concept.)
Claim 16 further recites following additional element
An apparatus for data processing, comprising: a processor; a memory including instructions executable by the processor; (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
a meta-feature extraction component configured to (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
and a meta-learner machine learning model configured to (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
wherein the meta-learner machine learning model comprises an artificial neural network (ANN) including a time-series meta-learner and a general meta-learner separate from the time-series meta-learner, (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 17, the rejection of claim 16 is incorporated herein. Further, claim 17 recites the following additional element:
a training component configured to update parameters of the meta-learner machine learning model based on a loss function. (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 18, the rejection of claim 16 is incorporated herein. Further, claim 18 recites the following additional element:
the meta-learner machine learning model comprises a general meta-learner and a time-series meta-learner. (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 19, the rejection of claim 18 is incorporated herein. Further, claim 19 recites the following additional element:
the time-series meta-learner comprises an long short-term memory (LSTM) model. (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 20, the rejection of claim 16 is incorporated herein. Further, claim 20 recites the following additional element:
a feature-embedding component configured to reduce a dimensionality of the plurality of meta-features. (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 3 – 6, 10 – 12, 14 – 15 are rejected under 35 U.S.C. 103 as being unpatentable over Talagala et al. (NPL: ”Meta-learning how to forecast time series”) in view of Talagala et al. (NPL: ”FFORMPP: Feature-based forecast model performance prediction”), further in view of Agrawal et al. (U.S. Pub. 2019/0095756 A1).
Regarding claim 1, Talagala (meta learn) teaches
A method for data processing, comprising: receiving a time-series dataset; (Pg. 9 3 Methodology section of Talagala (meta learn) states “In order to train our classification algorithm, we need a large collection of time series which are similar to those we will be forecasting. We assume that we have an essentially infinite population of time series, and we take a sample of them in order to train the classification algorithm denoted as the “observed sample”. The new time series we wish to forecast can be thought of as additional draws from the same population.”)
computing a time-series meta-feature vector based on the time-series dataset; (Pg. 10 3 Methodology section of Talagala (meta learn) states “From each training period we compute a range of time series features, and fit a selection of candidate models. The calculated features form the input vector to the classification algorithm.”)
selecting the forecasting model from a plurality of forecasting models based on the performance score; (Pg. 10 3 Methodology of Talagala (meta learn) states “From each training period we compute a range of time series features, and fit a selection of candidate models. The calculated features form the input vector to the classification algorithm. Using the fitted models, we generate forecasts and identify the “best” model for each time series based on a forecast error measure (e.g., MASE) calculated over the test period.”)
and generating predicted time-series data based on the time-series dataset using the selected forecasting model. (Pg. 10 3 Methodology of Talagala (meta learn) states “Using the fitted models, we generate forecasts and identify the “best” model for each time series based on a forecast error measure (e.g., MASE) calculated over the test period.” Generated forecast are predicted time series data using the selected model.”)
Talagala (meta learn) does not explicitly teach that
generating a performance score for a forecasting model using a meta-learner machine learning model that takes the time-series meta-feature vector as input;
wherein the meta-learner machine learning model comprises an artificial neural network (ANN)
including a time-series meta-learner and a general meta-learner separate from the time-series meta-learner,
wherein the time-series meta-learner is trained to predict model performance based on relationships across different windows of time of the time-series dataset,
and wherein the general meta-learner is trained to predict model performance based on a single time window of the time-series dataset;
However, Talagala (FFORMPP) teaches that
generating a performance score for a forecasting model using a meta-learner machine learning model that takes the time-series meta-feature vector as input; (Pg. 7 2 Methodology section of Talagala (FFORMPP) states “We treat the forecast model selection problem as a multi-class ranking problem, where predictors are features, and the outcome consists of forecast errors (MASE) calculated over all available models. In particular, an instance i is a tuple (f1i, f2i, · · · , fmi, e1i, e2i, · · · , eni), where predictors (f1i, f2i , · · · , fmi) is a vector of m features which is the input to our algorithm, and the outcome (e1i , e2i, · · · , eni), is a vector of MASE values of the all n forecast models.”)
Agrawal teaches that
wherein the meta-learner machine learning model comprises an artificial neural network (ANN) ([0073] of Agrawal states “Each of meta-models 151-153 is itself an instance of trainable regression algorithm, although not the same algorithm for which the meta-models are trained for. For example, meta-models 151-153 may each be a distinct neural network that is already trained to predict the performance of algorithm 121, which may be support vector machine instead of a neural network. Training of meta-models is discussed later herein.” [0093] of Agrawal states “For example, meta-models 151-153 may each be an already trained neural network that takes a subset of hyperparameter values and a subset of meta-feature values as stimulus inputs, shown as dashed arrows entering meta-models 151-153. Training of meta-models is discussed later herein.” [0065] of Agrawal states “In an embodiment, the trained regressors are distinctly configured artificial neural networks. In an embodiment, the trained regressors are contained within algorithm-specific ensembles. Techniques are also provided herein for optimal training of regressors and/or ensembles.”)
including a time-series meta-learner and a general meta-learner separate from the time-series meta-learner, ([0003] of Agrawal states “There are hundreds of machine learning algorithms. Training and testing each one to find the best performing might not be feasible. Automatic approaches to selective training typically ultimately use a single regressor/classifier for predicting algorithm performance, which causes different algorithms to interfere with each other in the selection model, thereby lowering accuracy. These approaches also do not consider algorithm hyperparameters, which can significantly affect algorithm performance and behavior.” [0072] of Agrawal states “Computer 100 creates or obtains meta-models for each of algorithms 121-123 to quickly and accurately predict the performance of each algorithm. For example, computer 100 may create meta-models 151-153 as performance predictors of algorithm 121.” [0073] of Agrawal states “Each of meta-models 151-153 is itself an instance of trainable regression algorithm, although not the same algorithm for which the meta-models are trained for.” [0080] of Agrawal states “For each actual configuration alternative or set of related configuration alternatives, computer 100 has a separate meta-model, such as 151-153.” [0065] of Agrawal states “In an embodiment, the trained regressors are distinctly configured artificial neural networks. In an embodiment, the trained regressors are contained within algorithm-specific ensembles. Techniques are also provided herein for optimal training of regressors and/or ensembles.” Agrawal teaches using separate neural network meta models for performance prediction rather than a single shared regressor/classifier. It would have been obvious to modify Talagala’s time series forecasting meta learning framework to include separate ANN meta learner components because Agrawal teaches that separate performance prediction meta models reduce interference and improve prediction accuracy.)
wherein the time-series meta-learner is trained to predict model performance based on relationships across different windows of time of the time-series dataset, (Pg. 1 Abstract section of Talagala (FFORMPP) states “We model the forecast error as a function of time series features calculated from the historical time series with an efficient Bayesian multivariate surface regression approach.” [0081] of Agrawal states “Each of meta-models 151-153 was trained to predict how a particular configuration (or set of related configurations) of algorithm 121 will perform for a variety of datasets that are similar or dissimilar to inference dataset 110. Related configurations are those that have identical or similar values for a subset of hyperparameters 181-184.” Talagala teaches predicting forecast model error from historical time series features. Agrawal teaches that using a single regressor/classifier for performance prediction can cause interference and lower accuracy, and teaches using separate meta-models. It would have been obvious to implement this temporal performance prediction aspect as the claimed separate time series meta learner.)
and wherein the general meta-learner is trained to predict model performance based on a single time window of the time-series dataset; (Pg. 10 3 Methodology of Talagala (meta learn) states “From each training period we compute a range of time series features, and fit a selection of candidate models. The calculated features form the input vector to the classification algorithm. Using the fitted models, we generate forecasts and identify the “best” model for each time series based on a forecast error measure (e.g., MASE) calculated over the test period.” [0064] of Agrawal states “In an embodiment, a computer derives meta-feature values from an inference dataset by, for each meta-feature, deriving a respective meta-feature value from the inference dataset. For each trainable algorithm and each regression meta-model that is respectively associated with the algorithm, a respective score is calculated by invoking the meta-model based on at least one of: a) a respective subset of meta-feature values, and/or b) hyperparameter values of a respective subset of hyperparameters of the algorithm. One or more of the algorithms are selected based on the respective scores. Based on the inference dataset, the one or more algorithms may be invoked to obtain a result.” [0003] of Agrawal states “Automatic approaches to selective training typically ultimately use a single regressor/classifier for predicting algorithm performance, which causes different algorithms to interfere with each other in the selection model, thereby lowering accuracy.” Talagala teaches selecting a forecast model based on features calculated from a time series, and Agrawal teaches deriving meta feature values and invoking a meta model to calculate a score. It would have been obvious to implement this feature based performance prediction aspect as the claimed separate general meta learner.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings from Talagala (meta learn), Talagala (FFORMPP), and Agrawal. Talagala (meta learn) framework introduces forecast model selection with meta learning where time series features are computed and used to evaluate performance metric with forecast model selection. Talagala (FFORMPP) builds on similar framework and teaches using time series features as inputs to a meta learning model that predicts forecast loss for a pool of forecasting models. Agrawal teaches trained meta model regressors, including distinctly configured artificial neural networks, for predicting algorithm performance, and further teaches that separate meta models avoid interference caused by a single shared regressor/classifier. One with ordinary skill in the art would be motivated to incorporate the teachings of Talagala (meta learn), Talagala (FFORMPP), and Agrawal to separately model different aspects of forecasting model performance prediction and improve prediction accuracy. It would have been a predictable use of known ANN meta models for known purpose of predicting model performance scores.
Regarding claim 3, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Talagala (meta learn), Agrawal, and Talagala (FFORMPP) teaches
computing a plurality of meta-features based on the time-series dataset; and generating the time-series meta-feature vector based on the plurality of meta-features. (Pg. 4 2.1 Time series features section of Talagala (meta learn) states “Rather than work with the time series directly at the level of individual observations, we propose analyzing time series via an associated “feature space”. A time series feature is any measurable characteristic of a time series.” Pg. 10 3 Methodology section of Talagala (meta learn) states “From each training period we compute a range of time series features, and fit a selection of candidate models. The calculated features form the input vector to the classification algorithm.”)
Regarding claim 4, the rejection of claim 3 is incorporated herein. Furthermore, the combination of Talagala (meta learn), Agrawal, and Talagala (FFORMPP) teaches
the plurality of meta-features include an aggregate statistic of the time-series dataset. (Pg. 7 2.4 Forecast-model selection using meta-learning section of Talagala (meta learn) states “Shah (1997) used the following features to classify time series: the number of observations, the ratio of the number of turning points to the length of the series, the ratio of number of step changes, skewness, kurtosis, the coefficient of variation, autocorrelations at lags 1–4, and partial autocorrelations at lag 2–4.”)
Regarding claim 5, the rejection of claim 3 is incorporated herein. Furthermore, the combination of Talagala (meta learn) and Talagala (FFORMPP) teaches
performing a principal component analysis on the plurality of meta-features to obtain the time-series meta-feature vector. (Pg. 6 2.2 What makes features useful for forecast-model selection? Section of Talagala (meta learn) states “Most recently Kang, Hyndman & Smith-Miles (2017) applied principal component analysis to project a large collection of time series into a two dimensional feature space in order to visualize what makes a particular forecasting method perform well or not. The features they considered were spectral entropy, first-order auto-correlation coefficient, strength of trend, strength of seasonality, seasonal period and the optimal Box-Cox transformation parameter.”)
Regarding claim 6, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Talagala (meta learn), Agrawal, and Talagala (FFORMPP) teaches
generating first predicted performance data for each of the plurality of forecasting models using a time-series meta-learner of the meta-learner machine learning model, wherein the forecasting model is selected based on the first predicted performance data. (Pg. 7 2 Methodology of Talagala (FFORMPP) states “Our FFORMPP framework (Figure 2) consists of two phases: the offline phase and the online phase. We treat the forecast model selection problem as a multi-class ranking problem, where predictors are features, and the outcome consists of forecast errors (MASE) calculated over all available models. In particular, an instance i is a tuple (f1i, f2i, · · · , fmi, e1i, e2i, · · · , eni), where predictors (f1i, f2i, · · · , fmi) is a vector of m features which is the input to our algorithm, and the outcome (e1i, e2i, · · · , eni), is a vector of MASE values of the all n forecast models” For each time series, meta learner outputs a vector of MASE values one per forecasting models. These are predicted performance data for each of the plurality of forecasting models, produced by a timeseries meta learner.)
Regarding claim 10, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Talagala (meta learn), Agrawal, and Talagala (FFORMPP) teaches
receiving a time-series training set; and training the selected forecasting model based on the time-series training set, wherein the predicted time-series data is generated based on the training. (Pg. 11 Algorithm 1 of Talagala (meta learn) states The FFORMS framework Offline phase – train the classifier
PNG
media_image1.png
778
648
media_image1.png
Greyscale
Pg. 10 3 Methodology section of Talagala (meta learn) states “From each training period we compute a range of time series features, and fit a selection of candidate models. The calculated features form the input vector to the classification algorithm.”)
Regarding claim 11, the combination of Talagala (meta learn), Agrawal, and Talagala (FFORMPP) teaches
A method for data processing, comprising: identifying a training set comprising a plurality of time-series datasets, a plurality of forecasting models, and ground-truth performance data for the plurality of forecasting models applied to each of the plurality of time-series datasets; (Pg. 11 Algorithm 1 of Talagala (meta learn) The FFORMS framework Offline phase – train the classifier and Pg. 10 3 Methodology of Talagala (meta learn) states “From each training period we compute a range of time series features, and fit a selection of candidate models. The calculated features form the input vector to the classification algorithm.”)
generating predicted performance data for the plurality of forecasting models applied to each of the plurality of time-series datasets using a meta-learner machine learning model, wherein the meta-learner machine learning model comprises an artificial neural network (ANN) including a time-series meta-learner and a general meta-learner separate from the time-series meta-learner; (Pg. 7 2 Methodology section of Talagala (FFORMPP) states “Our FFORMPP framework (Figure 2) consists of two phases: the offline phase and the online phase. We treat the forecast model selection problem as a multi-class ranking problem, where predictors are features, and the outcome consists of forecast errors (MASE) calculated over all available models. In particular, an instance i is a tuple (f1i, f2i, · · · , fmi, e1i, e2i, · · · , eni), where predictors (f1i, f2i, · · · , fmi) is a vector of m features which is the input to our algorithm, and the outcome (e1i, e2i, · · · , eni), is a vector of MASE values of the all n forecast models” [0073] of Agrawal states “Each of meta-models 151-153 is itself an instance of trainable regression algorithm, although not the same algorithm for which the meta-models are trained for. For example, meta-models 151-153 may each be a distinct neural network that is already trained to predict the performance of algorithm 121, which may be support vector machine instead of a neural network. Training of meta-models is discussed later herein.” [0093] of Agrawal states “For example, meta-models 151-153 may each be an already trained neural network that takes a subset of hyperparameter values and a subset of meta-feature values as stimulus inputs, shown as dashed arrows entering meta-models 151-153. Training of meta-models is discussed later herein.” [0065] of Agrawal states “In an embodiment, the trained regressors are distinctly configured artificial neural networks. In an embodiment, the trained regressors are contained within algorithm-specific ensembles. Techniques are also provided herein for optimal training of regressors and/or ensembles.” [0003] of Agrawal states “There are hundreds of machine learning algorithms. Training and testing each one to find the best performing might not be feasible. Automatic approaches to selective training typically ultimately use a single regressor/classifier for predicting algorithm performance, which causes different algorithms to interfere with each other in the selection model, thereby lowering accuracy. These approaches also do not consider algorithm hyperparameters, which can significantly affect algorithm performance and behavior.” [0072] of Agrawal states “Computer 100 creates or obtains meta-models for each of algorithms 121-123 to quickly and accurately predict the performance of each algorithm. For example, computer 100 may create meta-models 151-153 as performance predictors of algorithm 121.” [0080] of Agrawal states “For each actual configuration alternative or set of related configuration alternatives, computer 100 has a separate meta-model, such as 151-153.” Agrawal teaches using separate neural network meta models for performance prediction rather than a single shared regressor/classifier. It would have been obvious to modify Talagala’s time series forecasting meta learning framework to include separate ANN meta learner components because Agrawal teaches that separate performance prediction meta models reduce interference and improve prediction accuracy.)
comparing the predicted performance data to the ground-truth performance data; and updating parameters of the meta-learner machine learning model based on the comparison to train the time-series meta-learner to predict model performance based on relationships across different windows of time of the time-series dataset and to train the general meta-learner to predict model performance based on a single time window of the time-series dataset. (Pg. 6 of Talagala (FFORMPP) states “We utilise an efficient Bayesian multivariate surface regression model to train the meta learner… The multivariate surface regression model jointly models the relationship between forecast errors and time series features for multiple forecasting models.” Pg. 4 of Talagala (FFORMPP) states “The Bayesian multivariate surface regression approach proposed by Li & Villani (2013) is used to estimate the forecast error for each model in the pool.” In supervised regression, features and ground truth errors learn parameters by comparing predictions to ground truth and updating parameters to minimize discrepancy. FFORMPP uses Bayesian multivariate regression to estimate forecast errors for each model. Pg. 1 Abstract section of Talagala (FFORMPP) states “We model the forecast error as a function of time series features calculated from the historical time series with an efficient Bayesian multivariate surface regression approach.” [0081] of Agrawal states “Each of meta-models 151-153 was trained to predict how a particular configuration (or set of related configurations) of algorithm 121 will perform for a variety of datasets that are similar or dissimilar to inference dataset 110. Related configurations are those that have identical or similar values for a subset of hyperparameters 181-184.” Talagala teaches predicting forecast model error from historical time series features. Agrawal teaches that using a single regressor/classifier for performance prediction can cause interference and lower accuracy, and teaches using separate meta-models. It would have been obvious to implement this temporal performance prediction aspect as the claimed separate time series meta learner. Pg. 10 3 Methodology of Talagala (meta learn) states “From each training period we compute a range of time series features, and fit a selection of candidate models. The calculated features form the input vector to the classification algorithm. Using the fitted models, we generate forecasts and identify the “best” model for each time series based on a forecast error measure (e.g., MASE) calculated over the test period.” [0064] of Agrawal states “In an embodiment, a computer derives meta-feature values from an inference dataset by, for each meta-feature, deriving a respective meta-feature value from the inference dataset. For each trainable algorithm and each regression meta-model that is respectively associated with the algorithm, a respective score is calculated by invoking the meta-model based on at least one of: a) a respective subset of meta-feature values, and/or b) hyperparameter values of a respective subset of hyperparameters of the algorithm. One or more of the algorithms are selected based on the respective scores. Based on the inference dataset, the one or more algorithms may be invoked to obtain a result.” [0003] of Agrawal states “Automatic approaches to selective training typically ultimately use a single regressor/classifier for predicting algorithm performance, which causes different algorithms to interfere with each other in the selection model, thereby lowering accuracy.” Talagala teaches selecting a forecast model based on features calculated from a time series, and Agrawal teaches deriving meta feature values and invoking a meta model to calculate a score. It would have been obvious to implement this feature based performance prediction aspect as the claimed separate general meta learner.)
Regarding claim 12, the rejection of claim 11 is incorporated herein. Furthermore, the combination of Talagala (meta learn), Agrawal, and Talagala (FFORMPP) teaches
computing a loss function based on the predicted performance data and the ground-truth performance data, wherein the parameters of the meta-learner machine learning model are based on the loss function. (Pg. 4 of Talagala (FFORMPP) states “We propose an algorithm to rank forecast models by simultaneously predicting the forecast errors.” Pg. 6 of Talagala (FFORMPP) states “We utilise an efficient Bayesian multivariate surface regression model to train the meta learner… The multivariate surface regression model jointly models the relationship between forecast errors and time series features for multiple forecasting models.” A Bayesian multivariate regression predicting errors from features necessarily defines a likelihood comparing predicted errors to the ground truth errors. That is like a loss function based on predicted performance data and ground truth performance data. )
Regarding claim 14, the rejection of claim 11 is incorporated herein. Furthermore, the combination of Talagala (meta learn), Agrawal, and Talagala (FFORMPP) teaches
applying each of the plurality of forecasting models to each of the plurality of time-series datasets to obtain the ground-truth performance data. (Pg. 11 Algorithm 1 and Pg. 10 3 Methodology of Talagala (meta learn) states “Using the fitted models, we generate forecasts and identify the “best” model for each time series based on a forecast error measure (e.g., MASE) calculated over the test period.” Models are fitted and forecast errors are computed. Each model is applied to each time series dataset, producing ground truth performance data.)
Regarding claim 15, the rejection of claim 14 is incorporated herein. Furthermore, the combination of Talagala (meta learn), Agrawal, and Talagala (FFORMPP) teaches
training a forecasting model of the plurality of forecasting models on each of the plurality of time-series datasets to obtain a trained forecasting model, wherein the ground-truth performance data is based on the trained forecasting model. (Pg. 11 Algorithm 1 and Pg. 10 3 Methodology of Talagala (meta learn) states “Using the fitted models, we generate forecasts and identify the “best” model for each time series based on a forecast error measure (e.g., MASE) calculated over the test period.” Talagala fit models to the training period. MASE on the test period are computed from the trained models’ forecasts (ground truth performance data based on trained forecasting models))
Claims 7 – 9, 13, 16 – 18 are rejected under 35 U.S.C. 103 as being unpatentable over Talagala et al. (NPL: ”Meta-learning how to forecast time series”) in view of Talagala et al. (NPL: ”FFORMPP: Feature-based forecast model performance prediction”), Agrawal et al. (U.S. Pub. 2019/0095756 A1), further in view of Chen et al. (U.S. Pub. 2022/0036246 A1).
Regarding claim 7, the rejection of claim 6 is incorporated herein. The combination of Talagala (meta learn), Agrawal, Talagala (FFORMPP) does not explicitly teach
generating second predicted performance data for each of the plurality of forecasting models using a general meta-learner of the meta-learner machine learning model, wherein the forecasting model is selected based on the second predicted performance data.
However, Chen teaches
generating second predicted performance data for each of the plurality of forecasting models using a general meta-learner of the meta-learner machine learning model, wherein the forecasting model is selected based on the second predicted performance data. ([0046] of Chen states “The learner component 112 can employ one or more meta transfer learning techniques to identify the one or more machine learning pipelines of interest.” [0044] of Chen states “For example, the learner component 112 can identify one or more machine learning pipelines from the pipeline library 120 to be employed in analyzing the time series data” [0051] of Chen states “Additionally, the learner component 112 can rank the identified machine learning pipelines in an order based on the defined evaluation metric. For example, the learner component 112 can rank the identified machine learning pipelines in order based on the predicted accuracy of the machine learning pipelines on the time series data.”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Talagala (meta learn), (FFORMPP), Agrawal with Chen. Talagala (meta learn) framework introduces forecast model selection with meta learning where time series features are computed and used to evaluate performance metric with forecast model selection. Talagala (FFORMPP) builds on similar framework and teaches using time series features as inputs to a meta learning model that predicts forecast loss for a pool of forecasting models. Agrawal teaches trained meta model regressors, including distinctly configured artificial neural networks, for predicting algorithm performance, and further teaches that separate meta models avoid interference caused by a single shared regressor/classifier. Chen teaches an automated timeseries analysis system with a server including a processor, memory, a timeseries component, and a learner component. Chen’s learner component uses meta transfer learning over meta data and rank candidate machine learning pipelines based on performance and evaluation metrics. Chen further uses feature extraction, PCA, and evaluation metrics. One with ordinary skill in the art would be motivated to incorporate the teachings of Chen with the combination of Talagala (meta learn), (FFORMPP), Agrawal to get more flexible system that uses meta features to predict performance, select and rank forecasting pipelines, tune hyperparameters, and update learner parameters based on loss functions from evaluation metrics.
Regarding claim 8, the rejection of claim 7 is incorporated herein. The combination of Talagala (meta learn), (FFORMPP), Agrawal, and Chen teaches
providing the second predicted performance data as an input to the time-series meta-learner. ([0051] of Chen states “Additionally, the learner component 112 can rank the identified machine learning pipelines in an order based on the defined evaluation metric. For example, the learner component 112 can rank the identified machine learning pipelines in order based on the predicted accuracy of the machine learning pipelines on the time series data.” Chen teaches producing predicted accuracy/metrics and FFORMPP teaches timeseries meta learner that takes feature vectors as input. Obvious that predicted performance measures are useful meta data for ranking pipeless, to treat the predicted accuracy / performance data from the general meta learner as additional feature input to the time-series meta-learner in FFORMPP.)
Regarding claim 9, the rejection of claim 1 is incorporated herein. The combination of Talagala (meta learn), (FFORMPP), Agrawal, and Chen teaches
identifying a plurality of hyperparameters for each of the plurality of forecasting models; and selecting a hyperparameter from the plurality of hyperparameters using the meta-learner machine learning model, wherein the predicted time-series data is based on the selected hyperparameter ([0067] of Chen states “FIG. 5 illustrates a diagram of the example, non-limiting system 100 further comprising hyperparameter component 502” [0068] of Chen states ”In various embodiments, the hyperparameter component 502 can employ a hyperparameter optimization to select a set of optimal hyperparameters for the identified machine learning pipelines. Thereby, the hyperparameter component 502 can select a set of hyperparameters used to control the automated machine learning process executed by the time series analysis component 108. Example hyperparameter optimization approaches that can be employed by the hyperparameter component 502 can include, but are not limited to: grid search, random search, gradient-based optimization, Bayesian optimization, evolutionary optimization, population-based training, alternating direction method of multipliers (“ADMM”), a combination thereof, and/or the like.” Hyperparameter component handles config over a set of hyperparameters for pipelines. Learner uses meta data and predicted accuracy/performance to rank pipelines. )
Regarding claim 13, the rejection of claim 12 is incorporated herein. The combination of Talagala (meta learn), (FFORMPP), Agrawal, and Chen teaches
computing a time-series loss term based on an output of a time-series meta-learner; and computing a general loss term based on an output of a general meta-learner, wherein the loss function comprises the time-series loss term and the general loss term. (Pg. 4 of Talagala (FFORMPP) states “We propose an algorithm to rank forecast models by simultaneously predicting the forecast errors. The rankings allow the user to identify a subset of forecasting models.” [0046] of Chen states “The learner component 112 can employ one or more meta transfer learning techniques to identify the one or more machine learning pipelines of interest... These observations can be captured as meta-data associated with the machine learning pipelines. The meta-data can regard how well the machine learning pipeline accomplished a given machine learning task with respect to one or more evaluation metrics (e.g., accuracy of predictions and/or classifications)” FFORMPP’s regression gives a timeseries specific prediction error for each forecasting model and Chen’s learner uses meta data and evaluation metrics across pipelines/tasks. Combine these for a loss function comprised of a timeseries loss term and a general meta learning loss term.)
Regarding claim 16, the combination of Talagala (meta learn), (FFORMPP), Agrawal, and Chen teaches
a processor; a memory including instructions executable by the processor; ([0036] of Chen states “Also, the server 102 can comprise or otherwise be associated with at least one memory 114. The server 102 can further comprise a system bus 116 that can couple to various components such as, but not limited to, the time series analysis component 108 and associated components, memory 114 and/or a processor 118.”)
a meta-feature extraction component configured to compute a plurality of meta-features based on a time-series dataset; (Pg. 4 of Talagala (meta) stateas “Rather than work with the time series directly at the level of individual observations, we propose analysing time series via an associated “feature space”. A time series feature is any measurable characteristic of a time series.” Pg. 10 of Talagala (meta) states “From each training period we compute a range of time series features, and fit a selection of candidate models. The calculated features form the input vector to the classification algorithm.”)
and a meta-learner machine learning model configured to select a forecasting model from a plurality of forecasting models based on the time-series dataset. (Pg. 2 of Talagala (meta) states “We present a general framework for forecast-model selection using meta-learning. A random forest is used to identify the best forecasting method using only time series features.” Pg. 10 of Talagala (meta) states “Using the fitted models, we generate forecasts and identify the “best” model for each time series based on a forecast error measure (e.g., MASE) calculated over the test period.” Pg. 7 of Talagala (FFORMPP) states “We treat the forecast model selection problem as a multi-class ranking problem, where predictors are features, and the outcome consists of forecast errors (MASE) calculated over all available models. In particular, an instance i is a tuple (f1i, f2i, · · · , fmi, e1i, e2i, · · · , eni), where predictors (f1i, f2i, · · · , fmi) is a vector of m features which is the input to our algorithm, and the outcome (e1i, e2i, · · · , eni), is a vector of MASE values of the all n forecast models. A description of offline and online phases are as follows.” [0046] of Chen states “The learner component 112 can employ one or more meta transfer learning techniques to identify the one or more machine learning pipelines of interest... The meta-data can regard how well the machine learning pipeline accomplished a given machine learning task with respect to one or more evaluation metrics (e.g., accuracy of predictions and/or classifications).” Talagala teaches meta learner ml model that takes timeseries meta features as input and selects the best forecasting model from a plurality based on performance. Chen gives a learner component that ranks and selects pipelines/models based on evaluation metrics. )
wherein the meta-learner machine learning model comprises an artificial neural network (ANN) including a time-series meta-learner and a general meta-learner separate from the time-series meta-learner, wherein the time-series meta-learner is trained to predict model performance based on relationships across different windows of time of the time-series dataset, and wherein the general meta-learner is trained to predict model performance based on a single time window of the time-series dataset. ([0073] of Agrawal states “Each of meta-models 151-153 is itself an instance of trainable regression algorithm, although not the same algorithm for which the meta-models are trained for. For example, meta-models 151-153 may each be a distinct neural network that is already trained to predict the performance of algorithm 121, which may be support vector machine instead of a neural network. Training of meta-models is discussed later herein.” [0093] of Agrawal states “For example, meta-models 151-153 may each be an already trained neural network that takes a subset of hyperparameter values and a subset of meta-feature values as stimulus inputs, shown as dashed arrows entering meta-models 151-153. Training of meta-models is discussed later herein.” [0065] of Agrawal states “In an embodiment, the trained regressors are distinctly configured artificial neural networks. In an embodiment, the trained regressors are contained within algorithm-specific ensembles. Techniques are also provided herein for optimal training of regressors and/or ensembles.” [0003] of Agrawal states “There are hundreds of machine learning algorithms. Training and testing each one to find the best performing might not be feasible. Automatic approaches to selective training typically ultimately use a single regressor/classifier for predicting algorithm performance, which causes different algorithms to interfere with each other in the selection model, thereby lowering accuracy. These approaches also do not consider algorithm hyperparameters, which can significantly affect algorithm performance and behavior.” [0072] of Agrawal states “Computer 100 creates or obtains meta-models for each of algorithms 121-123 to quickly and accurately predict the performance of each algorithm. For example, computer 100 may create meta-models 151-153 as performance predictors of algorithm 121.” [0080] of Agrawal states “For each actual configuration alternative or set of related configuration alternatives, computer 100 has a separate meta-model, such as 151-153.” Agrawal teaches using separate neural network meta models for performance prediction rather than a single shared regressor/classifier. It would have been obvious to modify Talagala’s time series forecasting meta learning framework to include separate ANN meta learner components because Agrawal teaches that separate performance prediction meta models reduce interference and improve prediction accuracy. Pg. 1 Abstract section of Talagala (FFORMPP) states “We model the forecast error as a function of time series features calculated from the historical time series with an efficient Bayesian multivariate surface regression approach.” [0081] of Agrawal states “Each of meta-models 151-153 was trained to predict how a particular configuration (or set of related configurations) of algorithm 121 will perform for a variety of datasets that are similar or dissimilar to inference dataset 110. Related configurations are those that have identical or similar values for a subset of hyperparameters 181-184.” Talagala teaches predicting forecast model error from historical time series features. Agrawal teaches that using a single regressor/classifier for performance prediction can cause interference and lower accuracy, and teaches using separate meta-models. It would have been obvious to implement this temporal performance prediction aspect as the claimed separate time series meta learner. Pg. 10 3 Methodology of Talagala (meta learn) states “From each training period we compute a range of time series features, and fit a selection of candidate models. The calculated features form the input vector to the classification algorithm. Using the fitted models, we generate forecasts and identify the “best” model for each time series based on a forecast error measure (e.g., MASE) calculated over the test period.” [0064] of Agrawal states “In an embodiment, a computer derives meta-feature values from an inference dataset by, for each meta-feature, deriving a respective meta-feature value from the inference dataset. For each trainable algorithm and each regression meta-model that is respectively associated with the algorithm, a respective score is calculated by invoking the meta-model based on at least one of: a) a respective subset of meta-feature values, and/or b) hyperparameter values of a respective subset of hyperparameters of the algorithm. One or more of the algorithms are selected based on the respective scores. Based on the inference dataset, the one or more algorithms may be invoked to obtain a result.” [0003] of Agrawal states “Automatic approaches to selective training typically ultimately use a single regressor/classifier for predicting algorithm performance, which causes different algorithms to interfere with each other in the selection model, thereby lowering accuracy.” Talagala teaches selecting a forecast model based on features calculated from a time series, and Agrawal teaches deriving meta feature values and invoking a meta model to calculate a score. It would have been obvious to implement this feature based performance prediction aspect as the claimed separate general meta learner.)
Regarding claim 17, the rejection of claim 16 is incorporated herein. The combination of Talagala (meta learn), (FFORMPP), Agrawal, and Chen teaches
a training component configured to update parameters of the meta-learner machine learning model based on a loss function. (Pg. 7 2.4 Forecast-model selection using meta-learning section of Talagala (meta learn) states “Forecast-model selection problem. For a given time series x ∈ P, with features f(x) ∈ F, find the selection mapping S(f(x)) into the algorithm space A, such that the selected algorithm α ∈ A minimizes forecast accuracy error metric y(α(x)) ∈ Y on the test set of the time series.“
Pg. 9 3 Methodology section of Talagala (meta learn) states “Our proposed FFORMS framework, presented in Figure 4, builds on this preceding research. The offline and online phases are shown in blue and red respectively. A classification algorithm (the meta-learner) is trained during the offline phase and is then used to select an appropriate forecast model for a new time series in the online phase.” Pg. 10 of Talagala (meta learn) states “From each training period we compute a range of time series features, and fit a selection of candidate models. The calculated features form the input vector to the classification algorithm. Using the fitted models, we generate forecasts and identify the “best” model for each time series based on a forecast error measure (e.g., MASE) calculated over the test period. The models deemed “best” form the output labels for the classification algorithm.” [0032] of Chen states “Machine learning models can learn through training, wherein data with known outcomes is inputted into the computer model, outputs regarding the data are compared to the known outcomes, and/or the weights of the computer model are autonomous adjusted based on the comparison to replicate the known outcomes.” [0046] of Chen states “The meta-data can regard how well the machine learning pipeline accomplished a given machine learning task with respect to one or more evaluation metrics (e.g., accuracy of predictions and/or classifications).” Talagala’s framework defines the meta learner in terms of minimizing an accuracy error metric and explicitly trains a classification meta-learner using “best” models determined by a forecast error measure (loss function over performance). Chen teaches that ML models are trained by comparing outputs to known outcomes and adjusting weights based on that comparison, i.e. using a loss derived from evaluation metrics. )
Regarding claim 18, the rejection of claim 16 is incorporated herein. The combination of Talagala (meta learn), (FFORMPP), Agrawal, and Chen teaches
the meta-learner machine learning model comprises a general meta-learner and a time-series meta-learner. (Pg. 9 3 Methodology section of Talagala (meta learn) states “Our proposed FFORMS framework, presented in Figure 4, builds on this preceding research. The offline and online phases are shown in blue and red respectively. A classification algorithm (the meta-learner) is trained during the offline phase and is then used to select an appropriate forecast model for a new time series in the online phase.” Pg. 10 section of Talagala (meta learn) states “From each training period we compute a range of time series features, and fit a selection of candidate models. The calculated features form the input vector to the classification algorithm. Using the fitted models, we generate forecasts and identify the “best” model for each time series based on a forecast error measure (e.g., MASE) calculated over the test period.” [0046] of Chen states “The meta-data can regard how well the machine learning pipeline accomplished a given machine learning task with respect to one or more evaluation metrics (e.g., accuracy of predictions and/or classifications).” [0048] of Chen states “One or more meta transfer learning algorithms executed by the learner component 112 can compare the meta-data of the candidate machine learning pipelines with one or more characteristics of the time series data subject to analysis.”)
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Talagala et al. (NPL: ”Meta-learning how to forecast time series”) in view of Talagala et al. (NPL: ”FFORMPP: Feature-based forecast model performance prediction”), Agrawal et al. (U.S. Pub. 2019/0095756 A1), further in view of Schwiep et al. (U.S. Pub. 2022/0292308 A1).
Regarding claim 2, the rejection of claim 1 is incorporated herein. The combination of Talagala (meta learn), (FFORMPP), Agrawal, and Chen does not teach
dividing the time-series dataset into a plurality of time windows of the time-series dataset; and identifying a time window of the plurality of time windows of the time-series dataset, wherein the forecasting model is selected based on the identified time window.
Schwiep teaches that
dividing the time-series dataset into a plurality of time windows; and identifying a time window of the plurality of time windows, wherein the forecasting model is selected based on the identified time window. ([0058] of Schwiep states “In certain examples, a user can configure the problem or experiment as a time series problem, for example, by specifying a feature derivation window 502 (FDW) and/or a forecast window 506 (FW), according to modeling requirements. In general, the feature derivation window 502 can be a period of time before a forecast point 504 (a time at which a forecast is made) within which features can be derived for the time series.” [0059] of Schwiep states “The system 100 can provide, via the graphical user interface, an indication of a forecast point 504 at or between the feature derivation window 502 (e.g., a first window) and the forecast window 506 (e.g., a second window). The system 100 can provide a forecast window user interface element 510 with input text boxes through which a user can adjust the forecast window 506.”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Talagala (meta learn), (FFORMPP), Agrawal with Schwiep. Talagala (meta learn) framework introduces forecast model selection with meta learning where time series features are computed and used to evaluate performance metric with forecast model selection. Talagala (FFORMPP) builds on similar framework and teaches using time series features as inputs to a meta learning model that predicts forecast loss for a pool of forecasting models. Agrawal teaches trained meta model regressors, including distinctly configured artificial neural networks, for predicting algorithm performance, and further teaches that separate meta models avoid interference caused by a single shared regressor/classifier. Schwiep teaches timeseries modeling with configurable feature derivation windows and performs feature reduction to obtain a reduced feature set. One with ordinary skill in the art would be motivated to incorporate the teachings of Schwiep into the combination of Talagala (meta learn), (FFORMPP), and Agrawal to manage feature dimensionality, avoid overfitting, and improve computational efficiency of the system. It would have been predictable to divide the timeseries into windows for feature computation and provide feature reduction component that reduces the dimensionality of the plurality of meta features.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Talagala et al. (NPL: ”Meta-learning how to forecast time series”) in view of Talagala et al. (NPL: ”FFORMPP: Feature-based forecast model performance prediction”), Agrawal et al. (U.S. Pub. 2019/0095756 A1), Chen et al. (U.S. Pub. 2022/0036246 A1), further in view of Schwiep et al. (U.S. Pub. 2022/0292308 A1).
Regarding claim 20, the rejection of claim 16 is incorporated herein. The combination of Talagala (meta learn), (FFORMPP), Agrawal, and Chen teaches
reduce a dimensionality of the plurality of meta-features. (Pg. 8-9 of Talagala (meta learn) states “They used a feature set F comprising nine features: strength of trend, strength of seasonality, serial correlation, nonlinearity, skewness, kurtosis, self-similarity, chaos and periodicity… The authors further reduced the dimensionality of time series by performing principal component analysis on the features.”)
However, the combination does not explicitly teach
a feature-embedding component configured to reduce a dimensionality
Schwiep teaches that
a feature-embedding component configured to reduce a dimensionality ([0122] of Schwiep states “In various examples, the systems and methods described herein can provide and utilize a large set of models for training on a specific dataset. Models can be added or considered based on a large set of characteristics inferred during exploratory data analysis and feature engineering. The systems and methods can build both regular non-time series regression and classification models as well as time series specialized models.” [0123] of Schwiep states “Non-time series models may be possible to train because of the innovative feature engineering and feature reduction capabilities of the feature engineering module 106.” Talagala teaches PCA on time series features to reduce dimensionality and Schwiep introduce a component responsible for reducing the feature space.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Talagala (meta learn), (FFORMPP), Agrawal, Chen with Schwiep. Talagala (meta learn) framework introduces forecast model selection with meta learning where time series features are computed and used to evaluate performance metric with forecast model selection. Talagala (FFORMPP) builds on similar framework and teaches using time series features as inputs to a meta learning model that predicts forecast loss for a pool of forecasting models. Agrawal teaches trained meta model regressors, including distinctly configured artificial neural networks, for predicting algorithm performance, and further teaches that separate meta models avoid interference caused by a single shared regressor/classifier. Chen provides a general meta transfer learning framework and Schwiep teaches performing feature reduction to obtain a reduced feature set for modeling. One with ordinary skill in the art would be motivated to incorporate the teachings of Schwiep into Talagala (meta learn), (FFORMPP), Agrawal, and Chen to control feature dimensionality, reduce overfitting, and improve the combined system when training meta learners on timeseries features. It would have been predictable to include feature embedding component in the system for feature reduction of timeseries meta features.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Talagala et al. (NPL: ”Meta-learning how to forecast time series”) in view of Talagala et al. (NPL: ”FFORMPP: Feature-based forecast model performance prediction”), Agrawal et al. (U.S. Pub. 2019/0095756 A1), Chen et al. (U.S. Pub. 2022/0036246 A1), further in view of Rao et al. (U.S. Pub. 2020/0242483).
Regarding claim 19, the rejection of claim 18 is incorporated herein. The combination of Talagala (meta learn), (FFORMPP), Agrawal, and Chen teaches
[ML model] comprises an long short-term memory (LSTM) model. ([0032] of Chen states “Example machine learning models can include, but are not limited to: neural network models, perceptron (“P”), feed forward (“FF”), radial basis network (“RBF”), deep feed forward (“DFF”), recurrent neural network (“RNN”), long/short term memory (“LSTM”),”)
However, the combination does not explicitly teach
the time-series meta-learner comprises a long short-term memory (LSTM) model.
Rao teaches that
the time-series meta-learner comprises a long short-term memory (LSTM) model. ([0016] of Rao states ”FIG. 11 is a workflow diagram of possible prediction algorithms applied to transformed data series groups for time series forecasting, in accordance with one embodiment.” [0137] of Rao states “In one embodiment, for non-sparse time series, a possible prediction algorithm is an auto-regressive approach including Autoregressive Integrated Moving Average (ARIMA) plus Holt-Winters Exponential Smoothing, Recurrent Neural Network-long short-term memory (RNN-LSTM) plus Markov Chain Monte Carlo/Dropout (MCM), RNN-LSTM plus Gaussian Process, hybrid of ARIMA plus RNN, and other auto-regressive approaches as discussed herein, or as known in the art at the time of filing, or as developed, or becomes available, after the time of filing.” [0144] of Rao states “As a specific illustrative example, cell 1225 may be represented by a transform of “Box_Cox” and a prediction algorithm of “Arima.” As a specific illustrative example, cell 1235 may be represented by a transform of “Monthly” and a prediction algorithm of “LSTM.””)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Talagala (meta learn), (FFORMPP), Agrawal, Chen with Rao. Talagala (meta learn) framework introduces forecast model selection with meta learning where time series features are computed and used to evaluate performance metric with forecast model selection. Talagala (FFORMPP) builds on similar framework and teaches using time series features as inputs to a meta learning model that predicts forecast loss for a pool of forecasting models. Agrawal teaches trained meta model regressors, including distinctly configured artificial neural networks, for predicting algorithm performance, and further teaches that separate meta models avoid interference caused by a single shared regressor/classifier. Chen provides a general meta transfer learning framework and Rao teaches a meta model decision engine for timeseries forecasting that selects among different prediction methods. One with ordinary skill in the art would be motivated to incorporate the teachings of Rao into Talagala (meta learn), (FFORMPP), and Chen. LSTM is well known in the art for timeseries data as the Talagala and Chen’s system are regarding the timeseries specific meta learner. It would have been predictable to use LSTM for better usage with the temporal dependencies in timeseries features.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BYUNGKWON HAN whose telephone number is (571)272-5294. The examiner can normally be reached M-F: 9:00AM-6PM PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at (571)272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BYUNGKWON HAN/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights