DETAILED ACTION
This Office Action is in response to Applicant's Communication received on 04/16/2026 for application number 17/812,282.
Claims 1-5 and 7-21 are presented for examination. Claims 1, 8, and 18 are independent claims.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 04/16/2026 has been entered.
Response to Amendment
The amendment filed on 04/16/2026 has been entered.
Claims 1, 8, 17, and 18 have been amended. Claim 21 is new. Claims 1-5 and 7-21 are pending in the application.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-5 and 7-21 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for pre-AIA the inventor(s), at the time the application was filed, had possession of the claimed invention.
Independent Claims 1, 8, and 18 recite “identifying one or more motifs within the selection of the plurality of time intervals”. The amendment imposes a sequence of operations not described anywhere in the specification - i.e., displaying time intervals, receiving user selection of time intervals and identifying motifs within that selection. The specification, in contrast, describes two clearly separate and alternative paths for creating classes - user class and subsequence class: [0071] The classes 107 may include user classes 107A that were selected by the entity and subsequence classes 107B that correspond to the top k motifs 106; [0034] the user class 107A may be a subsequence, or multiple subsequences, of time intervals from the time series data 104 that are selected for an entity by a user or administrator. The class component 110 may create one or more subsequence classes 107B, which may include the top k motifs 106 within the time series data 104; [0036] the class component 110 may provide a user interface through which the user or administrator may view the time intervals of the time series data 104 along with selected external data 108. The class component 110 may receive a user selection 102 of time intervals from the time series data 104 and may use the selected time intervals to create the user class 107A for the entity. Nothing in the specification describes using a user selected subset of time intervals for motif identification. At best, the specification suggests that the motif identification step is operated on the time series data. Therefore, the language - identifying one or more motifs within the selection of the plurality of time intervals - constitutes new matter. (See also 37 C.F.R. 1.121(f), MPEP 608.04, 706.03(o)).
Independent Claims 1, 8, and 18 recite “creating each class of the set of classes comprises:... creating the class from the selection of the plurality of time intervals and the identified one or more motifs associated with the class”. The amendment creates a hybrid class - one that is defined by both the user selected time intervals and motifs identified within that selection. The specification, in contrast, describes two clearly separate and alternative classes - user class and subsequence class: [0071] The classes 107 may include user classes 107A that were selected by the entity and subsequence classes 107B that correspond to the top k motifs 106; [0034] the user class 107A may be a subsequence, or multiple subsequences, of time intervals from the time series data 104 that are selected for an entity by a user or administrator. The class component 110 may create one or more subsequence classes 107B, which may include the top k motifs 106 within the time series data 104; [0036] the class component 110 may provide a user interface through which the user or administrator may view the time intervals of the time series data 104 along with selected external data 108. The class component 110 may receive a user selection 102 of time intervals from the time series data 104 and may use the selected time intervals to create the user class 107A for the entity. Nothing in the specification describes using a user selected subset of time intervals for motif identification. Therefore, the language - creating the class from the selection of the plurality of time intervals and the identified one or more motifs associated with the class - constitutes new matter. (See also 37 C.F.R. 1.121(f), MPEP 608.04, 706.03(o)).
Claims 2-8, 10-11, 15-17, 19, and 21-24 are also rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as being dependent on parent claims failing to comply with the written description requirement.
Dependent Claims 17 and 21 recite “the one or more motifs are identified by calculating a similarity score between different subsequences of the plurality of subsequences using a Euclidean distance metric”. The specification describes: [0029] The STUMPY tool takes as an input the time series data 104 and computes a matrix profile for the time series data 104. This matrix profile is then used to determine the motifs 106 for the entity; [0031] the motifs 106 may be ranked based on how closely the pattern corresponding to the motif 106 fits each instance of the motif in the time series data 104. There is no mention of calculating a similarity score using a Euclidean distance metric in the specification. Therefore, the language - the one or more motifs are identified by calculating a similarity score between different subsequences of the plurality of subsequences using a Euclidean distance metric - constitutes new matter. (See also 37 C.F.R. 1.121(f), MPEP 608.04, 706.03(o)).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5 are rejected under 35 U.S.C. 103 as being unpatentable over Yocum et al. (US 2021/0271925 A1 hereinafter Yocum) in view of Law (US 2020/0258157 A1).
Regarding Claim 1, Yocum teaches a method for selecting a forecasting algorithm for a class ([0023] for a given schedule of individual days/ points in time for which predictions are to be made, the system may select the best model by forecast group, prediction horizon, and model class for a particular day) comprising:
receiving time series data by a computing device, wherein the time series data comprises a plurality of time intervals and each time interval is associated with an interval value ([0013] models are trained to predict time series data, which may include the average handle time and call volume; [0015] the call volume identifies the number of phone calls that are expected to be received during a specified time interval; [0023] for a given schedule of individual days/ points in time for which predictions are to be made, the system may select the best model by forecast group, prediction horizon, and model class for a particular day; predictions are then made with a set of models to generate a piecewise forecast using the best model for a particular day; [0032] the repository (150) is a computing system that may include multiple computing devices in accordance with the computing system (500) and the nodes (522) and (524); the data in the repository (150) include multiple variables, parameters, historical time series data (i.e., time series data comprising plurality of time intervals), and forecasts that are generated (i.e., interval value) and used by the components of the system - thus, comprising plurality of time intervals and each time interval associated with forecasts/ interval value; [0075] the nodes may include functionality to receive requests from the client device and transmit responses to the client device; the client device may be a computing system; [0076] the computing system or group of computing systems include functionality to perform a variety of operations disclosed);
receiving a plurality of forecasting algorithms by the computing device ([0033] a user may control the model discovery service by selecting when the model discovery service is executed, selecting the model horizons used by the model discovery service, and maintaining the connections between the services of the server application and the databases of the repository that store the data used by the system; [0037] the process generates predictions used to generate estimated headcounts from models with multiple horizons and classes; in Step 202, models are obtained corresponding to model horizons; [0043] multiple models may be trained having multiple model horizons and classes; a model discovery service may have a model space of different models that are trained; the model space may include time series models that have different classes, different lags, and be of different types; the different classes of models may include in-season models, out-of-season models, and all-season models; [0076] the computing system or group of computing systems include functionality to perform a variety of operations disclosed);
creating a set of classes by the computing device, wherein each class in the set of classes is associated with a plurality of subsequences of the time series data, wherein each of the plurality of subsequences comprises a time interval of the plurality of time intervals ([0020] a model class identifies, for a model, the class of the model; the classes may relate to seasonality, which is how the model performs with respect to a particular period of time; example classes may include “all-season”, “in-season”, and “out-of-season” (i.e., set of classes); different classes may be trained and scored differently during training to identify the best performing model - thus, the set of classes trained/ created for the model; [0023] for a given schedule of individual days/ points in time for which predictions are to be made, the system may select the best model by forecast group, prediction horizon, and model class for a particular day; [0037] the models may include lags from target time series; a target time series is a time series for which a model is used to generate predictions; [0039] as a part of generating the values and for each day to be predicted, the model having the model horizon with a lowest value that is greater than or equal to an ordinal value of the day to be predicted may be selected for that day to generate a piecewise forecast for the time series variable being predicted - thus, each class/ particular period of time is associated with plurality of subsequences of the time series data/ individual days comprising plurality of time intervals; [0076] the computing system or group of computing systems include functionality to perform a variety of operations disclosed);
for each class of the set of classes, selecting a forecasting algorithm from the plurality of forecasting algorithms based on the subsequences of the time series data associated with the class by the computing device ([0020] a model class identifies, for a model, the class of the model; the classes may relate to seasonality, which is how the model performs with respect to a particular period of time; example classes may include “all-season”, “in-season”, and “out-of-season”; [0023] for a given schedule of individual days/ points in time for which predictions are to be made, the system may select the best model by forecast group, prediction horizon, and model class for a particular day; [0025] select the best scoring models according to the future time range to forecast; [0039] as a part of generating the values and for each day to be predicted, the model having the model horizon with a lowest value that is greater than or equal to an ordinal value of the day to be predicted may be selected for that day to generate a piecewise forecast for the time series variable being predicted);
receiving a request to forecast the interval value for a future time interval by the computing device ([0025] select the best scoring models according to the future time range to forecast, and provide this selection of models to the demand prediction service; [0058] FIG. 4 illustrates how the piecewise model selection displayed on the graphical user interface (400); [0063] the row (410) displays the date for the predicted days; the dates range from January 5 to January 13 (i.e., future time interval); [0064] the row (412) displays a piecewise prediction forecast generated based on multiple models; [0037] the process generates predictions used to generate estimated headcounts (i.e., forecast interval value) from models with multiple horizons and classes);
determining a class of the set of classes that is associated with the future time interval by the computing device ([0020] a model class identifies, for a model, the class of the model; the classes may relate to seasonality, which is how the model performs with respect to a particular period of time; example classes may include “all-season”, “in-season”, and “out-of-season”; [0023] for a given schedule of individual days/ points in time for which predictions are to be made, the system may select the best model by forecast group, prediction horizon, and model class for a particular day; [0025] select the best scoring models according to the future time range to forecast; [0051] for each of the identified dates, the class that applies to that date is identified; the classes may include in-season, out-of-season, and all-season classes, and one of the classes may be assigned to each date for which a prediction will be generated; [0058] FIG. 4 illustrates how the piecewise model selection displayed on the graphical user interface (400); [0060] the row (404) displays the classes of the models used to generate the predictions of the row (412); [0063] the row (410) displays the date for the predicted days; the dates range from January 5 to January 13 (i.e., future time interval) - thus, determining a class associated with the future time interval);
using the forecasting algorithm selected for the determined class to predict the interval value for the future time interval by the computing device ([0023] for a given schedule of individual days/ points in time for which predictions are to be made, the system may select the best model by forecast group, prediction horizon, and model class for a particular day; [0025] select the best scoring models according to the future time range to forecast; [0037] the process generates predictions used to generate estimated headcounts (i.e., forecast interval value) from models with multiple horizons and classes; [0058] FIG. 4 illustrates how the piecewise model selection displayed on the graphical user interface (400); [0064] the row (412) displays a piecewise prediction forecast generated based on multiple models; the model “T(1,A)” is used to generate the predicted value of “8” for the predicted day “0” that corresponds to the date of January 5 using a horizon of 1 and a class of A;[0042] the estimated headcount generated using the Erlang-C traffic modeling formula with the values from the piecewise forecast and displayed on a client computing device); and
providing the predicted interval value for the future time interval by the computing device ([0037] the process generates predictions used to generate estimated headcounts (i.e., forecast interval value) from models with multiple horizons and classes; [0058] FIG. 4 illustrates how the piecewise model selection displayed on the graphical user interface (400); [0064] the row (412) displays a piecewise prediction forecast generated based on multiple models; the model “T(1,A)” is used to generate the predicted value of “8” for the predicted day “0” that corresponds to the date of January 5 using a horizon of 1 and a class of A; [0042] the estimated headcount generated using the Erlang-C traffic modeling formula with the values from the piecewise forecast and displayed on a client computing device).
However, Yocum fails to expressly teach wherein creating each class of the set of classes comprises: displaying some or all of the plurality of time intervals in a user interface by the computing device; receiving a selection of the plurality of time intervals associated with the class from among the displayed some or all of the plurality of time intervals through the user interface by the computing device; identifying one or more motifs within the selection of the plurality of time intervals by the computing device, wherein each motif of the one or more motifs represents a repeating pattern in the time series data; and creating the class from the selection of the plurality of time intervals and the identified one or more motifs associated with the class by the computing device.
In the same field of endeavor, Law teaches wherein creating each class of the set of classes comprises: displaying some or all of the plurality of time intervals in a user interface by the computing device ([0097] in FIG. 6, a user interface (UI) 400 uses the matrix profile 320 and the matrix profile index 360 of FIG. 3 to allow a user to graphically investigate patterns of interest; x-axis of a data view 408 of the time-series data 300); receiving a selection of the plurality of time intervals associated with the class from among the displayed some or all of the plurality of time intervals through the user interface by the computing device ([0101] a slider bar 440 indicates which subsequence (i.e., time interval) is currently selected as the reference subsequence; the slider bar 440 may also permit the user to directly select the reference subsequence; [0102] the selected reference subsequence may be adjusted as the user moves their pointer across the data view 408 or the matrix profile view 404); identifying one or more motifs within the selection of the plurality of time intervals by the computing device, wherein each motif of the one or more motifs represents a repeating pattern in the time series data ([0112] boxes 528-1 and 528-2 indicate the locations of the selected subsequences (i.e., time intervals); as the box 528-1 is adjusted, the matrix profile index may be used to identify the closest corresponding subsequence and then the box 528-2 would be placed to indicate the corresponding subsequence; [0121] the pattern engine 624 calculates matrix profiles and matrix profile indices from selected time-series data; [0162] motif is a repeated sub-sequence of a time series; [0165] stumpy will be used to identify motifs (patterns) and anomalies/novelties with two different time series datasets; [0168] time series motifs are approximately repeated subsequences found within a longer time series; the stump function is configured to take in any time series and compute the matrix profile along with the matrix profile indices; finding time series motifs with the matrix profile and matrix profile indices - thus, matrix profile is used to identify motifs/ repeated subsequences from the selected subsequences/ time intervals in the time series); and creating the class from the selection of the plurality of time intervals and the identified one or more motifs associated with the class by the computing device ([0115] if the user identifies a pattern of interest, the user can provide a name for that pattern at UI element 560 and then save the named pattern to a pattern library 564 using UI element 568).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated wherein creating each class of the set of classes comprises: displaying some or all of the plurality of time intervals in a user interface by the computing device; receiving a selection of the plurality of time intervals associated with the class from among the displayed some or all of the plurality of time intervals through the user interface by the computing device; identifying one or more motifs within the selection of the plurality of time intervals by the computing device, wherein each motif of the one or more motifs represents a repeating pattern in the time series data; and creating the class from the selection of the plurality of time intervals and the identified one or more motifs associated with the class by the computing device, as taught by Law into Yocum. Doing so would be desirable because it would allow for identifying favorable trends and making favorable predictions (Law [0004]), thereby enhancing user experience.
As to dependent Claim 2, Yocum and Law teach all the limitations of Claim 1. Yocum further teaches wherein the predicted interval value is one of a communication volume, an average handling time, or a shrinkage ([0037] the process generates predictions used to generate estimated headcounts (i.e., forecast interval value) from models with multiple horizons and classes; [0017] the headcount identifies how many representatives are needed to be able to handle the expected amount of calls).
As to dependent Claim 3, Yocum and Law teach all the limitations of Claim 1. Yocum further teaches wherein one or more of scheduling one or more workers to work during the future time interval based on the predicted interval value and generating a hiring plan for the future time interval ([0037] the process generates predictions used to generate estimated headcounts from models with multiple horizons and classes; [0017] the headcount identifies how many representatives are needed to be able to handle the expected amount of calls; the headcount identifies the number of representatives needed over a period of time).
As to dependent Claim 4, Yocum and Law teach all the limitations of Claim 1. Yocum further teaches wherein selecting the forecasting algorithm from the plurality of forecasting algorithms based on the subsequences of the time series data associated with the class comprises selecting the forecasting algorithm with a minimum associated forecast error when predicting the interval value for time intervals from the plurality of subsequences of the time series data associated with the class ([0045] the training data used to train the models may include historical time series data for at least two years; [0046] cross validation metrics are generated; different classes of models may use different data for testing and generating cross validation metrics; in-season models may be tested against in-season data, out-of-season model may be tested against out-of-season data, and all-season models may be tested against all-season data (i.e., models based on the subsequences of the time series data associated with the class/ days associated with the season); the cross validation metrics may include a mean absolute scaled error (MASE), a weighted mean absolute percentage error (wMAPE) for the models, mean absolute percentage error (MAPE), root mean square error (RMSE), explained variance (R2), and mean absolute error (MAE); [0047] models are selected based on cross validation metrics; the model discovery service may select the model with the lowest mean absolute scaled error or the lowest weighted mean absolute percentage error (i.e., minimum associated error)).
As to dependent Claim 5, Yocum and Law teach all the limitations of Claim 1. Yocum further teaches wherein each class of the set of classes is one of a user class or a subsequence class ([0023] for a given schedule of individual days/ points in time for which predictions are to be made, the system may select the best model by forecast group, prediction horizon, and model class for a particular day; [0020] a model class identifies, for a model, the class of the model; the classes may relate to seasonality, which is how the model performs with respect to a particular period of time, such as, Summer, Fall, tax season, etc.; example classes may include “all-season”, “in-season”, and “out-of-season” (equivalent to user class/ subsequence class)).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Yocum in view of Law, further in view of Saleh et al. (US 2023/0252267 A1 hereinafter Saleh).
As to dependent Claim 7, Yocum and Law teach all the limitations of Claim 1. However, Yocum and Law fail to expressly teach wherein receiving a set of external data values by the computing device, wherein each external data value in the set of external data values is associated with a time interval of the plurality of time intervals; and for at least one class in the set of classes, selecting the plurality of subsequences of the time series data for the at least one class based on the set of external data values.
In the same field of endeavor, Saleh teaches wherein receiving a set of external data values by the computing device, wherein each external data value in the set of external data values is associated with a time interval of the plurality of time intervals ([0013] different data records may also include different feature data at different points in time or timesteps, such as data occurring on a particular day of week, a selected time period or time in the past, in a particular month or other time; training data may be used for these features, as well as additional external features that may be specifically selected for a time series forecasting task; external features may include those associated with additional customer data, fraud data, transaction data, a macro-economical feature, a trend in an e-commerce industry, a pandemic effect feature, a total payment volume migration feature, or a combination thereof; these features may provide additional time-based data that may have a temporal nature and affect predictive forecasting at future timesteps/times - thus, receiving a set of external data value/ external feature data associated with time interval); and for at least one class in the set of classes, selecting the plurality of subsequences of the time series data for the at least one class based on the set of external data values ([0012] utilize a trained DNN model, such as one using an LSTM architecture, that includes an attention mechanism to focus on particular past timesteps and/or time-based/temporal data (i.e., time series data) when training the DNN to predict future timesteps and corresponding forecasts; [0013] different data records may also include different feature data at different points in time or timesteps, such as data occurring on a particular day of week, a selected time period or time in the past, in a particular month or other time (i.e., subsequences of the time series data for a class); training data may be used for these features, as well as additional external features that may be specifically selected for a time series forecasting task; external features may include those associated with additional customer data, fraud data, transaction data, a macro-economical feature, a trend in an e-commerce industry, a pandemic effect feature, a total payment volume migration feature, etc.; these features may provide additional time-based data that may have a temporal nature and affect predictive forecasting at future timesteps/times; [0021] when training the DNN model for predictive forecasts of a future traits, features, or variables, additional external features utilized to provide additional accuracy, temporal relevance, and/or feature dependence; external feature may therefore be used to provide additional context or factors that may assist in explaining past trends and predicting future forecasts for a particular variable; external features correspond to segments of temporal data that may affect the main input attribute or feature - thus, selecting the subsequences of the time series data/ temporal data for particular points in time/ class corresponding to the external data values/external feature data for training and predicting forecasts).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated wherein receiving a set of external data values by the computing device, wherein each external data value in the set of external data values is associated with a time interval of the plurality of time intervals; and for at least one class in the set of classes, selecting the plurality of subsequences of the time series data for the at least one class based on the set of external data values, as taught by Saleh into Yocum and Law. Doing so would be desirable because using external features would provide additional context or factors that may assist in explaining past trends and predicting future forecasts for a particular variable and provide additional accuracy, temporal relevance, and/or feature dependence (Saleh [0021]).
Claims 8-11, 14-17, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Yocum in view of Law, further in view of Mitra et al. (US 2023/0075453 A1 hereinafter Mitra).
Regarding Claim 8, Yocum teaches a method ([0023] for a given schedule of individual days/ points in time for which predictions are to be made, the system may select the best model by forecast group, prediction horizon, and model class for a particular day) comprising:
receiving time series data by a computing device, wherein the time series data comprises a plurality of time intervals and each time interval is associated with an interval value ([0013] models are trained to predict time series data, which may include the average handle time and call volume; [0015] the call volume identifies the number of phone calls that are expected to be received during a specified time interval; [0023] for a given schedule of individual days/ points in time for which predictions are to be made, the system may select the best model by forecast group, prediction horizon, and model class for a particular day; predictions are then made with a set of models to generate a piecewise forecast using the best model for a particular day; [0032] the repository (150) is a computing system that may include multiple computing devices in accordance with the computing system (500) and the nodes (522) and (524); the data in the repository (150) include multiple variables, parameters, historical time series data (i.e., time series data comprising plurality of time intervals), and forecasts that are generated (i.e., interval value) and used by the components of the system - thus, comprising plurality of time intervals and each time interval associated with forecasts/ interval value; [0075] the nodes may include functionality to receive requests from the client device and transmit responses to the client device; the client device may be a computing system; [0076] the computing system or group of computing systems include functionality to perform a variety of operations disclosed);
receiving a plurality of forecasting algorithms by the computing device ([0033] a user may control the model discovery service by selecting when the model discovery service is executed, selecting the model horizons used by the model discovery service, and maintaining the connections between the services of the server application and the databases of the repository that store the data used by the system; [0037] the process generates predictions used to generate estimated headcounts from models with multiple horizons and classes; in Step 202, models are obtained corresponding to model horizons; [0043] multiple models may be trained having multiple model horizons and classes; a model discovery service may have a model space of different models that are trained; the model space may include time series models that have different classes, different lags, and be of different types; the different classes of models may include in-season models, out-of-season models, and all-season models; [0076] the computing system or group of computing systems include functionality to perform a variety of operations disclosed);
creating a set of classes by the computing device, wherein each class in the set of classes is associated with a plurality of subsequences of the time series data, wherein each of the plurality of subsequences comprises a time interval of the plurality of time intervals ([0020] a model class identifies, for a model, the class of the model; the classes may relate to seasonality, which is how the model performs with respect to a particular period of time; example classes may include “all-season”, “in-season”, and “out-of-season” (i.e., set of classes); different classes may be trained and scored differently during training to identify the best performing model - thus, the set of classes trained/ created for the model; [0023] for a given schedule of individual days/ points in time for which predictions are to be made, the system may select the best model by forecast group, prediction horizon, and model class for a particular day; [0037] the models may include lags from target time series; a target time series is a time series for which a model is used to generate predictions; [0039] as a part of generating the values and for each day to be predicted, the model having the model horizon with a lowest value that is greater than or equal to an ordinal value of the day to be predicted may be selected for that day to generate a piecewise forecast for the time series variable being predicted - thus, each class/ particular period of time is associated with plurality of subsequences of the time series data/ individual days comprising plurality of time intervals; [0076] the computing system or group of computing systems include functionality to perform a variety of operations disclosed);
training each forecasting algorithm to predict the interval value using a portion of the time series data by the computing device ([0036] training models and predicting call center volume (i.e., interval value); [0038] the time series variable values are predictions generated with trained models; the values may correspond to the days to be predicted; [0045] the training data used to train the models may include historical time series data for at least two years; additional data beyond two years may be included and more or less data may be used; the initial training of the models may use a subset of data (i.e., portion of time series data), such as the trailing two years of data; then after the models are selected, the selected models may be trained with the additional training data that was not used prior to model selection); and
training a selection model by the computing device using the received time series data, the set of classes, ([0025] the model discovery service (equivalent to selection model) is a set of programs executing on the server computing system as part of the server application; the model discovery service identifies and trains the models used to generate predictions of time series variable values that may be used as inputs to generate headcount predictions; the model discovery service may receive input data from the repository, train the models, score the models, select the best scoring models according to the future time range to forecast, and provide this selection of models to the demand prediction service; [0043] multiple models may be trained having multiple model horizons and classes; a model discovery service may have a model space of different models that are trained; the model space may include time series models that have different classes, different lags, and be of different types; the different classes of models may include in-season models, out-of-season models, and all-season models; [0032] the repository is a computing system that may include multiple computing devices in accordance with the computing system and the nodes; the data in the repository include multiple variables, parameters, historical time series data (i.e., time series data), and forecasts that are generated and used by the components of the system - thus, the model discovery service including model space of different models is trained using time series data and classes/ particular periods of time).
However, Yocum fails to expressly teach wherein creating each class of the set of classes comprises: displaying some or all of the plurality of time intervals in a user interface by the computing device; receiving a selection of the plurality of time intervals associated with the class from among the displayed some or all of the plurality of time intervals through the user interface by the computing device; identifying one or more motifs within the selection of the plurality of time intervals by the computing device, wherein each motif of the one or more motifs represents a repeating pattern in the time series data; and creating the class from the selection of the plurality of time intervals and the identified one or more motifs associated with the class by the computing device.
In the same field of endeavor, Law teaches wherein creating each class of the set of classes comprises: displaying some or all of the plurality of time intervals in a user interface by the computing device ([0097] in FIG. 6, a user interface (UI) 400 uses the matrix profile 320 and the matrix profile index 360 of FIG. 3 to allow a user to graphically investigate patterns of interest; x-axis of a data view 408 of the time-series data 300); receiving a selection of the plurality of time intervals associated with the class from among the displayed some or all of the plurality of time intervals through the user interface by the computing device ([0101] a slider bar 440 indicates which subsequence (i.e., time interval) is currently selected as the reference subsequence; the slider bar 440 may also permit the user to directly select the reference subsequence; [0102] the selected reference subsequence may be adjusted as the user moves their pointer across the data view 408 or the matrix profile view 404); identifying one or more motifs within the selection of the plurality of time intervals by the computing device, wherein each motif of the one or more motifs represents a repeating pattern in the time series data ([0112] boxes 528-1 and 528-2 indicate the locations of the selected subsequences (i.e., time intervals); as the box 528-1 is adjusted, the matrix profile index may be used to identify the closest corresponding subsequence and then the box 528-2 would be placed to indicate the corresponding subsequence; [0121] the pattern engine 624 calculates matrix profiles and matrix profile indices from selected time-series data; [0162] motif is a repeated sub-sequence of a time series; [0165] stumpy will be used to identify motifs (patterns) and anomalies/novelties with two different time series datasets; [0168] time series motifs are approximately repeated subsequences found within a longer time series; the stump function is configured to take in any time series and compute the matrix profile along with the matrix profile indices; finding time series motifs with the matrix profile and matrix profile indices - thus, matrix profile is used to identify motifs/ repeated subsequences from the selected subsequences/ time intervals in the time series); and creating the class from the selection of the plurality of time intervals and the identified one or more motifs associated with the class by the computing device ([0115] if the user identifies a pattern of interest, the user can provide a name for that pattern at UI element 560 and then save the named pattern to a pattern library 564 using UI element 568).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated wherein creating each class of the set of classes comprises: displaying some or all of the plurality of time intervals in a user interface by the computing device; receiving a selection of the plurality of time intervals associated with the class from among the displayed some or all of the plurality of time intervals through the user interface by the computing device; identifying one or more motifs within the selection of the plurality of time intervals by the computing device, wherein each motif of the one or more motifs represents a repeating pattern in the time series data; and creating the class from the selection of the plurality of time intervals and the identified one or more motifs associated with the class by the computing device, as taught by Law into Yocum. Doing so would be desirable because it would allow for identifying favorable trends and making favorable predictions (Law [0004]), thereby enhancing user experience.
However, Yocum and Law fail to expressly teach wherein for each time interval of the plurality of time intervals of the time series data that is not in the portion: for each forecasting algorithm of the plurality of forecasting algorithms: predicting the interval value for the time interval using the forecasting algorithm by the computing device; and determining a difference between the interval value associated with the time interval in the time series data and the predicted interval value for the time interval by the computing device; and the determined differences for each forecasting algorithm for each time interval of the plurality of time intervals of the time series data.
In the same field of endeavor, Mitra teaches wherein for each time interval of the plurality of time intervals of the time series data that is not in the portion: for each forecasting algorithm of the plurality of forecasting algorithms: predicting the interval value for the time interval using the forecasting algorithm by the computing device ([0008] the system performs forecasting using the machine learning based model by determining a training data set based on the time series data; the training dataset includes a training subset and a test subset; the system trains the machine learning based model using the training subset of the training dataset and evaluates the machine learning based model using the test subset of the training dataset; [0047] the training dataset D includes a training subset T1 and a test subset T2; the model generation module 130 trains 420 the machine learning based model using the training subset T1; the model generation module 130 evaluates 430 the machine learning based model using the test subset T2; [0057] the system trains all machine learning models on S11 and evaluates their performance on S1); and determining a difference between the interval value associated with the time interval in the time series data and the predicted interval value for the time interval by the computing device ([0036] a model metric represents a criterion for evaluating machine learning based models; a model metric represents a function or an expression used for determining a difference between data forecasted using a model with observed data (or labelled data)); and training a selection model by the computing device using the determined differences for each forecasting algorithm for each time interval of the plurality of time intervals of the time series data ([0057] the system trains all machine learning models on S11 and evaluates their performance on S12; based on the performance, the system selects top model from each pool; the top models from each pool (selected based on the evaluation in previous round) is trained on S21, and the system evaluates their performance on S22; based on the performance, the system selects the TOP model across all pools; [0058] the model generation module 130 (equivalent to selection model) trains 720 each machine learning based model from the pool P1 of machine learning based models using training subset S11; the model generation module 130 evaluates 730 each machine learning based model from the pool 610 of machine learning based models using the given model metric and using the test subset S12; [0060] the model generation module 130 evaluates 760 each machine learning based model from the pool P2 of machine learning based models using the given model metric and using training subset S22; [0061] the model generation module 130 selects the best model for forecasting time series data for the given application or based on a given model metric from the pool P2 of machine learning based models; [0036] a model metric represents a criterion for evaluating machine learning based models; a model metric represents a function or an expression used for determining a difference between data forecasted using a model with observed data (or labelled data) - thus, training using the determined differences for each model/ forecasting algorithm for each time interval of the plurality of time intervals of the time series data).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated wherein for each time interval of the plurality of time intervals of the time series data that is not in the portion: for each forecasting algorithm of the plurality of forecasting algorithms: predicting the interval value for the time interval using the forecasting algorithm by the computing device; and determining a difference between the interval value associated with the time interval in the time series data and the predicted interval value for the time interval by the computing device; and training a selection model by the computing device using the determined differences for each forecasting algorithm for each time interval of the plurality of time intervals of the time series data, as taught by Mitra into Yocum and Law. Doing so would be desirable because it would enable the system to automatically select a model from different models based on previously known applications (Mitra [0070]).
As to dependent Claim 9, Yocum, Law, and Mitra teach all the limitations of Claim 8. Yocum further teaches wherein receiving a request to forecast the interval value at a future time interval ([0025] select the best scoring models according to the future time range to forecast, and provide this selection of models to the demand prediction service; [0058] FIG. 4 illustrates how the piecewise model selection displayed on the graphical user interface (400); [0063] the row (410) displays the date for the predicted days; the dates range from January 5 to January 13 (i.e., future time interval); [0064] the row (412) displays a piecewise prediction forecast generated based on multiple models; [0037] the process generates predictions used to generate estimated headcounts (i.e., forecast interval value) from models with multiple horizons and classes); using the selection model to select a forecasting algorithm of the plurality of forecasting algorithms for the future time interval ([0023] for a given schedule of individual days/ points in time for which predictions are to be made, the system may select the best model by forecast group, prediction horizon, and model class for a particular day; [0025] the model discovery service may receive input data from the repository, train the models, score the models, select the best scoring models according to the future time range to forecast, and provide this selection of models to the demand prediction service; [0037] the process generates predictions used to generate estimated headcounts (i.e., forecast interval value) from models with multiple horizons and classes; [0058] FIG. 4 illustrates how the piecewise model selection displayed on the graphical user interface (400); [0064] the row (412) displays a piecewise prediction forecast generated based on multiple models; the model “T(1,A)” is used to generate the predicted value of “8” for the predicted day “0” that corresponds to the date of January 5 using a horizon of 1 and a class of A;[0042] the estimated headcount generated using the Erlang-C traffic modeling formula with the values from the piecewise forecast and displayed on a client computing device); using the selected forecasting algorithm to predict the interval value for the future time interval ([0023] for a given schedule of individual days/ points in time for which predictions are to be made, the system may select the best model by forecast group, prediction horizon, and model class for a particular day; [0025] select the best scoring models according to the future time range to forecast; [0037] the process generates predictions used to generate estimated headcounts (i.e., forecast interval value) from models with multiple horizons and classes; [0058] FIG. 4 illustrates how the piecewise model selection displayed on the graphical user interface (400); [0064] the row (412) displays a piecewise prediction forecast generated based on multiple models; the model “T(1,A)” is used to generate the predicted value of “8” for the predicted day “0” that corresponds to the date of January 5 using a horizon of 1 and a class of A;[0042] the estimated headcount generated using the Erlang-C traffic modeling formula with the values from the piecewise forecast and displayed on a client computing device); and providing the predicted interval value for the future time interval ([0037] the process generates predictions used to generate estimated headcounts (i.e., forecast interval value) from models with multiple horizons and classes; [0058] FIG. 4 illustrates how the piecewise model selection displayed on the graphical user interface (400); [0064] the row (412) displays a piecewise prediction forecast generated based on multiple models; the model “T(1,A)” is used to generate the predicted value of “8” for the predicted day “0” that corresponds to the date of January 5 using a horizon of 1 and a class of A; [0042] the estimated headcount generated using the Erlang-C traffic modeling formula with the values from the piecewise forecast and displayed on a client computing device).
As to dependent Claim 10, Yocum, Law, and Mitra teach all the limitations of Claim 9. Yocum further teaches wherein one or more of scheduling one or more workers to work during the future time interval based on the predicted interval value and generating a hiring plan for the future time interval ([0037] the process generates predictions used to generate estimated headcounts from models with multiple horizons and classes; [0017] the headcount identifies how many representatives are needed to be able to handle the expected amount of calls; the headcount identifies the number of representatives needed over a period of time).
As to dependent Claim 11, Yocum, Law, and Mitra teach all the limitations of Claim 8. Yocum further teaches wherein the interval value is one of a communication volume, an average handling time, or a shrinkage ([0037] the process generates predictions used to generate estimated headcounts (i.e., forecast interval value) from models with multiple horizons and classes; [0017] the headcount identifies how many representatives are needed to be able to handle the expected amount of calls).
As to dependent Claim 14, Yocum, Law, and Mitra teach all the limitations of Claim 8. Yocum further teaches wherein the selection model comprises a decision tree ([0025] the model discovery service (equivalent to selection model) is a set of programs executing on the server computing system as part of the server application; the model discovery service identifies and trains the models used to generate predictions of time series variable values that may be used as inputs to generate headcount predictions; the model discovery service may score the models, select the best scoring models according to the future time range to forecast - thus, comprising a decision tree).
As to dependent Claim 15, Yocum, Law, and Mitra teach all the limitations of Claim 8. Yocum further teaches wherein for one or more classes of the set of classes, some or all of the subsequences of the plurality of subsequences of the time series data associated with the class are selected by an entity computing device ([0023] for a given schedule of individual days/ points in time for which predictions are to be made, the system may select the best model by forecast group, prediction horizon, and model class for a particular day; [0025]the model discovery service identifies and trains the models used to generate predictions of time series variable values that may be used as inputs to generate headcount predictions; the model discovery service may receive input data from the repository, train the models, score the models, select the best scoring models according to the future time range to forecast, and provide this selection of models to the demand prediction service; [0039] as a part of generating the values and for each day to be predicted, the model having the model horizon with a lowest value that is greater than or equal to an ordinal value of the day to be predicted may be selected for that day to generate a piecewise forecast for the time series variable being predicted (i.e., each class/ schedule is associated with plurality of subsequences of the time series data/ individual days comprising plurality of time intervals); [0043] multiple models may be trained having multiple model horizons and classes (i.e., set of classes); a model discovery service may have a model space of different models that are trained; the model space may include time series models that have different classes, different lags, and be of different types; the different classes of models may include in-season models, out-of-season models, and all-season models; [0037] the models may include lags from target time series; a target time series is a time series for which a model is used to generate predictions - thus, the subsequences of the time series data associated with the class of the model are selected by the system including model discovery service/ entity computing device).
As to dependent Claim 16, Yocum, Law, and Mitra teach all the limitations of Claim 8. Yocum further teaches wherein the classes in the set of classes comprise one or more of user classes or subsequence classes ([0023] for a given schedule of individual days/ points in time for which predictions are to be made, the system may select the best model by forecast group, prediction horizon, and model class for a particular day; [0020] a model class identifies, for a model, the class of the model; the classes may relate to seasonality, which is how the model performs with respect to a particular period of time, such as, Summer, Fall, tax season, etc.; example classes may include “all-season”, “in-season”, and “out-of-season” (equivalent to user class/ subsequence class)).
As to dependent Claim 17, Yocum, Law, and Mitra teach all the limitations of Claim 8. Law further teaches wherein the one or more motifs are identified by calculating a similarity score between different subsequences of the plurality of subsequences using a Euclidean distance metric ([0075] the candidate subsequence that was most similar to (has the shortest z-normalized Euclidean distance to) the subsequence 332 is chosen and a measure of its similarity to the subsequence 332 can be recorded within the matrix profile 320; [0070] the similarity measure used by the matrix profile 320 is a z-normalized Euclidean distance; [0168] time series motifs are approximately repeated subsequences found within a longer time series; the stump function is configured to take in any time series and compute the matrix profile along with the matrix profile indices; finding time series motifs with the matrix profile and matrix profile indices).
As to dependent Claim 21, Yocum, Law, and Mitra teach all the limitations of Claim 8. Law further teaches wherein identifying the one or more motifs comprises calculating a similarity score between different subsequences of the plurality of subsequences using a Euclidean distance metric ([0075] the candidate subsequence that was most similar to (has the shortest z-normalized Euclidean distance to) the subsequence 332 is chosen and a measure of its similarity to the subsequence 332 can be recorded within the matrix profile 320; [0070] the similarity measure used by the matrix profile 320 is a z-normalized Euclidean distance; [0168] time series motifs are approximately repeated subsequences found within a longer time series; the stump function is configured to take in any time series and compute the matrix profile along with the matrix profile indices; finding time series motifs with the matrix profile and matrix profile indices).
Claims 12-13 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Yocum in view of Law and Mitra, further in view of Saleh et al. (US 2023/0252267 A1 hereinafter Saleh).
As to dependent Claim 12, Yocum, Law, and Mitra teach all the limitations of Claim 8. However, Yocum, Law, and Mitra fail to expressly teach wherein receive a set of external data values, wherein each external value in the set of external values is associated with a time interval of the plurality of time intervals; create at least one class based on the set of external data values.
In the same field of endeavor, Saleh teaches wherein receive a set of external data values, wherein each external value in the set of external values is associated with a time interval of the plurality of time intervals ([0013] different data records may also include different feature data at different points in time or timesteps, such as data occurring on a particular day of week, a selected time period or time in the past, in a particular month or other time; training data may be used for these features, as well as additional external features that may be specifically selected for a time series forecasting task; external features may include those associated with additional customer data, fraud data, transaction data, a macro-economical feature, a trend in an e-commerce industry, a pandemic effect feature, a total payment volume migration feature, or a combination thereof; these features may provide additional time-based data that may have a temporal nature and affect predictive forecasting at future timesteps/times - thus, receiving a set of external data value/ external feature data associated with time interval); create at least one class based on the set of external data values ([0012] utilize a trained DNN model, such as one using an LSTM architecture, that includes an attention mechanism to focus on particular past timesteps and/or time-based/temporal data (i.e., time series data) when training the DNN to predict future timesteps and corresponding forecasts; [0013] different data records may also include different feature data at different points in time or timesteps, such as data occurring on a particular day of week, a selected time period or time in the past, in a particular month or other time (i.e., class); training data may be used for these features, as well as additional external features that may be specifically selected for a time series forecasting task; external features may include those associated with additional customer data, fraud data, transaction data, a macro-economical feature, a trend in an e-commerce industry, a pandemic effect feature, a total payment volume migration feature, etc.; these features may provide additional time-based data that may have a temporal nature and affect predictive forecasting at future timesteps/times; [0015] the DNN model trained for a predictive score, classification, or output variable associated with input features - thus, creating class/ classification in training the model based on the external data value/ external feature data associated with time interval ).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated wherein receive a set of external data values, wherein each external value in the set of external values is associated with a time interval of the plurality of time intervals; create at least one class based on the set of external data values, as taught by Saleh into Yocum, Law, and Mitra. Doing so would be desirable because using external features would provide additional context or factors that may assist in explaining past trends and predicting future forecasts for a particular variable and provide additional accuracy, temporal relevance, and/or feature dependence (Saleh [0021]).
As to dependent Claim 13, Yocum, Law, Mitra, and Saleh teach all the limitations of Claim 12. Saleh further teaches wherein training the selection model using the received time series data, the set of classes, the determined differences for each forecasting algorithm for each time interval of the plurality of time intervals of the time series data, and the external data ([0012] utilize a trained DNN model, such as one using an LSTM architecture, that includes an attention mechanism to focus on particular past timesteps and/or time-based/temporal data (i.e., time series data) when training the DNN to predict future timesteps and corresponding forecasts; [0013] different data records may also include different feature data at different points in time or timesteps, such as data occurring on a particular day of week, a selected time period or time in the past, in a particular month or other time (i.e., set of classes); training data may be used for these features, as well as additional external features that may be specifically selected for a time series forecasting task; external features may include those associated with additional customer data, fraud data, transaction data, a macro-economical feature, a trend in an e-commerce industry, a pandemic effect feature, a total payment volume migration feature, etc.; these features may provide additional time-based data that may have a temporal nature and affect predictive forecasting at future timesteps/times; [0021] when training the DNN model for predictive forecasts of a future traits, features, or variables, additional external features utilized to provide additional accuracy, temporal relevance, and/or feature dependence; external feature may therefore be used to provide additional context or factors that may assist in explaining past trends and predicting future forecasts for a particular variable; external features correspond to segments of temporal data that may affect the main input attribute or feature - thus, training the selection model using the received time series data, the set of classes and the external data). Mitra further teaches wherein training the selection model using the determined differences for each forecasting algorithm for each time interval of the plurality of time intervals of the time series data ([0057] the system trains all machine learning models on S11 and evaluates their performance on S12; based on the performance, the system selects top model from each pool; the top models from each pool (selected based on the evaluation in previous round) is trained on S21, and the system evaluates their performance on S22; based on the performance, the system selects the TOP model across all pools; [0058] the model generation module 130 (equivalent to selection model) trains 720 each machine learning based model from the pool P1 of machine learning based models using training subset S11; the model generation module 130 evaluates 730 each machine learning based model from the pool 610 of machine learning based models using the given model metric and using the test subset S12; [0060] the model generation module 130 evaluates 760 each machine learning based model from the pool P2 of machine learning based models using the given model metric and using training subset S22; [0061] the model generation module 130 selects the best model for forecasting time series data for the given application or based on a given model metric from the pool P2 of machine learning based models; [0036] a model metric represents a criterion for evaluating machine learning based models; a model metric represents a function or an expression used for determining a difference between data forecasted using a model with observed data (or labelled data) - thus, training using the determined differences for each model/ forecasting algorithm for each time interval of the plurality of time intervals of the time series data).
Regarding Claim 18, Yocum teaches a system comprising:
one or more processors ([0067] computing system include one or more computer processors); and
a computer-readable medium storing computer-executable instructions that when executed by the one or more processors cause the system ([0067] computing system include one or more computer processors, non-persistent storage (e.g., volatile memory, such as random access memory (RAM), cache memory), persistent storage (e.g., a hard disk, an optical drive such as a compact disk (CD) drive or digital versatile disk (DVD) drive, a flash memory, etc.); [0071] software instructions in the form of computer readable program code to perform embodiments of the invention may be stored on a non-transitory computer readable medium; the software instructions may correspond to computer readable program code that, when executed by a processor(s), is configured to perform one or more embodiments of the invention) to:
receive time series data, wherein the time series data comprises a plurality of time intervals and each time interval is associated with an interval value ([0013] models are trained to predict time series data, which may include the average handle time and call volume; [0015] the call volume identifies the number of phone calls that are expected to be received during a specified time interval; [0023] for a given schedule of individual days/ points in time for which predictions are to be made, the system may select the best model by forecast group, prediction horizon, and model class for a particular day; predictions are then made with a set of models to generate a piecewise forecast using the best model for a particular day; [0032] the repository (150) is a computing system that may include multiple computing devices in accordance with the computing system (500) and the nodes (522) and (524); the data in the repository (150) include multiple variables, parameters, historical time series data (i.e., time series data comprising plurality of time intervals), and forecasts that are generated (i.e., interval value) and used by the components of the system - thus, comprising plurality of time intervals and each time interval associated with forecasts/ interval value);
receive a plurality of forecasting algorithms ([0033] a user may control the model discovery service by selecting when the model discovery service is executed, selecting the model horizons used by the model discovery service, and maintaining the connections between the services of the server application and the databases of the repository that store the data used by the system; [0037] the process generates predictions used to generate estimated headcounts from models with multiple horizons and classes; in Step 202, models are obtained corresponding to model horizons; [0043] multiple models may be trained having multiple model horizons and classes; a model discovery service may have a model space of different models that are trained; the model space may include time series models that have different classes, different lags, and be of different types; the different classes of models may include in-season models, out-of-season models, and all-season models; [0076] the computing system or group of computing systems include functionality to perform a variety of operations disclosed);
create at least one class, wherein the at least one class is associated with a plurality of subsequences of the time series data, wherein each of the plurality of subsequences comprises a time interval of the plurality of time intervals (([0020] a model class identifies, for a model, the class of the model; the classes may relate to seasonality, which is how the model performs with respect to a particular period of time; example classes may include “all-season”, “in-season”, and “out-of-season” (i.e., set of classes); different classes may be trained and scored differently during training to identify the best performing model - thus, the class is trained/ created for the model; [0023] for a given schedule of individual days/ points in time for which predictions are to be made, the system may select the best model by forecast group, prediction horizon, and model class for a particular day; [0037] the models may include lags from target time series; a target time series is a time series for which a model is used to generate predictions; [0039] as a part of generating the values and for each day to be predicted, the model having the model horizon with a lowest value that is greater than or equal to an ordinal value of the day to be predicted may be selected for that day to generate a piecewise forecast for the time series variable being predicted - thus, the class/ particular period of time is associated with plurality of subsequences of the time series data/ individual days comprising plurality of time intervals); and
train each forecasting algorithm of the plurality of forecasting algorithms to predict the interval value using a portion of the time series data ([0036] training models and predicting call center volume (i.e., interval value); [0038] the time series variable values are predictions generated with trained models; the values may correspond to the days to be predicted; [0045] the training data used to train the models may include historical time series data for at least two years; additional data beyond two years may be included and more or less data may be used; the initial training of the models may use a subset of data (i.e., portion of time series data), such as the trailing two years of data; then after the models are selected, the selected models may be trained with the additional training data that was not used prior to model selection);
train a selection model using the received time series data, the at least one class ([0025] the model discovery service (equivalent to selection model) is a set of programs executing on the server computing system as part of the server application; the model discovery service identifies and trains the models used to generate predictions of time series variable values that may be used as inputs to generate headcount predictions; the model discovery service may receive input data from the repository, train the models, score the models, select the best scoring models according to the future time range to forecast, and provide this selection of models to the demand prediction service; [0043] multiple models may be trained having multiple model horizons and classes; a model discovery service may have a model space of different models that are trained; the model space may include time series models that have different classes, different lags, and be of different types; the different classes of models may include in-season models, out-of-season models, and all-season models; [0032] the repository is a computing system that may include multiple computing devices in accordance with the computing system and the nodes; the data in the repository include multiple variables, parameters, historical time series data (i.e., time series data), and forecasts that are generated and used by the components of the system - thus, the model discovery service including model space of different models is trained using time series data and classes).
However, Yocum fails to expressly teach wherein creating the at least one class comprises: displaying some or all of the plurality of time intervals in a user interface; receiving a selection of the plurality of time intervals associated with the at least one class from among the displayed some or all of the plurality of time intervals through the user interface; identifying one or more motifs within the selection of the plurality of time intervals by the computing device, wherein each motif of the one or more motifs represents a repeating pattern in the time series data; and creating the at least one class from the selection of the plurality of time intervals and the identified one or more motifs associated with the at least one class.
In the same field of endeavor, Law teaches wherein creating the at least one class comprises: displaying some or all of the plurality of time intervals in a user interface ([0097] in FIG. 6, a user interface (UI) 400 uses the matrix profile 320 and the matrix profile index 360 of FIG. 3 to allow a user to graphically investigate patterns of interest; x-axis of a data view 408 of the time-series data 300); receiving a selection of the plurality of time intervals associated with the at least one class from among the displayed some or all of the plurality of time intervals through the user interface ([0101] a slider bar 440 indicates which subsequence (i.e., time interval) is currently selected as the reference subsequence; the slider bar 440 may also permit the user to directly select the reference subsequence; [0102] the selected reference subsequence may be adjusted as the user moves their pointer across the data view 408 or the matrix profile view 404); identifying one or more motifs within the selection of the plurality of time intervals by the computing device, wherein each motif of the one or more motifs represents a repeating pattern in the time series data ([0112] boxes 528-1 and 528-2 indicate the locations of the selected subsequences (i.e., time intervals); as the box 528-1 is adjusted, the matrix profile index may be used to identify the closest corresponding subsequence and then the box 528-2 would be placed to indicate the corresponding subsequence; [0121] the pattern engine 624 calculates matrix profiles and matrix profile indices from selected time-series data; [0162] motif is a repeated sub-sequence of a time series; [0165] stumpy will be used to identify motifs (patterns) and anomalies/novelties with two different time series datasets; [0168] time series motifs are approximately repeated subsequences found within a longer time series; the stump function is configured to take in any time series and compute the matrix profile along with the matrix profile indices; finding time series motifs with the matrix profile and matrix profile indices - thus, matrix profile is used to identify motifs/ repeated subsequences from the selected subsequences/ time intervals in the time series); and creating the at least one class from the selection of the plurality of time intervals and the identified one or more motifs associated with the at least one class ([0115] if the user identifies a pattern of interest, the user can provide a name for that pattern at UI element 560 and then save the named pattern to a pattern library 564 using UI element 568).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated wherein creating the at least one class comprises: displaying some or all of the plurality of time intervals in a user interface; receiving a selection of the plurality of time intervals associated with the at least one class from among the displayed some or all of the plurality of time intervals through the user interface; identifying one or more motifs within the selection of the plurality of time intervals by the computing device, wherein each motif of the one or more motifs represents a repeating pattern in the time series data; and creating the at least one class from the selection of the plurality of time intervals and the identified one or more motifs associated with the at least one class, as taught by Law into Yocum. Doing so would be desirable because it would allow for identifying favorable trends and making favorable predictions (Law [0004]), thereby enhancing user experience.
However, Yocum and Law fail to expressly teach wherein for each time interval of the plurality of time intervals of the time series data that is not in the portion: for each forecasting algorithm of the plurality of forecasting algorithms: predict the interval value for the time interval using the forecasting algorithm; and determine a difference between the interval value associated with the time interval in the time series data and the predicted interval value for the time interval; training the selection model by the computing device using the determined differences for each forecasting algorithm for each time interval of the plurality of times intervals of the time series data.
In the same field of endeavor, Mitra teaches wherein for each time interval of the plurality of time intervals of the time series data that is not in the portion: for each forecasting algorithm of the plurality of forecasting algorithms: predicting the interval value for the time interval using the forecasting algorithm by the computing device ([0008] the system performs forecasting using the machine learning based model by determining a training data set based on the time series data; the training dataset includes a training subset and a test subset; the system trains the machine learning based model using the training subset of the training dataset and evaluates the machine learning based model using the test subset of the training dataset; [0047] the training dataset D includes a training subset T1 and a test subset T2; the model generation module 130 trains 420 the machine learning based model using the training subset T1; the model generation module 130 evaluates 430 the machine learning based model using the test subset T2; [0057] the system trains all machine learning models on S11 and evaluates their performance on S1); and determining a difference between the interval value associated with the time interval in the time series data and the predicted interval value for the time interval by the computing device ([0036] a model metric represents a criterion for evaluating machine learning based models; a model metric represents a function or an expression used for determining a difference between data forecasted using a model with observed data (or labelled data)); and training a selection model by the computing device using the determined differences for each forecasting algorithm for each time interval of the plurality of time intervals of the time series data ([0057] the system trains all machine learning models on S11 and evaluates their performance on S12; based on the performance, the system selects top model from each pool; the top models from each pool (selected based on the evaluation in previous round) is trained on S21, and the system evaluates their performance on S22; based on the performance, the system selects the TOP model across all pools; [0058] the model generation module 130 (equivalent to selection model) trains 720 each machine learning based model from the pool P1 of machine learning based models using training subset S11; the model generation module 130 evaluates 730 each machine learning based model from the pool 610 of machine learning based models using the given model metric and using the test subset S12; [0060] the model generation module 130 evaluates 760 each machine learning based model from the pool P2 of machine learning based models using the given model metric and using training subset S22; [0061] the model generation module 130 selects the best model for forecasting time series data for the given application or based on a given model metric from the pool P2 of machine learning based models; [0036] a model metric represents a criterion for evaluating machine learning based models; a model metric represents a function or an expression used for determining a difference between data forecasted using a model with observed data (or labelled data) - thus, training using the determined differences for each model/ forecasting algorithm for each time interval of the plurality of time intervals of the time series data).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated wherein for each time interval of the plurality of time intervals of the time series data that is not in the portion: for each forecasting algorithm of the plurality of forecasting algorithms: predicting the interval value for the time interval using the forecasting algorithm by the computing device; and determining a difference between the interval value associated with the time interval in the time series data and the predicted interval value for the time interval by the computing device; and training a selection model by the computing device using the determined differences for each forecasting algorithm for each time interval of the plurality of time intervals of the time series data, as taught by Mitra into Yocum and Law. Doing so would be desirable because it would enable the system to automatically select a model from different models based on previously known applications (Mitra [0070]).
However, Yocum, Law, and Mitra fail to expressly teach wherein receive a set of external data values, wherein each external value in the set of external values is associated with a time interval of the plurality of time intervals; create at least one class based on the set of external data values.
In the same field of endeavor, Saleh teaches wherein receive a set of external data values, wherein each external value in the set of external values is associated with a time interval of the plurality of time intervals ([0013] different data records may also include different feature data at different points in time or timesteps, such as data occurring on a particular day of week, a selected time period or time in the past, in a particular month or other time; training data may be used for these features, as well as additional external features that may be specifically selected for a time series forecasting task; external features may include those associated with additional customer data, fraud data, transaction data, a macro-economical feature, a trend in an e-commerce industry, a pandemic effect feature, a total payment volume migration feature, or a combination thereof; these features may provide additional time-based data that may have a temporal nature and affect predictive forecasting at future timesteps/times - thus, receiving a set of external data value/ external feature data associated with time interval); create at least one class based on the set of external data values ([0012] utilize a trained DNN model, such as one using an LSTM architecture, that includes an attention mechanism to focus on particular past timesteps and/or time-based/temporal data (i.e., time series data) when training the DNN to predict future timesteps and corresponding forecasts; [0013] different data records may also include different feature data at different points in time or timesteps, such as data occurring on a particular day of week, a selected time period or time in the past, in a particular month or other time (i.e., class); training data may be used for these features, as well as additional external features that may be specifically selected for a time series forecasting task; external features may include those associated with additional customer data, fraud data, transaction data, a macro-economical feature, a trend in an e-commerce industry, a pandemic effect feature, a total payment volume migration feature, etc.; these features may provide additional time-based data that may have a temporal nature and affect predictive forecasting at future timesteps/times; [0015] the DNN model trained for a predictive score, classification, or output variable associated with input features - thus, creating class/ classification in training the model based on the external data value/ external feature data associated with time interval).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated wherein receive a set of external data values, wherein each external value in the set of external values is associated with a time interval of the plurality of time intervals; create at least one class based on the set of external data values, as taught by Saleh into Yocum, Law, and Mitra. Doing so would be desirable because using external features would provide additional context or factors that may assist in explaining past trends and predicting future forecasts for a particular variable and provide additional accuracy, temporal relevance, and/or feature dependence (Saleh [0021]).
As to dependent Claim 19, Yocum, Law, Mitra, and Saleh teach all the limitations of Claim 18. Yocum further teaches wherein receive a request to forecast the interval value at a future time interval ([0025] select the best scoring models according to the future time range to forecast, and provide this selection of models to the demand prediction service; [0058] FIG. 4 illustrates how the piecewise model selection displayed on the graphical user interface (400); [0063] the row (410) displays the date for the predicted days; the dates range from January 5 to January 13 (i.e., future time interval); [0064] the row (412) displays a piecewise prediction forecast generated based on multiple models; [0037] the process generates predictions used to generate estimated headcounts (i.e., forecast interval value) from models with multiple horizons and classes); use the selection model to select a forecasting algorithm of the plurality of forecasting algorithms for the future time interval ([0023] for a given schedule of individual days/ points in time for which predictions are to be made, the system may select the best model by forecast group, prediction horizon, and model class for a particular day; [0025] the model discovery service may receive input data from the repository, train the models, score the models, select the best scoring models according to the future time range to forecast, and provide this selection of models to the demand prediction service; [0037] the process generates predictions used to generate estimated headcounts (i.e., forecast interval value) from models with multiple horizons and classes; [0058] FIG. 4 illustrates how the piecewise model selection displayed on the graphical user interface (400); [0064] the row (412) displays a piecewise prediction forecast generated based on multiple models; the model “T(1,A)” is used to generate the predicted value of “8” for the predicted day “0” that corresponds to the date of January 5 using a horizon of 1 and a class of A;[0042] the estimated headcount generated using the Erlang-C traffic modeling formula with the values from the piecewise forecast and displayed on a client computing device); use the selected forecasting algorithm to predict the interval value for the future time interval ([0023] for a given schedule of individual days/ points in time for which predictions are to be made, the system may select the best model by forecast group, prediction horizon, and model class for a particular day; [0025] select the best scoring models according to the future time range to forecast; [0037] the process generates predictions used to generate estimated headcounts (i.e., forecast interval value) from models with multiple horizons and classes; [0058] FIG. 4 illustrates how the piecewise model selection displayed on the graphical user interface (400); [0064] the row (412) displays a piecewise prediction forecast generated based on multiple models; the model “T(1,A)” is used to generate the predicted value of “8” for the predicted day “0” that corresponds to the date of January 5 using a horizon of 1 and a class of A;[0042] the estimated headcount generated using the Erlang-C traffic modeling formula with the values from the piecewise forecast and displayed on a client computing device); and provide the predicted interval value for the future time interval ([0037] the process generates predictions used to generate estimated headcounts (i.e., forecast interval value) from models with multiple horizons and classes; [0058] FIG. 4 illustrates how the piecewise model selection displayed on the graphical user interface (400); [0064] the row (412) displays a piecewise prediction forecast generated based on multiple models; the model “T(1,A)” is used to generate the predicted value of “8” for the predicted day “0” that corresponds to the date of January 5 using a horizon of 1 and a class of A; [0042] the estimated headcount generated using the Erlang-C traffic modeling formula with the values from the piecewise forecast and displayed on a client computing device).
As to dependent Claim 20, Yocum, Law, Mitra, and Saleh teach all the limitations of Claim 18. Yocum further teaches wherein each class in the set of classes comprise one or more of a user class or a subsequence class ([0023] for a given schedule of individual days/ points in time for which predictions are to be made, the system may select the best model by forecast group, prediction horizon, and model class for a particular day; [0020] a model class identifies, for a model, the class of the model; the classes may relate to seasonality, which is how the model performs with respect to a particular period of time, such as, Summer, Fall, tax season, etc.; example classes may include “all-season”, “in-season”, and “out-of-season” (equivalent to user class/ subsequence class)).
Response to Arguments
35 U.S.C. §103: In the remarks, applicant argues that the cited references do not teach the features "identifying one or more motifs within the selection of the plurality of time intervals by the computing device, wherein each motif of the one or more motifs,” as recited in amended independent claims 1, 8, and 18. The cited references do not teach the features recited in new dependent claim 21.
Applicant's arguments with respect to the 103 rejections have been considered, but are moot in view of new ground of rejection made under 35 U.S.C. § 103.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Applicant is required under 37 CFR § 1.111(c) to consider these references fully when responding to this action.
Kotorov et al. (US 2019/0188201 A1) teaches: displaying the time-series data on an interactive line chart component, selecting a time sequence subset from the time-series data displayed on the interactive line chart, converting data points from the selected time sequence subset into query parameters, generating a search query against the time-series data to retrieve a set of time sequences matching the query parameters, generating a similarity score for each member of the set of time sequences to the time sequence subset, and displaying a motif on the interactive line chart formed by the time sequences with a similarity score satisfying a threshold condition (see Abstract).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to REJI KARTHOLY whose telephone number is (571)272-3432. The examiner can normally be reached on Monday - Thursday from 7:30 am to 3:30 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch, can be reached at telephone number 571-272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center for authorized users only. Should you have questions about access to Patent Center, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/REJI KARTHOLY/Primary Examiner, Art Unit 2143
Prosecution Insights