DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Applicant filed a response dated March 27, 2026 in which claims 1, 8, 19, and 20 have been amended; and claims 2-3, 10, and 14-16 have been canceled. Therefore, claims 1, 4-9, 11-13, and 17-20 are currently pending in the application.
Priority
Application 18/228,971 was filed on August 1, 2023 and claims priority to GREECE 20230100614 July 25, 2023.
Examiner Request
The Applicant is requested to indicate where in the specification there is support for amendments to claims should Applicant amend. The purpose of this is to reduce potential 35 U.S.C. § 112(a) or § 112 1st paragraph issues that can arise when claims are amended without support in the specification. The Examiner thanks the Applicant in advance.
Claim Rejections - 35 USC § 101
35 U.S.C. § 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 4-9, 11-13, and 17-20 are rejected under 35 U.S.C. § 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more. (MPEP 2106). The claims are directed to a method, system, and apparatus which is one of the statutory categories of invention (Step 1: YES). The recitation of the claimed invention is analyzed as follows, in which the abstract elements are boldfaced.
Claim 1 recites the limitations of:
A method for determining clustering relevance in a volatile data environment and adjusting clustering composition for improved accuracy, the method comprising: receiving, from a plurality of source devices and via a network, time series data inputs forming a time series dataset, wherein the time series data inputs correspond to information in a volatile environment;
plotting, via a processor, the time series data inputs forming the time series dataset;
performing, via the processor, a plurality of pre-processing operations on the received time series data inputs, wherein the plurality of pre-processing operations includes identifying extra values and removing extraneous data and outliers;
plotting, via the processor, a linear graph based on the pre-processed time series data inputs;
generating, by executing a machine learning (ML) algorithm via the processor and based on the plotted linear graph, grand truth data values for a target period as an initial prediction, wherein the grand truth data values include an initial predicted data value outputted by the ML algorithm, wherein the prediction generated based on the linear graph is more accurate than a prediction generated using the time series data inputs;
clustering, via the processor, the plotted time series dataset for generating a plurality of data clusters, wherein the plurality of data clusters are of differing shapes and size, wherein each of the plurality of data clusters represent differing cluster characteristics,
wherein the clustering is performed based on data correlation of individual data values included in the time series dataset, and
wherein the clustering includes clustering of adjacent datapoints in the plotted time series dataset when the adjacent datapoints are located within a reference distance of one another;
training, a plurality of ML algorithms using the plurality of data clusters present in the time series dataset for which the linear graph was plotted, wherein each of the plurality of ML algorithms is trained using a different data cluster among the plurality of data clusters, wherein the plurality of ML algorithms form a managing ML algorithm model to be applied to a future time series dataset,
wherein applying the ML algorithm model to the future time series dataset includes selectively applying two or more ML algorithms among the plurality of ML algorithms,
wherein one or more of the plurality of ML algorithms include one or more sub-ML models, and
wherein each of the plurality of ML algorithms is trained separately and independently from one another as each of the plurality of data clusters has differing correlations from one another such that only select ML algorithms among the plurality of ML algorithms are retrained when new time series data inputs received modifies only select data clusters among the plurality of data clusters;
applying, by executing the managing ML algorithm model formed by the plurality of ML algorithms separately and independently trained based on the plurality of data clusters, to the time series dataset for predicting at least one future data value;
comparing, via the processor, differences between at least one grand truth data value among the grand truth data values for the target period and the at least one future data value for estimating clustering error at a future time interval,
wherein a divergence between the at least one future data value and the at least one grand truth data value indicates a presence of material error and incorrectness in the clustering, and
wherein an alignment between the at least one future data value and the at least one grand truth data value indicates an absence of the material error and correctness in the clustering;
determining, by the processor, the clustering error at a plurality of future time intervals;
determining, by the processor, average error values by dividing estimated clustering errors at the plurality of future time intervals by a number of predicted values;
plotting, by the processor, the average error values determined;
performing, by the processor, an error/correlation analysis based on the plotted average error values based on a magnitude of a respective error and proximate distance from other plotted average error values for determining whether a particular data value is correctly assigned to a data cluster among the plurality of data clusters, wherein the error/correlation analysis is achieved by:
retrieving a function of each data value in a cluster, computing a correlation of each pair of functions indicating how correlated two data values are, measuring the correlation through Pearson's coefficient, and based on the measured correlations, determining whether a data value correctly belongs to its assigned cluster or not;
adjusting, via the processor, composition of at least one of the plurality of data clusters
to increase intra-cluster correlation and to reduce out-of-cluster correlation based on the clustering error and
applying at least one corresponding ML algorithm among the plurality of ML algorithms to the adjusted composition of the at least one of the plurality of data clusters,
wherein the adjusting includes relocating one or more data values from one cluster to another cluster, removing one or more data values from at least one cluster, and setting the one or more removed data values in one or more clusters;
modifying weighting on clustering in response to the adjusting of the composition of the at least one of the plurality of data clusters; and
retraining, among the plurality of ML algorithms and based on the modified weighting on the clustering, only the select ML algorithms corresponding to the adjusted clusters among the plurality of data clusters.
The claim as a whole recites a method that, under its broadest reasonable interpretation, covers collecting, inputting, analyzing, and transmitting data to facilitate adjusting data correspondence with respect to time in data clusters of a dataset in a volatile data environment and adjusting composition of such clusters to reflect changing data correspondence, e.g., pricing in bond markets and stock markets. As disclosed in the specification, [0017] the time series dataset includes multiple bond prices. [0084] Examples of such may include generation of bond pricing or predicting stock pricing that account for unobservable environmental changes. … In an example, bond price outputs may represent bond prices that are adjusted to the new unobservable conditions (e.g., market conditions) with one or more explanations highlighting the reasons of pricing strategy modification.
This is a fundamental economic practice of a financial transaction; a commercial interaction, such as for business relations; and managing personal behavior or relationships or interactions between people, which are certain methods of organizing human activity.
The claims also recite the use of a machine learning (ML) algorithm, a managing ML algorithm model, a plurality of ML algorithms, and one or more sub-ML models to determine composition of data clusters based on an estimated clustering error. This is a mathematical calculation or concept.
In the alternative, the machine learning (ML) algorithm, a managing ML algorithm model, a plurality of ML algorithms, and one or more sub-ML models are considered a technology that is recited at a high level of generality and merely applied as a tool to implement the abstract idea.
Thus, the claims recite an abstract idea. (Step 2A, prong 1: YES).
Moreover, the judicial exception is not integrated into a practical application. Other than reciting “a plurality of source devices and via a network”, “a processor”, “machine learning (ML) algorithm,” “a managing ML algorithm model,” “a plurality of ML algorithms,” and “one or more sub-ML models” to perform the steps of “plotting”, “performing”, “generating”, “clustering”, “training”, “applying”, “comparing”, “determining”, “retrieving”, “adjusting”, and “modifying”, nothing in the claim elements preclude the steps from practically being a certain method for organizing human activity or mathematical calculation. The claim as a whole does not integrate the exception into a practical application. The claim merely describes how to generally “apply” the concept of collecting, inputting, analyzing, and transmitting data to facilitate adjusting data correspondence with respect to time in data clusters of a dataset in a volatile data environment and adjusting composition of such clusters to reflect changing data correspondence, e.g., pricing in bond markets and stock markets and completing mathematical calculations in a computer environment. The additional computer elements recited in the claim limitations are recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception utilizing generic computer components.
For example, the Specification at [0043] discloses “[0043] In a networked deployment, the computer system 102 may operate in the capacity of a server or as a client user computer in a server-client user network environment, a client user computer in a cloud computing environment, or as a peer computer system in a peer-to-peer (or distributed) network environment. The computer system 102, or portions thereof, may be implemented as, or incorporated into, various devices, such as a personal computer, a tablet computer, a set-top box, a personal digital assistant, a mobile device, a palmtop computer, a laptop computer, a desktop computer, a communications device, a wireless smart phone, a personal trusted device, a wearable device, a global positioning satellite (GPS) device, a web appliance, or any other machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while a single computer system 102 is illustrated, additional embodiments may include any collection of systems or sub-systems that individually or jointly execute instructions or perform functions. The term system shall be taken throughout the present disclosure to include any collection of systems or sub-systems that individually or jointly execute a set, or multiple sets, of instructions to perform one or more computer functions.”
Furthermore, the Specification at “[00101] More specifically, machine learning/artificial intelligence and pattern recognition may include supervised learning algorithms such as, for example, k-medoids analysis, regression analysis, decision tree analysis, random forest analysis, k-nearest neighbors analysis, logistic regression analysis, 5-fold cross-validation analysis, balanced class weight analysis, and the like. In another exemplary embodiment, machine learning analytical techniques may include unsupervised learning algorithms such as, for example, Apriori analysis, K-means clustering analysis, etc. In another exemplary embodiment, machine learning analytical techniques may include reinforcement learning algorithms such as, for example, Markov Decision Process analysis, and the like. [00102] In another exemplary embodiment, the ML or AI model may be based on a machine learning algorithm. The machine learning algorithm may include at least one from among a process and a set of rules to be followed by a computer in calculations and other problem-solving operations such as, for example, a linear regression algorithm, a logistic regression algorithm, a decision tree algorithm, and/or a Naive Bayes algorithm. [00103] In another exemplary embodiment, the ML or AI model may include training models such as, for example, a machine learning model which is generated to be further trained on additional data. Once the training model has been sufficiently trained, the training model may be deployed onto various connected systems to be utilized. In another exemplary embodiment, the training model may be sufficiently trained when model assessment methods such as, for example, a holdout method, a K-fold-cross-validation method, and a bootstrap method determine that at least one of the training model's least squares error rate, tine positive rate, true negative rate, false positive rate, and false negative rates are within predetermined ranges. [00104] In another exemplary embodiment, the training model may be operable, i.e., actively utilized by an organization, while continuing to be trained using new data. In another exemplary embodiment, the ML or AI models may be generated using at least one from among an artificial neural network technique, a decision tree technique, a support vector machines technique, a Bayesian network technique, and a genetic algorithms technique.”
Thus, the specification supports that general purpose computers or computer components are utilized to implement the steps of the abstract idea.
Merely implementing the abstract idea on a generic computer is not a practical application of the abstract idea. The claim as a whole, in viewing the additional elements both individually and in combination, does not integrate the judicial exception into a practical application. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea. (Step 2A prong two: No)
The claim does not include additional elements, when considered both individually and as an ordered combination, that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of using “a plurality of source devices and via a network”, “a processor”, “machine learning (ML) algorithm,” “a managing ML algorithm model,” “a plurality of ML algorithms,” and “one or more sub-ML models” to perform the steps of “plotting”, “performing”, “generating”, “clustering”, “training”, “applying”, “comparing”, “determining”, “retrieving”, “adjusting”, and “modifying”, amounts to no more than mere instructions to apply the exception using generic computer component. The claim merely describes how to generally “apply” the concept of collecting, inputting, analyzing, and transmitting data to facilitate adjusting data correspondence with respect to time in data clusters of a dataset in a volatile data environment and adjusting composition of such clusters to reflect changing data correspondence, e.g., pricing in bond markets and stock markets and completing mathematical calculations in a computer environment. Thus, even when viewed as a whole, nothing in the claim adds significantly more (i.e. an inventive concept) to the abstract idea. Such additional elements are determined to not contain an inventive concept according to MPEP 2106.05(f). It should be noted that (1) the “recitation of claim limitations that attempt to cover any solution to an identified problem with no restriction on how the result is accomplished and no description of the mechanism for accomplishing the result, does not provide significantly more because this type of recitation is equivalent to the words “apply it”, and (2) “Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice, commercial interaction, or managing personal behavior or relationships or interactions between people, mental process, or mathematical calculation) does not integrate a judicial exception into a practical application or provide significantly more”.
Claims 19 and 20 are substantially similar to claim 1, thus, they are rejected on similar grounds.
Claim 19 recites the additional elements of “A system to determine clustering relevance in a volatile data environment and adjust clustering composition for improved accuracy, the system comprising: a memory; a display; and a processor, wherein the system is configured to perform:”
Claim 20 recites the additional elements of “A non-transitory computer readable storage medium that stores a computer program for determining clustering relevance in a volatile data environment and adjusting clustering composition for improved accuracy, the computer program, when executed by a processor, causing a system to perform a plurality of processes comprising:”
Furthermore, while claim 4 recites the additional elements of “a combination of neural network models and linear models”, in view of the specification, as explained above, the broadest reasonable interpretation of these elements require a mathematical calculation. Therefore, the limitations fall into the mathematical concepts groupings of abstract ideas.
For similar reasons as explained above with regard to claim 1, under Step 2A, prong two, these additional elements are merely applying generic computer components to implement the abstract idea. Under Step 2B, when viewing the additional elements individually and in combination, the additional elements do not amount to an inventive concept amounting to significantly more than the judicial exception itself as the claimed computer-related technologies are mere tools for implementing the abstract idea as explained with regard to claim 1.
Dependent claims 4-9, 11-13, and 17-18 merely limit the abstract idea and do not recite any further additional elements beyond the cited abstract idea and the elements addressed above, thus, they do not amount to significantly more. The dependent claims are abstract for the reasons presented above because there are no additional elements that integrate the abstract idea into a practical application or are sufficient to amount to significantly more than the judicial exception when considered both individually and as an ordered combination. Thus, the dependent claims are directed to an abstract idea. (Step 2B: No)
Therefore, claims 1, 4-9, 11-13, and 17-20 are not patent-eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. §§ 102 and 103 (or as subject to pre-AIA 35 U.S.C. §§ 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4-9, 11, 13, 17, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Schwiep, U.S. Patent Application Publication Number 2022/0292308; Saha, U.S. Patent Application Publication Number 2023/0049418; in view of Yang, U.S. Patent Application Publication Number 2021/0003974; in view of Qiao, U.S. Patent Application Publication Number 2007/0097965; in view of Sridharan, U.S. Patent Application Publication Number 2020/0311749; in view of Yanchenko, U.S. Patent Application Publication Number 2024/0211835; in view of Webber, U.S. Patent Application Publication Number 2022/0004565.
As per claim 1,
Schwiep explicitly teaches:
A method for determining clustering relevance in a volatile data environment and adjusting clustering composition for improved accuracy, the method comprising: receiving, from a plurality of source devices and via a network, time series data inputs forming a time series dataset, wherein the time series data inputs correspond to information in a volatile environment;
(Schwiep US20220292308 at paras. 41-43, 92-94) (“[0044] Referring to FIG. 1, in certain examples, the systems and methods described herein provide a complete technological solution for a large-scale data science workflow that includes several independent modules or components for data processing and exploration, feature engineering and reduction, model development and selection, and model deployment and monitoring. In brief overview, the system 100 can include, interface, access, or otherwise use a data processing module 104. The data processing module 104 can be provided that ingests training data 102 (e.g., from one or more files and/or databases) and performs data processing and/or segmentation. Training data 102 can be referred to as first data or a first dataset. The system 100 can include, interface, access, or otherwise use a feature engineering module 106. The feature engineering module 106 can receive the processed/segmented data and perform feature engineering, feature reduction, and/or data partitioning. The system 100 can include, interface, access, or otherwise use a model development module 108. The features and partitioned data can be provided to the model development module 108 that develops and trains one or more predictive models. The system 100 can include, interface, access, or otherwise use a model management module 110. The model management module 110 can deploy the models for end users and can monitor model performance and output model results. The deployed models can receive new data 112 (prediction data) and make time series predictions based on the new data. New data 112 can be referred to as second data or a second dataset. Once the training data 102 and/or new data 112 is received by the system (e.g., fed over an API), the data can be ingested and processed automatically.")
plotting, via a processor, the time series data inputs forming the time series dataset;
(Schwiep US20220292308 at paras. 52-54) ("[0052] For example, the system 100 can provide, for display via a graphical user interface presented on a display device coupled to a computing device, a prompt 406 to split the first dataset by segments. The system 100 can receive, via the graphical user interface from the computing device, an indication to split the first dataset by segments. For example, the system 100 can receive the indication via a selection of button 402. The system 100 can split, responsive to the indication, the first dataset into segments. [0053] Additionally or alternatively, differences between such products or time series can be expressed in terms of seasonality (e.g., of single SKU sales). For example, FIG. 3 is a plot of a search engine trend for a query “buy beer” vs. “buy tea.” Unlike the buy tea curve, the buy beer curve has obvious spikes on the 5th and 6th of December, which correspond to Saturday and Sunday, respectively. In other words, beer sales may spike on weekends while tea sales may be relatively constant all week. Given the different characteristics for such time series (e.g., in terms of frequency content and/or magnitudes), it can be difficult to make accurate predictions for such time series using a single model. [0054] One reason for such difficulty can relate to feature engineering. For example, an individual series (e.g., for a single product) in a set of multiple time series (e.g., for different products) may need a unique seasonality handling (e.g., due to different seasonality periods or frequencies) and/or trend handling (e.g., a linear trend vs. a logistic trend). Another reason for the difficulty of developing a model for a variety of time series is that the model may try to learn each datapoint as accurately as it can, across multiple time series. The resulting model may learn an average effect rather than learning each individual series correctly.")
performing, via the processor, a plurality of pre-processing operations on the received time series data inputs, wherein the plurality of pre-processing operations includes identifying extra values and removing extraneous data and outliers;
(Schwiep US20220292308 at paras. 41-43, 92-94) (“[0042] As used herein, a “modeling blueprint” (or “blueprint”) refers to a computer-executable set of pre-processing operations, model-building operations, and postprocessing operations to be performed to develop a model based on the input data. Blueprints may be generated “on-the-fly” based on any suitable information including, without limitation, the size of the user data, features types, feature distributions, etc. Blueprints may be capable of jointly using multiple (e.g., all) data types, thereby allowing the model to learn the associations between image features, as well as between image and non-image features." "[0093] In various examples, the feature engineering module 106 can perform several data quality checks on datasets. The predictive power of resulting models can depend on a quality of the data/features used during training, thus the feature engineering module 106 can perform several time series specific data quality checks. For example, the feature engineering module 106 may look for any of the following data quality issues: inconsistent gaps between time steps in data (e.g., due to inconsistent measurement times or gaps in available data); lagged features that have already be derived by users (e.g., such features can be detected and flagged, marked as do-not-derive, or can be removed from the dataset); leading or trailing zeros (e.g., at a beginning or end of a column); a new series in validation data (e.g., a time series in validation data that was not present in training data). Results from the data quality checks can be displayed for the user, as shown in FIG. 17.")
plotting, via the processor, a linear graph based on the pre-processed time series data inputs;
(Schwiep US20220292308 at paras. 41-43, 92-94) ("[0092] A feature over time plot can be created for each feature, as shown in FIG. 16. [0093] In various examples, the feature engineering module 106 can perform several data quality checks on datasets. The predictive power of resulting models can depend on a quality of the data/features used during training, thus the feature engineering module 106 can perform several time series specific data quality checks. For example, the feature engineering module 106 may look for any of the following data quality issues: inconsistent gaps between time steps in data (e.g., due to inconsistent measurement times or gaps in available data); lagged features that have already be derived by users (e.g., such features can be detected and flagged, marked as do-not-derive, or can be removed from the dataset); leading or trailing zeros (e.g., at a beginning or end of a column); a new series in validation data (e.g., a time series in validation data that was not present in training data). Results from the data quality checks can be displayed for the user, as shown in FIG. 17.")
generating, by executing a machine learning (ML) algorithm via the processor and based on the plotted linear graph, grand truth data values for a target period as an initial prediction, wherein the grand truth data values include an initial predicted data value outputted by the ML algorithm, wherein the prediction generated based on the linear graph is more accurate than a prediction generated using the time series data inputs;
(Schwiep US20220292308 at paras. 39-46, 76-78, 92-95, 127-129) ("[0076] The feature engineering module 106 can also derive statistics in several rolling windows depending on FDW size (e.g., min, max, median, mean, and/or standard deviation) and/or latest and seasonal naive baseline features. Naive baseline features can be determined by selecting values from history to forecast future values, based on different strategies. For example, a naive latest prediction can use the latest history value to predict rows or values in the forecast window. Naive seasonal prediction can extract a previous season's target value in the history to predict values in the forecast window. For example, for a given Monday-Friday dataset, a naive latest prediction for a Monday can use a target value from a preceding Friday as the prediction for Monday. For a naive 7-day prediction, the feature engineering module 106 can use a target value from a previous Monday as the prediction for a next Monday. If a multiplicative or exponential trend is detected in the dataset or series, the naive prediction can be in log scale." "[0093] In various examples, the feature engineering module 106 can perform several data quality checks on datasets. The predictive power of resulting models can depend on a quality of the data/features used during training, thus the feature engineering module 106 can perform several time series specific data quality checks. For example, the feature engineering module 106 may look for any of the following data quality issues: inconsistent gaps between time steps in data (e.g., due to inconsistent measurement times or gaps in available data); lagged features that have already be derived by users (e.g., such features can be detected and flagged, marked as do-not-derive, or can be removed from the dataset); leading or trailing zeros (e.g., at a beginning or end of a column); a new series in validation data (e.g., a time series in validation data that was not present in training data). Results from the data quality checks can be displayed for the user, as shown in FIG. 17." "[0128] In various examples, modeling accuracy for such time series can be improved significantly through the use of the segmented modeling approach described herein. Segmented modeling can use a combination of models that are trained on data series having different characteristics. For example, one model may be used to make predictions for time series that have a weekly seasonality (e.g., patterns in the time series repeat each week), and another model may be used to make predictions for time series that have a monthly seasonality. Additional models can be used to make predictions for other time series, such as zero-inflated time series and/or time series having extreme magnitudes.")
clustering, via the processor, the plotted time series dataset for generating a plurality of data clusters, wherein the plurality of data clusters are of differing shapes and size, wherein each of the plurality of data clusters represent differing cluster characteristics,
(Schwiep US20220292308 at paras. 8-10, 89-91) ("[0090] Referring again to FIG. 2, once the feature derivation process has been performed, the feature engineering module 106 can begin a feature reduction process in which one or more features that are redundant or not impactful can be removed or ignored from further consideration. Feature reduction can be performed using proprietary algorithms based on GBT (Gradient Boosting-based Tree) and/or SHAP (SHapley Additive exPlanations) algorithms. Feature reduction can involve fitting a Light Gradient Boosting Machine model (LGBM) on all derived features (e.g., in a matrix having a size or shape of num_observations×num_features). A tree explainer (e.g., TreeExplainer from a SHAP library) can be fit using the LGBM model. Next, using the tree explainer, shapley values can be obtained for each observation from the data. The resulting shapley values can be provided in a matrix (e.g., having a size or shape of num_observations×num_features). Each value in the shapley values matrix can provide a measure of how much a feature contributes to a prediction. For example, the shapley value in element (i,j) of the matrix can be a contribution that feature j has on the prediction for observation (i). Next, a mean(abs(shapley values)) is calculated (e.g., in a direction along the rows) to obtain a single vector (of shape num_features) in which each value j is a mean contribution that feature j has on all the predictions. The vector can then be normalized to sum to 1 by dividing each value by a total sum of all elements in this vector, and the normalized vector can be sorted (e.g., from largest to smallest). Finally, features corresponding to large values in the normalized vector can be retained and all other features can be eliminated or removed from further consideration. For example, the top features (e.g., features with the highest contributions) that sum to a specified threshold (e.g., 0.98) can be retained and used as a reduced set of features.")
training, a plurality of ML algorithms using the plurality of data clusters present in the time series dataset for which the linear graph was plotted, wherein each of the plurality of ML algorithms is trained using a different data cluster among the plurality of data clusters, wherein the plurality of ML algorithms form a managing ML algorithm model to be applied to a future time series dataset,
(Schwiep US20220292308 at paras. 8-10, 43-45) (“[0008] In one aspect, the subject matter of this disclosure relates to a computer-implemented method. The method includes: providing a training dataset including a first plurality of time series having a plurality of time series characteristics, at least one time series from the first plurality of time series having a unique combination of the time series characteristics; identifying a plurality of predictive models for the first plurality of time series based on the time series characteristics; training the plurality of predictive models using the training dataset to generate a combined model including the plurality of predictive models; providing a prediction dataset including a second plurality of time series corresponding to the first plurality of time series; and using the combined model to make predictions for the second plurality of time series.")
wherein one or more of the plurality of ML algorithms include one or more sub-ML models, and
(Schwiep US20220292308 at paras. 8-10, 43-45) (“[0008] In one aspect, the subject matter of this disclosure relates to a computer-implemented method. The method includes: providing a training dataset including a first plurality of time series having a plurality of time series characteristics, at least one time series from the first plurality of time series having a unique combination of the time series characteristics; identifying a plurality of predictive models for the first plurality of time series based on the time series characteristics; training the plurality of predictive models using the training dataset to generate a combined model including the plurality of predictive models; providing a prediction dataset including a second plurality of time series corresponding to the first plurality of time series; and using the combined model to make predictions for the second plurality of time series.")
wherein each of the plurality of ML algorithms is trained separately and independently from one another as each of the plurality of data clusters has differing correlations from one another such that only select ML algorithms among the plurality of ML algorithms are retrained when new time series data inputs received modifies only select data clusters among the plurality of data clusters;
(Schwiep US20220292308 at paras. 8-10, 43-45) (“[0009] In certain examples, the plurality of time series characteristics can include seasonality, frequency content, average target values, maximum target values, minimum target values, a number of zero values, or any combination thereof. Identifying the plurality of predictive models can include mapping each time series in the first plurality of time series to at least one model in the plurality of predictive models. Mapping each time series can include: clustering the time series in the first plurality of time series into a plurality of groups, wherein each group in the plurality of groups includes common or similar characteristics from the time series characteristics; and assigning each group to a respective model from the plurality of predictive models. Using the plurality of predictive models can include mapping each time series in the second plurality of time series to at least one model in the plurality of predictive models.")
applying, by executing the managing ML algorithm model formed by the plurality of ML algorithms separately and independently trained based on the plurality of data clusters, to the time series dataset for predicting at least one future data value;
(Schwiep US20220292308 at paras. 8-10, 43-45) (“[0009] In certain examples, the plurality of time series characteristics can include seasonality, frequency content, average target values, maximum target values, minimum target values, a number of zero values, or any combination thereof. Identifying the plurality of predictive models can include mapping each time series in the first plurality of time series to at least one model in the plurality of predictive models. Mapping each time series can include: clustering the time series in the first plurality of time series into a plurality of groups, wherein each group in the plurality of groups includes common or similar characteristics from the time series characteristics; and assigning each group to a respective model from the plurality of predictive models. Using the plurality of predictive models can include mapping each time series in the second plurality of time series to at least one model in the plurality of predictive models.")
determining, by the processor, the clustering error at a plurality of future time intervals;
(Schwiep US20220292308 at paras. 47-49, 109-112) (“[0048] FIG. 2 is a screenshot of an example graphical user interface illustrating the data ingestion and segmentation capability. In one example, an ingested dataset can include one or more series identifiers (IDs) that identify one or more time series in the dataset. Additionally or alternatively, the data processing module 104 can determine whether data is regularly spaced over time (e.g., all observations appear at regularly spaced time steps). A regularly spaced time series can be or include, for example, a series in which each observation appears at the same time unit, such as every hour, every day, or every week. The data processing module 104 can accommodate series that have semi-regular spacing, such as time series that have inconsistent time units and/or occasional spacing errors. This can be useful with weekly data, for example, when observations may be recorded on different days of week, depending on the week. In such instances, the spacing may vary between 1 and 7 days and the dataset can still be detected or processed as though the dataset has regular spacing. For example, time values can be adjusted to make the spacing regular. Additionally or alternatively, interpolation can be performed to adjust variable values to reflect a uniform spacing.")
determining, by the processor, average error values by dividing estimated clustering errors at the plurality of future time intervals by a number of predicted values;
(Schwiep US20220292308 at paras. 47-49, 109-112) (“[0110] FIG. 22 includes a plot of model accuracy over time. The plot can allow a user to validate how well a model fits actual values in validation sets over time. Accuracy over time can be computed per series and per forecast distance and can provide an ability to validate individual time series (e.g., for products and/or product categories) and individual prediction horizons. FIG. 22 shows the accuracy over time plot for the model recommended in FIG. 18 (using the RMSE metric). In general, the model predicts values that are close to a mean target value but misses many of the peak values, which may not be ideal for the use case." "[0112] Accuracy over time (AOT) plots can also be used to view predictions vs. actuals for different backtests and/or series types (e.g., SKUs, categories, departments, etc.), explore different prediction horizons (e.g., forecast window lengths), and/or assess whether the model is overfitting or underfitting. For example, FIG. 24 is an accuracy over time plot on training data for one of the XGB models lower on the leaderboard. The results in the figure indicate that the model is underfitting the data.")
plotting, by the processor, the average error values determined;
(Schwiep US20220292308 at paras. 47-49, 109-112) (“[0110] FIG. 22 includes a plot of model accuracy over time. The plot can allow a user to validate how well a model fits actual values in validation sets over time. Accuracy over time can be computed per series and per forecast distance and can provide an ability to validate individual time series (e.g., for products and/or product categories) and individual prediction horizons. FIG. 22 shows the accuracy over time plot for the model recommended in FIG. 18 (using the RMSE metric). In general, the model predicts values that are close to a mean target value but misses many of the peak values, which may not be ideal for the use case." "[0112] Accuracy over time (AOT) plots can also be used to view predictions vs. actuals for different backtests and/or series types (e.g., SKUs, categories, departments, etc.), explore different prediction horizons (e.g., forecast window lengths), and/or assess whether the model is overfitting or underfitting. For example, FIG. 24 is an accuracy over time plot on training data for one of the XGB models lower on the leaderboard. The results in the figure indicate that the model is underfitting the data.")
applying at least one corresponding ML algorithm among the plurality of ML algorithms to the adjusted composition of the at least one of the plurality of data clusters,
(Schwiep US20220292308 at paras. 4-17, 101-103) ("[0004] In various examples, the time series in the training data and/or the prediction data can be clustered into a plurality of groups having common time series characteristics. For example, two or more time series in the training data that have similar time series characteristics (e.g., similar seasonalities and/or magnitudes) can be added to the same group. Additionally or alternatively, the training data and/or the prediction data can be provided with identifiers (e.g., provided by users) that can be used to form the groups and/or identify the time series that belong in each group. Each group in the plurality of groups can be assigned to a respective model from the plurality of predictive models." "[0009] In certain examples, the plurality of time series characteristics can include seasonality, frequency content, average target values, maximum target values, minimum target values, a number of zero values, or any combination thereof. Identifying the plurality of predictive models can include mapping each time series in the first plurality of time series to at least one model in the plurality of predictive models. Mapping each time series can include: clustering the time series in the first plurality of time series into a plurality of groups, wherein each group in the plurality of groups includes common or similar characteristics from the time series characteristics; and assigning each group to a respective model from the plurality of predictive models. Using the plurality of predictive models can include mapping each time series in the second plurality of time series to at least one model in the plurality of predictive models." "[0016] The plurality of characteristics can include at least one of seasonality, frequency content, average target values, maximum target values, minimum target values, or a number of zero values. The one or more processors can map each time series in the plurality of time series to at least one model in the plurality of models to select the plurality of models. The one or more processors can cluster the time series in the plurality of time series into a plurality of groups. Each group in the plurality of groups comprises common or similar characteristics from the characteristics. The one or more processors can assign each group to a respective model from the plurality of models to select the plurality of models." “[0102] In certain examples, after the best models have been identified and trained for a dataset, the model development module 108 can create an average blender of a small subset (e.g., 3 total) of the best models and/or can select a best model that will be recommended for deployment. The recommended model is preferably one of the best models from the leaderboard. In FIG. 18, for example, the recommended model is a temporal hierarchical model on a reduced features list. The temporal hierarchical model can utilize a temporal hierarchical modeler with an Elastic Net modeler for a first stage and XGBoost for a second stage. The model can implement a two-level temporal hierarchical model, as follows: the first stage fits the average target aggregated to a coarse time scale one level above the time step for the dataset; and the second stage fits an allocation of the average aggregated target to each time step. The net prediction in this example can be a predicted aggregated average multiplied by the predicted allocation. The best model in the figure in terms of prediction accuracy is the second model in the leaderboard (e.g., an AVG Blender model); however, this is not the recommended model because it is not as fast (e.g., longer prediction times). In some examples, the model development module 108 can recommend models that are fast, in addition to being accurate. Blender models (e.g., models trained using features from multiple feature lists) are generally slower and, as a result, may not be the best model. Once a model has been recommended, the recommended model can be retrained on a full set of training data (e.g., for all features in the feature list for the model). The retrained model may then be ready for deployment. At this point, the user can deploy the model or may choose a different model (e.g., based on a manual model selection process).")
retraining, among the plurality of ML algorithms [and based on the modified weighting on the clustering, only the select ML algorithms] corresponding to the adjusted clusters among the plurality of data clusters.
(Schwiep US20220292308 at paras. 4-17, 101-103) ("[0004] In various examples, the time series in the training data and/or the prediction data can be clustered into a plurality of groups having common time series characteristics. For example, two or more time series in the training data that have similar time series characteristics (e.g., similar seasonalities and/or magnitudes) can be added to the same group. Additionally or alternatively, the training data and/or the prediction data can be provided with identifiers (e.g., provided by users) that can be used to form the groups and/or identify the time series that belong in each group. Each group in the plurality of groups can be assigned to a respective model from the plurality of predictive models." "[0009] In certain examples, the plurality of time series characteristics can include seasonality, frequency content, average target values, maximum target values, minimum target values, a number of zero values, or any combination thereof. Identifying the plurality of predictive models can include mapping each time series in the first plurality of time series to at least one model in the plurality of predictive models. Mapping each time series can include: clustering the time series in the first plurality of time series into a plurality of groups, wherein each group in the plurality of groups includes common or similar characteristics from the time series characteristics; and assigning each group to a respective model from the plurality of predictive models. Using the plurality of predictive models can include mapping each time series in the second plurality of time series to at least one model in the plurality of predictive models." "[0010] An aspect of this technical solution is directed to a system. The system can include one or more processors, coupled to memory. The one or more processors can identify a first dataset that includes a plurality of time series having a plurality of characteristics. A first time series of the plurality of time series can include one or more characteristics of the plurality of characteristics that are different from characteristics of a second time series of the plurality of time series. The one or more processors can select, based at least in part on the plurality of characteristics, a plurality of models. The one or more processors can train, via machine learning, the plurality of models with the first dataset. The one or more processors can generate a model based at least in part on a combination of the plurality of models. The one or more processors can deploy the model to output one or more predictions responsive to a second dataset. The second dataset can be different from the first dataset and can have at least one of the plurality of characteristics." "[0017] An aspect of this technical solution is directed to a method. The method can be performed by one or more processors, coupled to memory. The method can include the one or more processors identifying a first dataset with a plurality of time series having a plurality of characteristics. A first time series of the plurality of time series can include one or more characteristics of the plurality of characteristics that are different from characteristics of a second time series of the plurality of time series. The method can include the one or more processors selecting, based at least in part on the plurality of characteristics, a plurality of models. The method can include the one or more processors training, via machine learning, the plurality of models with the first dataset. The method can include the one or more processors generating a model based at least in part on a combination of the plurality of models. The method can include the one or more processors deploying the model to output one or more predictions responsive to a second dataset. The second dataset can be different from the first dataset and have at least one of the plurality of characteristics." “[0102] In certain examples, after the best models have been identified and trained for a dataset, the model development module 108 can create an average blender of a small subset (e.g., 3 total) of the best models and/or can select a best model that will be recommended for deployment. The recommended model is preferably one of the best models from the leaderboard. In FIG. 18, for example, the recommended model is a temporal hierarchical model on a reduced features list. The temporal hierarchical model can utilize a temporal hierarchical modeler with an Elastic Net modeler for a first stage and XGBoost for a second stage. The model can implement a two-level temporal hierarchical model, as follows: the first stage fits the average target aggregated to a coarse time scale one level above the time step for the dataset; and the second stage fits an allocation of the average aggregated target to each time step. The net prediction in this example can be a predicted aggregated average multiplied by the predicted allocation. The best model in the figure in terms of prediction accuracy is the second model in the leaderboard (e.g., an AVG Blender model); however, this is not the recommended model because it is not as fast (e.g., longer prediction times). In some examples, the model development module 108 can recommend models that are fast, in addition to being accurate. Blender models (e.g., models trained using features from multiple feature lists) are generally slower and, as a result, may not be the best model. Once a model has been recommended, the recommended model can be retrained on a full set of training data (e.g., for all features in the feature list for the model). The retrained model may then be ready for deployment. At this point, the user can deploy the model or may choose a different model (e.g., based on a manual model selection process).")
Schwiep does not explicitly teach, however, Saha does teach:
wherein the clustering is performed based on data correlation of individual data values included in the time series dataset, and
(Saha US20230049418 at paras. 3-5, 63-66) ("[0003] Some embodiments involve improving the quality of information output by a machine learning model by minimizing correlation of features in training data used to train the machine learning model. Highly correlated features can inadvertently impact a machine learning model’s output as adjustments to one of the correlated features can affect the impact of the other correlated feature(s). By minimizing how correlated the features are in training data used to train the machine learning model, the trained machine learning model may reduce or even eliminate the effects of multicollinearity on the machine learning model’s output. [0004] In some embodiments, datasets including a plurality of features may be obtained. Correlation scores may be computed based on the datasets, where the correlation scores indicate a correlation between features. Based on the computed correlation scores, a plurality of feature clusters can be generated, where features in each feature cluster lack correlation with features included in other feature clusters. A machine learning model may be selected based on a set of input features associated with the machine learning model and the feature clusters. The set of input features may represent features that are capable of being used as input for the machine learning model. A subset of the obtained datasets may be selected based on the set of input features associated with the machine learning model, and training data for training the machine learning model may be generated based on the selected subset of datasets." "[0063] In some embodiments, computer system 102 may be configured to compute a CoV to identify volatility in time series data. As an example, with reference to FIG. 9A, plot 900 may represent time series data 902 over a rolling window of 12 months. The use of 12 months as the rolling window is merely exemplary, and other temporal durations may be used as the rolling window (e.g., 1 month, 3 months, 6 months, 24 months, etc.). For each rolling window, a CoV may be computed for the time series data included in that portion. As an example, window 904 of plot 900 may represent one 12 month window within time series data 902. Within the time period of window 904, the standard deviation of the data may be 95.7 and the mean value of the data may be 1905.8. The units of the data in the example are arbitrary, and may represent a value relevant to the model to be trained. For example, the value may refer to a pixel value for a computer vision model, a credit limit for a financial model, or other values. The CoV is defined as the ratio of the standard deviation to the mean. Therefore, within window 904, the CoV is approximately 0.0502.)
retrieving a function of each data value in a cluster, computing a correlation of each pair of functions indicating how correlated two data values are, measuring the correlation through Pearson's coefficient, and based on the measured correlations, determining whether a data value correctly belongs to its assigned cluster or not;
(Saha US20230049418 at paras. 34-44) ("[0034] For each feature included in the datasets, a correlation score may be computed. The correlation score may indicate how well correlated each feature is to each other feature. In some embodiments, the correlation score may be represented using a Pearson score, computed using a Pearson Correlation Coefficient. In some embodiments, the correlation score may be represented using a Spearman Coefficient score computed using a Spearman Correlation Coefficient. In some embodiments, the correlation score may be represented using a Variance Inflation Factor (VIF) computed by determining how much a variance of an estimated regression coefficient is increased due to collinearity. [0035] The features included in the datasets may be clustered together based on the correlation scores. For example, features that have a high correlation score, indicating the features are strongly correlated, are placed in a same cluster, whereas features that are not correlated with one another are located in different clusters. Therefore, each cluster can contain features correlated with one another, and which lack correlation with features included in each other cluster. In some embodiments, the features included within a given cluster may also be ranked based on the respective correlation scores. For example, two features that have a high correlation score may be ranked higher than two features that have a low correlation score." "[0040] Scoring subsystem 112 may be configured to compute a correlation score for each of the features represented by the obtained datasets. The correlation score may indicate how well correlated each feature is to each other feature. In some embodiments, the correlation score may be represented using a Pearson score, computed using a Pearson Correlation Coefficient. In some embodiments, the correlation score may be represented using a Spearman Coefficient score computed using a Spearman Correlation Coefficient. In some embodiments, the correlation score may be represented using a Variance Inflation Factor (VIF) computed by determining how much a variance of an estimated regression coefficient is increased due to collinearity." "[0043] Returning to FIG. 1, clustering subsystem 114 may cluster features together based on the correlation scores. The result of the clustering may be feature clusters each including features that are uncorrelated with one another. As an example, with reference to FIG. 4, clustering subsystem 114 may be configured to identify, for each feature, one or more correlated features at 402. Correlated features may be identified based on correlation scores 310, which indicate whether one feature is correlated to another feature. Identifying correlated features may include identifying how strongly or weakly two features are to one another, but, in particular, identifying whether two features are correlated.")
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Schwiep and Saha, because it allows for improving the quality of information output by a machine learning model by minimizing correlation of features in training data used to train the machine learning model. (Saha at Abstract and paras. 2-5).
Schwiep and Saha do not explicitly teach, however, Yang does teach:
wherein the clustering includes clustering of adjacent datapoints in the plotted time series dataset when the adjacent datapoints are located within a reference distance of one another;
(Yang US20210003974 at Fig. 2, paras. 97-99, 142-144, 162-164) ("[0163] Learned knowledge via embedding may be transferred by following the above methods. After embeddings are created for nodes, these learnings may be transferred from one problem component to another. For instance, if a new ML device 2330 is introduced where there is no prior operational guide attached to it, a set of operational rules may be suggested based on the same idea of vertex nomination where the new nodes' nearest neighbors in the embedding space may be identified and used to suggest the rules. The nearest neighbor may be found using a distance function, for example, a Euclidian or Cosine distance. Thus, if the example ML device 2330 is closest in distance to dishwashers in the embedding space, the ML device 2330 might be suggested to operate using the same parameters as dishwashers in the environment. In sample embodiments, a human would have the power to change or discard the suggestion. Also, the distance function may be applied in higher dimensions based on the representation of the data by the ML device 2330 as embeddings are usually more than 3 dimensions.")
wherein a divergence between the at least one future data value and the at least one grand truth data value indicates a presence of material error and incorrectness in the clustering, and
(Yang US20210003974 at paras. 137-139) ("[0138] In further sample embodiments, the active and passive monitoring methods may be combined to detect faults and discrepancies in the system. For example, the passive monitor status may serve as the grand truth, and the difference between the active monitor status and the passive monitor status may be used to detect faulty devices. Discrepancies of collected data between different devises also may be used to detect anomalies.")
wherein an alignment between the at least one future data value and the at least one grand truth data value indicates an absence of the material error and correctness in the clustering;
(Yang US20210003974 at paras. 137-139) ("[0138] In further sample embodiments, the active and passive monitoring methods may be combined to detect faults and discrepancies in the system. For example, the passive monitor status may serve as the grand truth, and the difference between the active monitor status and the passive monitor status may be used to detect faulty devices. Discrepancies of collected data between different devises also may be used to detect anomalies.")
performing, by the processor, an error/correlation analysis based on the plotted average error values based on a magnitude of a respective error and proximate distance from other plotted average error values for determining whether a particular data value is correctly assigned to a data cluster among the plurality of data clusters, wherein the error/correlation analysis is achieved by:
(Yang US20210003974 at Fig. 2, paras. 97-99, 142-144) ("[0098] FIG. 6 illustrates the respective stacks of the LSTM 500, including an LSTM autoencoder 600 that receives past values as inputs 610, processes the inputs 610 through multiple LSTM layers 620, and feeds the results into a second LSTM stack 630 that averages incoming vectors and concatenates with new inputs 640 using multiple LSTM layers 650. Input features to the model are the capability clearing price, performance clearing price, mileage ratio, performance score of previous period, and mean of the regulation signal amplitude. It will be appreciated by those skilled in the art that autoencoders are neural networks used to learn an efficient representation (encoding) of the data by solving f(x)=x, wheref (x), or the encoded data, has less dimensionality than x. ")
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Schwiep, Saha, and Yang, because it allows for an improved system utilizing a machine learning model to provide active and passive monitoring methods may be combined to detect faults and discrepancies in the system and discrepancies of collected data between different devises also may be used to detect anomalies. (Yang at Abstract and paras. 3-19, 138).
Schwiep, Saha, and Yang do not explicitly teach, however, Qiao does teach:
adjusting, via the processor, composition of at least one of the plurality of data clusters
(Qiao US20070097965 at paras. 12-18) ("[0012] In one embodiment, the cross validation module iteratively cross validates and optimizes the data conversions. The cross validation module may direct the cluster module to relocate data clusters, direct the conversion selection module to select an alternate conversion function, direct the norm selection module to select an alternate norm function, and direct the weight module to modify the weight assigned to a data cluster for each conversion function in order to optimize the data interpolations." "[0018] An interpolation module converts each data cluster. In a certain embodiment, a cross validation module iteratively employs a function such as a visual color difference equation as an error control. The cross validation module may optimize conversion parameters including the center selection/clustering algorithm, conversion function, norm functions, and weights. In one embodiment, the validation module cross validates and optimizes the conversion parameters off-line.")
wherein the adjusting includes relocating one or more data values from one cluster to another cluster, removing one or more data values from at least one cluster, and setting the one or more removed data values in one or more clusters;
(Qiao US20070097965 at paras. 12-18) ("[0012] In one embodiment, the cross validation module iteratively cross validates and optimizes the data conversions. The cross validation module may direct the cluster module to relocate data clusters, direct the conversion selection module to select an alternate conversion function, direct the norm selection module to select an alternate norm function, and direct the weight module to modify the weight assigned to a data cluster for each conversion function in order to optimize the data interpolations." "[0018] An interpolation module converts each data cluster. In a certain embodiment, a cross validation module iteratively employs a function such as a visual color difference equation as an error control. The cross validation module may optimize conversion parameters including the center selection/clustering algorithm, conversion function, norm functions, and weights. In one embodiment, the validation module cross validates and optimizes the conversion parameters off-line.")
modifying weighting on clustering in response to the adjusting of the composition of the at least one of the plurality of data clusters; and
(Qiao US20070097965 at paras. 12-18) ("[0012] In one embodiment, the cross validation module iteratively cross validates and optimizes the data conversions. The cross validation module may direct the cluster module to relocate data clusters, direct the conversion selection module to select an alternate conversion function, direct the norm selection module to select an alternate norm function, and direct the weight module to modify the weight assigned to a data cluster for each conversion function in order to optimize the data interpolations." "[0018] An interpolation module converts each data cluster. In a certain embodiment, a cross validation module iteratively employs a function such as a visual color difference equation as an error control. The cross validation module may optimize conversion parameters including the center selection/clustering algorithm, conversion function, norm functions, and weights. In one embodiment, the validation module cross validates and optimizes the conversion parameters off-line.")
and based on the modified weighting on the clustering, only the select ML algorithms...
(Qiao US20070097965 at paras. 12-18) ("[0012] In one embodiment, the cross validation module iteratively cross validates and optimizes the data conversions. The cross validation module may direct the cluster module to relocate data clusters, direct the conversion selection module to select an alternate conversion function, direct the norm selection module to select an alternate norm function, and direct the weight module to modify the weight assigned to a data cluster for each conversion function in order to optimize the data interpolations." "[0018] An interpolation module converts each data cluster. In a certain embodiment, a cross validation module iteratively employs a function such as a visual color difference equation as an error control. The cross validation module may optimize conversion parameters including the center selection/clustering algorithm, conversion function, norm functions, and weights. In one embodiment, the validation module cross validates and optimizes the conversion parameters off-line.")
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Schwiep, Saha, Yang, and Qiao, because it allows for an improved system that interpolates data using variable granularity sub-sets and interpolation functions optimized for each sub-set. Beneficially, such an apparatus, system, and method would improve the effectiveness of data interpolation. (Qiao at Abstract and paras. 2-22).
Schwiep, Saha, Yang, and Qiao do not explicitly teach, however, Sridharan does teach:
wherein applying the ML algorithm model to the future time series dataset includes selectively applying two or more ML algorithms among the plurality of ML algorithms,
(Sridharan US20200311749 at paras. 17, 23, 39, 42) ("[0017] A system, method, and computer-readable medium are disclosed for using machine learning to improve forecasting of market behavior. Certain aspects of the invention reflect an appreciation that is common for many organizations to employ various forecasting approaches in an attempt to cope with the uncertainty of the future. Likewise, various aspects of the invention reflect an appreciation that while certain forecasting approaches may inherently be based upon qualitative experience, knowledge and judgment, their accuracy typically involves some degree of quantitative, statistical, and predictive analysis. Certain aspects of the invention recognize that there are a substantial number of different types of quantitative, statistical, and predictive analyses that may be applied to historical data in order to predict future market behavior. Such analyses may include application of predictive models to historical data." "[0023] To address shortcomings of current prediction models, certain embodiments of the present invention recognize that a stacked prediction model may be generated from multiple prediction models. In certain embodiments, multiple prediction models are applied to time-series sequenced data representing market behavior for an item of interest. In certain embodiments, the item of interest may be the sales, purchases, production data, etc. of a product or service. Certain embodiments involve determining the error associated with the application of the multiple prediction models to the time-series sequenced data. In certain embodiments, a stacked prediction model is generated. In certain embodiments, the stacked prediction model uses at least two of the multiple prediction models, which includes a weighting factor for each of the prediction models used in the stacked prediction model. In certain embodiments, the weighting factor for each of the prediction models in the stacked prediction model employs an inversion of the respective error in the prediction model. In certain embodiments, the stacked prediction model is used to forecast market behavior, consolidating the benefits associated with multiple prediction models while ensuring that prediction models of the stack having lower error rates are given a higher weighting than prediction models of the stack having higher error rates. In certain embodiments, the stacked prediction model is more capable of accurately predicting market behavior than a single deterministic model. In certain embodiments, the stacked prediction model may be automatically reevaluated in response to changes in the error in the stacked prediction model over time." "[0042] In certain embodiments, at least two prediction models are selected for inclusion in the stacked prediction model at operation 408. In certain embodiments, more than two prediction models may be included in the stacked prediction model. In certain embodiments, the determination as to which prediction models are included in the stacked prediction model may be determined based on the error associated with each prediction model, where only prediction models having errors below a certain level are included in the stacked prediction model. In certain embodiments, the error associated with the prediction model may be presented to a user, who may then select which of the prediction models are to be included in the stacked prediction model."
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Schwiep, Saha, Yang, Qiao, and Sridharan, because it allows for an improved system for generating and using a stacked prediction model to forecast market behavior. (Sridharan at Abstract and paras. 1-6).
Schwiep, Saha, Yang, Qiao, and Sridharan do not explicitly teach, however, Yanchenko does teach:
comparing, via the processor, differences between at least one grand truth data value among the grand truth data values for the target period and the at least one future data value for estimating clustering error at a future time interval,
(Yanchenko US20240211835 at paras. 73-81) ("[0073] The feature extractor 414 may extract forecast data properties, such as the same type of features as computed for the time series but applied to the forecast over time, as the forecasts also comprise time series (i.e., the forecast at each time points). This can include the same sort of features, such as the mean and standard deviation of each forecast for a given time series in some specified time window relative to the current point. This also includes some additional features unique to the forecast time series, such as statistics of the forecast values for future time widows relative to a time point, and statistics and measurements of the forecast errors in different time periods. For example, a feature could be the mean squared error of a forecast for a time series in the past week which is obtained by measuring the difference between the forecast and the ground truth time series values for each time point in the past week relative to a given time point and taking the mean of the square of these values. Another feature could be the average bias of the forecast in a past time window such is in the past few days or past month, where the bias is the difference between the forecast and the true values. These can also be extended across the hierarchy of time series in different ways, such as providing individual features for each time series, or summary statistics of features or aggregations across different hierarchy groups or the entire hierarchy (such as average error in the last week across all time series in the hierarchy). Similar features can also be computed and included for different reconciliation models applied to the data." "[0075] In performing this fitting of the reconciliation model(s) 432 to the training datasets and forecast data, the ML reconciliation adaptation computer model 420 may predict, for each group of historical data and/or time points in the historical data, a corresponding reconciliation model 432 to be used to reconcile forecast data. The dynamical reconciliation tool/model 440 may then apply the predicted reconciliation model 432 to the forecast data corresponding to those time periods to generate reconciled forecast data 455. The reconciled forecast data 455 is analyzed by the machine learning logic 425 including a comparison to the corresponding ground truth data 406 for that time period to generate a loss or error in accordance with a loss formula. The machine learning logic 425 determines, based on the loss/error, an adjustment to operational parameters of the ML reconciliation adaptation computer model 420 to reduce the loss/error. This process is continued until the loss/error is below a given threshold, a predetermined number of iterations or epochs are executed, or other convergence criteria are satisfied. Once the model 420 reaches convergence, the resulting reconciliation models for the input features representing structural changes in hierarchical datasets and/or forecast data are identified and mapped to these input features in the reconciliation model mapping data structure 445." "[0081] As shown in FIG. 4B, in accordance with a first process flow 470, in a first operation 471, historical hierarchical data and forecast data are obtained, and a dataset is generated in which each data point comprises a combination of the historical hierarchical data and the forecast data for the horizon. The historical hierarchical data may be considered ground truth data or “actual” data, i.e., values that are actually observed. Thus, historically, for each point in time, there may be forecast for values at that point in time based on data collected up to some previous time, as well as the actual time series values at that time, which were the values that were being predicted by the forecast. Thus, the historical hierarchical data may include both the previous forecasts and the actual observed data, such that the actual observed data may serve as a ground truth for machine learning training.")
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Schwiep, Saha, Yang, Qiao, Sridharan, and Yanchenko, because it allows for an improved data processing apparatus and method and more specifically to an improved computing tool and improved computing tool operations/functionality for automatically and dynamically adapting a hierarchical reconciliation process for time series forecasting. (Yanchenko at Abstract and paras. 1-4).
Schwiep, Saha, Yang, Qiao, Sridharan, and Yanchenko do not explicitly teach, however, Webber does teach:
to increase intra-cluster correlation and to reduce out-of-cluster correlation based on the clustering error and
(Webber US20220004565 at paras. 38-40) ("[0039] In addition to identifying differences with the published clustering, confidence metrics can be computed for each proposed cluster, based both on the strength of the internal connections within the cluster, and on the strength of inter-cluster connections, similar to the intra-cluster and inter-cluster metrics used in the Davies-Bouldin index. For example, pairwise correlation can be calculated based on similarity of record fields of all pairs within a cluster (intra-cluster), and the same calculation may be run on across different clusters (inter-cluster). These confidences may be calculated during creation of the proposed clustering. Preconfigured, or dynamically adjustable, thresholds may be set to identify clusters likely warranting manual review. These metrics are computed as part of cluster-level metadata when preparing for review. Clusters with poor intra-cluster scores indicate a weak cluster that has a risk of incorrectly containing unrelated records. Clusters with poor inter-cluster scores indicate a weak separation from other clusters that has a risk of incorrectly separating related records. Clusters that show a significant worsening in either score are likely to warrant manual review. Clusters with high scores are less likely to warrant manual review, and clusters with confidence over particular thresholds can be excluded from review.")
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Schwiep, Saha, Yang, Qiao, Sridharan, Yanchenko, and Webber, because it outlines tools and methods to represent proposed changes to clusterings for ease of review, as well as tools to help subject matter experts identify clusters that warrant review versus those that do not. These tools make overall assessment of proposed clustering changes and targeted curation practical at large scale. Use of these tools and method enables efficient data management operations when dealing with extreme scale, such as where entity resolution involves clusterings created from data sources involving millions of entities. (Webber at Abstract and paras. 3-8).
As per claim 4,
Schwiep explicitly teaches:
wherein the managing ML model includes a combination of neural network models and linear models.
(Schwiep US20220292308 at paras.54, 73, 96-99, 123, 145) (“[0008] In one aspect, the subject matter of this disclosure relates to a computer-implemented method. The method includes: providing a training dataset including a first plurality of time series having a plurality of time series characteristics, at least one time series from the first plurality of time series having a unique combination of the time series characteristics; identifying a plurality of predictive models for the first plurality of time series based on the time series characteristics; training the plurality of predictive models using the training dataset to generate a combined model including the plurality of predictive models; providing a prediction dataset including a second plurality of time series corresponding to the first plurality of time series; and using the combined model to make predictions for the second plurality of time series." "[0097] Referring again to FIG. 1, in various examples, the model development module 108 can use a multi-stage autopilot (also referred to as simply “autopilot”) to automatically select or pick one or more models (e.g., a best model) from a variety of available models. According to a “no free lunch theorem,” there is generally no best model that is going to be accurate or successful for every possible dataset. Some datasets may be dominated by linear models and others may be dominated by trees or neural nets; however, training every possible model is time consuming and usually not possible. Multi-stage autopilot can be used to efficiently try a large number models and pick the best one.")
As per claim 5,
Schwiep does not explicitly teach, however, Saha does teach:
wherein a data value included in the time series dataset is determined not to belong in an assigned cluster when an error value of the data value is above a reference threshold.
(Saha US20230049418 at paras. 33-36, 44-48) ("[0034] For each feature included in the datasets, a correlation score may be computed. The correlation score may indicate how well correlated each feature is to each other feature. In some embodiments, the correlation score may be represented using a Pearson score, computed using a Pearson Correlation Coefficient. In some embodiments, the correlation score may be represented using a Spearman Coefficient score computed using a Spearman Correlation Coefficient. In some embodiments, the correlation score may be represented using a Variance Inflation Factor (VIF) computed by determining how much a variance of an estimated regression coefficient is increased due to collinearity. [0035] The features included in the datasets may be clustered together based on the correlation scores. For example, features that have a high correlation score, indicating the features are strongly correlated, are placed in a same cluster, whereas features that are not correlated with one another are located in different clusters. Therefore, each cluster can contain features correlated with one another, and which lack correlation with features included in each other cluster. In some embodiments, the features included within a given cluster may also be ranked based on the respective correlation scores. For example, two features that have a high correlation score may be ranked higher than two features that have a low correlation score." "In some embodiments, at 406, a cluster plot 410 may be generated including a visual depiction of the feature clusters identified from clustering data 408. As an example, with reference to FIG. 5 , cluster plot 500 includes two feature clusters, feature cluster A and feature cluster B. Each of feature clusters A and B include one or more features. The features included in feature cluster A each have some correlation to one another based on a correlation score for each of those features being non-zero. Similarly, the features included in feature cluster B each have some correlation to one another based on a correlation score for each of those features being non-zero. On the other hand, features included within feature cluster A are determined to have no correlation, or less than a threshold amount of correlation (e.g., correlation score being less than a threshold correlation score), with respect to features included within feature cluster B. For instance, a correlation score SA1,A2 between two features, feature A1 and A2, both included within feature cluster A, may be greater than a threshold correlation score. As an example, SA1,A2 may be greater than Sthreshold, where Sthreshold = 0.2 or less, 0.1 or less, 0.01 or less, 0.001 or less, or other values, or 0. As another example, correlation score S1,2 may be non-zero (e.g., SA1,A2 ≠ 0). However, a correlation score between two features, feature A1 of feature cluster A and feature B1 of feature cluster B may be less than a threshold correlation score. As an example, SA1,B1 may be less than Sthreshold, As another example, correlation score SA1,B1 may be zero (e.g., SA1,B1 = 0).")
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Schwiep, Saha, Yang, Qiao, Sridharan, Yanchenko, and Webber, because it allows for improving the quality of information output by a machine learning model by minimizing correlation of features in training data used to train the machine learning model. (Saha at Abstract and paras. 2-5).
As per claim 6,
Schwiep does not explicitly teach, however, Saha does teach:
wherein a data value included in the time series dataset is determined not to belong in an assigned cluster when a distance of an error value of the data value is beyond a reference distance from an error value of another data value in the assigned cluster.
(Saha US20230049418 at paras. 33-36, 44-48) ("[0034] For each feature included in the datasets, a correlation score may be computed. The correlation score may indicate how well correlated each feature is to each other feature. In some embodiments, the correlation score may be represented using a Pearson score, computed using a Pearson Correlation Coefficient. In some embodiments, the correlation score may be represented using a Spearman Coefficient score computed using a Spearman Correlation Coefficient. In some embodiments, the correlation score may be represented using a Variance Inflation Factor (VIF) computed by determining how much a variance of an estimated regression coefficient is increased due to collinearity. [0035] The features included in the datasets may be clustered together based on the correlation scores. For example, features that have a high correlation score, indicating the features are strongly correlated, are placed in a same cluster, whereas features that are not correlated with one another are located in different clusters. Therefore, each cluster can contain features correlated with one another, and which lack correlation with features included in each other cluster. In some embodiments, the features included within a given cluster may also be ranked based on the respective correlation scores. For example, two features that have a high correlation score may be ranked higher than two features that have a low correlation score." "In some embodiments, at 406, a cluster plot 410 may be generated including a visual depiction of the feature clusters identified from clustering data 408. As an example, with reference to FIG. 5 , cluster plot 500 includes two feature clusters, feature cluster A and feature cluster B. Each of feature clusters A and B include one or more features. The features included in feature cluster A each have some correlation to one another based on a correlation score for each of those features being non-zero. Similarly, the features included in feature cluster B each have some correlation to one another based on a correlation score for each of those features being non-zero. On the other hand, features included within feature cluster A are determined to have no correlation, or less than a threshold amount of correlation (e.g., correlation score being less than a threshold correlation score), with respect to features included within feature cluster B. For instance, a correlation score SA1,A2 between two features, feature A1 and A2, both included within feature cluster A, may be greater than a threshold correlation score. As an example, SA1,A2 may be greater than Sthreshold, where Sthreshold = 0.2 or less, 0.1 or less, 0.01 or less, 0.001 or less, or other values, or 0. As another example, correlation score S1,2 may be non-zero (e.g., SA1,A2 ≠ 0). However, a correlation score between two features, feature A1 of feature cluster A and feature B1 of feature cluster B may be less than a threshold correlation score. As an example, SA1,B1 may be less than Sthreshold, As another example, correlation score SA1,B1 may be zero (e.g., SA1,B1 = 0).")
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Schwiep, Saha, Yang, Qiao, Sridharan, Yanchenko, and Webber, because it allows for improving the quality of information output by a machine learning model by minimizing correlation of features in training data used to train the machine learning model. (Saha at Abstract and paras. 2-5).
As per claim 7,
Schwiep does not explicitly teach, however, Saha does teach:
wherein a data value included in the time series dataset is determined to belong in an assigned cluster when (i) when an error value of the data value is at or below a reference threshold, and (ii) a distance of the error value of the data value is within a reference distance from an error value of another data value in the assigned cluster.
(Saha US20230049418 at paras. 33-36, 44-48) ("[0034] For each feature included in the datasets, a correlation score may be computed. The correlation score may indicate how well correlated each feature is to each other feature. In some embodiments, the correlation score may be represented using a Pearson score, computed using a Pearson Correlation Coefficient. In some embodiments, the correlation score may be represented using a Spearman Coefficient score computed using a Spearman Correlation Coefficient. In some embodiments, the correlation score may be represented using a Variance Inflation Factor (VIF) computed by determining how much a variance of an estimated regression coefficient is increased due to collinearity. [0035] The features included in the datasets may be clustered together based on the correlation scores. For example, features that have a high correlation score, indicating the features are strongly correlated, are placed in a same cluster, whereas features that are not correlated with one another are located in different clusters. Therefore, each cluster can contain features correlated with one another, and which lack correlation with features included in each other cluster. In some embodiments, the features included within a given cluster may also be ranked based on the respective correlation scores. For example, two features that have a high correlation score may be ranked higher than two features that have a low correlation score." "In some embodiments, at 406, a cluster plot 410 may be generated including a visual depiction of the feature clusters identified from clustering data 408. As an example, with reference to FIG. 5 , cluster plot 500 includes two feature clusters, feature cluster A and feature cluster B. Each of feature clusters A and B include one or more features. The features included in feature cluster A each have some correlation to one another based on a correlation score for each of those features being non-zero. Similarly, the features included in feature cluster B each have some correlation to one another based on a correlation score for each of those features being non-zero. On the other hand, features included within feature cluster A are determined to have no correlation, or less than a threshold amount of correlation (e.g., correlation score being less than a threshold correlation score), with respect to features included within feature cluster B. For instance, a correlation score SA1,A2 between two features, feature A1 and A2, both included within feature cluster A, may be greater than a threshold correlation score. As an example, SA1,A2 may be greater than Sthreshold, where Sthreshold = 0.2 or less, 0.1 or less, 0.01 or less, 0.001 or less, or other values, or 0. As another example, correlation score S1,2 may be non-zero (e.g., SA1,A2 ≠ 0). However, a correlation score between two features, feature A1 of feature cluster A and feature B1 of feature cluster B may be less than a threshold correlation score. As an example, SA1,B1 may be less than Sthreshold, As another example, correlation score SA1,B1 may be zero (e.g., SA1,B1 = 0).")
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Schwiep, Saha, Yang, Qiao, Sridharan, Yanchenko, and Webber, because it allows for improving the quality of information output by a machine learning model by minimizing correlation of features in training data used to train the machine learning model. (Saha at Abstract and paras. 2-5).
As per claim 8,
Schwiep explicitly teaches:
further comprising: plotting, via the processor, the grand truth data values for the target period.
(Schwiep US20220292308 at paras. 47-49, 109-112) (“[0110] FIG. 22 includes a plot of model accuracy over time. The plot can allow a user to validate how well a model fits actual values in validation sets over time. Accuracy over time can be computed per series and per forecast distance and can provide an ability to validate individual time series (e.g., for products and/or product categories) and individual prediction horizons. FIG. 22 shows the accuracy over time plot for the model recommended in FIG. 18 (using the RMSE metric). In general, the model predicts values that are close to a mean target value but misses many of the peak values, which may not be ideal for the use case." "[0112] Accuracy over time (AOT) plots can also be used to view predictions vs. actuals for different backtests and/or series types (e.g., SKUs, categories, departments, etc.), explore different prediction horizons (e.g., forecast window lengths), and/or assess whether the model is overfitting or underfitting. For example, FIG. 24 is an accuracy over time plot on training data for one of the XGB models lower on the leaderboard. The results in the figure indicate that the model is underfitting the data.")
As per claim 9,
Schwiep does not explicitly teach, however, Saha does teach:
further comprising: outputting, via the processor, at least one of: an explanation of why the composition of at least one of the plurality of data clusters was adjusted, a new clustering strategy with a stronger weight on new data characteristics, and new data values based on the adjusting of the composition of at least one of the plurality of data clusters.
(Saha US20230049418 at paras. 21-24) ("[0022] As an example, a machine learning model may take inputs and provide outputs. In some embodiments, the outputs may be fed back to the machine learning model as inputs to train the machine learning model (e.g., alone or in conjunction with user indications of the accuracy of the outputs, labels associated with the inputs, or with other reference feedback information). In some embodiments, the machine learning model may update its configurations (e.g., weights, biases, or other parameters) based on its assessment of its predictions (e.g., the outputs) and reference feedback information (e.g., user indication of accuracy, reference labels, or other information). In some embodiments, where the machine learning model is a neural network, connection weights may be adjusted to reconcile differences between the neural network’s prediction and the reference feedback. Some embodiments include one or more neurons (or nodes) of the neural network requiring that their respective errors be sent backward through the neural network to facilitate the update process (e.g., backpropagation of error). Updates to the connection weights may, for example, be reflective of the magnitude of error propagated backward after a forward pass has been completed. In this way, for example, the machine learning model may be trained to generate better predictions.")
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Schwiep, Saha, Yang, Qiao, Sridharan, Yanchenko, and Webber, because it allows for improving the quality of information output by a machine learning model by minimizing correlation of features in training data used to train the machine learning model. (Saha at Abstract and paras. 2-5).
As per claim 11,
Schwiep does not explicitly teach, however, Saha does teach:
further comprising: modifying a weight for the clustering based on the adjusting of the composition of at least one of the plurality of data clusters.
(Saha US20230049418 at paras. 21-24) ("[0022] As an example, a machine learning model may take inputs and provide outputs. In some embodiments, the outputs may be fed back to the machine learning model as inputs to train the machine learning model (e.g., alone or in conjunction with user indications of the accuracy of the outputs, labels associated with the inputs, or with other reference feedback information). In some embodiments, the machine learning model may update its configurations (e.g., weights, biases, or other parameters) based on its assessment of its predictions (e.g., the outputs) and reference feedback information (e.g., user indication of accuracy, reference labels, or other information). In some embodiments, where the machine learning model is a neural network, connection weights may be adjusted to reconcile differences between the neural network’s prediction and the reference feedback. Some embodiments include one or more neurons (or nodes) of the neural network requiring that their respective errors be sent backward through the neural network to facilitate the update process (e.g., backpropagation of error). Updates to the connection weights may, for example, be reflective of the magnitude of error propagated backward after a forward pass has been completed. In this way, for example, the machine learning model may be trained to generate better predictions.")
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Schwiep, Saha, Yang, Qiao, Sridharan, Yanchenko, and Webber, because it allows for improving the quality of information output by a machine learning model by minimizing correlation of features in training data used to train the machine learning model. (Saha at Abstract and paras. 2-5).
As per claim 13,
Schwiep does not explicitly teach, however, Saha does teach:
wherein the adjusting indicates a change in correlation of the time series dataset over time.
(Saha US20230049418 at paras. 21-24) ("[0022] As an example, a machine learning model may take inputs and provide outputs. In some embodiments, the outputs may be fed back to the machine learning model as inputs to train the machine learning model (e.g., alone or in conjunction with user indications of the accuracy of the outputs, labels associated with the inputs, or with other reference feedback information). In some embodiments, the machine learning model may update its configurations (e.g., weights, biases, or other parameters) based on its assessment of its predictions (e.g., the outputs) and reference feedback information (e.g., user indication of accuracy, reference labels, or other information). In some embodiments, where the machine learning model is a neural network, connection weights may be adjusted to reconcile differences between the neural network’s prediction and the reference feedback. Some embodiments include one or more neurons (or nodes) of the neural network requiring that their respective errors be sent backward through the neural network to facilitate the update process (e.g., backpropagation of error). Updates to the connection weights may, for example, be reflective of the magnitude of error propagated backward after a forward pass has been completed. In this way, for example, the machine learning model may be trained to generate better predictions." "[0024] Datasets 202 may be pulled from dataset database 132 responsive to a data pull request, such as an API call. After obtaining datasets 202, process 200 may include a quality check 210 that parses datasets 202 and identifies the different types of features included therein. For example, datasets 202 may include metadata indicating the different types of features. In some embodiments, quality check 210 may also identify redundant variables. A redundant variable refers to a variable that can appear multiple times within datasets 202 but refers to a same entity or feature type. For example, the feature “account identifier” may appear multiple times within datasets 202, each being associated with other, different features. For instance, the feature “account identifier” may be associated with both of the features “FICO score” and “credit limit.” Quality check 210 may identify these instances of redundant features, and in some embodiments, flag the particular feature that is redundant. Additionally, quality check 210 may be configured to identify issues associated with some or all of the features included in datasets 202. In some embodiments, variable level monitoring (VLM) may be performed for all of the features, which may also be referred to herein interchangeably as “variables,” to identify any issues that may be present. Quality check 210 may execute a package of quality check programs to identify various issues that may be present within datasets 202. As an example, quality check 210 may determine whether a null set is present within datasets 202 with respect to any of the features expected to be included within datasets 202. Additionally, quality check 210 may generate plots of each feature over time to determine changes that may have occurred to the feature over a given time range of datasets 202. In some embodiments, quality check 210 may be configured to filter out features that may have issues (e.g., abnormal distribution over time, null set, etc.).")
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Schwiep, Saha, Yang, Qiao, Sridharan, Yanchenko, and Webber, because it allows for improving the quality of information output by a machine learning model by minimizing correlation of features in training data used to train the machine learning model. (Saha at Abstract and paras. 2-5).
As per claim 17,
Schwiep does not explicitly teach, however, Saha does teach:
wherein each data value included in the time series dataset includes a plurality of dimensions utilized in the clustering.
(Saha US20230049418 at paras. 46-48) ("[0047] In some embodiments, a size, shape, color, or other manner of display, of each of feature clusters A and B may be determined based on a number of features included within that feature cluster. For example, the greater the number of features included within a given feature cluster, the greater the size of that feature cluster within the cluster plot.")
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Schwiep, Saha, Yang, Qiao, Sridharan, Yanchenko, and Webber, because it allows for improving the quality of information output by a machine learning model by minimizing correlation of features in training data used to train the machine learning model. (Saha at Abstract and paras. 2-5).
Claims 19 and 20 are substantially similar to claims 1, thus, they are rejected on similar grounds.
Claims 12 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Schwiep, U.S. Patent Application Publication Number 2022/0292308; Saha, U.S. Patent Application Publication Number 2023/0049418; in view of Yang, U.S. Patent Application Publication Number 2021/0003974; in view of Qiao, U.S. Patent Application Publication Number 2007/0097965; in view of Sridharan, U.S. Patent Application Publication Number 2020/0311749; in view of Yanchenko, U.S. Patent Application Publication Number 2024/0211835; in view of Webber, U.S. Patent Application Publication Number 2022/0004565; in view of Magdelinic, U.S. Patent Application Publication Number 2020/0167869.
As per claim 12,
Schwiep, Saha, Yang, Qiao, Sridharan, Yanchenko, and Webber do not explicitly teach, however, Magdelinic does teach:
wherein the time series dataset includes a plurality of bond prices.
(Magdelinic US20200167869 at paras. 164-166) ("[0165] Predictive issuance analytics engine 38 sources raw trading and fundamental data via automated scripts executed at predetermined intervals e.g. every 24 hours (step 4000). The data sources include Thomson Reuters (primary and secondary bond issuance and trading levels, secondary pricing data, outstanding securities, historical bond issuance), S&P Global Market Intelligence (company fundamental data), DBRA, S&P, Moody's, Fitch (company ratings, company credit rating, and macro market data). Thomson Reuters (company sector information), SEDAR (Canada), EDGAR (USA), public filings (prospectus filings), Central Banks/Treasuries, public sources (Macro Market Data), including various other sources. This raw data is pre-processed (step 4002) and the trading data and fundamental data is structured and mapped to the appropriate issuer ID, and stored in databases 27 (step 4004). The data is systematically scrubbed for anomalies and null values. Finally, a set of key input factors are generated based on the raw input. These include but are not limited to factors that measure recent issuance, issuance frequency, maturity schedule gap, propensity for specific tenors (step 4006). These factors are divided between sector and company specific and are used as inputs to the machine learning models.")
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Schwiep, Saha, Yang, Qiao, Sridharan, Yanchenko, Webber, and Magdelinic, because it allows for improved methods for supporting multiple functions such as communication, information management, deal execution, stakeholder collaboration, pricing calculation, securities offering and issuance, and analytics for issuers, investors, and dealers and computing device to measure best-fit correlations with respect to a company's fundamental valuation and secondary market pricing for the company's at least one financial instrument across sector peers and markets conditions and generate an at least one financial instrument pricing output, in real-time. (Magdelinic at Abstract and paras. 2-20).
As per claim 18,
Schwiep, Saha, Yang, Qiao, Sridharan, Yanchenko, and Webber do not explicitly teach, however, Magdelinic does teach:
wherein the plurality of dimensions include maturity, ticker and industry type.
(Magdelinic US20200167869 at paras. 164-166) ("[0165] Predictive issuance analytics engine 38 sources raw trading and fundamental data via automated scripts executed at predetermined intervals e.g. every 24 hours (step 4000). The data sources include Thomson Reuters (primary and secondary bond issuance and trading levels, secondary pricing data, outstanding securities, historical bond issuance), S&P Global Market Intelligence (company fundamental data), DBRA, S&P, Moody's, Fitch (company ratings, company credit rating, and macro market data). Thomson Reuters (company sector information), SEDAR (Canada), EDGAR (USA), public filings (prospectus filings), Central Banks/Treasuries, public sources (Macro Market Data), including various other sources. This raw data is pre-processed (step 4002) and the trading data and fundamental data is structured and mapped to the appropriate issuer ID, and stored in databases 27 (step 4004). The data is systematically scrubbed for anomalies and null values. Finally, a set of key input factors are generated based on the raw input. These include but are not limited to factors that measure recent issuance, issuance frequency, maturity schedule gap, propensity for specific tenors (step 4006). These factors are divided between sector and company specific and are used as inputs to the machine learning models.")
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Schwiep, Saha, Yang, Qiao, Sridharan, Yanchenko, Webber, and Magdelinic, because it allows for improved methods for supporting multiple functions such as communication, information management, deal execution, stakeholder collaboration, pricing calculation, securities offering and issuance, and analytics for issuers, investors, and dealers and computing device to measure best-fit correlations with respect to a company's fundamental valuation and secondary market pricing for the company's at least one financial instrument across sector peers and markets conditions and generate an at least one financial instrument pricing output, in real-time. (Magdelinic at Abstract and paras. 2-20).
Response to Arguments
Applicant’s arguments filed on March 27, 2026 have been fully considered but are not persuasive for the following reasons:
With respect to Applicant’s arguments as to the § 101 rejections for now pending claims 1, 4-9, 11-13, and 17-20, Examiner notes the following:
Applicant argues that the amended features would integrate the abstract idea into a practical application.
Examiner notes that the stated problem that a “conventional ML model is unable to generate an accurate prediction based on volatile data environments with unstable data relationships” is not a technical problem, and the claimed solution is not a technical solution. In the claim, correcting a dataset for providing a more accurate output is part of the abstract idea, as it is merely involves data manipulation and data analysis. Furthermore, the data manipulation and analysis could be completed mentally or manually by paper or pen.
Additionally, the additional elements of the computer system - “a plurality of source devices and via a network”, “a processor”, “machine learning (ML) algorithm,” “a managing ML algorithm model,” “a plurality of ML algorithms,” and “one or more sub-ML models” to perform the steps of “plotting”, “performing”, “generating”, “clustering”, “training”, “applying”, “comparing”, “determining”, “retrieving”, “adjusting”, and “modifying”, in all steps is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component. The claims at issue covers collecting, inputting, analyzing, and transmitting data to facilitate adjusting data correspondence with respect to time in data clusters of a dataset in a volatile data environment and adjusting composition of such clusters to reflect changing data correspondence, e.g., pricing in bond markets and stock markets and completing mathematical calculations. The claims invoke the “a plurality of source devices and via a network”, “a processor”, “machine learning (ML) algorithm,” “a managing ML algorithm model,” “a plurality of ML algorithms,” and “one or more sub-ML models” to perform the steps of “plotting”, “performing”, “generating”, “clustering”, “training”, “applying”, “comparing”, “determining”, “retrieving”, “adjusting”, and “modifying” merely as tools to execute the abstract idea. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a certain method of organizing human activity or mental process or mathematical calculation) does not integrate a judicial exception into a practical application. (MPEP 2106.05 (f))
With respect to Applicant’s arguments as to the § 103 rejections for now pending claims 1, 4-9, 11-13, and 17-20, Examiner notes that the arguments are moot in light of the new grounds for rejection above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure and is available for review on Form PTO-892 Notice of References Cited.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MERRITT J HASBROUCK whose telephone number is (571)272-3109. The examiner can normally be reached M-F 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Christine Tran can be reached on 571-272-8103. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MERRITT J HASBROUCK/Examiner, Art Unit 3695
/CHRISTINE M Tran/Supervisory Patent Examiner, Art Unit 3695
Prosecution Insights