Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
2. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
3. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office Action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
4. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
5. Claims 1-6, 8-14, and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication No. 2023/0113287 (“Aggarwal”) in view of Non-Patent Literature “Embracing Automated Retraining” (“Longo”).
Regarding claim 1, AGGARWAL teaches A computing system (FIG. 1) comprising:
a storage configured to store a plurality of time series models and a data set (FIG. 10’s element 1018, and elements contained therein, and see also [0027] discussing that “Portions of data or information used by or generated by the systems and subsystems 108, 110, 124, and 130 as part of its processing may be stored in a persistent memory data store ...”, which clearly encompasses models directed to modelling time series data (FIG. 4’s step 408 and [0067]) a time series dataset (FIG. 2’s step 202 and [0062])); and
a processor (FIG. 10’s element 1004, and [0027] and [0034]’s mentions of “processing units (e.g., processors, cores)”) configured to
divide the data set into a k folds of data, where k is greater than two ([0003] teaching the splitting of a time series dataset into a parameterized number of groups or folds (see also [0029] and [0062]-[0063]), where the number of folds (which the Examiner equates with the recited k folds) can be more than two as detailed in [0063]),
execute the plurality of time series models on a newest fold and an oldest fold from among the k folds of data to train/retrain the plurality of time series models ([0067] discussing the evaluation of a set of models, which the Examiner understands to model time series data, for each generated fold, which would be inclusive of say a first and last fold as created (which the Examiner equates with the newest and oldest folds as recited)),
determine a plurality of error values for the plurality of time series models, respectively, based on the newest fold and the oldest fold ([0067]’s discussion of an “accuracy metric” for the model as applied to a fold, which the Examiner understands to be done for each fold and hence a first/oldest and last/newest fold, and per [0049] the accuracy metric is an error evaluation which the Examiner reasons is representative of a gap between a model’s output and an expected output (as [0003]’s discussion of validation suggests)), and
store the plurality of error values within the storage ([0027]: “Portions of data or information used by or generated by the systems and subsystems 108, 110, 124 and 130 as part of its processing may be stored in a persistent memory data store such as performance metrics for evaluated models data store 126.”).
As discussed above, Aggarwal teaches a fold-based time series model training aspect. That said, Aggarwal does not teach the further limiting feature of the processor to execute the plurality of time series models on a newest fold and an oldest fold from among the k folds of data to dynamically retrain the plurality of time series models. At best, Aggarwal’s process seems to be explicitly request-driven, per FIG. 2’s step 202. Rather, the Examiner relies upon LONGO to teach what Aggarwal otherwise lacks, see e.g., Longo’s page 2 bottom section details two core approaches to automated model retraining, including a dynamic one that is triggered based on model performance metrics, such as to “prevent models from going stale, and optimize the compute cost” (page 3’s 2nd full paragraph), with a more elaborative metrics-based dynamic retraining discussion is provided on page 4.
Aggarwal and Longo both relate to training models with an aim to promote model performance/accuracy, and hence are similarly directed and therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Longo’s performance metric-based dynamic retraining aspect into Aggarwal’s time series model training framework, with a reasonable expectation of success, for purposes of keeping Aggarwal’s models relevant, useful/applicable, and not stale in a timely and responsive manner per Longo’s remedial dynamic aspect.
Regarding claim 2, Aggarwal in view of Longo teaches the computing system of claim 1, as discussed above. The aforementioned references further teach the additional limitation wherein each fold of data includes a first subset of data for training and a second subset of data for validation (Aggarwal’s [0003], and also [0029], teaching: “... A cross-validation technique is generally identified by a number of cross-validation parameters that may be used to estimate/validate the performance of a trained model. By way of example, a cross-validation parameter may identify the number of groups or folds that a given time series dataset may be split into, where each fold consists of a training set on which the model is trained and a validation set on which the performance of the trained model is evaluated.”). The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 3, Aggarwal in view of Longo teaches the computing system of claim 1, as discussed above. The aforementioned references further teach the additional limitation wherein the processor is configured to select a time series model from among the plurality of time series models for additional retraining based on an error value of the selected time series model, and execute the selected time series model on an additional fold from among the k folds of data to further retrain the selected time series model (as discussed per claim 1, Longo’s dynamic retraining aspect, triggered based on a model performance metric not meeting an acceptable threshold (page 4), which when incorporated into Aggarwal’s framework would retrain the underperforming models for additional training to improve performance/accuracy). The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 4, Aggarwal in view of Longo teaches the computing system of claim 3, as discussed above. The aforementioned references teach the further limitations wherein the processor is configured to select an additional fold from among the k folds, execute the selected time series model on the additional fold to further retrain the selected time series model, determine an error value for the selected time series model based on the further retraining, and determine whether or not to additionally retrain the selected time series model based on the error value (as discussed per claim 1, Aggarwal’s [0067] teaches the application of the model to evaluate each fold (and hence inclusive of an additional fold), and to the extent that Aggarwal trains its model this way, Aggarwal modified in view of Longo then permits a retraining of its model in a similar way (i.e., training the model as Aggarwal teaches but again as Longo suggests), and hence in providing retraining, the Examiner reasons that further accuracy evaluation (inclusive of an error/loss consideration) is involved, and that Longo teaches the notion of further training/retraining based on accuracy computation/measurement). The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 5, Aggarwal in view of Longo teach the computing system of claim 4, as discussed above. The aforementioned references teach the additional limitations wherein the processor is configured to select a second additional fold from among the k folds, execute the selected time series model on the second additional fold to even further retrain the selected time series model, determine an additional error value for the selected time series model based on the even further retraining, and determine whether or not to additionally retrain the selected time series model based on the additional error value (same rationale provided above per claim 4, as this claim is merely a further instance of what claim 4 teaches). The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 6, Aggarwal in view of Longo teaches the computing system of claim 1, as discussed above. The aforementioned references teach the additional limitations wherein the processor is further configured to identify a second time series model from among the plurality of time series models to stop retraining based on an error value of the second time series model, and terminate retraining of the second time series model (as discussed per claim 1, Aggarwal contemplates training many models ([0067]: “set of models”, and hence permits the Examiner’s mappings to additional model instances such as the presently recited second one), and where Aggarwal and Longo when combined evaluate the training/retraining of models in view of model accuracy, and Longo specifically teaches this in relation to a threshold that is used to determine whether to train further or not (i.e., the Examiner reasons if the threshold is met then no further training is necessary)). The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 8, Aggarwal in view of Longo teaches the computing system of claim 1, as discussed above. The aforementioned references further teach the additional limitation wherein the processor is configured to execute a time series model from among the plurality of time series models on two folds from the k folds to generate two predicted outputs, compare the two predicted outputs to two expected outputs to generate two fold error values, and compare the two fold error values to determine whether to further retrain the time series model (as discussed per claim 1, Longo’s dynamic retraining aspect, triggered based on a model performance metric not meeting an acceptable threshold (page 4), which when incorporated into Aggarwal’s framework would retrain the underperforming models for additional training to improve performance/accuracy, and where the Examiner reasons such a performance/accuracy evaluation is representative of a gap between a model’s output and an expected output (as [0003]’s discussion of validation suggests) and would intuitively be performed on a per-model for models needing retraining upon evaluation and on a per-fold basis if training and validation is performed in accordance with Aggarwal’s approach). The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 9, the claim includes the same or similar limitations as claim 1 discussed above, and is therefore rejected under the same rationale.
Regarding claim 10, the claim includes the same or similar limitations as claim 2 discussed above, and is therefore rejected under the same rationale.
Regarding claim 11, the claim includes the same or similar limitations as claim 3 discussed above, and is therefore rejected under the same rationale.
Regarding claim 12, the claim includes the same or similar limitations as claim 4 discussed above, and is therefore rejected under the same rationale.
Regarding claim 13, the claim includes the same or similar limitations as claim 5 discussed above, and is therefore rejected under the same rationale.
Regarding claim 14, the claim includes the same or similar limitations as claim 6 discussed above, and is therefore rejected under the same rationale.
Regarding claim 16, the claim includes the same or similar limitations as claim 8 discussed above, and is therefore rejected under the same rationale.
Regarding claim 17, the claim includes the same or similar limitations as claim 1 discussed above, and is therefore rejected under the same rationale.
Regarding claim 18, the claim includes the same or similar limitations as claim 2 discussed above, and is therefore rejected under the same rationale.
Regarding claim 19, the claim includes the same or similar limitations as claim 3 discussed above, and is therefore rejected under the same rationale.
Regarding claim 20, the claim includes the same or similar limitations as claim 4 discussed above, and is therefore rejected under the same rationale.
6. Claims 7 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Aggarwal in view of Longo and further in view of U.S. Patent Application Publication No. 2023/0153394 (“Ahuja”).
Regarding claim 7, Aggarwal in view of Longo teaches the computing system of claim 1, as discussed above. The aforementioned references do not teach the additional limitations wherein the processor is configured to select a fold with a newest timestamp from among the k folds as the newest fold and select a fold with a oldest timestamp from among the k folds as the oldest fold. At best, Aggarwal teaches that the application of its model to many folds, including those that are earlier and later among a span of folds, the folds encompassing time series data that has an ordering, and that a cutoff is determined for each fold (FIG. 5, [0063]-[0066]), such that the Examiner reasons that a fold has an associated time span relative to other folds, and because of that one fold could be understood to precede or follow another fold as considered temporally. That said, Aggarwal and also Longo do not explicitly teach a timestamp such that as to provide the data its ordering, although it would seem obvious that time series data would inherently feature a value ordered in accordance with / as a function of time. Hence, the Examiner further relies upon AHUJA to teach what Aggarwal and Longo otherwise lack, see e.g., Ahuja’s [0020], [0034], [0037], [0048], and [0107] teaching the association of data in a timeseries with timestamps explicitly.
Aggarwal and Ahuja both relate to working with timeseries models, e.g., their training, their use, and their management, and hence are similarly directed and therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement’s Aggarwal’s time series model training framework and specifically its timeseries data to feature timestamps to specifically serve as a basis for ordering the data, with a reasonable expectation of success, such that this time-ordered data can be understood, evaluated, and grasped as a function of its time in a time format that is well known and widely used.
Regarding claim 15, the claim includes the same or similar limitations as claim 7 discussed above, and is therefore rejected under the same rationale.
Conclusion
7. The prior art made of record and not relied upon is considered pertinent to Applicants’ disclosure:
US 2019/0102693 (Yates)
Non-Patent Literature “WaveBound: Dynamic Error Bounds for Stable Time Series Forecasting” (Cho)
Non-Patent Literature “To retrain, or not to retrain? Let's get analytical about ML model updates” (EvidentlyAI)
8. Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHOURJO DASGUPTA whose telephone number is (571)272-7207. The examiner can normally be reached M-F 8am-5pm CST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tamara Kyle can be reached at 571 272 4241. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHOURJO DASGUPTA/Primary Examiner, Art Unit 2144
Prosecution Insights