Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Detailed Action
This Final Office Action is responsive to Applicants’ Reply as received 7/6/26, including amendments and arguments. Claims 1-20 remain pending, of which claims 1, 11, and 20 are independent.
The Examiner withdraws the prior grounds for rejection of claims 5 and 15 under 35 U.S.C. 112(b). However, based on the amendment to those same aforementioned claims, a new grounds of rejection is presented. See the Examiner’s rejection below under 35 U.S.C. 112(b).
Claim Rejections - 35 USC § 112
4. The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
5. Claims 5 and 15 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
The aforementioned rejected claims as amended now recite, in part, a limitation akin to “wherein the path length is based on an aggregate path length determined using observed drift periods in a historical training dataset used during training of the sample synthesis model.” However, the Examiner notes that these claims depend from the independent claims, which as amended now recite in part a limitation for “determining a path length for the drift period, the path length including a cumulative deviation determined using distance values over a plurality of successive time increments during a drift-build portion of the drift period.”
The Examiner believes the path length determination, as recited in amended claims 5 and 15, is in tension with the path length determination as recited in the independent claims. For example, the path length determination in the independent claims appears to be related to an inference time determination, such that a drift is subject to detection in real-time, based on actual data that has been considered from a start time to a present time of the claim’s performance. However, rather differently in the Examiner’s understanding, the path length determination of the dependent claims 5 and 15 involve a measure that is based on “observed drift periods in a historical training dataset” that is (i) an entirely different dataset than the data operative in claim 1’s path length determination for example and (ii) appears to be old/ “historical” / “training” data that would not necessarily be the same as the data being subject to measure/evaluation for drift path length in the independent claims for example. Said another way, the independent claims and the dependent claims feature two different definitions for the same path length determination, and further the definitions appear to be based on entirely different data. On this basis, the Examiner reasons that the recited determination of a path length for a drift period is vague and indefinite, when one considers these rejected dependent claims together with the independent claims from which they depend.
Claim Rejections - 35 USC § 103
6. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
7. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
8. Claims 1-9 and 11-20 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication No. 2023/0129390 (“Fusco”) in view of Non-Patent Literature “How to generate synthetic data for model monitoring” (“Widmann”) and further in view of U.S. Patent Application Publication No. 2020/0372302 (“Tadepalli”).
Regarding claim 1, FUSCO teaches A system (FIGs. 1-2 showing a cloud computing node, having a computer system/server (Fig. 1), as situated inside of a cloud computing environment (Fig. 2), where the server is associated with a performance management service (as discussed per [0063] and shown in FIG. 4)) comprising:
at least one processing device (the node / server, as discussed above in relation to FIGs. 1-2 and 4, e.g. see FIG. 4 showing explicitly the performance management service hosted inside a server as discussed per [0063] for example) including a processor (FIG. 4’s element 420) coupled to a memory (FIG. 4’s element 430);
the at least one processing device being configured to implement the following steps:
detecting a drift in a dataset, the drift including a drift period having a start time, wherein the dataset pertains to a machine learning (ML)-based model ([0017]: “... the present invention provides for managing performance of a data processing system in a computing environment, by one or more processors, is depicted. A drift may be dynamically detected in one or more machine learning models generating a plurality of predictions and deployed in a computing system. A plurality of metrics and data may be collected of the one or more machine learning models based on the drift. One or more additional machine learning models may be trained based of the drift and the plurality of metrics and data.”);
determining a path length for the drift period, the path length including a cumulative deviation determined using distance values over a plurality of successive time increments during a drift-build portion of the drift period ([0020]: “The performance management service may detect and measure drift, where the drift is a change in the data and/or model performance distribution. The performance management service may determine/compute one or more drift metrics based on changes in model performance or distance between data samples or their statistics.”, where the Examiner reasons a distance of deviation in the model or its data is clearly being considered; and further [0087]: “After some time, a drift is detected (e.g., drift event 630 is detected) where a drift model drift time-series is above a drift threshold 610, which means the error metric of model M0 higher than expected. During the drift event 630, the sampling rate 612 for collecting retrain data has increased but still substantially low as data are further away from where M0 was trained (e.g., model uncertainty high, distribution distance from original training data high).”).
Responsive to Applicants’ argument, Examiner’s comment specifically regarding this particular limitation provided just above: the Examiner notes that any drift as understood in relation to a model relative to its data is understood to be a distance/measurement, i.e., “distance values”, and particularly one that accumulates / is “cumulative”. In the reference for example, a drift/deviation grows over time, e.g., “successive time increments”, until the threshold is reached, in which case some further processing is engaged to address the drift.
Fusco does not teach the further limitations for:
obtaining one or more synthetic post-drift samples generated for a period following the start time using an ML-based sample synthesis model that is trained based on one or more pre-drift samples observed during a period preceding the start time and on the path length for the drift period; and
predicting a drift period duration for the dataset based on the synthetic samples.
Rather, the Examiner relies upon Widmann and Tadepalli to teach what Fusco lack.
Regarding the limitation for obtaining, Widmann teaches, on the 2nd page, that once a model drifts and the drift is detected, steps may be taken by a model monitoring application, e.g. an alarm and model retraining. However, Widmann (2nd page still) teaches that such a monitoring application needs training and, for that, it then needs current and future data – for which, it teaches the generation of synthetic data to stand in for future data to train the monitoring application on. For example, see Widmann’s later pages with different headings for generating synthetic data with and without drift. See also Widmann’s 3rd page discussing that the synthetic data is generated from existing data, i.e., existing data that would be understood to precede drift, or existing data that could easily be selected to ensure no drift is associated with, and in either case, existing data that the Examiner reasons that Fusco would have available from sampling/collection opportunities in the FIGs. 6A-6B timeline prior to / left of moment 630 (i.e., the moment when the drift is detected).
Both Fusco and Widmann relate to model management frameworks that detect for drift so that the model can be managed to be maintained by remedy measures. Hence, they are similarly directed and therefore analogous. It would have been obvious to one of ordinary skill in the art to improve upon Fusco’s model drift detection by training a monitoring application adjacent to Fusco’s model, as Widmann contemplates, such that the monitoring application has been trained to detect drift as benefitting from the synthetic data generation approach Widmann teaches.
Fusco and Widmann contemplate various predictive metrics and measures that help their frameworks remedy a drift scenario. However, regarding the limitation for predicting, neither Fusco nor Widmann teach specifically teaches a drift duration period as predicted based on generate synthetic data. Rather, Tadepalli’s [0033]-[0035] teach determining behavior changes in a model / its data, as can be machine-learned ([0033]), such that for any given state in a model, a future state can be predicted/forecasted ([0034]), and moreover, the forecasting can assume a time series progression of states, such that forecasts can be associated with a confidence score ([0035]).
Like Fusco and Widmann, Tadepalli teaches a manner of forecasting a state of a model. Hence, the references are similarly directed and therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Tadepalli’s modeling approach to a model as Fusco etc. contemplate, the modeling have current/prior data per Fusco and future data per Fusco in view of Widmann, such that using that information, a future state of the model (e.g., its drift condition/state as Fusco and Widmann consider) could be predicted at future times using Tadepalli’s approach. Hence, knowing a future drift condition at a future time in this manner, it is a simple matter of knowing in which future time intervals the drift will still persist and in which the drift won’t exist, and determining a prediction for a drift duration from those types of forecasting steps.
Regarding claim 2, Fusco in view of Widmann and further in view of Tadepalli teach the system of claim 1, as discussed above. The aforementioned references teach the additional limitations wherein predicting the drift period duration further comprises: determining a first confidence value based on the pre-drift samples observed during the period preceding the start time and a second confidence value based on the synthetic post-drift samples generated by the sample synthesis model; and estimating the drift period duration using an ML-based drift model that is trained based on the first and second confidence values (Fusco’s [0078] discusses the sampling of live data to essentially monitor for “model uncertainty” (i.e., a confidence value as recited) as a sign of drift, and hence the Examiner reasons that model uncertainty is being evaluated along the time axis of Fusco’s FIGs. 6A-6B, and if Fusco is modified in view of Widmann and Tadepalli, then there is some representation of future values per Widmann’s synthetic data generation, and the forecasting to determine future state as Tadepalli’s [0034]-[0035] teaches is based on confidence in the model’s predictions). The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 3, Fusco in view of Widmann and further in view of Tadepalli teach the system of claim 2, as discussed above. The aforementioned references teach the additional limitations wherein the at least one processing device is further configured to implement the following steps:
upon observing one or more actual post-drift samples immediately following the start time:
replacing a subset of the synthetic post-drift samples with the actual post-drift samples as the actual post-drift samples are observed, to define an updated post-drift sample set for the period following the start time (receiving actual/measured data, as monitoring/sampling for drift persists and hence time progresses along the time axis in Fusco’s FIGs. 6A-6B, and the Examiner reasons that it would be obvious to replace each synthetic data instance with an actual/measured data instances as that actual/measured data is collected in time), and
determining a subsequent second confidence value based on the updated post-drift sample set, and estimating an updated drift period duration for the dataset using the drift model that is iteratively retrained based on the first confidence value and on the subsequent second confidence value (essentially repeating the steps discussed above per claim 2, which are taught by the prior art, once the underlying variables have changed – the Examiner reasons it would be obvious to do so by a monitoring application such as Fusco modified in view of Widmann would provide, since it would essentially be monitoring and capable of reevaluating drift at every time moving along the time axis as shown in Fusco’s FIGs. 6A-6B).
The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 4, Fusco in view of Widmann and further in view of Tadepalli teach the system of claim 2, as discussed above. The aforementioned references teach the additional limitations wherein the model or the drift model comprises a classifier model or a regression model (Fusco’s [0012] teaching that the model itself being monitored for drift may be a classifier model). The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 5, Fusco in view of Widmann and further in view of Tadepalli teach the system of claim 1, as discussed above. The aforementioned references teach the additional limitations wherein the path length is based on determining an aggregate path length determined using observed drift periods in a historical training dataset used during training of the sample synthesis model (Fusco’s FIGs. 6A-6B showing the monitoring of model drift as a function of time, which Fusco also details is measurable as a function of distance ([0020], [0022], and [0076]-[0087]: “distance” as being measured/evaluated), and where the drift is measured in these ways, as Fusco [0083] teaches, to determine when the drift is far enough to warrant an action, e.g., changing the model version). The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 6, Fusco in view of Widmann and further in view of Tadepalli teach the system of claim 1, as discussed above. The aforementioned references teach the additional limitations wherein the path length comprises a period following the start time, and the period is determined based on a magnitude of the drift (Fusco’s FIGs. 6A-6B showing the monitoring of model drift as a function of time, which Fusco also details is measurable as a function of distance ([0020], [0022], and [0076]-[0087]: “distance” as being measured/evaluated), and where the drift is measured in these ways, as Fusco [0083] teaches, to determine when the drift is far enough to warrant an action, e.g., changing the model version). The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 7, Fusco in view of Widmann and further in view of Tadepalli teach the system of claim 6, as discussed above. The aforementioned references teach the additional limitations wherein the magnitude measures an amount of change in a distribution underlying the dataset, and the magnitude comprises a distance metric between a start and an end time (Fusco’s FIGs. 6A-6B showing the monitoring of model drift as a function of time, which Fusco also details is measurable as a function of distance ([0020], [0022], and [0076]-[0087]: “distance” as being measured/evaluated), and where the drift is measured in these ways, as Fusco [0083] teaches, to determine when the drift is far enough to warrant an action, e.g., changing the model version). The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 8, Fusco in view of Widmann and further in view of Tadepalli teach the system of claim 1, as discussed above. The aforementioned references teach the additional limitations wherein the at least one processing device is further configured to implement the following step: in response to predicting the drift period duration, managing the model, e.g., Fusco’s [0074]-[0092] contemplates retraining the model if appropriate based on the drift. The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 9, Fusco in view of Widmann and further in view of Tadepalli teach the system of claim 8, as discussed above. The aforementioned references teach the additional limitations wherein managing the model comprises: retraining the model using one or more newly observed actual samples to generate a new version of the model, and deploying the new version of the model to an edge node (Fusco’s [0083] and also FIG. 5 step 536 and also FIG. 7 element 708). The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 11, the claim includes the same or similar limitations as discussed above in relation to claim 1, and is therefore rejected under the same rationale.
Regarding claim 12, the claim includes the same or similar limitations as discussed above in relation to claim 2, and is therefore rejected under the same rationale.
Regarding claim 13, the claim includes the same or similar limitations as discussed above in relation to claim 3, and is therefore rejected under the same rationale.
Regarding claim 14, the claim includes the same or similar limitations as discussed above in relation to claim 4, and is therefore rejected under the same rationale.
Regarding claim 15, the claim includes the same or similar limitations as discussed above in relation to claim 5, and is therefore rejected under the same rationale.
Regarding claim 16, the claim includes the same or similar limitations as discussed above in relation to claim 6, and is therefore rejected under the same rationale.
Regarding claim 17, the claim includes the same or similar limitations as discussed above in relation to claim 7, and is therefore rejected under the same rationale.
Regarding claim 18, the claim includes the same or similar limitations as discussed above in relation to claim 8, and is therefore rejected under the same rationale.
Regarding claim 19, the claim includes the same or similar limitations as discussed above in relation to claim 9, and is therefore rejected under the same rationale.
Regarding claim 20, the claim includes the same or similar limitations as discussed above in relation to claim 1, and is therefore rejected under the same rationale. Specifically, see Fusco’s [0090] and [0094]-[0098] for discussions of implementations reading on the recited non-transitory processor-readable storage medium.
9. Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Fusco in view of Widmann and Tadepalli and further in view of Non-Patent Literature “Generative Adversarial Nets for Synthetic Time Series Data” (“Jansen”).
Regarding claim 10, Fusco in view of Widmann and further in view of Tadepalli teach the system of claim 1, as discussed above. The aforementioned references do not specifically teach the additional limitation wherein the sample synthesis model comprises an artificial neural network. Rather, see Jansen’s GAN network/system that generates the same type of data (e.g., time-series) that Widmann, and Fusco as modified in view of Widmann for example.
Fusco and Widmann contemplate a similar type of data as used in modelling as Jansen. Hence, they are similarly directed and therefore analogous. More specifically, both Widmann and Jansen explicitly relate to synthetic data generation as relating to that data type as used in modelling. Jansen provides for a well-known type of machine learning approach to do what Widmann only teaches in a function sense on a more generalized level. Hence, the Examiner believes it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement Widmann’s synthetic data generation step using a framework such as Jansen.
Response to Arguments
Applicants’ arguments filed 7/6/26 have been fully considered but they are not persuasive.
In the Reply, page 9, Applicants argue that the cited prior art does not teach amended claim limitation per the independent claims for determining a path length for the drift period, the path length including a cumulative deviation determined using distance values over a plurality of successive time increments during a drift-build portion of the drift period.
In maintaining a rejection using the same reference, the Examiner cites to Fusco:
[0020]: “The performance management service may detect and measure drift, where the drift is a change in the data and/or model performance distribution. The performance management service may determine/compute one or more drift metrics based on changes in model performance or distance between data samples or their statistics.”, where the Examiner reasons a distance of deviation in the model or its data is clearly being considered; and
[0087]: “After some time, a drift is detected (e.g., drift event 630 is detected) where a drift model drift time-series is above a drift threshold 610, which means the error metric of model M0 higher than expected. During the drift event 630, the sampling rate 612 for collecting retrain data has increased but still substantially low as data are further away from where M0 was trained (e.g., model uncertainty high, distribution distance from original training data high).”).
The Examiner reasons that any drift as understood in relation to a model relative to its data is understood to be a distance/measurement, i.e., “distance values”, and particularly one that accumulates / is “cumulative”. In the reference for example, a drift/deviation grows over time, e.g., “successive time increments”, until the threshold is reached, in which case some further processing is engaged to address the drift. Hence, based on this reasoning, the Examiner believes Fusco still reads on the limitation as amended and as argued.
Conclusion
The prior art made of record and not relied upon is considered pertinent to Applicants’ disclosure:
US 2020/0012900 WALTERS: FIGs. 19-20, and especially FIG. 19’s loop which continually monitors for drift based on incoming data, and [0192] considers that at any given time, the processing window may comprise of both synthetic and actual data or entirely one of either.
The Examiner reasons this teaching per [0192], as applied to drift detection, is similar to the subject matter in Applicants’ claim 3.
US 2022/0036201 TAMIR: particularly FIGs. 5-6, contemplating a window-based drift detection approach.
CN 113033643 A: Abstract discussing what is essentially an automated model drift detection framework that measures drift in real-time streaming data and engages in model retraining if appropriate based on the measured drift.
Non-Patent Literatures “A Review on Real Time Data Stream Classification and Adapting To Various Concept Drift Scenarios” (DONGRE), “Literature Review on Phenomenon of Concept Drift and its Handling Approaches” (NAYAK), “Literature Review on Phenomenon of Concept Drift and its Handling Approaches” (PERSANGHADER), “Autoregressive based Drift Detection Method” (MAYAKI), “Real-time Drift Detection on Time-series Data” (RAMANAN), and “A Novel Concept Drift Detection Method for Incremental Learning in Nonstationary Environments” (YANG) also teach a window-based approach to drift detection.
Non-Patent Literature “Unsupervised Drift Detection on High-speed Data Streams” (SOUZA)
THIS ACTION IS MADE FINAL. Applicants are reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHOURJO DASGUPTA whose telephone number is (571)272-7207. The examiner can normally be reached M-F 8am-5pm CST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tamara Kyle can be reached at 571 272 4241. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHOURJO DASGUPTA/Primary Examiner, Art Unit 2144
Prosecution Insights