Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1, 6, 8, 10, 13, 15, 17-19 were amended. Claim 21 is new. Claim 20 is canceled. Claims 1-19, and 21 are pending and are examined herein.
Claims 1-19, and 21 are rejected under 35 U.S.C. 103.
Response to Amendment
The amendment filed December 18th, 2025 has been entered. Claims 1, 6, 8, 10, 13, 15, 17-19 were amended. Claim 21 is new. Claim 20 is canceled. Claims 1-19, and 21 are pending and are examined herein. Applicant’s amendments to the claims have overcome each and every objection previously set forth in the Non-Final Rejection Office Action mailed September 23th, 2025.
Response to Arguments
Applicant's arguments filed December 18th, 2025 regarding the 35 U.S.C. 101 rejection of claims 1-19, and 21 have been fully considered and are persuasive. The 35 U.S.C. 101 rejection of claims 1-19, and 21 has been withdrawn.
Applicant's arguments filed December 18th, 2025 regarding the rejections under 35 U.S.C. 103 have been fully considered and are persuasive. The cited references do not fairly teach or suggest the claim as amended. However, new references, Babu et al. (U.S. Pub. 2019/0102700 A1) and Kolar et al. (U.S. Pub. 2021/0184958 A1) are introduced in the below 35 U.S.C. 103 rejection to teach the new features.
Claim Objections
Claim 10 is objected to because of the following informalities:
In line 28, “automatically deploy the candidate model on production run-time machines of the network, in please of the currently deployed model”. “In place of the” seems appropriate here.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1 – 19, 21 are rejected under 35 U.S.C. 103 as being unpatentable over Babu et al. (U.S. Pub. 2019/0102700 A1), Kolar et al. (U.S. Pub. 2021/0184958 A1), further in view of Shemer et al. (U.S. Pub. 11595282).
Regarding Claim 1, Babu teaches
generating, by a processor, a candidate model by training a machine learning (ML) algorithm on training data … acquired between deployment of a currently deployed model generated by training the ML algorithm … and a current time; ([0061] of Babu states ”The machine learning platform disclosed herein may manage different machine learning models or different versions of a machine learning model, and facilitate the evaluation of the machine learning models and the selection and deployment of the machine learning model for production. The machine learning platform may also collect and report information, such as scores of the model applied to the input data or statistics about the usage of a model, which may be used to improve the model or the selection of the model by a selector. In some cases, the machine learning platform may use the selector to select an appropriate model for a give dataset from a number of available models.” [0063] of Babu states “The data from data flow may be input into model generator 120 through a user interface (UI) 122. Feature vectors may then be extracted from the data and used to train the ML model. The trained model may be saved to model store 110. In some embodiments, model generator 120 may retrieve an existing model from model store 110, retrain the existing model using incoming data to generate a new model (e.g., a new version of a model), and save the new model to model store 110.” [0087] of Babu states “For instance, an application may use a model group that includes a first model M1. When a new model M2 is created later to replace first model M1, the application may seamlessly transition to using the new model M2, without knowing the underlying change. Data can be used for scoring against a group” [0081] of Babu states ”Models published for scoring may be refreshed as new models are available with new training data. This would allow transparently retraining models with recent data to prevent model drift.” Babu teaches that new model versions are generated by training on new data and training candidate versions on later acquired data. It also teaches temporally sequenced deployment and replacement cycle.)
executing, by the processor, the candidate model ... applying testing data … to the candidate model ([0046] of Babu states “As used herein, scoring using a model may involve generating predictions for given data using the model. The predictions can be categorical predictions or numeric values based on the type (classification vs. regression) of the model.” [0044] of Babu states “The different versions of a model might differ in their hyper-parameters or other parameters, and may be trained using different training data. The different versions may be evaluated against various test datasets to identify one or more of them to deploy in a production environment.” [0076] of Babu states “The dynamically selected models may be applied to real time input data to generete inference results. At 460, score data may be generated based on the inference results. The score data may be feedback to the machine learning platform to improve one or more models or to improve the the selector that dynamically selects the models.” Babu teaches evaluating model versions against test datasets and generating predictions.)
executing, by the processor, a plurality of previously deployed models and the currently deployed model to generate respective prediction by applying the testing data to each of the plurality of previously deployed models and the currently deployed model, wherein the plurality of previously deployed models are generated… ([0044] of Babu states “The different versions of a model might differ in their hyper-parameters or other parameters, and may be trained using different training data. The different versions may be evaluated against various test datasets to identify one or more of them to deploy in a production environment.” [0061] of Babu states ”The machine learning platform disclosed herein may manage different machine learning models or different versions of a machine learning model, and facilitate the evaluation of the machine learning models and the selection and deployment of the machine learning model for production.” [0086] of Babu states “Multiple versions of a model can be created by creating these models with the same name. The different versions may have different model IDs.” [0075] of Babu states “In some embodiments, the machine learning platform may support periodically publishing a model at a given frequency to implement continuous learning of a model using new data in new time windows.” [0077] of Babu states “In some embodiments, the machine learning platform may provide API(s) for supporting simultaneous testing of several versions of a model. The API may allow defining rules for distributing input data among the different versions of models. The rules may support, for example, percentage-based partitioning and/or attribute value-based partitioning of input data. A scoring response may include the version of the model that was used to score the request. The version information may be used in the application layer to gather statistics regarding effectiveness of the model.” [0129] of Babu states “For example, the scheme for using the selected at least one ML model to analyze the input data may include analyzing a same portion of the input data by each of the selected at least one ML model, and selecting, from results of analyzing the same portion of the input data by the selected at least one ML model, a most common result as a result for the portion of the input data.” Babu teaches that a model group contains multiple versions of a model and also teaches evaluating multiple versions simultaneously against the same test data. Also, prediction output is tracked per version. Combining with Kolar that these “model may be configured to detect problems in a wireless network, predict tunnel failures” teaches that these data and prediction are accounting network failure prediction.)
… a plurality of performance distributions derived from the respective predictions generated by the plurality of previously deployed models; ([0080] of Babu states ”In some embodiments, the machine learning platform may provide API(s) for reporting statistics on the usage of a model, such as the number of requests served by the model in a given time window, and the distribution of predictions by category for classification models and binning for regression models.” [0082] of Babu states ”The machine learning platform may be automated. For example, the machine learning platform may be configured to run testing or update models at a certain frequency, such as nightly or weekly. The machine learning platform may be configured to support unit tests and end-to-end tests. The machine learning platform may be configured to support both golden sets and thresholds (notion of test pass fail under some conditions). The machine learning platform may also support correctness as well as performance tests.” Babu teaches generating prediction distribution statistics for each model versions.)
determining, by the processor, a criteria … ([0076] of Babu states “The dynamically selected models may be applied to real time input data to generete inference results. At 460, score data may be generated based on the inference results. The score data may be feedback to the machine learning platform to improve one or more models or to improve the the selector that dynamically selects the models.” [0103] of Babu states “At 740, it may be determined whether the score meet certain criteria, such as greater than a threshold. If the score meets the criteria, there may be no change to the models and the selector. The one or more models and/or the combining strategy may continue to be used for the input data at 720 and 730 until a score fails to meet the criteria at 740, at which point, the processing may move to 750. If the score does not meet the criteria, the model selector and/or some models in the model group may be revised at 750, and the processing may then proceed to 710 to re-perform the processing at 710-740.” Babu teaches a deployment decision criteria derived from model scoring outputs.)
and automatically deploying, by the processor, the candidate model… in place of the currently deployed model… ([0078] of Babu states “Batch data may be analyzed with a very high throughput, for example, in the order of a million or more records per second. There may be no downtime when model versions are being swapped.” [0089] of Babu states “In some embodiments, the comparison may be visualized using the ML platform. Based on the results of this evaluation, one or more of these models can be chosen to be deployed for scoring.” [0090] of Babu states “As described above, in some embodiments, the machine learning platform may be used to deploy a model group and a selector in a production environment, and the selector may learn to dynamically select the model(s) from the model group in the production environment in different contexts or for different input data based on a score determined using certain scoring metrics, such as certain business goals.” Babu teaches threshold triggered model version update in production.)
wherein each of the currently deployed model, the candidate model, and the plurality of previously deployed models … ([0044] of Babu states “The different versions of a model might differ in their hyper-parameters or other parameters, and may be trained using different training data. The different versions may be evaluated against various test datasets to identify one or more of them to deploy in a production environment.” [0061] of Babu stataes ”The machine learning platform disclosed herein may manage different machine learning models or different versions of a machine learning model, and facilitate the evaluation of the machine learning models and the selection and deployment of the machine learning model for production.“ [0076] of Babu states “The dynamically selected models may be applied to real time input data to generete inference results. At 460, score data may be generated based on the inference results. The score data may be feedback to the machine learning platform to improve one or more models or to improve the the selector that dynamically selects the models.”)
Babu does not explicitly teach that the
data indicative of second network operating conditions of a network … on first network operating conditions of the network…
… network failure prediction data by applying testing data indicative of current network operating conditions…
calculating, by the processor, a plurality of quality assurance metrics for the candidate model from the generated network failure prediction data;
by training the ML algorithm on historical network operating conditions of the network acquired prior to the first network operating conditions;
determining, by the processor, a plurality of thresholds based on a plurality of performance distributions …
… based on the respective predictions generated by the currently deployed model.;
… on production run-time machines of the network…, based on a comparison of the plurality of quality assurance metrics with the plurality of thresholds and a comparison of the plurality of quality assurance metrics with the criteria,
… are configured to predict failure scenarios of the network by detecting anomalous issues occurring on the network from real-time data from network functions.
However, Kolar teaches that
data indicative of second network operating conditions of a network … on first network operating conditions of the network… ([0061] of Kolar states “The machine learning model is used to monitor one or more computer networks based on data collected from the one or more computer networks. The service also tracks performance metrics associated with training of the machine learning model. The service determines that a degradation of the performance of the machine learning model is anomalous, based on the tracked performance of the machine learning model and performance metrics associated with training of the model.” [0104] of Kolar states “In another embodiment, the service may determine whether a drift in the performance of the machine learning model between times at which the model the model is trained is anomalous.” Kolar teaches that ML model trains and operates on data reflecting the condition of a monitored network and there could be temporal gap between these network data.)
… network failure prediction data by applying testing data indicative of current network operating conditions… ([0102] of Kolar states “In various embodiments, the machine learning model is used to monitor one or more computer networks based on data collected from the one or more computer networks. For example, the model may be configured to detect problems in a wireless network, predict tunnel failures in an SD-WAN or other network, classify devices in the network(s) by device type, or the like. In further embodiments, the model may take the form of an anomaly detector, a classifier, or other form of machine learning model. Accordingly, the tracked performance may be indicative of a percentage of anomalies raised by the anomaly detector for the one or more networks, indicative of a detection probability of the classifier, a recall or precision of the model, etc.” Kolar teaches applying the ML model to current network data to generate network failure prediction.)
calculating, by the processor, a plurality of quality assurance metrics for the candidate model from the generated network failure prediction data; ([0083] of Kolar states “The model(s), when registered with ML Ops service 408, will be associated with the model performance metrics to be monitored during model training. For example, in the case of a classification model, the model may be registered with service 408 to monitor its precision, recall, area under curve (AUC)… A complex model may register with service 408 to monitor multiple metrics. For example, if two models of model inference engine 526 are used to predict the anomaly bands (e.g., top and bottom prediction bands), then quantile loss and R-squared metrics can be monitored for each regressor.” [0102] of Kolar states “Accordingly, the tracked performance may be indicative of a percentage of anomalies raised by the anomaly detector for the one or more networks, indicative of a detection probability of the classifier, a recall or precision of the model, etc.” Kolar ties these various metrics to the model’s network failure prediction outputs.)
by training the ML algorithm on historical network operating conditions of the network acquired prior to the first network operating conditions; ([0061] of Kolar states “The machine learning model is used to monitor one or more computer networks based on data collected from the one or more computer networks. The service also tracks performance metrics associated with training of the machine learning model. The service determines that a degradation of the performance of the machine learning model is anomalous, based on the tracked performance of the machine learning model and performance metrics associated with training of the model.” [0104] of Kolar states “In another embodiment, the service may determine whether a drift in the performance of the machine learning model between times at which the model the model is trained is anomalous. In further embodiments, the service may further track distribution changes in the data collected from the one or more networks (e.g., the data from the one or more networks consumed by the model) and base the determination on these tracked changes.” Kolar teaches that the model trains on data reflecting network conditions over time.)
determining, by the processor, a plurality of thresholds based on a plurality of performance distributions … ([0100] of Kolar states ”In a further embodiment, MPA 502 may also include data-caused inference degradation analyzer 514 that is responsible for determining whether any data distribution changes (e.g., as indicated by metrics 528 from DCD 522) are responsible for any degradation in the inference accuracy metrics 530 from IA 524. For example, analyzer 514 may discretize the inference accuracy metrics 530 (e.g., across all deployments) into ‘Positive’ and ‘Negative’ classes, where the ‘Positive’ class indicates severe drops in the inference accuracy metric(s) and the ‘Negative’ class indicates normal changes. In turn, analyzer 514 may train a decision tree using the classes and the data distribution change metrics 528 (e.g., median, 75th percentile, etc.) to identify rules that can be used for purposes of root-causing model performance issues.” [0101] of Kolar states ”For example, if the severe inference accuracy drop occurs when the data distribution exhibits a median>threshold_1 AND 75th percentile>threshold_2, then analyzer 514 may infer that the model of engine 526 may not be effective to use under such situations.” Kolar teaches deriving a plurality of thresholds from statistical properties of performance distributions.)
… [performance metrics] based on the respective predictions generated by the currently deployed model.; ([0061] of Kolar states “The service determines that a degradation of the performance of the machine learning model is anomalous, based on the tracked performance of the machine learning model and performance metrics associated with training of the model. The service initiates a corrective measure for the degradation of the performance, in response to determining that the degradation of the performance is anomalous.” [0102] of Kolar states “Accordingly, the tracked performance may be indicative of a percentage of anomalies raised by the anomaly detector for the one or more networks, indicative of a detection probability of the classifier, a recall or precision of the model, etc.” Kolar teaches that the system determines anomalous degradation by comparing the currently deployed model’s tracked prediction performance against various metrics. Under BRI, criterion derived from the currently deployed model’s tracked prediction outputs correspond to “criteria based on the respective predictions generated by the currently deployed model.”)
… based on a comparison of the plurality of quality assurance metrics with the plurality of thresholds and a comparison of the plurality of quality assurance metrics with the criteria, ([0101] of Kolar states “For example, if the severe inference accuracy drop occurs when the data distribution exhibits a median>threshold_1 AND 75th percentile>threshold_2, then analyzer 514 may infer that the model of engine 526 may not be effective to use under such situations… Other corrective measures that analyzer 514 may initiate could also entail raising an alert to the UI or initiating model retraining or reselection (e.g., by sending an action 518 to model selection engine 506).” [0061] of Kolar states “The service determines that a degradation of the performance of the machine learning model is anomalous, based on the tracked performance of the machine learning model and performance metrics associated with training of the model. The service initiates a corrective measure for the degradation of the performance, in response to determining that the degradation of the performance is anomalous.” Kolar teaches that when model quality metrics fail the threshold conditions, corrective action including model replacement is initiated.)
… are configured to predict failure scenarios of the network… ([0102] of Kolar states “In various embodiments, the machine learning model is used to monitor one or more computer networks based on data collected from the one or more computer networks. For example, the model may be configured to detect problems in a wireless network, predict tunnel failures in an SD-WAN or other network, classify devices in the network(s) by device type, or the like. In further embodiments, the model may take the form of an anomaly detector, a classifier, or other form of machine learning model. Accordingly, the tracked performance may be indicative of a percentage of anomalies raised by the anomaly detector for the one or more networks, indicative of a detection probability of the classifier, a recall or precision of the model, etc.”)
Shemer teaches that
automatically… on production run-time machines of the network…, (Col 4 Lines 56 – 63 of Shemer states “In some examples of the disclosed technology, metrics, parameters, or properties of the network collected or analyzed to detect network anomalies can be compared to a past model based on customer behavior and alerting the user when a significant change occurs in real time. In some examples, the alert can be based on a tunable or adjustable threshold over which an alert is generated.” Shemer teaches that automatically taking an action when the network state meets a predetermined threshold condition, with the model deployed on production network machine.)
…by detecting anomalous issues occurring on the network from real-time data from network functions (Col 2 Lines 20 – 27 of Shemer states “obtaining network parameters in real time, selecting a first model to determine a state of the network, selecting a second model to detect a state of the network upon detecting a change in a network pattern, wherein detecting the change is based on at least a current network parameter, evaluating in real time the state of the network, based on the obtained network parameters, using at least one of the first model or second model,” Col 3 Lines 38 – 51 of Shemer states “A second model or multiple models are selected and used to evaluate the state of the network. The first model and a second model can be used to evaluate the current state of the network and can be selected or adjusted according to the presence of a condition internal to the network or a condition external to the network. The computer readable medium containing program instructions can include machine learning of the first model comprises generating weights for network parameters and evaluating the network involves evaluating multiple network parameters simultaneously. The predetermined state can be an anomalous state. A cause for the anomalous network state can be determined using the evaluated current state of the network and the data related to the network parameters.” Col 4 Lines 56 – 63 of Shemer states “In some examples of the disclosed technology, metrics, parameters, or properties of the network collected or analyzed to detect network anomalies can be compared to a past model based on customer behavior and alerting the user when a significant change occurs in real time. In some examples, the alert can be based on a tunable or adjustable threshold over which an alert is generated.” Shemer teaches obtaining network parameters in real time and evaluating the current state of the network using ML models to detect anomalous conditions.)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings of Babu, Kolar, and Shemer because all references are directed to using ML models in a live network environment while accounting for changing network conditions and model performance degradation over time. Shemer teaches the real time network operational context, including obtaining network parameters in real time, evaluating network state, and taking action in response to detected network conditions. Babu teaches managing multiple versions of ML models, evaluating and comparing versions using test data, publishing a selected version for scoring, and swapping versions in production. Kolar teaches monitoring deployed network models using tracked performance metrics, percentile-based threshold, and corrective measures such as alerting, retraining, or reselection when model performance becomes anomalous. One with the ordinary skill in the art would have been motivated to incorporate the teachings of Shemer, Kolar into the teachings of Babu so that deployment decisions would be made using objective quality metrics derived from the monitored model behavior. The combination would have yielded the predictable result of an automated model quality assurance workflow that evaluates candidate network failure models against deployed model performance, apply explicit threshold/criteria checks, and replace or update production models in response to changing network conditions without interrupting live network monitoring.
Regarding Claim 2, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Babu, Kolar, and Shemer teaches
automatically deploying the candidate model in response to the plurality of quality assurance metrics satisfying the plurality of thresholds and the plurality of quality assurance metrics satisfying the criteria. ([0013] of Babu states “updating, during the analyzing, the model selector or the model group based upon determining that the score is below a threshold value. The model group may include one or more machine learning (ML) models, where each ML model in the model group may be configured to perform a same function.” [0089] of Babu states “In some embodiments, the comparison may be visualized using the ML platform. Based on the results of this evaluation, one or more of these models can be chosen to be deployed for scoring.” Under BRI, “update when score falls below threshold” in Babu is when a candidate’s quality assurance metrics satisfy the thresholds, that candidate is selected for deployment. Therefore, Babu teaches this automatic selection and deployment process. [0101] of Kolar states ”For example, if the severe inference accuracy drop occurs when the data distribution exhibits a median>threshold_1 AND 75th percentile>threshold_2, then analyzer 514 may infer that the model of engine 526 may not be effective to use under such situations.” Kolar teaches where deployment occurs when both the threshold and the criteria are satisfied simultaneously.)
Regarding Claim 3, the rejection of claim 2 is incorporated herein. Furthermore, the combination of Babu, Kolar, and Shemer teaches
generating an alert in response to one or more of: at least one of the plurality of quality assurance metrics failing to satisfy at least one of the plurality the thresholds and at least one of the plurality of quality assurance metrics failing to satisfy the criteria. ([0061] of Kolar states “The service determines that a degradation of the performance of the machine learning model is anomalous, based on the tracked performance of the machine learning model and performance metrics associated with training of the model.“ [0101] of Kolar states “Other corrective measures that analyzer 514 may initiate could also entail raising an alert to the UI or initiating model retraining or reselection (e.g., by sending an action 518 to model selection engine 506).” Col 4 Lines 56 – 63 of Shemer states “In some examples of the disclosed technology, metrics, parameters, or properties of the network collected or analyzed to detect network anomalies can be compared to a past model based on customer behavior and alerting the user when a significant change occurs in real time. In some examples, the alert can be based on a tunable or adjustable threshold over which an alert is generated.”)
Regarding Claim 4, the rejection of claim 3 is incorporated herein. Furthermore, the combination of Babu, Kolar, and Shemer teaches
responsive to the alert, generating a visualization comprising a graphical user interface configured to display issue detection results from the candidate model and issue detection results of the currently deployed model, wherein a notification of the alert is provided by the graphical user interface. ([0065] of Kolar states “If ML Ops service 408 detects either condition, service 408 may initiate corrective measures such as sending an alert to a user interface (UI) or instructions back to the monitoring service. For example, service 408 may send instructions 412 to wireless network assurance service 402 that adjust how or when its machine learning model assesses its collected data (e.g., by disabling the model under certain conditions, etc.). In further cases, instructions 412 may even trigger model retraining.“ [0101] of Kolar states “Other corrective measures that analyzer 514 may initiate could also entail raising an alert to the UI or initiating model retraining or reselection (e.g., by sending an action 518 to model selection engine 506).” Column 8 lines 63 – 67 of Shemer states “Alerts 240 and 250 illustrated in FIG. 2A can provide alerts to a user through user interface 200. Alerts 240 and 250 can be generated by a user device or displayed responsive to alerts received through the technology described herein. In some examples, alerts 240 and 250 can be interacted with by a user. Upon receiving a response from the alert, one or more components of cloud 101 can take an action responsive to the response.” [0074] of Babu states “In some embodiments, the machine learning platform may provide API(s) for evaluating a model using test data. In some embodiments, the machine learning platform may provide API(s) for comparing different models, comparing different versions of a model, or comparing a rule-based model (or manually crafted model) and a machine learning model… In some embodiments, the comparison may be visualized using the ML platform.”)
Regarding Claim 5, the rejection of claim 4 is incorporated herein. Furthermore, the combination of Babu, Kolar, and Shemer teaches
deploying the candidate model responsive to an input that is based on the visualization of the candidate model and the currently deployed model. ([0075] of Babu states ”In some embodiments, the machine learning platform may provide API(s) for publishing a version of a model for scoring. In some embodiments, the API(s) may specify whether a given version is the default version for scoring.” [0056] of Kolar states “Cloud service 302 may also include output and visualization interface 318 configured to provide sensory data to a network administrator or other user via one or more user interface devices (e.g., an electronic display, a keypad, a speaker, etc.). For example, interface 318 may present data indicative of the state of the monitored network, current or predicted issues in the network (e.g., the violation of a defined rule, etc.), insights or suggestions regarding a given condition or issue in the network, etc. Cloud service 302 may also receive input parameters from the user via interface 318 that control the operation of system 300 and/or the monitored network itself. For example, interface 318 may receive an instruction or other indication to adjust/retrain one of the models of analyzer 312 from interface 318 (e.g., the user deems an alert/rule violation as a false positive).”)
Regarding Claim 6, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Babu, Kolar, and Shemer teaches
the plurality of quality assurance metrics for the candidate model comprises a value for the candidate model indicative of the network failure prediction data generated by the candidate model and a scale range of the network failure prediction data generated by the candidate model. ([0102] of Kolar states “Accordingly, the tracked performance may be indicative of a percentage of anomalies raised by the anomaly detector for the one or more networks, indicative of a detection probability of the classifier, a recall or precision of the model, etc.” [0080] of Kolar states “DCD 522 may regularly compute the data distribution change metrics 528 for the data consumed by model inference engine 526 and transmit metrics 528 to model performance analyzer (MPA) 502 on a push, pull, or periodic basis. For example, data distribution change metrics 528 may be of the form<customer, timestamp, feature, distribution-difference, confidence, min, percentile-25, percentile-50, percentile-75, max>, where the last few metrics showcase the approximate distribution of the variables by using the 25th, 50th and 75th percentiles with the maximum and minimum values of the data feature.” [0101] of Kolar states “For example, if the severe inference accuracy drop occurs when the data distribution exhibits a median>threshold_1 AND 75th percentile>threshold_2, then analyzer 514 may infer that the model of engine 526 may not be effective to use under such situations” Kolar teaches that QA metrics for a network failure prediction model comprise both a point value metric and a distribution range metric. [0080] of Babu states “In some embodiments, the machine learning platform may provide API(s) for reporting statistics on the usage of a model, such as the number of requests served by the model in a given time window, and the distribution of predictions by category for classification models and binning for regression models.”)
Regarding Claim 7, the rejection of claim 6 is incorporated herein. Furthermore, the combination of Babu, Kolar, and Shemer teaches
the plurality of thresholds comprises: a first threshold based on a first performance distribution of values indicative of prediction data associated with each of the plurality of previously deployed models, and a second threshold based on a second performance distribution of scale ranges of the prediction data associated with each of the plurality of previously deployed models. ([0101] of Kolar states “For example, if the severe inference accuracy drop occurs when the data distribution exhibits a median>threshold_1 AND 75th percentile>threshold_2, then analyzer 514 may infer that the model of engine 526 may not be effective to use under such situations. In turn, analyzer 514 may propagate such a rule to model inference engine 526 as data check actions 534, which engine 526 uses to assess the distribution characteristics of the data and determines whether or not to pause use of the model (e.g., on a daily basis).” Under BRI, “a first threshold based on a first performance distribution of values” map precisely onto threshold_1 derived from the distribution of value metrics from prior model versions and “a second threshold based on a second performance distribution of scale ranges” onto threshold_2 derived from the distribution of scale range metrics from prior model versions. [0100] of Kolar states “In a further embodiment, MPA 502 may also include data-caused inference degradation analyzer 514 that is responsible for determining whether any data distribution changes (e.g., as indicated by metrics 528 from DCD 522) are responsible for any degradation in the inference accuracy metrics 530 from IA 524. For example, analyzer 514 may discretize the inference accuracy metrics 530 (e.g., across all deployments) into ‘Positive’ and ‘Negative’ classes, where the ‘Positive’ class indicates severe drops in the inference accuracy metric(s) and the ‘Negative’ class indicates normal changes. In turn, analyzer 514 may train a decision tree using the classes and the data distribution change metrics 528 (e.g., median, 75th percentile, etc.) to identify rules that can be used for purposes of root-causing model performance issues.”)
Regarding Claim 8, the rejection of claim 7 is incorporated herein Furthermore, the combination of Babu, Kolar, and Shemer teaches
the criteria comprises a scale range of the respective predictions associated with the currently deployed model. ([0080] of Kolar states “For example, data distribution change metrics 528 may be of the form<customer, timestamp, feature, distribution-difference, confidence, min, percentile-25, percentile-50, percentile-75, max>, where the last few metrics showcase the approximate distribution of the variables by using the 25th, 50th and 75th percentiles with the maximum and minimum values of the data feature.” [0061] of Kolar states “The service also tracks performance metrics associated with training of the machine learning model. The service determines that a degradation of the performance of the machine learning model is anomalous, based on the tracked performance of the machine learning model and performance metrics associated with training of the model. The service initiates a corrective measure for the degradation of the performance, in response to determining that the degradation of the performance is anomalous.” Kolar teaches using the currently deployed model’s prediction scale range as the deployment criterion. Under BRI, “the criteria comprises a scale range of the respective predictions associated with the currently deployed model” reads on Kolar’s use of the deployed model’s tracked prediction distribution range. [0046] of Babu states “As used herein, scoring using a model may involve generating predictions for given data using the model. The predictions can be categorical predictions or numeric values based on the type (classification vs. regression) of the model.” [0077] of Babu states “A scoring response may include the version of the model that was used to score the request. The version information may be used in the application layer to gather statistics regarding effectiveness of the model.”)
Regarding Claim 9, the rejection of claim 8 is incorporated herein. Furthermore, the combination of Babu, Kolar, and Shemer teaches
determining that the candidate model satisfies a first condition where the value of the candidate model is less than value corresponding to a first percentile of the first performance distribution; determining that the candidate model satisfies a second condition where the scale range of the candidate model is less than a scale range corresponding to a second percentile of the second performance distribution; (([0101] of Kolar states “For example, if the severe inference accuracy drop occurs when the data distribution exhibits a median>threshold_1 AND 75th percentile>threshold_2, then analyzer 514 may infer that the model of engine 526 may not be effective to use under such situations. In turn, analyzer 514 may propagate such a rule to model inference engine 526 as data check actions 534, which engine 526 uses to assess the distribution characteristics of the data and determines whether or not to pause use of the model (e.g., on a daily basis).” [0100] of Kolar states “In a further embodiment, MPA 502 may also include data-caused inference degradation analyzer 514 that is responsible for determining whether any data distribution changes (e.g., as indicated by metrics 528 from DCD 522) are responsible for any degradation in the inference accuracy metrics 530 from IA 524. For example, analyzer 514 may discretize the inference accuracy metrics 530 (e.g., across all deployments) into ‘Positive’ and ‘Negative’ classes, where the ‘Positive’ class indicates severe drops in the inference accuracy metric(s) and the ‘Negative’ class indicates normal changes. In turn, analyzer 514 may train a decision tree using the classes and the data distribution change metrics 528 (e.g., median, 75th percentile, etc.) to identify rules that can be used for purposes of root-causing model performance issues.” [0102] of Kolar states “Accordingly, the tracked performance may be indicative of a percentage of anomalies raised by the anomaly detector for the one or more networks, indicative of a detection probability of the classifier, a recall or precision of the model, etc.” )
and determining that the candidate model satisfies a third condition where the scale range of the candidate model overlaps with the scale range of the currently deployed model, ([0129] of Babu states “For example, the scheme for using the selected at least one ML model to analyze the input data may include analyzing a same portion of the input data by each of the selected at least one ML model, and selecting, from results of analyzing the same portion of the input data by the selected at least one ML model, a most common result as a result for the portion of the input data.” [0080] of Kolar states “For example, data distribution change metrics 528 may be of the form<customer, timestamp, feature, distribution-difference, confidence, min, percentile-25, percentile-50, percentile-75, max>, where the last few metrics showcase the approximate distribution of the variables by using the 25th, 50th and 75th percentiles with the maximum and minimum values of the data feature.” Babu teaches applying the same test data to both the candidate and currently deployed model to get their respective predictions. Kolar teaches computing the distribution-difference metric between those prediction distribution. Under BRI, a distribution-difference within acceptable bounds establishes that the scale ranges overlap.)
wherein the candidate model is automatically deployed responsive to determining the candidate model satisfied the first, second, and third conditions. ([0101] of Kolar states “For example, if the severe inference accuracy drop occurs when the data distribution exhibits a median>threshold_1 AND 75th percentile>threshold_2, then analyzer 514 may infer that the model of engine 526 may not be effective to use under such situations… Other corrective measures that analyzer 514 may initiate could also entail raising an alert to the UI or initiating model retraining or reselection (e.g., by sending an action 518 to model selection engine 506).” Under BRI, “the candidate model is automatically deployed responsive to determining the candidate model satisfies the first, second, and third conditions” reads with Kolar’s multi condition gate. When all conditions are met, model reselection is initiated, which corresponds to the automatic deployment. )
Regarding Claim 21, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Babu, Kolar, and Shemer teaches
prior to generating the candidate model, predicting by the currently deployed model deployed on the production run-time machines, failure scenarios of the network by detecting anomalous issues occurring on the network from real-time data from network functions, ([0061] of Kolar states “Specifically, according to one or more embodiments of the disclosure as described in detail below, a service tracks performance of a machine learning model over time. The machine learning model is used to monitor one or more computer networks based on data collected from the one or more computer networks. The service also tracks performance metrics associated with training of the machine learning model.” [0064] of Kolar states ”For example, as shown, ML Ops service 408 may oversee the operation of a wireless network assurance service 402 that uses machine learning to monitor a wireless network, a software-defined wide area network (SD-WAN) assurance service 404 that uses machine learning to monitor an SD-WAN (e.g., to predict tunnel failures, etc.), a device classification service 406 that uses machine learning to classify devices in a network by device type, based on their behaviors, and/or any other machine learning-based network services.” [0046] of Kolar states ”To support the operations of the network, there may be any number of network services and control plane functions 310. For example, functions 310 may include routing topology and network metric collection functions such as, but not limited to, routing protocol exchanges, path computations, monitoring services (e.g., NetFlow or IPFIX exporters), etc.”)
wherein deploying the candidate model on the production run-time machines comprises predicting, by the candidate model, failure scenarios of the network by detecting anomalous issues occurring on the network from real-time data from the network functions. ([0081] of Babu states “Models published for scoring may be refreshed as new models are available with new training data. This would allow transparently retraining models with recent data to prevent model drift.” [0078] of Babu states “Batch data may be analyzed with a very high throughput, for example, in the order of a million or more records per second. There may be no downtime when model versions are being swapped.” [0064] of Kolar states “For example, as shown, ML Ops service 408 may oversee the operation of a wireless network assurance service 402 that uses machine learning to monitor a wireless network, a software-defined wide area network (SD-WAN) assurance service 404 that uses machine learning to monitor an SD-WAN (e.g., to predict tunnel failures, etc.), a device classification service 406 that uses machine learning to classify devices in a network by device type, based on their behaviors, and/or any other machine learning-based network services.” Oracle teaches that when a new model is deployed, it immediately begins serving the same function as its predecessor. Kolar teaches that once a candidate model is deployed into MLOps platform, it inherits the same prediction configuration and predict tunnel failures from real time network data.)
Claims 10 – 16 recite substantially similar subject matter to claim 1 – 3, 6 – 9 respectively and are rejected with the same rationale, mutatis mutandis.
Claim 17 – 19 recite substantially similar subject matter to claims 1 – 3 respectively expressed in the context of a non-transitory computer-readable medium and are rejected with the same rationale, mutatis mutandis.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BYUNGKWON HAN whose telephone number is (571) 272-5294. The examiner can normally be reached M-F: 9:00AM-6PM PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at (571)272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BYUNGKWON HAN/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights