DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 03/16/2026, for application 17/938,806 has been entered.
This Office Action is in response to the Amendment filed on 03/16/2026. In the instant amendment: Claims 1, 2, 4, 5, 7, 8, 9, 12, 13, 15, 17, 18 have been amended. Claims 10-11, 20 are cancelled, claim 23 newly added. Claims 1-9, 12-19, 21-23 have been examined and are pending.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 03/16/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements have been considered by the examiner.
Response to Arguments
Applicants' arguments in the instant Amendment, filed on 03/16/2026, with respect to limitations listed below, have been fully considered but they are not persuasive.
Applicant Argues: Abbaszadeh in view of Olgiati fails to disclose “performing, by the anomaly detector, an anomaly detection process to determine whether the data is anomalous by at least,” “performing a first action set in response to determining that the data is anomalous to modify operations of a device that provides, at least in part, the computer-implemented services,” “performing, by the anomaly detector, a data drift detection process separate from the anomaly detection process to determine whether the continuous inference model has adapted to data drift by at least,” and “performing a second action set separate from the first action set in response to determining that the continuous inference model has adapted to the data drift to modify the operations of the device that provides, at least in part, the computer- implemented services” of amended claim 1. See Remarks at 12-13 (emphasis added).
The examiner respectfully disagrees because these arguments are not persuasive.
In regards to, “performing, by the anomaly detector, an anomaly detection process to determine whether the data is anomalous…,” Abbaszadeh discloses “At S210, an abnormal detection computer platform 150 may receive from a plurality or real time monitoring nodes 110, monitoring node signal values 112. Then at S212, the abnormal detection computer platform 150 may compute an anomaly score 132/126 for at least one of a given monitoring node 110 and the system 100, respectively. Note that module 120 contains one detection model and uses one threshold for anomaly detection at the system level, but module 130 may contain one or more detection models and use one or more thresholds for anomaly location for each individual node (e.g., a separate model and threshold per node.” See Abbaszadeh ¶¶ [0033]-[0034] (emphasis added). Thus, Abbaszadeh discloses calculating an anomaly score based on the received monitoring data, and declaring an anomaly based on threshold-based criteria.
In regards to, “performing a first action set in response to determining that the data is anomalous to modify operations of a device that provides, at least in part, the computer-implemented services,” Abbaszadeh teaches “The decision boundary can then be used to detect abnormal operation (e.g., as might occur during cyber-attacks)… At S520, an attack detection platform computer may then generate, based on the received series of current values, a set of current feature vectors. At S530, an abnormal detection model may be executed to transmit an abnormal alert signal based on the set of current feature vectors, an anomaly score 126/132, the adaptive threshold 140 and a decision boundary when appropriate (e.g., when a cyber-attack is detected). According to some embodiments, one or more response actions may be performed when an abnormal alert signal is transmitted. For example, the system might automatically shut down all or a portion of the cyber-physical system (e.g., to let the detected potential cyber-attack be further investigated). As other examples, one or more parameters might be automatically modified, a software application might be automatically triggered to capture data and/or isolate possible causes, etc.” See Abbaszadeh ¶ [0067] (emphasis added). Thus, Abbaszadeth teaches detecting an anomaly or cyber-attack using a threshold-based model, and taking a remedial action (e.g., isolate suspected network nodes) in response to the detected anomaly or cyber-attack.
In regards to, “performing, by the anomaly detector, a data drift detection process separate from the anomaly detection process to determine whether the continuous inference model has adapted to data drift …,” Abbaszadeth teaches using a machine-learning based continuous inference model for anomaly detection decision1. See Abbaszedeth ¶¶ [0054], [0069] (“The continuous process 310 may not switch between a finite/pre-set number of values, as with the discrete process 308, but instead may generate any value that may be continuously adapted using one of a static model 318 or a dynamic model 320. The continuous process 310 may be a model-based approach in which modeling for the adaptive threshold 140 may become part of the abnormal detection model/module training. Moreover, multiple algorithmic methods (e.g., support vector machines or machine learning techniques) may be used to generate decision boundaries.”) (emphasis added).
Furthermore, Olgiati teaches about detecting deviation in machine learning model inference outputs (subject to threshold performance requirements), and to retrain/update the machine learning model to optimize performance and limit the effect of data drift:
The analysis system 170 may determine the correctness of inputs, outputs, and intermediate steps of the inference production and/or the quality of deployed machine learning models. The analysis system 170 may include a component for automated problem detection 172 that attempts to find one or more types of problems, anomalies, or other flaws in a model or its input data. As will be discussed in greater detail with respect to FIG. 6 through FIG. 11, the analysis may automatically detect problems or anomalies such as models that fail golden examples, outliers in input data, inference data distribution changes, label distribution changes, label changes for individual entities, ground truth discrepancies, and/or other forms of data drift or model drift. In some embodiments, the analysis may utilize training data 112 and/or testing data 114.
The analysis may be performed according to thresholds that determine whether a given observation about the model rises to the level of a problem that may require intervention. As shown in 570, if a problem was detected, then one or more actions may be initiated by the analysis system to remediate the problem. The problem remediation may initiate one or more actions to improve a model or its use in generating inferences… In one embodiment, the analysis may result in automatically initiating retraining of machine learning models based on the problem detection.
See Olgiati ¶¶ [0027], [0046] (emphasis added).
Here, Olgiati teaches a threshold-based criteria for analyzing the performance of machine-learning modes. When inference output from such models falls below a performance threshold, data drift or model drift is detected, and retraining or other remedial actions can be applied. With Abbaszedeth’s machine learning model for detecting network anomaly, and Olgiatis’s criteria for monitoring and retraining machine learning model in response to detected model/data drift and performance degradation, the combination of Abbaszedeth and Olgiati teaches “performing, by the anomaly detector, a data drift detection process separate from the anomaly detection process to determine whether the continuous inference model has adapted to data drift …” of amended claim 1.
In regards to, “performing a second action set separate from the first action set in response to determining that the continuous inference model has adapted to the data drift to modify the operations of the device that provides, at least in part, the computer- implemented services,” Olgiati teaches a threshold-based criteria for analyzing the performance of machine-learning modes. As discussed above, supra, when inference output from such models falls below a performance threshold, data drift or model drift is detected, and retraining or other remedial actions can be applied. See also Olgiati ¶¶ [0027], [0046]. The retrained and updated machine learning model can perform further inference outputs. Thus, with Abbaszedeth’s machine learning model for detecting network anomaly, and Olgiatis’s criteria for monitoring and retraining machine learning model in response to model/data drift and degradation of model performance, the combination of Abbaszedeth and Olgiati teaches “performing a second action set separate from the first action set in response to determining that the continuous inference model has adapted to the data drift to modify the operations of the device that provides, at least in part, the computer- implemented services” of amended claim 1.
Thus, in conclusion, applicant’s argument is unpersuasive and the rejection of amended claims 1 is maintained. Rejection of claims 12 and 17, which recite similar matters, is similarly maintained.
Applicant Argues: Abbaszadeh in view of Olgiati and Green fails to disclose “wherein classifying the data using the quantized inference model comprises: quantizing the data obtained from the data collector to obtained a quantized version of the data” of amended claim 4. See Remarks at 14.
The examiner respectfully disagrees because these arguments are not persuasive.
In regards to, “quantizing the data obtained from the data collector to obtained a quantized version of the data,” Green teaches “the computer system enables the user to determine an appropriate quantization for data sampling, analysis, and output within these containerized applications in order to conserve memory and/or computing resources on corresponding devices. In particular, the computer system can: generate a first set of containerized applications including data sampling, data processing, and data output instructions at a first quantization … [A]pply outputs from the first set of containerized applications at the first quantization in a first instance of a machine learning model; and apply outputs from the second set of containerized applications in a second instance of the machine learning model, thereby enabling the user to compare results of the model and/or insights based on the model at each quantization, identify a desired quantization based on these results, memory considerations, etc., and select a corresponding set of containers to retain on and/or uninstall from the set of devices.” See Green ¶ [0027] (emphasis added).
Because Green teaches using different quantized data inputs, sampling, and/or model quantization for evaluating and comparing model performance, Green teaches at least “quantizing the data obtained from the data collector to obtained a quantized version of the data” of amended claim 4. Thus, in conclusion, applicant’s argument is unpersuasive and the rejection of amended claims 4 is maintained. Rejection of claim 15, which recite similar matters, is similarly maintained.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically discloses as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 12, 17, 21 and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Abbaszadeh et al. (“Abbaszadeh,” US 20220329613, filed April 12, 2021) in view of Olgiati et al. (“Olgiati,” US 20210097433, published April 21, 2021).
Regarding claim 1, Abbaszadeh discloses A method of providing computer-implemented services, the method comprising:
obtaining, by an anomaly detector, data from a data collector (Abbaszadeh FIG. 3B, [0037], [0043]. The normal space data source 320 might store, for each of the plurality of monitoring nodes 310, a series of normal values over time that represent normal operation of a cyber-physical system (e.g., generated by a model or collected from actual sensor data as illustrated by the dashed line in FIG. 3A). The abnormal space data source 330 might store, for each of the monitoring nodes 310, a series of abnormal values that represent abnormal operation of the cyber-physical system (e.g., when the system is experiencing a cyber-attack or a fault). FIG. 3B illustrates a block diagram including an adaptive thresholding system 145 according to some embodiments. The monitoring node raw values 112 may be received by the system 150, and in particular a data pre-processing module 302.);
performing, by the anomaly detector, an anomaly detection process to determine whether the data is anomalous by at least (Abbaszadeh [0033]-[0034]. At S210, an abnormal detection computer platform 150 may receive from a plurality or real time monitoring nodes 110, monitoring node signal values 112. Then at S212, the abnormal detection computer platform 150 may compute an anomaly score 132/126 for at least one of a given monitoring node 110 and the system 100, respectively. Note that module 120 contains one detection model and uses one threshold for anomaly detection at the system level, but module 130 may contain one or more detection models and use one or more thresholds for anomaly location for each individual node (e.g., a separate model and threshold per node.):
classifying, by the anomaly detector, the data using a continuous inference model and an anomaly threshold to obtain a first classification, the first classification specifying whether the data is considered anomalous or non-anomalous (Abbaszadeh [0054], [0064], [0103]. The continuous process 310 may be a model-based approach in which modeling for the adaptive threshold 140 may become part of the abnormal detection model/module training. It is noted that the prediction may use a Gaussian distribution or any other continuous distribution, which may be determined by fitting the anomaly scores to a statistical distribution. For example, the processor 1710 may receive from a detection model an anomaly score for each monitoring node and, then compare that anomaly score to an adaptive threshold to determine whether each monitoring node has a local status of “normal” or “abnormal.” The processor 1710 may also receive, from a detection model an anomaly score for the whole system and then compare that anomaly score to an adaptive threshold to determine whether the system has a global status of “normal” or “abnormal”.2 The processor 1710 may then output, for each monitoring node, a local status of “normal” or “abnormal,”. The processor 1710 may also output a global status of “normal” or “abnormal” for the system.); and
performing a first action set in response to determining that the data is anomalous to modify operations of a device that provides, at least in part, the computer-implemented services (Abbaszadeh [0067]. The decision boundary can then be used to detect abnormal operation (e.g., as might occur during cyber-attacks)… At S520, an attack detection platform computer may then generate, based on the received series of current values, a set of current feature vectors. At S530, an abnormal detection model may be executed to transmit an abnormal alert signal based on the set of current feature vectors, an anomaly score 126/132, the adaptive threshold 140 and a decision boundary when appropriate (e.g., when a cyber-attack is detected). According to some embodiments, one or more response actions may be performed when an abnormal alert signal is transmitted. For example, the system might automatically shut down all or a portion of the cyber-physical system (e.g., to let the detected potential cyber-attack be further investigated). As other examples, one or more parameters might be automatically modified, a software application might be automatically triggered to capture data and/or isolate possible causes, etc.); and
performing, by the anomaly detector, [a data drift detection process separate from the anomaly detection process to determine whether the continuous inference model has adapted to data drift by at least] (Abbaszedeth [0054], [0069]. The continuous process 310 may not switch between a finite/pre-set number of values, as with the discrete process 308, but instead may generate any value that may be continuously adapted using one of a static model 318 or a dynamic model 320. The continuous process 310 may be a model-based approach in which modeling for the adaptive threshold 140 may become part of the abnormal detection model/module training. Moreover, multiple algorithmic methods (e.g., support vector machines or machine learning techniques) may be used to generate decision boundaries.”):
classifying, by the anomaly detector, the data using a quantized inference model and the anomaly threshold to obtain a second classification, the second classification specifying whether the data is considered anomalous or non-anomalous (Abbaszadeh [0050], [0103]. The discrete process 308 may also be referred to as a “multi-level” threshold process. The discrete process 308 may quantize the threshold into multiple levels, where each level may correspond to a mode of operation, the region of operation. The multi-level thresholds may be computed using a rule-based logic or a machine learning, such as a decision tree. In this process 308, the threshold is switched among a finite (aka discrete) number of values. In one or more embodiments, the threshold levels associated with each trigger element 314 may be optimized offline as part of the model training process. For example, the processor 1710 may receive from a detection model an anomaly score for each monitoring node and, then compare that anomaly score to an adaptive threshold to determine whether each monitoring node has a local status of “normal” or “abnormal.”. The processor 1710 may also receive, from a detection model an anomaly score for the whole system and then compare that anomaly score to an adaptive threshold to determine whether the system has a global status of “normal” or “abnormal”. The processor 1710 may then output, for each monitoring node, a local status of “normal” or “abnormal,”. The processor 1710 may also output a global status of “normal” or “abnormal” for the system.).
Abbaszadeh does not explicitly disclose:
performing, by the anomaly detector, a data drift detection process separate from the anomaly detection process to determine whether the continuous inference model has adapted to data drift by at least:
making a first determination, by the anomaly detector, regarding whether the first classification specifies that the data is considered non-anomalous and the second classification specifies that the data is considered anomalous;
in a first instance of the first determination where the first classification specifies that the data is considered non-anomalous and the second classification specifies that the data is anomalous:
determining, by the anomaly detector, that the continuous inference model has adapted to the data drift;
and performing a second action set separate from the first action set in response to determining that the continuous inference model has adapted to the data drift to modify the operations of the device that provides, at least in part, the computer- implemented services.
However, in an analogous art, Olgiati discloses a method comprising the step of:
performing, by the anomaly detector, a data drift detection process separate from the anomaly detection process to determine whether the continuous inference model has adapted to data drift by at least (Olgiati [0027], [0045]-[0046]. The analysis system 170 may determine the correctness of inputs, outputs, and intermediate steps of the inference production and/or the quality of deployed machine learning models. The analysis system 170 may include a component for automated problem detection 172 that attempts to find one or more types of problems, anomalies, or other flaws in a model or its input data. As will be discussed in greater detail with respect to FIG. 6 through FIG. 11, the analysis may automatically detect problems or anomalies such as models that fail golden examples, outliers in input data, inference data distribution changes, label distribution changes, label changes for individual entities, ground truth discrepancies, and/or other forms of data drift or model drift. In some embodiments, the analysis may utilize training data 112 and/or testing data 114. Note that the inference generation shown in 500 may be performed continuously or regularly. The analysis may be performed according to thresholds that determine whether a given observation about the model rises to the level of a problem that may require intervention. As shown in 570, if a problem was detected, then one or more actions may be initiated by the analysis system to remediate the problem. The problem remediation may initiate one or more actions to improve a model or its use in generating inferences… In one embodiment, the analysis may result in automatically initiating retraining of machine learning models based on the problem detection.):
making a first determination, by the anomaly detector, regarding whether the first classification specifies that the data is considered non-anomalous and the second classification specifies that the data is considered anomalous (Olgiati FIG. 6, [0031], [0047]. The problem remediation 172 may initiate one or more actions to improve a model or its use in generating inferences, e.g., such that the inferences produced by the improved model (or the same model using an improved input data set) represent a higher degree of accuracy and/or usefulness for the application 190. In one embodiment, the analysis system 170 may automatically initiate retraining of machine learning models based on problem detection. FIG. 6 illustrates further aspects of the example system environment for automated problem detection for machine learning models, including golden example discrepancy analysis for a machine learning model, according to some embodiments. In one embodiment, the analysis 172 may automatically detect problems or anomalies such as models that fail verified or “golden” examples. Golden example discrepancy analysis 672 may use a repository of testing data 114 associated with one or more golden examples. The testing data 114 may be regularly executed in batch or against the endpoint to check that the model 135 continues to work as expected with the testing data 114, e.g., by comparing results of the inference production 152A to expected results 115 of the testing data. By monitoring the quality of the model using golden examples, the golden example discrepancy analysis 672 may detect inadvertent deployment of a faulty model, detect changes in the production environment (e.g., changes in a dependency that have impacted the model), and/or ensure that new versions of a model do not break fundamental use cases.);
in a first instance of the first determination where the first classification specifies that the data is considered non-anomalous and the second classification specifies that the data is anomalous (Olgiati FIG. 6, [0031], [0047]. The problem remediation 172 may initiate one or more actions to improve a model or its use in generating inferences, e.g., such that the inferences produced by the improved model (or the same model using an improved input data set) represent a higher degree of accuracy and/or usefulness for the application 190. In one embodiment, the analysis system 170 may automatically initiate retraining of machine learning models based on problem detection. FIG. 6 illustrates further aspects of the example system environment for automated problem detection for machine learning models, including golden example discrepancy analysis for a machine learning model, according to some embodiments. In one embodiment, the analysis 172 may automatically detect problems or anomalies such as models that fail verified or “golden” examples. Golden example discrepancy analysis 672 may use a repository of testing data 114 associated with one or more golden examples. The testing data 114 may be regularly executed in batch or against the endpoint to check that the model 135 continues to work as expected with the testing data 114, e.g., by comparing results of the inference production 152A to expected results 115 of the testing data. By monitoring the quality of the model using golden examples, the golden example discrepancy analysis 672 may detect inadvertent deployment of a faulty model, detect changes in the production environment (e.g., changes in a dependency that have impacted the model), and/or ensure that new versions of a model do not break fundamental use cases.):
determining, by the anomaly detector, that the continuous inference model has adapted to the data drift (Olgiati [0027], [0039], [0045]. A machine learning analysis system 170 may use the collected inference data in the data store 160 to perform automated analysis of inference production 152A. The analysis system 170 may determine the correctness of inputs, outputs, and intermediate steps of the inference production and/or the quality of deployed machine learning models. [T]he analysis may automatically detect problems or anomalies such as models that fail golden examples, outliers in input data, inference data distribution changes, label distribution changes, label changes for individual entities, ground truth discrepancies, and/or other forms of data drift or model drift. In some embodiments, the analysis may utilize training data 112 and/or testing data 114. The retrained model 126 may be automatically tested to produce a tested model 136 that may again be used to produce inferences using the inference system 140. The model may be periodically retrained using the automated analysis 170 so that the machine learning system 100 adapts to changes in input data, service dependencies, and so on. Note that the inference generation shown in 500 may be performed continuously or regularly.); and
performing a second action set separate from the first action set in response to determining that the continuous inference model has adapted to the data drift to modify the operations of the device that provides, at least in part, the computer- implemented services (Olgiati [0027],[0045]- [0046]. The analysis system 170 may determine the correctness of inputs, outputs, and intermediate steps of the inference production and/or the quality of deployed machine learning models. The analysis system 170 may include a component for automated problem detection 172 that attempts to find one or more types of problems, anomalies, or other flaws in a model or its input data. As will be discussed in greater detail with respect to FIG. 6 through FIG. 11, the analysis may automatically detect problems or anomalies such as models that fail golden examples, outliers in input data, inference data distribution changes, label distribution changes, label changes for individual entities, ground truth discrepancies, and/or other forms of data drift or model drift. In some embodiments, the analysis may utilize training data 112 and/or testing data 114. Note that the inference generation shown in 500 may be performed continuously or regularly. The analysis may be performed according to thresholds that determine whether a given observation about the model rises to the level of a problem that may require intervention. As shown in 570, if a problem was detected, then one or more actions may be initiated by the analysis system to remediate the problem. The problem remediation may initiate one or more actions to improve a model or its use in generating inferences… In one embodiment, the analysis may result in automatically initiating retraining of machine learning models based on the problem detection.).
Therefore, it would have been obvious to one of ordinary skill in the art on or before the effective date of the claimed invention to combine the embodiments of Abbaszadeh and Olgiati to include the step of: performing a first action set in response to the data drift to modify operation of a device that provides, at least in part, the computer implemented services. One would have been motivated to provide users with a means for evaluate machine learning inference model performance and retrain the model according to performance benchmarks. (See Olgiati [0031].)
Regarding claim 12, claim 12 is directed to a non-transitory computer readable medium corresponding to the method of claim 1. Claim 12 is similar to claim 1 and is therefore rejected under similar rationale.
Regarding claim 17, claim 17 is directed to a system corresponding to the method of claim 1. Claim 17 is similar to claim 1 and is therefore rejected under similar rationale.
Regarding claim 21, Abbaszadeh and Olgiati disclose the method of claim 1. Abbaszadeh further discloses wherein the continuous interference model is adapted to generate inferences over a continuous range of values, the quantized inference model is adapted to generate inferences over a quantized range of values, the continuous range and the quantized range having a same extent (Abbaszadeh [0049]-[0050], [0054]. The adaptive thresholding system 145 may apply either a discrete process 308 or a continuous process 310 to generate the adaptive threshold 140, as indicated in the chart 312 in FIG. 3E. The multi-level thresholds may be computed using a rule-based logic or a machine learning, such as a decision tree. In this process 308, the threshold is switched among a finite (aka discrete) number of values. With this process 308, the adaptive threshold 140 applied to the anomaly score 126/132 may switch between the pre-set/discrete threshold levels based on a trigger element 314 (i.e., the mode of operation, ambient level [of an electronic device or electro-mechanical device, see [0052]-[0053]], etc.)). The continuous process 310 [ ] may generate any value that may be continuously adapted using one of a static model 318 or a dynamic model 320. The continuous process 310 may be a model-based approach in which modeling for the adaptive threshold 140 may become part of the abnormal detection model/module training.).
Regarding claim 23, Abbaszadeh and Olgiati disclose the method of claim 1. Olgiati further discloses wherein the first action set comprises the data drift detection process, and the data drift detection process is only performed as part of the first action set when the data is determined to be anomalous during the anomaly detection process (Olgiati [0027], [0045]-[0046]. The analysis system 170 may determine the correctness of inputs, outputs, and intermediate steps of the inference production and/or the quality of deployed machine learning models. The analysis system 170 may include a component for automated problem detection 172 that attempts to find one or more types of problems, anomalies, or other flaws in a model or its input data. As will be discussed in greater detail with respect to FIG. 6 through FIG. 11, the analysis may automatically detect problems or anomalies such as models that fail golden examples, outliers in input data, inference data distribution changes, label distribution changes, label changes for individual entities, ground truth discrepancies, and/or other forms of data drift or model drift. In some embodiments, the analysis may utilize training data 112 and/or testing data 114. Note that the inference generation shown in 500 may be performed continuously or regularly. The analysis may be performed according to thresholds that determine whether a given observation about the model rises to the level of a problem that may require intervention. As shown in 570, if a problem was detected, then one or more actions may be initiated by the analysis system to remediate the problem. The problem remediation may initiate one or more actions to improve a model or its use in generating inferences… In one embodiment, the analysis may result in automatically initiating retraining of machine learning models based on the problem detection.).
The motivation is the same as that of claim 1 above.
Claims 2-8, 13-16, 18-19, 22 are rejected under 35 U.S.C. 103 as being unpatentable over Abbaszadeh et al. (“Abbaszadeh,” US 20220329613, filed April 12, 2021) in view of Olgiati et al. (“Olgiati,” US 20210097433, published April 21, 2021) and Green et al. (“Green,” US 20220171863, published June 2, 2022).
Regarding claim 2, Abbaszadeh and Olgiati disclose the method of claim 2.
Abbaszadeh further discloses making a second determination regarding whether the first classification specifies that the data is considered anomalous; in a first instance of the second determination where the first classification specifies that the data is considered anomalous (Abbaszadeh [0103]. For example, the processor 1710 may receive from a detection model an anomaly score for each monitoring node and, then compare that anomaly score to an adaptive threshold to determine whether each monitoring node has a local status of “normal” or “abnormal.”. The processor 1710 may also receive, from a detection model an anomaly score for the whole system and then compare that anomaly score to an adaptive threshold to determine whether the system has a global status of “normal” or “abnormal”. The processor 1710 may then output, for each monitoring node, a local status of “normal” or “abnormal,”. The processor 1710 may also output a global status of “normal” or “abnormal” for the system.).
Olgiati further discloses wherein performing the data drift detection process comprises:
in a second instance of the first determination where the first classification does not specify that the data is considered non-anomalous and the second classification does not specify that the data is considered anomalous (Olgiati FIG. 6, [0047]. FIG. 6 illustrates further aspects of the example system environment for automated problem detection for machine learning models, including golden example discrepancy analysis for a machine learning model, according to some embodiments. In one embodiment, the analysis 172 may automatically detect problems or anomalies such as models that fail verified or “golden” examples. Golden example discrepancy analysis 672 may use a repository of testing data 114 associated with one or more golden examples. The testing data 114 may be regularly executed in batch or against the endpoint to check that the model 135 continues to work as expected with the testing data 114, e.g., by comparing results of the inference production 152A to expected results 115 of the testing data. By monitoring the quality of the model using golden examples, the golden example discrepancy analysis 672 may detect inadvertent deployment of a faulty model, detect changes in the production environment (e.g., changes in a dependency that have impacted the model), and/or ensure that new versions of a model do not break fundamental use cases.):
performing a third action set in response to the data being considered anomalous (Olgiati [0031]. The analysis system 170 may include a component for automated problem remediation 174 that attempts to remediate, correct, or otherwise improve a detected problem. The problem remediation 172 may initiate one or more actions to improve a model or its use in generating inferences, e.g., such that the inferences produced by the improved model (or the same model using an improved input data set) represent a higher degree of accuracy and/or usefulness for the application 190. In one embodiment, the analysis system 170 may automatically notify users of detected problems and/or provide users with sufficient information for the user to examine the inputs, outputs, and intermediate steps of particular inferences. In one embodiment, the analysis system 170 may automatically initiate retraining of machine learning models based on problem detection.).
Abbaszadeh and Olgiati do not explicitly disclose:
in a second instance of the second determination where the first classification does not specify that the data is considered non-anomalous: discarding the data.
However, in an analogous art, Green discloses a method comprising the step of:
in a second instance of the second determination where the first classification does not specify that the data is considered non-anomalous: discarding the data (Green [0079]. More specifically, the processing module can implement data processing and/or storage functions such as flagging and/or discarding anomalous data (e.g., anomaly detection according to an autoregressive-moving-average model), (weighted) averaging of data captured over a particular sampling period, and/or reading and writing data to local memory on an embedded device.).
Therefore, it would have been obvious to one of ordinary skill in the art on or before the effective date of the claimed invention to combine the embodiments of Abbaszadeh, Olgiati and Green to include the step of: in a second instance of the second determination where the first classification does not specify that the data is considered non-anomalous: discarding the data. One would have been motivated to provide users with a means for determining and discarding analogous or frivolous (noisy) data during data sampling. (See Green [0079].)
Regarding claim 3, Abbaszadeh, Olgiati and Green disclose the method of claim 2.
Abbaszadeh further discloses:
wherein classifying the data using the continuous inference model comprises: obtaining a first inference using the continuous inference model and the data; making a third determination regarding whether the first inference is within an anomaly threshold; in a first instance of the third determination where the first inference is within the anomaly threshold, classifying the data as non-anomalous to obtain the first classification (Abbaszadeh [0035]. As described further above with respect to FIG. 1B, a decision boundary 122 is a mathematical representation of the detection model, such that the decision boundary 122 separates a normal state from an abnormal state for the monitoring node (with respect to the local status determination module) and for the system (with respect to the global status determination module). Embodiments may also include an additional turning parameter (threshold for the anomaly score). The distance and location of each point (in the feature space) is calculated with respect to the decision boundary 122 by computing the difference between the anomaly score 126/132 and the adaptive threshold 140. In a case the anomaly score 126/132 is bigger than the adaptive threshold 140, the point is outside the boundary 136, and hence, abnormal. The bigger the difference, the more the point is outside of the boundary 136 (the “more abnormal”). In a case the anomaly score 126/132 is smaller than the adaptive threshold 140, the point is inside the boundary 134, and hence, normal.); and
in a second instance of the third determination where the first inference is not within the anomaly threshold, classifying the data as anomalous to obtain the first classification (Abbaszadeh [0035]. As described further above with respect to FIG. 1B, a decision boundary 122 is a mathematical representation of the detection model, such that the decision boundary 122 separates a normal state from an abnormal state for the monitoring node (with respect to the local status determination module) and for the system (with respect to the global status determination module). Embodiments may also include an additional turning parameter (threshold for the anomaly score). The distance and location of each point (in the feature space) is calculated with respect to the decision boundary 122 by computing the difference between the anomaly score 126/132 and the adaptive threshold 140. In a case the anomaly score 126/132 is bigger than the adaptive threshold 140, the point is outside the boundary 136, and hence, abnormal. The bigger the difference, the more the point is outside of the boundary 136 (the “more abnormal”). In a case the anomaly score 126/132 is smaller than the adaptive threshold 140, the point is inside the boundary 134, and hence, normal.).
Regarding claim 4, Abbaszadeh and Olgiati disclose the method of claim 1.
Abbaszadeh further discloses:
wherein classifying the data using the quantized inference model comprises: obtaining a second inference using the quantized inference model [and the quantized version of the data] (Abbaszadeh [0050]. The discrete process 308 may also be referred to as a “multi-level” threshold process. The discrete process 308 may quantize the threshold into multiple levels, where each level may correspond to a mode of operation, the region of operation, one or more ambient levels. The multi-level thresholds may be computed using a rule-based logic or a machine learning, such as a decision tree. In this process 308, the threshold is switched among a finite (aka discrete) number of values.);
making a fourth determination regarding whether the second inference is within the anomaly threshold (Abbaszadeh [0103]. For example, the processor 1710 may receive from a detection model an anomaly score for each monitoring node and, then compare that anomaly score to an adaptive threshold to determine whether each monitoring node has a local status of “normal” or “abnormal.”. The processor 1710 may also receive, from a detection model an anomaly score for the whole system and then compare that anomaly score to an adaptive threshold to determine whether the system has a global status of “normal” or “abnormal”. The processor 1710 may then output, for each monitoring node, a local status of “normal” or “abnormal,”. The processor 1710 may also output a global status of “normal” or “abnormal” for the system.);
in a first instance of the fourth determination where the second inference is within the anomaly threshold, classifying the data as non-anomalous to obtain the second classification; and in a second instance of the fourth determination where the second inference is not within the anomaly threshold, classifying the data as anomalous to obtain the second classification (Abbaszadeh [0103]. For example, the processor 1710 may receive from a detection model an anomaly score for each monitoring node and, then compare that anomaly score to an adaptive threshold to determine whether each monitoring node has a local status of “normal” or “abnormal.”. The processor 1710 may also receive, from a detection model an anomaly score for the whole system and then compare that anomaly score to an adaptive threshold to determine whether the system has a global status of “normal” or “abnormal”. The processor 1710 may then output, for each monitoring node, a local status of “normal” or “abnormal,”. The processor 1710 may also output a global status of “normal” or “abnormal” for the system.).
Green further discloses: quantizing the data obtained from the data collector to obtain a quantized version of the data (Green [0023]. [T]he computer system enables the user to determine an appropriate quantization for data sampling, analysis, and output within these containerized applications in order to conserve memory and/or computing resources on corresponding devices. In particular, the computer system can: generate a first set of containerized applications including data sampling, data processing, and data output instructions at a first quantization … [A]pply outputs from the first set of containerized applications at the first quantization in a first instance of a machine learning model; and apply outputs from the second set of containerized applications in a second instance of the machine learning model, thereby enabling the user to compare results of the model and/or insights based on the model at each quantization, identify a desired quantization based on these results, memory considerations, etc., and select a corresponding set of containers to retain on and/or uninstall from the set of devices.).
Therefore, it would have been obvious to one of ordinary skill in the art on or before the effective date of the claimed invention to combine the embodiments of Abbaszadeh, Olgiati and Green to include the step of: quantizing the data obtained from the data collector to obtain a quantized version of the data. One would have been motivated to provide users with a means for quantized inputs and quantized models for optimizing model performance within the constraint of computing resources. (See Green [0023].)
Regarding claim 5, Abbaszadeh, Olgiati and Green disclose the method of claim 4.
Green further discloses wherein quantizing the data obtained from the data collector comprises: identifying a quantized data value corresponding to each data value of the data using a schema for quantizing data and a set of quantized data values; and obtaining the quantized version of the data using the quantized data value corresponding to each data value of the data (Green [0023]. [T]he computer system enables the user to determine an appropriate quantization for data sampling, analysis, and output within these containerized applications in order to conserve memory and/or computing resources on corresponding devices. In particular, the computer system can: generate a first set of containerized applications including data sampling, data processing, and data output instructions at a first quantization … [A]pply outputs from the first set of containerized applications at the first quantization in a first instance of a machine learning model; and apply outputs from the second set of containerized applications in a second instance of the machine learning model, thereby enabling the user to compare results of the model and/or insights based on the model at each quantization, identify a desired quantization based on these results, memory considerations, etc., and select a corresponding set of containers to retain on and/or uninstall from the set of devices.).
The motivation is the same as that of claim 4 above.
Regarding claim 6, Abbaszadeh, Olgiati and Green disclose the method of claim 5.
Green further discloses wherein the schema specifies a range of the data uniquely corresponding to each quantized data value of the set of quantized data values (Green [0023]. [T]he computer system enables the user to determine an appropriate quantization for data sampling, analysis, and output within these containerized applications in order to conserve memory and/or computing resources on corresponding devices. In particular, the computer system can: generate a first set of containerized applications including data sampling, data processing, and data output instructions at a first quantization … [A]pply outputs from the first set of containerized applications at the first quantization in a first instance of a machine learning model; and apply outputs from the second set of containerized applications in a second instance of the machine learning model, thereby enabling the user to compare results of the model and/or insights based on the model at each quantization, identify a desired quantization based on these results, memory considerations, etc., and select a corresponding set of containers to retain on and/or uninstall from the set of devices.).
The motivation is the same as that of claim 5 above.
Regarding claim 7, Abbaszadeh, Olgiati and Green disclose the method of claim 3.
Olgiati further discloses wherein the second action set comprises alerting a downstream consumer of the data drift (Olgiati [0027], [0041]. The analysis system 170 may include a component for automated problem detection 172 that attempts to find one or more types of problems, anomalies, or other flaws in a model or its input data. As will be discussed in greater detail with respect to FIG. 6 through FIG. 11, the analysis may automatically detect problems or anomalies such as models that fail golden examples, outliers in input data, inference data distribution changes, label distribution changes, label changes for individual entities, ground truth discrepancies, and/or other forms of data drift or model drift. The analysis 170 may compare two versions of a model to checkpoint previous versions and provide improved alerts. In some embodiments, the analysis 170 may alert a user or automatically initiate model retraining once the predictions differ between the current model and the previous model.).
The motivation is the same as that of claim 6 above.
Regarding claim 8, Abbaszadeh, Olgiati and Green disclose the method of claim 7.
Abbaszadeh further discloses continuous inference model or the quantized inferenced model (Abbaszadeh [0049], [0054]. The adaptive thresholding system 145 may apply either a discrete process 308 or a continuous process 310 to generate the adaptive threshold 140, as indicated in the chart 312 in FIG. 3E. The continuous process 310 may not switch between a finite/pre-set number of values, as with the discrete process 308, but instead may generate any value that may be continuously adapted using one of a static model 318 or a dynamic model 320.).
Olgiati further discloses wherein the second action set comprises initiating a re-training process to obtain an updated inference model using at least one of the continuous inference model or the quantized inferenced model (Olgiati [0027], [0041]. Olgiati [0027]. The analysis system 170 may include a component for automated problem detection 172 that attempts to find one or more types of problems, anomalies, or other flaws in a model or its input data. As will be discussed in greater detail with respect to FIG. 6 through FIG. 11, the analysis may automatically detect problems or anomalies such as models that fail golden examples, outliers in input data, inference data distribution changes, label distribution changes, label changes for individual entities, ground truth discrepancies, and/or other forms of data drift or model drift. The analysis 170 may compare two versions of a model to checkpoint previous versions and provide improved alerts. In some embodiments, the analysis 170 may alert a user or automatically initiate model retraining once the predictions differ between the current model and the previous model.).
The motivation is the same as that of claim 7 above.
Regarding claim 13, claim 13 is directed to a non-transitory computer readable medium corresponding to the method of claim 2. Claim 13 is similar to claim 2 and is therefore rejected under similar rationale.
Regarding claim 14, claim 14 is directed to a non-transitory computer readable medium corresponding to the method of claim 3. Claim 14 is similar to claim 3 and is therefore rejected under similar rationale.
Regarding claim 15, claim 15 is directed to a non-transitory computer readable medium corresponding to the method of claim 4. Claim 15 is similar to claim 4 and is therefore rejected under similar rationale.
Regarding claim 16, claim 16 is directed to a non-transitory computer readable medium corresponding to the method of claim 5. Claim 16 is similar to claim 5 and is therefore rejected under similar rationale.
Regarding claim 18, claim 18 is directed to a system corresponding to the method of claim 2. Claim 18 is similar to claim 2 and is therefore rejected under similar rationale.
Regarding claim 19, claim 19 is directed to a system corresponding to the method of claim 3. Claim 19 is similar to claim 3 and is therefore rejected under similar rationale.
Regarding claim 22, Abbaszadeh, and Olgiati disclose the method of claim 1. Abbaszadeh further discloses wherein the continuous interference model is based on a first training data set of values varying over the continuous range of values (Abbaszadeh [0031], [0054]. FIG. 1B is an example of a decision boundary 122 in a 3D feature space, and the points inside 134 and outside 136 the decision boundary 122. The models are trained using simulation and/or historical field data to be used as anomaly decision boundaries. The continuous process 310 [] may generate any value that may be continuously adapted using one of a static model 318 or a dynamic model 320. The continuous process 310 may be a model-based approach in which modeling for the adaptive threshold 140 may become part of the abnormal detection model/module training.).
Green further discloses the quantized inference model being based on a quantized version of the first training data that of quantized values that vary over the quantized range of values (Green [0023], [0088]. [T]he computer system enables the user to determine an appropriate quantization for data sampling, analysis, and output within these containerized applications in order to conserve memory and/or computing resources on corresponding devices. Additionally, due to the containerized nature of each containerized application, the computer system can securely update containerized applications with an updated machine learning model (e.g., a retrained set of weights or a more effective model) for execution by the embedded device. The gateway device or cloud computing system can then aggregate these data and train (or retrain) machine learning models based on these data and labelled outcomes or classifications associated with these data.).
Therefore, it would have been obvious to one of ordinary skill in the art on or before the effective date of the claimed invention to combine the embodiments of Abbaszadeh, Olgiati and Green to include the step of: the quantized inference model being based on a quantized version of the first training data that of quantized values that vary over the quantized range of values. One would have been motivated to provide users with a means for selecting a quantization of data and training and updating a AI model with the quantized data. (See Green [0088].)
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Abbaszadeh et al. (“Abbaszadeh,” US 20220329613, filed April 12, 2021) in view of Olgiati et al. (“Olgiati,” US 20210097433, published April 21, 2021), Green et al. (“Green,” US 20220171863, published June 2, 2022) and Isikdogan et al. (“Isikdogan,” US 20200293870, published Sept. 17, 2020).
Regarding claim 9, Abbaszadeh, Olgiati and Green disclose the method of claim 8. Abbaszadeh discloses continuous inference model and the quantized inference model, and the inference model being at least one of the inference model or the quantized inference model (Abbaszadeh [0054]. The continuous process 310 may not switch between a finite/pre-set number of values, as with the discrete process 308, but instead may generate any value that may be continuously adapted using one of a static model 318 or a dynamic model 320. The continuous process 310 may be a model-based approach in which modeling for the adaptive threshold 140 may become part of the abnormal detection model/module training.).
Abbaszadeh, Olgiati and Green do not explicitly disclose: wherein initiating a re-training process comprises: treating the data as training data; and freezing a portion of an inference model prior to re-training the inference model to obtain a frozen portion of the inference model, the freezing of the portion of the inference model renders the frozen portion of the inference model unaffected by re-training of the inference model; and modifying portions of the inference model that are not part of the frozen portion of the inference model based on the training data to obtain the updated inference model.
However, in an analogous art, Isikdogan discloses a method, comprising the steps of:
wherein initiating a re-training process comprises: treating the data as training data; and
freezing a portion of an inference model prior to re-training the inference model to obtain a frozen portion of the inference model, the freezing of the portion of the inference model renders the frozen portion of the inference model unaffected by re-training of the inference model; and modifying portions of the inference model that are not part of the frozen portion of the inference model based on the training data to obtain the updated inference model (Isikdogan [0027], [0082]-[0083], [0086]. Once trained, the deployed model may be operated in an inference phase to process data. In the inference phase, data to be analyzed (e.g., live data) is input to the model, and the model executes to create an output. Once the example model trainer 110 has retrained the model, … the example weight selector 120 selects weights in the machine learning model to freeze. (Block 920). In examples disclosed herein, the weight selector 120 selects a random set of weights in each layer until a ratio of frozen to non-frozen weights is met. However, any other methods to select weights to freeze can additionally and/or alternatively be used. The example weight freezer 130 freezes the weights selected by the example weight selector 120. (Block 925). In examples disclosed herein, the weight freezer 130 freezes weights by setting the value of the weight to a hard-coded scalar. However, any other methods to freeze a weight can additionally and/or alternatively be used. Once the model head has been attached to the model, the example model trainer 110 may train the model head and the non-frozen weights. (Block 937). In some examples, the model head and non-frozen weights are trained for a specified target task (e.g., image classification).).
Therefore, it would have been obvious to one of ordinary skill in the art on or before the effective date of the claimed invention to combine the embodiments of Abbaszadeh, Olgiati and Green to include the step of: freezing a portion of an inference model prior to re-training the inference model to obtain a frozen portion of the inference model, the freezing of the portion of the inference model renders the frozen portion of the inference model unaffected by re-training of the inference model; and modifying portions of the inference model that are not part of the frozen portion of the inference model based on the training data to obtain the updated inference model. One would have been motivated to provide users with a means for selecting a subset of AI model weights for training or retraining. (See Isikdogan [0086].)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EDWARD LONG whose telephone number is (571)272-8961. The examiner can normally be reached on Monday to Friday, 9 AM - 6 PM EST (Alternate Fridays).
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Luu Pham can be reached on 571 270 5002. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR.
Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EDWARD LONG/
Examiner, Art Unit 2439
/LUU T PHAM/ Supervisory Patent Examiner, Art Unit 2439
1 A machine learning model decision regarding an anomaly is an inference output from that model.
2 A machine learning model decision regarding an anomaly is an inference output from that model.
Prosecution Insights