DETAILED ACTION
This Office action is responsive to the Request for Continued Examination (RCE) filed under 37 CFR §1.53(d) for the instant application on May 26, 2026. The Applicants have properly set forth the RCE, which has been entered into the application, and an examination on the merits follows herewith.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 6-8, 10, 11, 13-15, 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication No. 2007/0005311 to Wegerich et al. (“Wegerich”), over U.S. Patent No. 8,903,747 to Eilebrecht et al. (“Eilebrecht”), and also over Chinese Invention Application Publication No. CN 110705598 A to Wang (“Wang”).
Regarding claim 1, Wegerich describes a method and system for the automated measurement of model performance in a data-driven equipment health monitoring system, the data-driven equipment monitoring health system being for use in the early detection of equipment problems and process problems in a mechanical or industrial engine or process (see e.g. paragraph 0006). Like claimed, Wegerich particularly teaches:
applying a perturbation to one test signal of a plurality of test signals input into a multivariate machine learning model (see e.g. paragraphs 0013-0014: Wegerich discloses that the equipment health monitoring system comprises an estimation engine that uses one or more models to generate estimates in response to receiving input observations. Wegerich discloses that the models can be developed using one or more machine learning techniques – see e.g. paragraphs 0014-0015. Moreover, Wegerich teaches that the models can particularly be “autoassociative” multivariate models that receive a plurality of input signals, e.g. from a plurality of sensors, and that generate a plurality of respective estimated signals corresponding to the plurality of input signals – see e.g. paragraphs 0003, 0013 and 0015-0016. In such implementations, the one or more machine learning models are considered multivariate machine learning models like claimed. Wegerich further discloses that the equipment health monitoring system also comprises a model performance module for generating performance metrics for the machine learning models used by or for potential use by the equipment health monitoring system – see e.g. paragraphs 0014 and 0019-0020. To generate such performance metrics for a machine learning model, Wegerich teaches applying a perturbation to individual signals, i.e. variables, input to the model – see e.g. paragraphs 0023-0025 and 0028. Accordingly, Wegerich teaches applying a perturbation to an individual signal, i.e. one test signal, of the plurality of signals input to a multivariate machine learning model.);
monitoring a plurality of estimate signals output from the multivariate machine learning model, wherein the estimate signals predict expected values of test signals other than the one test signal (see e.g. paragraphs 0014 and 0019-0020: like noted above, Wegerich discloses that the equipment health monitoring system comprises a model performance module for generating performance metrics for the machine learning models used by or for potential use by the equipment health monitoring system – see e.g. paragraphs 0014 and 0019-0020. Wegerich particularly discloses that one of these performance metrics, spillover, measures the relative amount that variables in a model deviate from normality when another variable is perturbed – see paragraph 0028. To determine the spillover, Wegerich teaches perturbing a variable, i.e. one test signal like claimed, input to a machine learning model and determining the amount that the estimated signals output from the machine learning model, with the exception of the estimated signal corresponding to the perturbed test signal, deviate from normality – see e.g. paragraphs 0028. Determining the amount that the estimated signals output from the machine learning model deviate from normal would entail monitoring the plurality of estimate signals output from the multivariate machine learning model, and wherein the estimate signals predict expected values of test signals other than the one perturbed test signal.);
checking for the perturbation in the estimate signals (see e.g. paragraph 0028: like noted above, Wegerich teaches identifying the amount the estimated signals output from the machine learning model deviate from normal when a signal with perturbed data is input to the machine learning model; the deviation is indicative of the perturbation in the estimated signals.); and
based on results of the checking for the perturbation, where the perturbation appears in one estimate signal of the plurality of estimate signals, determining a severity metric between the one test signal and the one estimate signal, wherein the severity metric quantifies an extent to which dependency adversely affects accuracy of the one estimate signal (see e.g. paragraphs 0006, 0014, 0019 and 0028: like noted above, Wegerich teaches that the performance metrics calculated for a machine learning model include spillover, which is based on the amount the estimated signals output from the machine learning model deviate from normal when a signal with perturbed data is input to the machine learning model. The calculated spillover metric is considered a severity metric between the one test signal and the one estimate signal; the spillover metric understandably quantifies an extent to which dependency adversely affects accuracy, i.e. a deviation from normality, of the one estimate signal.).
Accordingly, Wegerich teaches a computer-implemented method similar to that of claim 1, but does not explicitly disclose that the perturbations applied to the test input signal are “oscillating” like required by claim 1. Wegerich also does not explicitly teach evaluating the severity metric to determine that a mitigation technique should be applied to the multivariate machine learning model, and automatically implementing the mitigation technique to reduce the dependency in the machine learning model, wherein the mitigation technique comprises one or more of increasing a number of training vectors used to train the multivariate machine learning model, performing filtering operations on training signals and monitored signals to reduce noise, and changing a number of monitored signals, as is further required by claim 1.
Eilebrecht nevertheless describes a software optimization system that isolates an effect of a change in a control variable from the effects of changes in other variables: the system particularly applies an oscillating perturbation to a control variable input to a software system (i.e. by varying the control variable at a specific frequency), and then checks for the oscillating perturbation in an output variable (i.e. by using digital signal processing techniques) to identify the effect of the control variable on the output variable (see e.g. column 2, lines 27-48; column 3, line 52 – column 4, line 4; and column 4, line 28 – column 5, line 13).
It would have been obvious to one of ordinary skill in the art, having the teachings of Wegerich and Eilebrecht before the effective filing date of the claimed invention, to modify the method taught by Wegerich such that the perturbations applied to the test signal input to the machine learning model are oscillating like taught by Eilebrecht, and whereby the output responses (i.e. estimates) dependent on such inputs are identified by checking for the oscillating perturbation in the output response (i.e. estimate) signal. It would have been advantageous to one of ordinary skill to utilize such a combination because it can isolate the effects of the change of the input variable from the effects of other changes, as is taught by Eilebrecht (see e.g. column 3, line 52 – column 4, line 4).
Wang generally teaches: (i) determining a severity metric (i.e. accuracy) associated with a machine learning model; (ii) evaluating the severity metric to determine that a mitigation technique should be applied to the machine learning model (i.e. determining if the accuracy of the machine learning model reaches a preset threshold); and (iii) automatically implementing a mitigation technique to increase the accuracy of the machine learning model, wherein the mitigation technique comprises increasing a number of training vectors (i.e. using a latest sample set of training data) used to train the machine learning model (see e.g. the Abstract).
It would have been obvious to one of ordinary skill in the art, having the teachings of Wegerich, Eilebrecht and Wang before the effective filing date of the claimed invention, to modify the method taught by Wegerich and Eilebrecht so as to further evaluate the severity metric to determine that a mitigation technique should be applied to the machine learning model (i.e. the multivariate machine learning model), and automatically implement the mitigation technique to increase the accuracy of the machine learning model (i.e. to reduce the dependency in the machine learning model), wherein the mitigation technique comprises increasing a number of training vectors used to train the machine learning model (and thus comprises one or more of increasing a number of training vectors used to train the multivariate machine learning model, performing filtering operations on training signals and monitored signals to reduce noise, and changing a number of monitored signals), as is taught by Wang. It would have been advantageous to one of ordinary skill to utilize such a combination because it can save time and energy, as is taught by Wang (see e.g. the “Background” section on page 2 of the provided translation). Accordingly, Wegerich, Eilebrecht and Wang are considered to teach, to one of ordinary skill in the art, a computer-implemented method like that of claim 1.
As per claim 6, Wegerich further teaches that one estimate signal of the plurality of estimate signals can predict values for a second test signal (i.e. variable) input into the machine learning model, wherein the method generally comprises determining that the one estimate signal erroneously predicts values for the second test signal that at least partially mimic behavior of the one test signal, and indicating that the multivariate machine learning model is subject to spillover (see e.g. paragraphs 0015 and 0028). Accordingly, the above-described combination of Wegerich, Eilebrecht and Wang is further considered to teach a computer-implemented method like that of claim 6.
As per claim 7, Wegerich further teaches that the one estimate signal of the plurality of estimate signals predicts values for the one test signal (i.e. variable), wherein the method comprises determining that the one estimate signal erroneously predicts values that at least partially mimic behavior of the one test signal, and indicating that the multivariate machine learning model is subject to following in the evaluation of the dependency of the machine learning model (see e.g. paragraphs 0015 and 0023-0025). Accordingly, the above-described combination of Wegerich, Eilebrecht and Wang is further considered to teach a computer-implemented method like that of claim 7.
Regarding claim 8, Wegerich describes a method and system for the automated measurement of model performance in a data-driven equipment health monitoring system, the data-driven equipment monitoring health system being for use in the early detection of equipment problems and process problems in a mechanical or industrial engine or process (see e.g. paragraph 0006). Like claimed, Wegerich particularly teaches:
applying a perturbation to one test signal of a plurality of test signals input into a multivariate machine learning model (see e.g. paragraphs 0013-0014: Wegerich discloses that the equipment health monitoring system comprises an estimation engine that uses one or more models to generate estimates in response to receiving input observations. Wegerich discloses that the models can be developed using one or more machine learning techniques – see e.g. paragraphs 0014-0015. Moreover, Wegerich teaches that the models can particularly be “autoassociative” multivariate models that receive a plurality of input signals, e.g. from a plurality of sensors, and that generate a plurality of respective estimated signals corresponding to the plurality of input signals – see e.g. paragraphs 0003, 0013 and 0015-0016. In such implementations, the one or more machine learning models are considered multivariate machine learning models like claimed. Wegerich further discloses that the equipment health monitoring system also comprises a model performance module for generating performance metrics for the machine learning models used by or for potential use by the equipment health monitoring system – see e.g. paragraphs 0014 and 0019-0020. To generate such performance metrics for a machine learning model, Wegerich teaches applying a perturbation to individual signals, i.e. variables, input to the model – see e.g. paragraphs 0023-0025 and 0028. Accordingly, Wegerich teaches applying a perturbation to an individual signal, i.e. one test signal, of the plurality of signals input to a multivariate machine learning model.);
monitoring a plurality of estimate signals output from the multivariate machine learning model, wherein the estimate signals predict expected values of test signals other than the one test signal (see e.g. paragraphs 0014 and 0019-0020: like noted above, Wegerich discloses that the equipment health monitoring system comprises a model performance module for generating performance metrics for the machine learning models used by or for potential use by the equipment health monitoring system – see e.g. paragraphs 0014 and 0019-0020. Wegerich particularly discloses that one of these performance metrics, spillover, measures the relative amount that variables in a model deviate from normality when another variable is perturbed – see paragraph 0028. To determine the spillover, Wegerich teaches perturbing a variable, i.e. one test signal like claimed, input to a machine learning model and determining the amount that the estimated signals output from the machine learning model, with the exception of the estimated signal corresponding to the perturbed test signal, deviate from normality – see e.g. paragraphs 0028. Determining the amount that the estimated signals output from the machine learning model deviate from normal would entail monitoring the plurality of estimate signals output from the multivariate machine learning model, and wherein the estimate signals predict expected values of test signals other than the one perturbed test signal.);
checking for the perturbation in the estimate signals (see e.g. paragraph 0028: like noted above, Wegerich teaches identifying the amount the estimated signals output from the machine learning model deviate from normal when a signal with perturbed data is input to the machine learning model; the deviation is indicative of the perturbation in the estimated signals.); and
quantifying adverse effects of dependency on accuracy of the estimate signals with a severity metric based on results of checking for the perturbation (see e.g. paragraphs 0006, 0014, 0019 and 0028: like noted above, Wegerich teaches that the performance metrics calculated for a machine learning model include spillover, which is based on the amount the estimated signals output from the machine learning model deviate from normal when a signal with perturbed data is input to the machine learning model. The calculated spillover metric is considered a severity metric that quantifies adverse effects of dependency on accuracy, i.e. deviation from normality, of the estimate signals based on the identified perturbation.).
Wegerich suggests that such teachings can be implemented via computer-executable instructions (i.e. software) for execution by at least a processor of a computer (see e.g. paragraphs 0014 and 0032). The non-transitory computer-readable medium necessary for storing such software is considered a non-transitory computer-readable medium similar to that of claim 8. However, Wegerich does not disclose that the perturbations applied to the test input signal are “oscillating” like required by claim 8. Wegerich also does not explicitly teach automatically implementing a mitigation technique to reduce dependency in the machine learning model when the severity metric satisfies a threshold, wherein the mitigation technique comprises one or more of increasing a number of training vectors used to train the multivariate machine learning model, performing filtering operations on training signals and monitored signals to reduce noise, and changing a number of monitored signals, as is further required by claim 8.
Like noted above, Eilebrecht nevertheless describes a software optimization system that isolates an effect of a change in a control variable from the effects of changes in other variables: the system particularly applies an oscillating perturbation to a control variable input to a software system (i.e. by varying the control variable at a specific frequency), and then checks for the oscillating perturbation in an output variable (i.e. by using digital signal processing techniques) to identify the effect of the control variable on the output variable (see e.g. column 2, lines 27-48; column 3, line 52 – column 4, line 4; and column 4, line 28 – column 5, line 13).
It would have been obvious to one of ordinary skill in the art, having the teachings of Wegerich and Eilebrecht before the effective filing date of the claimed invention, to modify the non-transitory computer-readable medium taught by Wegerich such that the perturbations applied to the test signal input to the machine learning model are oscillating like taught by Eilebrecht, and whereby the output responses (i.e. estimates) dependent on such inputs are identified by checking for the oscillating perturbation in the output response (i.e. estimate) signal. It would have been advantageous to one of ordinary skill to utilize such a combination because it can isolate the effects of the change of the input variable from the effects of other changes, as is taught by Eilebrecht (see e.g. column 3, line 52 – column 4, line 4).
Wang generally teaches: (i) determining a severity metric (i.e. accuracy) associated with a machine learning model; and (ii) where the severity metric satisfies a threshold, automatically implementing a mitigation technique to increase the accuracy of the machine learning model, wherein the mitigation technique comprises increasing a number of training vectors (i.e. using a latest sample set of training data) used to train the machine learning model (see e.g. the Abstract).
It would have been obvious to one of ordinary skill in the art, having the teachings of Wegerich, Eilebrecht and Wang before the effective filing date of the claimed invention, to modify the non-transitory computer-readable medium taught by Wegerich and Eilebrecht so as to further automatically implement a mitigation technique to increase the accuracy of the machine learning model (i.e. to reduce dependency in the machine learning model) when the severity metric satisfies a threshold, wherein the mitigation technique comprises increasing a number of training vectors used to train the multivariate machine learning model (and thus comprises one or more of increasing a number of training vectors used to train the multivariate machine learning model, performing filtering operations on training signals and monitored signals to reduce noise, and changing a number of monitored signals), as is taught by Wang. It would have been advantageous to one of ordinary skill to utilize such a combination because it can save time and energy, as is taught by Wang (see e.g. the “Background” section on page 2 of the provided translation). Accordingly, Wegerich, Eilebrecht and Wang are considered to teach, to one of ordinary skill in the art, a non-transitory computer-readable medium like that of claim 8.
As per claim 10, Wegerich teaches determining a spillover metric based on the applied perturbation, wherein the spillover metric quantifies an extent to which the one test signal (i.e. a variable) influences an estimate signal output by the machine learning model (see e.g. paragraph 0028). Such a metric is considered a coupling coefficient like claimed between the test signal and the estimate signal. Wegerich further suggests that the spillover metric is presented (e.g. to an engineer) in an evaluation of the dependency (see e.g. paragraph 0014). As described above, it would have been obvious to modify the non-transitory computer-readable medium taught by Wegerich such that the perturbations applied to the test signal are oscillating like taught by Eilebrecht, and whereby the responses (i.e. estimates) output by the machine learning model are identified by checking for the oscillating perturbation in the output (i.e. estimate) signal. It thus follows that the spillover metric, which as noted above is determined based on the perturbation, would particularly be inferred based on the oscillating perturbation like claimed. Accordingly, the above-described combination of Wegerich, Eilebrecht and Wang is further considered to teach a non-transitory computer-readable medium like that of claim 10.
As per claim 11, it would have been obvious, as is described above, to modify the non-transitory computer-readable medium taught by Wegerich such that the perturbations applied to the one test signal are oscillating like taught by Eilebrecht, and whereby the responses (i.e. estimates) output by the machine learning model are identified by checking for the oscillating perturbation in the output (i.e. estimate) signal. Eilebrecht particularly teaches that the oscillating perturbation can be generated as a sinusoidal waveform (see e.g. column 3, lines 52-63; and column 4, lines 28-40). Accordingly, the above-described combination of Wegerich, Eilebrecht and Wang is further considered to teach a non-transitory computer-readable medium like that of claim 11.
As per claim 13, Wegerich further teaches that one estimate signal of the plurality of estimate signals can predict values for a second test signal (i.e. variable) input into the machine learning model, wherein the method generally comprises determining that the one estimate signal erroneously predicts values for the second test signal that at least partially mimic behavior of the one test signal, and indicating that the multivariate machine learning model is subject to spillover (see e.g. paragraphs 0015 and 0028). Accordingly, the above-described combination of Wegerich, Eilebrecht and Wang is further considered to teach a non-transitory computer-readable medium like that of claim 13.
As per claim 14, Wegerich further teaches that the one estimate signal of the plurality of estimate signals predicts values for the one test signal (i.e. variable), wherein the method comprises determining that the one estimate signal erroneously predicts values that at least partially mimic behavior of the one test signal, and indicating that the multivariate machine learning model is subject to following in the evaluation of the dependency of the machine learning model (see e.g. paragraphs 0015 and 0023-0025). Accordingly, the above-described combination of Wegerich, Eilebrecht and Wang is further considered to teach a non-transitory computer-readable medium like that of claim 14.
Regarding claim 15, and like noted above, Wegerich describes a method and system for the automated measurement of model performance in a data-driven equipment health monitoring system, the data-driven equipment monitoring health system being for use in the early detection of equipment problems and process problems in a mechanical or industrial engine or process (see e.g. paragraph 0006). Like claimed, Wegerich particularly teaches:
applying a probe signal to one input signal of a plurality of input signals that are input into a multivariate machine learning model (see e.g. paragraphs 0013-0014: Wegerich discloses that the equipment health monitoring system comprises an estimation engine that uses one or more models to generate estimates in response to receiving input observations. Wegerich discloses that the models can be developed using one or more machine learning techniques – see e.g. paragraphs 0014-0015. Moreover, Wegerich teaches that the models can particularly be “autoassociative” multivariate models that receive a plurality of input signals, e.g. from a plurality of sensors, and that generate a plurality of respective estimated signals corresponding to the plurality of input signals – see e.g. paragraphs 0003, 0013 and 0015-0016. In such implementations, the one or more machine learning models are considered multivariate machine learning models like claimed. Wegerich further discloses that the equipment health monitoring system also comprises a model performance module for generating performance metrics for the machine learning models used by or for potential use by the equipment health monitoring system – see e.g. paragraphs 0014 and 0019-0020. To generate such performance metrics for a machine learning model, Wegerich teaches applying a perturbation to individual signals, i.e. variables, input to the model – see e.g. paragraphs 0023-0025 and 0028. The perturbation is considered a “probe signal” like claimed, which is applied to one input signal of a plurality of input signals that are input into a multivariate machine learning model.);
monitoring a plurality of estimate signals output from the multivariate machine learning model, wherein the estimate signals predict expected values of input signals other than the one input signal (see e.g. paragraphs 0014 and 0019-0020: like noted above, Wegerich discloses that the equipment health monitoring system comprises a model performance module for generating performance metrics for the machine learning models used by or for potential use by the equipment health monitoring system – see e.g. paragraphs 0014 and 0019-0020. Wegerich particularly discloses that one of these performance metrics, spillover, measures the relative amount that variables in a model deviate from normality when another variable is perturbed – see paragraph 0028. To determine the spillover, Wegerich teaches perturbing a variable, i.e. one input signal like claimed, input to a machine learning model and determining the amount that the estimated signals output from the machine learning model, with the exception of the estimated signal corresponding to the perturbed input signal, deviate from normality – see e.g. paragraphs 0028. Determining the amount that the estimated signals output from the machine learning model deviate from normal would entail monitoring the plurality of estimate signals output from the multivariate machine learning model, and wherein the estimate signals predict expected values of input signals other than the one perturbed input signal.);
checking for the probe signal in one estimate signal of the plurality of estimate signals (see e.g. paragraphs 0025 and 0028: like noted above, Wegerich teaches identifying the amount the estimated signals deviate from normal when a signal with perturbed data is input to the machine learning model; the deviation is indicative of the perturbation in the estimated signals.); and
quantifying adverse effects of dependency on accuracy of the estimate signals with a severity metric based on results of checking for the probe signal (see e.g. paragraphs 0006, 0014, 0019 and 0028: like noted above, Wegerich teaches that the performance metrics calculated for a machine learning model include spillover, which is based on the amount the estimated signals output from the machine learning model deviate from normal when a signal with perturbed data is input to the machine learning model. The calculated spillover metric is considered a severity metric that quantifies adverse effects of dependency on accuracy, i.e. deviation from normality, of the estimate signals based on the identified perturbation.).
Wegerich suggests that such teachings can be implemented via computer-executable instructions (i.e. software) for execution by at least a processor of a computer system connected to a network (see e.g. paragraphs 0014 and 0032). Such a computer system, which would necessarily comprise a processor connected to at least one memory, at least one network interface for communicating with one or more networks, and a non-transitory computer readable medium storing the software for execution by the processor, is considered a computing system similar that of claim 15. However, Wegerich does not disclose or suggests that the probe signal applied to the input signal is “repeating” like required by claim 15. Wegerich also does not explicitly teach automatically implementing a mitigation technique to reduce dependency in the machine learning model when the severity metric satisfies a threshold, wherein the mitigation technique comprises one or more of increasing a number of training vectors used to train the multivariate machine learning model, performing filtering operations on training signals and monitored signals to reduce noise, and changing a number of monitored signals, as is further required by claim 15.
Eilebrecht nevertheless describes a software optimization system that isolates an effect of a change in a control variable from the effects of changes in other variables: the system particularly applies a repeating probe signal to a control variable input to a software system (i.e. by varying the control variable at a specific frequency), and then checks for the repeating probe signal in an output variable (i.e. by using digital signal processing techniques) to identify the effect of the control variable on the output variable (see e.g. column 2, lines 27-48; column 3, line 52 – column 4, line 4; and column 4, line 28 – column 5, line 13).
It would have been obvious to one of ordinary skill in the art, having the teachings of Wegerich and Eilebrecht before the effective filing date of the claimed invention, to modify the computing system taught by Wegerich such that the probe signal applied to the input signal input to the machine learning model is repeating like taught by Eilebrecht, and whereby the output responses (i.e. estimates) dependent on such inputs are identified by checking for the repeating probe signal in the output response (i.e. estimate) signal. It would have been advantageous to one of ordinary skill to utilize such a combination because it can isolate the effects of the change of the input variable from the effects of other changes, as is taught by Eilebrecht (see e.g. column 3, line 52 – column 4, line 4).
Wang generally teaches: (i) determining a severity metric (i.e. accuracy) associated with a machine learning model; and (ii) where the severity metric satisfies a threshold, automatically implementing a mitigation technique to increase the accuracy of the machine learning model, wherein the mitigation technique comprises increasing a number of training vectors (i.e. using a latest sample set of training data) used to train the machine learning model (see e.g. the Abstract).
It would have been obvious to one of ordinary skill in the art, having the teachings of Wegerich, Eilebrecht and Wang before the effective filing date of the claimed invention, to modify the computing system taught by Wegerich and Eilebrecht so as to further automatically implement a mitigation technique to increase the accuracy of the machine learning model (i.e. to reduce dependency in the machine learning model) when the severity metric satisfies a threshold, wherein the mitigation technique comprises increasing a number of training vectors used to train the multivariate machine learning model (and thus comprises one or more of increasing a number of training vectors used to train the multivariate machine learning model, performing filtering operations on training signals and monitored signals to reduce noise, and changing a number of monitored signals), as is taught by Wang. It would have been advantageous to one of ordinary skill to utilize such a combination because it can save time and energy, as is taught by Wang (see e.g. the “Background” section on page 2 of the provided translation). Accordingly, Wegerich, Eilebrecht and Wang are considered to teach, to one of ordinary skill in the art, a computing system like that of claim 15.
As per claim 19, Wegerich further teaches that the one estimate signal of the plurality of estimate signals can predict values for a second input signal (i.e. variable) input into the multivariate machine learning model, wherein the computing system further determines that the one estimate signal erroneously predicts values for the second input signal that at least partially mimic behavior of the one input signal, and indicates that the multivariate machine learning model is subject to spillover (see e.g. paragraphs 0015 and 0028). Accordingly, the above-described combination of Wegerich, Eilebrecht and Wang is further considered to teach a computing system like that of claim 19.
As per claim 20, Wegerich further teaches that one estimate signal of the plurality of estimate signals predicts values for the one input signal (i.e. variable), wherein the computing system further determines that the one estimate signal erroneously predicts values that at least partially mimic behavior of the one input signal, and indicates that the multivariate machine learning model is subject to following (see e.g. paragraphs 0015 and 0023-0025). Accordingly, the above-described combination of Wegerich, Eilebrecht and Wang is further considered to teach a computing system like that of claim 20.
Claims 2, 9 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Wegerich, Eilebrecht and Wang, which is described above, and also over the article entitled, “Multi-Frequency Sinusoidal Perturbation Method for Dynamic Characterization of Multi-Processor Computer Servers” by Schuster et al. (“Schuster”).
Regarding claims 2 and 9, Wegerich, Eilebrecht and Wang teach a computer-implemented method like that of claim 1 and a non-transitory computer-readable medium like that of claim 8, as are described above, and which entail applying an oscillating perturbation to one test signal input to a machine learning model and checking for the oscillating perturbation in a plurality of estimate signals output from the machine learning model. Wegerich suggests that the machine learning model is indicated as inaccurately predicting the estimate signal if the output is affected by the perturbed input (see e.g. paragraph 0025). Moreover, Eilebrecht generally teaches that digital signal processing techniques can be used to determine the effect of the input signal on the output signal (see e.g. column 4, line 28 – column 5, line 13). Wegerich, Eilebrecht and Wang, however, do not specifically teach: (a) performing a cross power spectral density transform on the one test signal and one estimate signal of the plurality of estimate signals; (b) examining the cross power spectral density at a frequency of the oscillating perturbation to determine whether a peak is present or absent at the frequency; and (c) in response to (i) determining that the peak is present, indicating that the multivariate machine learning model inaccurately predicts the estimate signal in an evaluation of dependency, and (ii) determining that the peak is absent, indicating that the multivariate machine learning model accurately predicts the estimate signal in the evaluation of dependency, as is required by claims 2 and 9.
Similar to Wegerich and Eilebrecht, Schuster generally teaches identifying a relationship between an input variable of a system and an output variable by applying an oscillating perturbation (i.e. a sinusoidal excitation) to the input variable and checking for the oscillating perturbation in the output variable:
Dynamic characterization of complex systems such as enterprise compute servers and web servers can be achieved by introducing perturbations in one or more “input” variables, and measuring the time-dependent responses in one or more “response” variables. We quantify this relationship between input variables and response variables with a “dynamic coupling coefficient,” which may be a function of load, or, more generally, may be a multivariate function of very many input variables. In a dynamically executing system such as a web server, distributed synthetic transaction generators can be employed for real-time continuous monitoring of system transaction latencies. These “canary tests” provide QoS performance metrics on a 24*7 basis as a dynamical function of system load. Specifically, in order to measure the impact of some performance parameter X on another performance parameter Y, the synthetic transactions introduce an (ideally small) perturbation in X, from which the resulting effect on parameter Y, if any, can be measured. As an example, one might compress a 10 Mbyte file and attempt to discern the temperature effect on one or more ASIC modules on a system board. Using time domain techniques, such a measurement would very likely be impossible on a large, multi-user, multi-cpu server, because of the extremely small effect one is seeking to discern and the poor signal-to-noise ratio.
The well-known sinusoidal excitation technique for estimation of transfer functions [1] allows us to translate this input-output effect to the frequency domain. The advantage of working with this technique is that we concentrate our effort on a few number of frequency points (the frequencies of the sinusoidal excitations) where the correlation or coupling between variables is clearly seen. The technique has been already adapted by one of the authors for the dynamic system characterization of chaotic, nonlinearly interacting physical variables in nuclear power plants [2, 3]. This method, used now for dynamical system characterization of large, multi-processor servers, is an elegant and powerful exploratory analysis tool to characterize complex system behavior, particularly the relationships among various dynamic system parameters.
(Pages 1-2; emphasis added).
Regarding the claimed invention, Schuster particularly teaches that checking for the oscillating perturbation can comprise: (a) performing a cross power spectral density transform (i.e. cross spectrum density, “CSD,” in the following excerpt) on the input variable and the output variable; (b) examining the cross power spectral density at a frequency of the oscillating perturbation to determine whether a peak is present or absent at the frequency; and (c) in response to (i) determining that the peak is present, concluding that the output variable is dependent on the input variable, and (ii) determining that the peak is absent, concluding that the output variable is independent of the input variable:
As an example of this technique, a sinusoidal perturbation with period of 15 minutes is generated in one of the input variables. This sinusoid has a very small amplitude in comparison with normal variations in user load patterns (typically < 1% of nominal variations). One or several synthetic client transactions (called the “canary” variables) are launched to execute typical user transactions (example: file compression, table lookup, inversion of a small matrix, sort a linear list). The response times for these canary tests are recorded to produce continuous time series that reflect QoS from the end-user perspective. In parallel with the canary tests, a large suite of system performance, throughput, and transaction latency variables as well as physical variables are recorded on a 24*7 basis using a continuous system telemetry harness, which has been separately developed by Sun Microsystems for high end UNIX ® servers.
Figure 5 plots the PSD of the response time of the synthetic client (canary variable) as a function of frequency. The stochastic noise (chaotic user load) associated with the canary signal is so high that the test period of 15 minutes is not discernable in the univariate PSD. This illustrates that characterization of typical web-server performance metrics is not amenable to univariate spectral decomposition calculations via conventional Fourier analysis because the signal-to-noise ratio is too small to discern the sinusoidal perturbation in PSD of the response variables. To overcome this limitation of conventional Fourier analysis methods, we employ the CSD, a bi-variate diagnostic technique that is highly sensitive, even to weakly coupled parameters with very poor signal-to-noise ratios, dramatically and selectively amplifying the input sinusoid harmonics in response variables such that the period of the sinusoidal perturbation in the control variable is readily apparent with excellent peak resolution and low noise and “side lobe” contamination [6]. Typical results are illustrated in Figure 5.
The presence of a peak in the CSD is evidence of a common periodicity and, hence, a cause-and-effect relationship between the sinusoidal perturbations in the load and the system variables as well as the canary variables. In the CSD subplot in Figure 5, a well-defined peak corresponding to the period of the sinusoid is readily observable, implying a common periodicity and, hence, a cause-and-effect relationship between the sinusoidal perturbation in the load and the synthetic client’s response time.
(Pages 13-14; emphasis added).
PNG
media_image1.png
523
514
media_image1.png
Greyscale
It would have been obvious to one of ordinary skill in the art, having the teachings of Wegerich, Eilebrecht, Wang and Schuster before the effective filing date of the claimed invention, to modify the method and non-transitory computer-readable medium taught by Wegerich, Eilebrecht and Wang such that the check for the oscillating perturbation comprises: (a) performing a cross power spectral density transform on the input variable (i.e. the one test signal) and the output variable (i.e. each estimate signal of the plurality of estimate signals); (b) examining the cross power spectral density at a frequency of the oscillating perturbation to determine whether a peak is present or absent at the frequency; and (c) in response to (i) determining that the peak is present, concluding that the output variable is dependent on the input variable (and thus indicating that the multivariate machine learning model inaccurately predicts the estimate signal in the evaluation of dependency), and (ii) determining that the peak is absent, concluding that the output variable is independent of the input variable (and thus indicating that the multivariate machine learning model accurately predicts the estimate signal in the evaluation of dependency), as is taught by Schuster. It would have been advantageous to one of ordinary skill to utilize such a cross power spectral density transform because it “is highly sensitive, even to weakly coupled parameters with very poor signal-to-noise ratios, dramatically and selectively amplifying the input sinusoid harmonics in response variables such that the period of the sinusoidal perturbation in the control variable is readily apparent with excellent peak resolution and low noise and ‘side lobe’ contamination,” as is taught by Schuster (see page 14; internal citation omitted). Accordingly, Wegerich, Eilebrecht, Schuster and Wang are considered to teach, to one of ordinary skill in the art, a computer-implemented method like that of claim 2 and a non-transitory computer-readable medium like that of claim 9.
Regarding claim 16, Wegerich, Eilebrecht and Wang teach a computing system like that of claim 15, as is described above, which applies a repeating probe signal to one input signal that is input to a multivariate machine learning model and checks for the repeating probe signal in one or more estimate signals output from the machine learning model. Wegerich suggests that the machine learning model is indicated as inaccurately predicting the estimate signal if the output is affected by the input (see e.g. paragraph 0025). Moreover, Eilebrecht generally teaches that digital signal processing techniques can be used to determine the effect of the input signal on the output signal (see e.g. column 4, line 28 – column 5, line 13). Wegerich, Eilebrecht and Wang, however, do not specifically teach: (a) performing a cross power spectral density transform on the one input signal and one estimate signal of the plurality of estimate signals; (b) examining the cross power spectral density at a frequency of the repeating probe signal to determine whether a peak is present or absent at the frequency; and (c) in response to (i) determining that the peak is present, indicating that the machine learning model inaccurately predicts the estimate signal in an evaluation of dependency, and (ii) determining that the peak is absent, indicating that the machine learning model accurately predicts the estimate signal in the evaluation of dependency, as is required by claim 16.
Like noted above, Schuster similarly teaches identifying a relationship between an input variable of a system and an output variable by applying a repeating probe signal (i.e. an oscillating perturbation) to the input variable and checking for the repeating probe signal in the output variable (see e.g. the portions of pages 1-2 of Schuster excerpted above). Regarding the claimed invention, Schuster particularly teaches that checking for the repeating probe signal can comprise: (a) performing a cross power spectral density transform (i.e. cross spectrum density, “CSD”) on the input variable and the output variable; (b) examining the cross power spectral density at a frequency of the repeating probe signal to determine whether a peak is present or absent at the frequency; and (c) in response to (i) determining that the peak is present, concluding that the output variable is dependent on the input variable, and (ii) determining that the peak is absent, concluding that the output variable is independent of the input variable (see e.g. the portions of pages 13-14 and FIG. 5 of Schuster excerpted above).
It would have been obvious to one of ordinary skill in the art, having the teachings of Wegerich, Eilebrecht, Wang and Schuster before the effective filing date of the claimed invention, to modify the computing system taught by Wegerich, Eilebrecht and Wang such that the check for the repeating probe signal comprises: (a) performing a cross power spectral density transform on the input variable (i.e. the input signal) and the output variable (i.e. each estimate signal of the plurality of estimate signals); (b) examining the cross power spectral density at a frequency of the repeating probe signal to determine whether a peak is present or absent at the frequency; and (c) in response to (i) determining that the peak is present, concluding that the output variable is dependent on the input variable (and thus indicating that the machine learning model inaccurately predicts the estimate signal in an evaluation of dependency), and (ii) determining that the peak is absent, concluding that the output variable is independent of the input variable (and thus indicating that the machine learning model accurately predicts the estimate signal in the evaluation of dependency), as is taught by Schuster. It would have been advantageous to one of ordinary skill to utilize such a cross power spectral density transform because it “is highly sensitive, even to weakly coupled parameters with very poor signal-to-noise ratios, dramatically and selectively amplifying the input sinusoid harmonics in response variables such that the period of the sinusoidal perturbation in the control variable is readily apparent with excellent peak resolution and low noise and ‘side lobe’ contamination,” as is taught by Schuster (see page 14; internal citation omitted). Accordingly, Wegerich, Eilebrecht, Wang and Schuster are considered to teach, to one of ordinary skill in the art, a computing system like that of claim 16.
Claims 3, 4 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Wegerich, Eilebrecht and Wang, which is described above, and also over U.S. Patent Application Publication No. 2016/0048125 to Cheta et al. (“Cheta”).
Regarding claim 3, Wegerich, Eilebrecht and Wang teach a computer-implemented method like that of claim 1, as is described above, and which entails applying an oscillating perturbation to one test signal input to a multivariate machine learning model and checking for the oscillating perturbation in a plurality of estimate signals output from the machine learning model. Wegerich further teaches presenting (e.g. to an engineer) an evaluation of dependency (e.g. spillover) between the input and output signals (see e.g. paragraph 0014). Moreover, Eilebrecht generally teaches that digital signal processing techniques can be used to determine the effect of the input signal on the output signal (see e.g. column 4, line 28 – column 5, line 13). Wegerich, Eilebrecht and Wang, however, do not explicitly teach inferring a coupling coefficient between the one test signal and one estimate signal of the plurality of estimate signals based on the oscillating perturbation, wherein the coupling coefficient is a ratio of output amplitude of the one estimate signal at a frequency of the oscillating perturbation to the input amplitude of the one test signal at the frequency of the oscillating perturbation, and presenting the coupling coefficient in the evaluation of the dependency, as is required by claim 3.
Similar to Wegerich, Eilebrecht and Wang, Cheta generally teaches applying an oscillating perturbation to a signal input to a system, monitoring a signal output from the system, and checking for the oscillating perturbation in the output signal (see e.g. paragraphs 0004, 0015, 0026-0027 and 0029-0031). Regarding the claimed invention, Cheta particularly teaches inferring a coupling coefficient (i.e. frequency response) between the input signal and the output signal based on the oscillating perturbation, wherein the coupling coefficient is a ratio of output amplitude of the output signal at a frequency of the oscillating perturbation to the input amplitude of the input signal at the frequency of the oscillating perturbation (see e.g. paragraphs 0034-0036).
It would have been obvious to one of ordinary skill in the art, having the teachings of Wegerich, Eilebrecht, Wang and Cheta before the effective filing date of the claimed invention, to modify the computer-implemented method taught by Wegerich, Eilebrecht and Wang so as to infer a coupling coefficient like taught by Cheta between the input signal (i.e. the one test signal) and the output signal (i.e. one estimate signal of the plurality of estimate signals) based on the oscillating perturbation, wherein the coupling coefficient is a ratio of output amplitude of the output signal at a frequency of the oscillating perturbation to the input amplitude of the input signal at the frequency of the oscillating perturbation. It would have been advantageous to one of ordinary skill to utilize such a combination, because the coupling coefficient is useful in characterizing the relationship between the input and output signals, as is evident from Cheta (see e.g. paragraphs 0054-0055). Accordingly, Wegerich, Eilebrecht, Wang and Cheta are considered to teach, to one of ordinary skill in the art, a computer-implemented method like that of claim 3.
Regarding claim 4, Wegerich, Eilebrecht and Wang teach a computer-implemented method like that of claim 1, as is described above, and which entails applying an oscillating perturbation to one test signal input to a multivariate machine learning model and checking for the oscillating perturbation in a plurality of estimate signals output from the machine learning model. Wegerich, Eilebrecht and Wang, however, do not explicitly teach generating the oscillating perturbation as a time series having corresponding indexes with the one test signal, wherein the oscillating perturbation is applied to the one test signal by adding values of the one test signal and the oscillating perturbation at the corresponding indexes, as is required by claim 4.
Like noted above, Cheta generally teaches applying an oscillating perturbation to a signal input to a system, monitoring a signal output from the system, and checking for the oscillating perturbation in the output signal (see e.g. paragraphs 0004, 0015, 0026-0027 and 0029-0031). Regarding the claimed invention, Cheta particularly teaches generating the oscillating perturbation as a time series (e.g. a waveform) having corresponding indexes (e.g. temporal periods, which are inherently related to the frequency of the waveform) with the input signal, wherein the oscillating perturbation is applied to the input signal by adding values of the input signal and the oscillating perturbation at the corresponding indexes (see e.g. paragraphs 0026-0027 and 0029).
It would have been obvious to one of ordinary skill in the art, having the teachings of Wegerich, Eilebrecht, Wang and Cheta before the effective filing date of the claimed invention, to modify the computer-implemented method taught by Wegerich, Eilebrecht and Wang so as to generate the oscillating perturbation as a time series having corresponding indexes with the input signal (i.e. the one test signal), wherein the oscillating perturbation is applied to the input signal by adding values of the input signal and the oscillating perturbation at the corresponding indexes, as is taught by Cheta. It would have been advantageous to one of ordinary skill to utilize such a combination, because it would enable the relationship between the input and output signals of the system to be identified during actual operation of the system, as is evident from Cheta (see e.g. paragraphs 0054-0055). Accordingly, Wegerich, Eilebrecht, Wang and Cheta are considered to teach, to one of ordinary skill in the art, a computer-implemented method like that of claim 4.
Regarding claim 21, Wegerich, Eilebrecht and Wang teach a computing system like that of claim 15, as is described above, and which entails quantifying adverse effects of dependency on accuracy of estimate signals with a severity metric. Wegerich, Eilebrecht and Wang, however, do not explicitly disclose that the severity metric is ratio of amplitude of a dominant frequency to amplitude at a frequency of the repeating probe signal in a cross power spectral density between the one input signal and the one estimate signal, as is required by claim 21.
Cheta generally teaches applying a repeating probe signal (i.e. an oscillating perturbation) to a signal input to a system, monitoring a signal output from the system, and checking for the oscillating perturbation in the output signal (see e.g. paragraphs 0004, 0015, 0026-0027 and 0029-0031). Regarding the claimed invention, Cheta particularly teaches determining a severity metric (i.e. a coherence) that is a ratio of amplitude (e.g. power) of a dominant frequency (i.e. at a frequency of the repeating probe signal) to amplitude at a frequency of the repeating probe signal in a cross power spectral density between the input signal and the output signal (see e.g. 0043-0045).
It would have been obvious to one of ordinary skill in the art, having the teachings of Wegerich, Eilebrecht, Wang and Cheta before the effective filing date of the claimed invention, to modify the computing system taught by Wegerich, Eilebrecht and Wang so as to determine a severity metric (i.e. coherence) that is a ratio of amplitude of a dominant frequency to amplitude at a frequency of the repeating probe signal in a cross power spectral density between the input signal and the output signal (i.e. the estimate signal), as is taught by Cheta. It would have been advantageous to one of ordinary skill to utilize such a combination, because such a severity metric is useful in characterizing the relationship between the input and output signals, as is evident from Cheta (see e.g. paragraphs 0043-0045). Accordingly, Wegerich, Eilebrecht, Wang and Cheta are considered to teach, to one of ordinary skill in the art, a computing system like that of claim 21.
Claims 5, 12 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Wegerich, Eilebrecht and Wang, which is described above, and also over the article entitled, “Determination of Sensor Quality of Calibration Using Advanced Data Analytics and Machine Learning Methods” by Cetiner et al. (“Cetiner”).
Regarding claims 5, 12 and 18, Wegerich, Eilebrecht and Wang teach a computer-implemented method like that of claim 1, a non-transitory computer-readable medium like that of claim 8, and a computing system like that of claim 15, as are described above, and which entail applying an oscillating perturbation (or repeating probe signal like in claim 15) to one test signal input to a machine learning model and checking for the oscillating perturbation in a plurality of estimate signals output from the machine learning model. Wegerich further suggests automatically selecting an amplitude of the perturbation/probe signal that is within a data range of the one test signal, and generating the perturbation to have the amplitude (see e.g. paragraph 0024). Wegerich, Eilebrecht and Wang, however, do not explicitly teach that the selected amplitude is within one standard deviation of the one input signal, as is required by claims 5, 12 and 18.
Similar to Wegerich, Eilebrecht and Wang, Cetiner teaches measuring sensitivity (e.g. auto-sensitivity, cross-sensitivity) in a multivariate machine learning model by applying a perturbation (i.e. artificial drift) to one test signal input to the model, and identifying the resulting effect in estimate signals output from the model (see e.g. section 3.2.2 “Performance Evaluation Metrics”). Cetiner suggests that the selected amplitude of the perturbation is within one standard deviation of the test signal (see e.g. section 3.2.2 “Performance Evaluation Metrics”).
It would have been obvious to one of ordinary skill in the art, having the teachings of Wegerich, Eilebrecht, Wang and Cetiner before the effective filing date of the claimed invention, to modify the method, non-transitory computer-readable medium and computing system taught by Wegerich, Eilebrecht and Wang such that the selected amplitude of the perturbation/probe signal is within one standard deviation of the one input signal (i.e. one test signal), as is taught by Cetiner. It would have been advantageous to one of ordinary skill to utilize such a combination, because it would enable the dependency (i.e. cross-sensitivity) between the input test signal and the output estimate signals to be identified, as is evident from Cetiner (see e.g. section 3.2.2 “Performance Evaluation Metrics”). Accordingly, Wegerich, Eilebrecht, Wang and Cetiner are considered to teach, to one of ordinary skill in the art, a computer-implemented method like that of claim 5, a non-transitory computer-readable medium like that of claim 12, and a computing system like that of claim 18.
Response to Arguments
The Examiner acknowledges the Applicant’s amendment to claims 1-8, 13-15, 19 and 20, cancellation of claim 17, and addition of new claim 21. In response to these amendments, the objections presented in the previous Office Action to claims 15-20 are respectfully withdrawn. Also in response to these amendments, the 35 U.S.C. § 101 rejections presented in the previous Office Action to claims 1-20 are respectfully withdrawn.
The Applicant’s arguments concerning the 35 U.S.C. § 103 rejections presented in the previous Office Action have been considered, but are moot in view of the new grounds of rejection presented above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BLAINE T BASOM whose telephone number is (571)272-4044. The examiner can normally be reached Monday-Friday, 9:00 am - 5:30 pm, EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Ell can be reached at (571)270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BTB/
9/30/2026
/MATTHEW ELL/Supervisory Patent Examiner, Art Unit 2141
Prosecution Insights