DETAILED ACTION
Status of Claims
This Office action is responsive to communications filed on 2026-03-25. Claim(s) 3-5 and 23-24 was/were cancelled. Claim(s) 28 was/were added. Claim(s) 1, 6-7, 9-12, 14-15, 19, and 25-28 are pending and are examined herein.
Claim(s) 1, 6-7, 9-12, 14-15, 19, and 25-28 is/are rejected under 35 USC 112(b).
Claim(s) 1, 6-7, 9-12, 14-15, 19, and 25-28 is/are rejected under 35 USC 103.
Notice of Pre-AIA or AIA Status
The present application, filed on or after 2013-03-16, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The attached information disclosure statement(s) (IDS), submitted on 2026-03-20, is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the attached information disclosure statement(s) is/are being considered by the examiner.
Response to Arguments
Regarding rejections under 35 USC 112(b), the applicant’s do not resolve the substance of the issues raised in the previous Office action, and the applicant provides no relevant remarks. Issues in the pending claims are described below.
Regarding rejections under 35 USC 103, the applicant’s arguments have been fully considered but they are unpersuasive.
Regarding claim 1 (which was amended to incorporate limitations from cancelled claims 3-4), the applicant’s remarks fail to clarify how the language of the claim avoids the disclosures of the references made of record for at least the following reasons:
The applicant asserts that “[t]wo points can establish linearity” but neither the meaning of this assertion, nor its relevance to the rejections as given, is clear to the examiner. It is evidently not true that “[t]wo points can establish linearity” since two points can be consistent with either a linear model or a non-linear model. For example, the two points (0, 1) and (1, 2) are both on the line y = x+1 and they are also both on the exponential curve y = 2^x, and it would not be clear from these two points alone which of these equations is the underlying model. (The examiner notes that what is true is that any two distinct points in the plane determine a line, but the property of being determined by two distinct points in the plane is not unique to lines: for example, it is also true that any two distinct points in the plane determine an exponential curve of the form y = a * b^x.)
The applicant asserts that “’a to another b’ is only two points which is insufficient to demonstrate nonlinearity” but again, neither the meaning of this assertion, nor its relevance to the rejections as given, is clear to the examiner. As described in the rejections in the previous Office action, any increase between two numbers can be exponential, which means, in particular, that the increases described in Aliferis can be regarded as being exponential. Moreover, even if the assertion were coherent (which the examiner does not concede), the examiner notes that the claim itself only discuses increasing a size from one iteration to the next (cf. 112(b) rejections), but an increase from one iteration to the next is also “only two points” so the applicant’s assertion would appear to rule impossible that such an increase be “exponential or super-exponential” as recited by the claim. See also: the next bullet point.
The applicant asserts that the example sequence of increasing sample sizes in Aliferis (namely, the one beginning with 100, 150, 200) “is linear” [remarks, page 11]. As noted in the previous Office action, it is not true that any sequence beginning with 100, 150, 200 is necessarily linear. Consider again, for example, the sequence given by x_n = 100*(2^n) - 50*(n^2) for each non-negative integer n. This is a non-linear sequence (it clearly includes an exponential term), and elementary arithmetic (i.e., plugging in n = 0, 1, and 2) shows that the first three terms of this sequence are x_0 = 100, x_1 = 150, and x_2 = 200.
Regarding claim 27, the applicant argues that there is no motivation to combine references. The examiner respectfully disagrees. The examiner maintains that it would have been obvious to a person of ordinary skill in the art before the effective filing date of the application to replace one type of classifier (e.g., a linear support vector machine) with another (e.g., a random forest) since different types of classifiers present different advantages.
The complete prior art mapping, updated in view of the applicant’s amendments is given below.
Examiner’s Remarks
The claims recite a plurality of old network packets and a plurality of recent network packets. While “old” and “recent” are subjective terms (cf. MPEP 2173.05(b)(IV)), the examiner notes that a more objective criterion for this subjective terminology may be found in a cancelled limitation of claim 11 as originally filed (“the first plurality of tuples is older than the second plurality of tuples”). In other words, the “plurality of old network packets” is interpreted as being a plurality of network packets all of which are older than the network packets in the “plurality of recent network packets”. Similarly, the claims were amended to recite the phrases a first label that indicates old network packets and a second label that indicates recent network packets. These two phrases are treated as units and interpreted as referring to labels for temporally contrasting classes of network packets as above.
Claim Rejections - 35 USC 112(b)
The following is a quotation of 35 USC 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 USC 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 1, 6-7, 9-12, 14-15, 19, and 25-28 is/are rejected under 35 USC 112(b) or 35 USC 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 USC 112, the applicant), regards as the invention.
Claims 1 and 19 were amended to incorporate limitations from cancelled dependent claims 3-4 and are rejected for indefiniteness for substantively the same reasons as given in the rejection of those dependent claims in the previous Office action. Specifically, claims 1 and 19 recite a “sequence of iterations” which ceases “in response to a comparison… that indicates data drift”. In particular, the claims do not actually require detecting data drift, and thus also do not require ceasing the sequence of iterations. In situations where no data drift is detected, it is not clear how to interpret exponentially or super-exponentially increasing a respective size of at least one plurality of network packets selected from a group consisting of the plurality of old network packets and the plurality of recent network packets. Namely, in situations where no data drift is detected, the claim would appear to require a non-terminating sequence of iterations, each iteration “increasing a respective size of at least one particular plurality of network packets” – but since there can only be finitely many network packets in the pluralities of old and recent network packets, a non-terminating sequence of such increases is impossible. (The examiner notes the specification describes ensuring that the sample size does not exceed the population size [specification, 0087], leading to further concerns about a lack of correspondence between the claims and the specification; cf. MPEP 2173.03.) The examiner notes that it is further unclear what the recited “increasing” is relative to (the cancelled dependent claims seemed to clarify that the “increasing” is relative to the selection made in the previous iteration, but this is not clear from the amended independent claims). Dependent claims 6-7, 9-12, 14-15, and 25-28 inherit the rejection.
Claim 7 recites that the “internals” of the anomaly detector be inaccessible. This is subjective terminology as model internals which are inaccessible to one person may not be inaccessible to another person, and the claim provides no clarification as to whom the model internals are required to be inaccessible to. MPEP 2173.05(b)(IV) indicates that, in the presence of subjective claim terminology, “[s]ome objective standard must be provided in order to allow the public to determine the scope of the claim. A claim term that requires the exercise of subjective judgment without restriction may render the claim indefinite.” The specification provides no objective criterion by which to determine whether model internals are inaccessible. It indicates only that a black-box model is one “whose internals are inaccessible or confusing” [specification, 0031], but model internals which are “inaccessible or confusing” to one person may not be “inaccessible or confusing” for another person. The claim is consequently indefinite. For the purpose of compact prosecution, the claim is interpreted broadly as encompassing any anomaly detector.
Claim Rejections - 35 USC 103
The following is a quotation of 35 USC 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 USC 102(b)(2)(C) for any potential 35 USC 102(a)(2) prior art against the later invention.
Claim(s) 1, 7, 9-12, 14-15, and 19 is/are rejected under 35 USC 103 as being unpatentable over Jason MAUGHAN et al. (US20170330109A1, published 2017-11-16; hereafter “Maughan”) in view of Sayan MUKHERJEE et al. (Permutation Tests for Classification, published 2003; hereafter “Mukherjee”), Konstantinos ALIFERIS et al. (US20140278339A1, published 2014-09-18; hereafter “Aliferis”), Abraham WALD (Sequential Tests of Statistical Hypotheses, published 1945; hereafter, “Wald”), and Tangqing LI et al. (Deep Unsupervised Anomaly Detection, published 2021-06-14; hereafter “Li”).
Claim 1
Maughan discloses:
A method comprising: ([Maughan, abstract]: Maughan discloses a method “for drift detection and correction” [Maughan, abstract].
performing in each iteration of a sequence of iterations: ([Maughan, figures 8-9]: The broadest reasonable interpretation of a “sequence of iterations” encompasses a sequence of length 1, i.e., a single iteration. In particular, one run-through of the method of drift detection [Maughan, figure 8 step 804 or figure 9 step 904] falls under the broadest reasonable interpretation of a “sequence of iterations” of the claim. The examiner notes that the combination of references below discloses a sequence of iterations having a plurality of iterations.)
[a) randomly selecting] a plurality of old network packets and a plurality of recent network packets; b) assigning: to each network packet in the plurality of old network packets, a first label that indicates old network packets, and to each network packet in the plurality of recent network packets, a second label that indicates recent network packets; ([Maughan, 0065, 0103]: Maughan discloses “data before a timestamp labeled with a ‘0’ and data after the timestamp labeled with a ‘1’” [Maughan, 0065]. Moreover, the data being labeled with these labels is indicated as being received by a data receiver module “over a data network” [Maughan, 0103]. In other words, this data falls under the broadest reasonable interpretation of a “network packet” as in the claim. The data before the timestamp maps to the “plurality of old network packets” of the claim and the data after the timestamp to the “plurality of recent network packets” of the claim. The label 0 maps to the “first label that indicates old network packets” of the claim and the label 1 maps to the “second label that indicates recent network packets” of the claim.)
e) supervised training a binary classifier by inferring, respectively from each network packet in a particular plurality of network packets, the first label that indicates old network packets or the second label that indicates recent network packets, wherein the particular plurality of network packets is selected from a group consisting of the permuted plurality of network packets and the combined plurality of network packets; ([Maughan, 0065]: Maughan discloses a “drift detection module 204 [which] may perform a binary classification” into the two classes labeled 0 and 1 as described above [Maughan, 0065]. In other words, the drift detection module maps to the “binary classifier” of the claim and the binary classification it performs maps to the “inferring” recited by the claim. Since the drift detection module uses the timestamp-based labels described above, its training is “supervised” as recited by the claim; the examiner notes that supervised training of a binary classifier is also disclosed in Mukherjee as discussed below. Since both the “plurality of old network packets” and the “plurality of recent network packets” are undergoing binary classification, either of these pluralities falls under the broadest reasonable interpretation of the “particular plurality of network packets… wherein the particular plurality of network packets is selected from a group consisting of the permuted plurality of network packets and the combined plurality of network packets” as recited by the claim.)
[unsupervised] retraining [an anomaly detector] in response to said ceasing; ([Maughan, 0042-0043, 0052, 0065, 0075]: Maughan discloses a “prediction module 202” which is “configured to apply a model to workload data to produce one or more predictive results” [Maughan, 0052]. The examiner notes that though Maughan indicates that the model can be any one of a variety of different types [Maughan, 0042-0043], it does not specifically mention the model being an anomaly detector. Nonetheless, Maughan discloses that the drift detection module determines that drift has occurred “if the drift detection module 204 can tell the difference between the [two] classes” described above [Maughan, 0065], and it discloses a “retrain module 302 [which] is configured to retrain the model… in response to the drift detection module detecting the drift phenomenon” [Maughan, 0075]. Since the “ceasing” of the claim happens “in response to a comparison… that indicates data drift”, the retraining disclosed by Maughan is in fact “in response to said ceasing” as required by the claim.)
wherein the method is performed by one or more computers. ([Maughan, 0031]: Maughan discloses that the methods disclosed therein “can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus… such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the” method disclosed therein [Maughan, 0031].)
As noted above, Maughan discloses drift being detected “if the drift detection module can tell the difference between the classes” [Maughan, 0065], though it does not distinctly disclose a method for determining whether there is a difference between the classes, and in particular, it does not distinctly disclose the use of a permutation test for this purpose. Also, while Maughan discloses retraining a model in response to drift detection [Maughan, 0075], it does not distinctly disclose the model being an anomaly detector whose retraining is unsupervised. In other words, Maughan does not distinctly disclose:
a) randomly selecting [a plurality of old network packets and a plurality of recent network packets]
including exponentially or super-exponentially increasing a respective size of at least one plurality of network packets selected from a group consisting of the plurality of old network packets and the plurality of recent network packets;
c) combining the plurality of old network packets and the plurality of recent network packets into a combined plurality of network packets;
d) permuting the labels of the combined plurality of network packets to generate a permuted plurality of network packets;
f) measuring: a first fitness score of the binary classifier by the binary classifier inferring, respectively from each network packet in the combined plurality of network packets tuples, the first label that indicates old network packets or the second label that indicates recent network packets, and a second fitness score of the binary classifier by the binary classifier inferring, respectively from each network packet in the permuted plurality of network packets, the first label that indicates old network packets or the second label that indicates recent network packets tuples;
ceasing the sequence of iterations in response to a comparison, in a last iteration of the sequence of iterations, of the first fitness score to the second fitness score that indicates data drift;
and unsupervised [retraining] an anomaly detector … and detecting, by the anomaly detector after said retraining, that a new network packet is anomalous;
Mukherjee is in the field of statistical analysis. It discusses a binary classifier [Mukherjee, section 3], which, in the combination, corresponds to the drift detection module performing binary classification as described in Maughan [Maughan, 0065]. The classifiers are implemented by training “linear Support Vector Machines” [Mukherjee, section 4], which is a supervised machine learning method as required by the claim. It also discusses observations x_k in R^n being paired with binary labels y_k for k = 1, …, l [Mukherjee, section 2 first paragraph], and in the combination, x_k of Mukherjee corresponds to the data in which drift is to be detected, and the binary labels y_k of Mukherjee correspond to the 0 and 1 labels of Maughan as described above [Maughan, 0065]. Mukherjee explicitly indicates that the test statistic T “can be estimated… using cross-validation” [Mukherjee, section 3], and cross-validation is a method which involves a plurality of iterations (cf. James as cited in the conclusion of a previous Office action). Moreover, Mukherjee discloses the use of a permutation test to tell whether there is a difference between the classes [Mukherjee, section 2]. In other words, Maughan in view of Mukherjee discloses:
a) randomly selecting [a plurality of old network packets and a plurality of recent network packets] ([Maughan, sections 2 and 4; Maughan, 0065]: As noted above, Mukherjee discloses a dataset {(x_k, y_k)}_{k = 1}^l where x_k are observations and y_k are binary labels [Mukherjee, section 2 first paragraph], and in the combination, the observations x_k are the data in which drift is to be detected and the binary labels y_k correspond to the labels 0 and 1 of Maughan [Maughan, 0065]. Mukherjee further discloses through their examples that the dataset are randomly selected from larger populations [Mukherjee, section 4]. For example, Mukherjee describes a dataset of observations from “50 patients diagnosed with dementia of the Alzheimer type and 50 normal controls of matched age” [Mukherjee, section 4.1 first paragraph], which means that the dataset is randomly selected from the larger populations of all people diagnosed with dementia of the Alzheimer type and of all normal controls. Similarly, Mukherjee also describes a dataset containing “48 samples of AML and 25 samples of ALL” [Mukherjee, section 4.1 paragraph beginning “Before proceeding”], which means that the dataset is randomly selected from the larger populations of all cases of AML (acute myeloid leukemia) and of all cases of ALL (acute lymphoblastic leukemia). In other words, the combination of Maughan and Mukherjee discloses that the “plurality of old network packets” and the “plurality of recent network packets” of the claim are “randomly select[ed” as recited by the claim.)
c) combining the plurality of old network packets and the plurality of recent network packets into a combined plurality of network packets; ([Mukherjee, section 2]: The full dataset {(x_k, y_k)}_{k = 1}^l maps to the “combined plurality” of the claim.)
d) permuting the labels of the combined plurality of network packets to generate a permuted plurality of network packets; ([Mukherjee, section 2]: Mukherjee discloses sampling a permutation π^m and using this to “permut[e] the labels” and obtain the dataset {(x_k, y_{π^m_k})}_{k = 1}^l [Mukherjee, section 2 page 4 paragraph beginning “Suppose”]. This {(x_k, y_{π^m_k})}_{k = 1}^l maps to the “permuted plurality” of the claim.)
f) measuring: a first fitness score of the binary classifier by the binary classifier inferring, respectively from each network packet in the combined plurality of network packets tuples, the first label that indicates old network packets or the second label that indicates recent network packets, ([Mukherjee, sections 2-3]: Mukherjee discloses “comput[ing] the test statistic for the actual labels t_0 = T(x_1, y_1, …, x_l, y_l)” [Mukherjee, section 2 page 4 paragraph beginning “Suppose”], where T is a an arbitrary statistic [Mukherjee, section 2 first paragraph centered equation], and then deriving a p-value p_0 [Mukherjee, section 2 page 4 paragraph beginning “Suppose”]. Either t_0 or p_0 can map to the “first fitness score” of the claim as recited.) and a second fitness score of the binary classifier by the binary classifier inferring, respectively from each network packet in the permuted plurality of network packets, the first label that indicates old network packets or the second label that indicates recent network packets tuples; ([Mukherjee, sections 2-3]: Mukherjee discloses “comput[ing] the statistic value for this permutation of labels t^m = T(x_1, y_{π^m_1}, …, x_l, y_{π^m_l})” [Mukherjee, section 2 page 4 paragraph beginning “Suppose”]. This step is repeated “M times (with index m = 1, …, M)” so, alternatively, the entire collection t^1, …, t^M can also be mapped to the “second fitness score” of the claim.)
a comparison, in a last iteration of the sequence of iterations, of the first fitness score to the second fitness score that indicates data drift; ([Mukherjee, section 2]: Mukherjee discloses using t^m to generate an “empirical cumulative distribution” hat{P}, computing a p-value p_0 for t_0 under hat{P}, and comparing p_0 to an “acceptable significance α” [Mukherjee, section 2 page 4 paragraph beginning “Suppose”]. This process falls under the broadest reasonable interpretation of “a comparison of the first fitness score to the second fitness score” as recited in the claim with the fitness scores being as mapped above. Since “the null hypothesis assumes that the two classes are indistinguishable” [Mukherjee, section 2 paragraph beginning “The procedure”], accepting (i.e., failing to reject) the null hypothesis using the method of Mukherjee becomes, in the combination with Maughan, an indication of data drift as recited in the claim.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the drift detection system of Maughan with the permutation test of Mukherjee because they are “useful in testing if the observed classification results are likely to be obtained by chance” [Mukherjee, section 3 paragraph beginning “To underscore”] and the combination would thus result in an effective drift detection system.
Maughan in view of Mukherjee might not distinctly disclose:
including exponentially or super-exponentially increasing a respective size of at least one plurality of network packets selected from a group consisting of the plurality of old network packets and the plurality of recent network packets;
ceasing the sequence of iterations in response to [a comparison… that indicates data drift]
and unsupervised [retraining] an anomaly detector … and detecting, by the anomaly detector after said retraining, that a new network packet is anomalous;
Aliferis is in the field of statistical analysis. In particular, Maughan in view of Mukherjee and Aliferis discloses:
including exponentially or super-exponentially increasing a respective size of at least one plurality of network packets selected from a group consisting of the plurality of old network packets and the plurality of recent network packets; ([Aliferis, 0090-0091]: Aliferis discloses a process of “generat[ing] random samples of increasing sample sizes. For example, for a starting point of sample size 1000 with 1000 repeats, this process would give 1000 randomly sampled data sets of sample size 100… Then, perform this repeated sampling for several increasing sample sizes (e.g., 100, 150, 200, …, total available sample)” [Aliferis, 0090]. These samples are then used to “estimate performance” of a model [Aliferis, 0091]. In the combination, the samples in the iterations disclosed by Maughan in view of Mukherjee and Wald are taken to have increasing sizes as disclosed by Aliferis. The examiner notes that any increase from one number a to another b falls under the broadest reasonable interpretation of an “exponential increase” since b can always be written as the ath power of some quantity (namely, b = (e^{ln(b)/a})^a), or, alternatively, b can always be written as some power of a (namely, b = a^{log_a(b)}). This implies that the increases described under the parent claim can be regarded as either “exponential” or as “super-exponential” (for example, an increase from 100 to 150 [Aliferis, 0090] is “exponential” by the logic presented here, but at the same time, the increase from 100 to 149 is also “exponential” by this logic, which means that the increase from 100 to 150 is “super-exponential”). Alternatively, the examiner notes that the sequence given by x_n = 100*2^n - 50*n^2 for each non-negative integer n is an exponential sequence (its dominant term is exponential) and elementary arithmetic (i.e., simply plugging in n = 0, 1, and 2) shows that the first three terms of this sequence are x_0 = 100, x_1 = 150, and x_2 = 200, i.e., the same as the first three terms in the example given in Aliferis. For yet another mapping of this limitation, the applicant is invited to consult Al-Omari as cited in the conclusion of a previous Office action.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the drift detection system disclosed by Maughan in view of Mukherjee with the statistical techniques of Aliferis because they allow for “determining sufficient sample size… taking into account the critical factors that affect the needed sample size [and] can be applied to practically any field where predictive modeling or causal modeling are desired” [Aliferis, abstract], so the system would be more robust overall.
Maughan in view of Mukherjee and Aliferis might not distinctly disclose:
ceasing the sequence of iterations in response to [a comparison… that indicates data drift]
and unsupervised [retraining] an anomaly detector … and detecting, by the anomaly detector after said retraining, that a new network packet is anomalous;
Wald is in the field of statistical analysis. Moreover, Maughan in view of Mukherjee, Aliferis, and Wald discloses:
ceasing the sequence of iterations in response to [a comparison… that indicates data drift] ([Wald, section A]: As noted above, Maughan in view of Mukherjee discloses a statistical method in which accepting a null hypothesis is an indication of data drift. Wald discloses a “sequential test of a statistical hypothesis” which entails repetition until a null hypothesis is either accepted or rejected (cf. “[i]f the first or second decision is made, the process is terminated”) [Wald, section A first paragraph]. In the combination, the statistical method of Maughan and Mukherjee is repeated to form a sequential test as in Wald. The combination thus discloses both a “sequence of iterations” having more than one iteration, as well as “ceasing the sequence of iterations in response to a comparison… that indicates data drift” as recited by the claim.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the drift detection system of Maughan in view of Mukherjee and Aliferis with sequential tests as disclosed by Wald because they have certain “optimum properties” [Wald, section A paragraph beginning “In this paper”], thereby resulting in a more robust statistical method.
Li is in the field of machine learning. Moreover, Maughan in view of Mukherjee, Aliferis, Wald, and Li discloses:
and unsupervised [retraining] an anomaly detector … and detecting, by the anomaly detector after said retraining, that a new network packet is anomalous; ([Li, abstract]: Li discloses a “method to detect anomalies in large datasets under a fully unsupervised setting” [Li, abstract]. In the combination, the model that is retrained in response to drift detection as in Maughan is taken to be an anomaly detector as in Li (and the data that is input into the anomaly detector of Li is the data described in Maughan), and the retraining of the anomaly detector is performed using the unsupervised method disclosed in Li. In other words, any of the data described in Maughan which is detected to be anomalous by the anomaly detector of Li after the unsupervised retraining disclosed by Maughan in view of Mukherjee and Li maps to the “new network packet” of the claim.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the drift detection system of Maughan in view of Mukherjee, Aliferis, and Wald with the unsupervised anomaly detector of Li because the method “outperforms state-of-the-art unsupervised techniques and is comparable to semi-supervised techniques in most cases” [Li, abstract], so the combination would couple the drift detection system with an effective and efficient anomaly detector.
Claim 7
Maughan in view of Mukherjee, Aliferis, Wald, and Li discloses the elements of the parent claim(s). It also discloses:
[The method of Claim 1 wherein] internals of said anomaly detector are inaccessible. ([Li, abstract]: The internals of the anomaly detector of Li fall under the broadest reasonable interpretation of being “inaccessible” in view of the 112(b) rejections.)
The same motivation to combine applies.
Claim 9
Maughan in view of Mukherjee, Aliferis, Wald, and Li discloses the elements of the parent claim(s). It also discloses:
[The method of Claim 1 wherein said permuting the labels of the combined plurality of network packets comprises at least one selected from a group consisting of:] assigning the label of a network packet of the combined plurality of network packets to a different network packet of the permuted plurality of network packets, ([Mukherjee, section 2]: The broadest reasonable interpretation of the claim requires just one of these. As noted above, the permutation process disclosed by Mukherjee assigns the label y_{π^m_k} to x_k. The label y_{π^m_k} originally belonged to x_{π^m_k}, so the label y_{π^m_k} is being assigned to “a different tuple” as recited by the claim.)
preserving, in the permuted plurality of network packets, respective frequencies of the first label and the second label that occur in the combined plurality of network packets, ([Mukherjee, section 2]: As noted above, the permutation process disclosed by Mukherjee simply permutes the labels using the permutation π^m. The application of a permutation necessarily preserves respective frequencies of the labels, as recited by the claim.)
reassigning the label of a network packet of the permuted plurality of network packets to a randomly selected one of the first label and the second label, ([Mukherjee, section 2]: As noted above, the permutation process of Mukherjee reassigns label y_{π^m_k} to x_k from its original assignment to x_{π^m_k}, where the permutation π^m is “sample[d]… from a uniform distribution over [the set of all permutations]” [Mukherjee, section 2]. In other words, y_{π^m_k} falls under the broadest reasonable interpretation of a “randomly selected one of the first label and the second label” as recited by the claim.)
and reassigning the label of a randomly selected network packet of the permuted plurality of network packets to a different label. ([Mukherjee, section 2]: As noted above, the permutation process of Mukherjee reassigns label y_{π^m_k} to x_k from its original assignment to x_{π^m_k}, where the permutation π^m is “sample[d]… from a uniform distribution over [the set of all permutations]” [Mukherjee, section 2]. This process can equivalently be described as a reassignment of the label y_j from x_j to x_{(π^m)^{-1}(j)}, where (π^m)^{-1} denotes the inverse permutation of π^m. Then x_{(π^m)^{-1}(j)} is the “randomly selected tuple” of the claim, and its original label y_{(π^m)^{-1}(j)} is reassigned to the “different label” y_j as recited by the claim.)
Claim 10
Maughan in view of Mukherjee, Aliferis, Wald, and Li discloses the elements of the parent claim(s). It also discloses:
[The method of Claim 1 wherein] said measuring entails cross validation. ([Mukherjee, section 3]: As noted under the parent claim, Mukherjee discloses that “the test error can be estimated… using cross-validation” [Mukherjee, section 3]. This means that both fitness scores already “entail[…] cross-validation” as recited by the claim.)
The same motivation to combine applies.
Claim 11
Maughan in view of Mukherjee, Aliferis, Wald, and Li discloses the elements of the parent claim(s). It also discloses:
[The method of Claim 1 wherein:] said retraining the anomaly detector uses at least the plurality of recent network packets; ([Maughan, 0163]: Maughan discloses that the “retrain module 302 retrains 914 the model using the new or modified training data” [Maughan, 0163].)
the plurality of old network packets is larger than the plurality of recent network packets. ([Maughan, 0065; Mukherjee, section 4.2 table 1]: As noted under the parent claim, Maughan discloses 0 being used as a label for before a timestamp (“plurality of old network packets”) and 1 for data after a timestamp (“plurality of recent network packets”) [Maughan, 0065]. In the combination, these labels correspond to the labels y_k of Mukherjee [Mukherjee, section 2]. Moreover, Mukherjee discloses a number of specific situations which include several situations (“Lymphoma outcome”, “Brain cancer outcome”, “Breast cancer outcome”, “Medullo vs. glioma”, “AML vs ALL”, and “Tumor vs norm”) where the number of “pos” samples is distinct from the number of “neg” samples. Depending on whether “pos” or “neg” that is taken to correspond to the binary label 1 of Maughan, the combination of references discloses “the plurality of old packets [being] larger than the plurality of recent network packets” as recited by the claim.)
The same motivation to combine applies.
Claim 12
Maughan in view of Mukherjee, Aliferis, Wald and Li discloses the elements of the parent claim(s). It also discloses:
[The method of Claim 1 wherein:] the first fitness score is a first plurality of scores; ([Aliferis, 0090-0091]: As noted under claim 2, Aliferis discloses a process of “generat[ing] random samples” [Aliferis, 0090] which are then used to “estimate performance” of a model [Aliferis, 0091]. In the combination, the drift detection method of Maughan in view of Mukherjee and Li to this sequence of random samples. The sequence of values t_0 obtained in this way then map to the “first plurality of scores” of the claim. The examiner notes that this combination also multiplies the number of scores in the “second plurality” as mapped above, since there one set of values t^1, …, t^M for every random sample generated.)
the second fitness score is a second plurality of scores; ([Mukherjee, section 2]: As noted under the parent claim, the collection t^1, …, t^M [Mukherjee, section 2 page 4 paragraph beginning “Suppose”] can be mapped to the “second fitness score” of the claim. The values t^1, …, t^M map to the “second plurality” of the claim.)
[said comparison of the first fitness score to the second fitness score comprises] one selected from a group consisting of: detecting that an average of the first plurality of scores falls within a particular quantile of the second plurality of scores, and performing a t-test. ([Mukherjee, section 2; Aliferis, 0105]: The broadest reasonable interpretation of this claim requires only one of the two conditions be satisfied. Mukherjee discloses detecting that t_0 falls within a particular quantile of t^1, …, t^M (namely, a particular percentile that is determined by the significance level α). Aliferis discloses “calculating the average performances for every sample size cutoff” [Aliferis, 0105]. In the combination, the t_0 values obtained as described above are the performances to be averaged, and it is these average performances that are compared against the “second plurality” as described above.)
Claim 14
Maughan in view of Mukherjee, Aliferis, Wald, and Li discloses the elements of the parent claim(s). It also discloses:
[The method of Claim 1 wherein] the anomaly detector accepts, as input, an integer feature having a range that includes a gap. ([Maughan, 0087, 0093, 0096, 0105, 0107]: Maughan indicates that “a feature may comprise a column, category, data type, attribute, characteristic, label, or other grouping of data” [Maughan, 0105] and that “certain learned functions may accept instances of one or more features as input” [Maughan, 0107]. It also discusses “a range or interval for a feature” [Maughan, 0087]. Maughan gives several concrete examples of features, including the “age” of a patient [Maughan, 0093 and 0096], which is an integer feature. Any range of integers includes gaps, namely, the gaps that lie between the integers.)
The same motivation to combine applies.
Claim 15
Maughan in view of Mukherjee, Aliferis, Wald, and Li discloses the elements of the parent claim(s). It also discloses:
[The method of Claim 1 wherein:] the anomaly detector accepts a plurality of features as input; ([Maughan, 0105, 0107]: Maughan discusses features, indicating that “certain learned functions may accept instances of one or more features as input” [Maughan, 0107]. These input features are the “plurality of features” of the claim.)
said data drift is based on at least one selected from a group consisting of: a first feature of the plurality of features that is correlated with a second feature of the plurality of features, and more than one feature of the plurality of features. ([Maughan, 0070]: The broadest reasonable interpretation of the claim requires just one of these. The system disclosed in Maughan “indicat[es] which feature(s) include one or more drifted values” [Maughan, 0070]. These “drifted features” [Maughan, 0070] map to the “more than one of the plurality of features” of the claim.)
The same motivation to combine applies.
Claim 19
Maughan discloses:
One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause: ([Maughan, 0022]: Maughan discloses that “aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable storage media having computer readable program code embodied thereon” [Maughan, 0022].)
performing in each iteration of a sequence of iterations: ([Maughan, figures 8-9]: The broadest reasonable interpretation of a “sequence of iterations” encompasses a sequence of length 1, i.e., a single iteration. In particular, one run-through of the method of drift detection [Maughan, figure 8 step 804 or figure 9 step 904] falls under the broadest reasonable interpretation of a “sequence of iterations” of the claim. The examiner notes that the combination of references below discloses a sequence of iterations having a plurality of iterations.)
[a) randomly selecting] a plurality of old network packets and a plurality of recent network packets; b) assigning: to each network packet in the plurality of old network packets, a first label that indicates old network packets, and to each network packet in the plurality of recent network packets, a second label that indicates recent network packets; ([Maughan, 0065, 0103]: Maughan discloses “data before a timestamp labeled with a ‘0’ and data after the timestamp labeled with a ‘1’” [Maughan, 0065]. Moreover, the data being labeled with these labels is indicated as being received by a data receiver module “over a data network” [Maughan, 0103]. In other words, this data falls under the broadest reasonable interpretation of a “network packet” as in the claim. The data before the timestamp maps to the “plurality of old network packets” of the claim and the data after the timestamp to the “plurality of recent network packets” of the claim. The label 0 maps to the “first label that indicates old network packets” of the claim and the label 1 maps to the “second label that indicates recent network packets” of the claim.)
e) supervised training a binary classifier by inferring, respectively from each network packet in a particular plurality of network packets, the first label that indicates old network packets or the second label that indicates recent network packets, wherein the particular plurality of network packets is selected from a group consisting of the permuted plurality of network packets and the combined plurality of network packets; ([Maughan, 0065]: Maughan discloses a “drift detection module 204 [which] may perform a binary classification” into the two classes labeled 0 and 1 as described above [Maughan, 0065]. In other words, the drift detection module maps to the “binary classifier” of the claim and the binary classification it performs maps to the “inferring” recited by the claim. Since the drift detection module uses the timestamp-based labels described above, its training is “supervised” as recited by the claim; the examiner notes that supervised training of a binary classifier is also disclosed in Mukherjee as discussed below. Since both the “plurality of old network packets” and the “plurality of recent network packets” are undergoing binary classification, either of these pluralities falls under the broadest reasonable interpretation of the “particular plurality of network packets… wherein the particular plurality of network packets is selected from a group consisting of the permuted plurality of network packets and the combined plurality of network packets” as recited by the claim.)
[unsupervised] retraining [an anomaly detector] in response to said ceasing; ([Maughan, 0042-0043, 0052, 0065, 0075]: Maughan discloses a “prediction module 202” which is “configured to apply a model to workload data to produce one or more predictive results” [Maughan, 0052]. The examiner notes that though Maughan indicates that the model can be any one of a variety of different types [Maughan, 0042-0043], it does not specifically mention the model being an anomaly detector. Nonetheless, Maughan discloses that the drift detection module determines that drift has occurred “if the drift detection module 204 can tell the difference between the [two] classes” described above [Maughan, 0065], and it discloses a “retrain module 302 [which] is configured to retrain the model… in response to the drift detection module detecting the drift phenomenon” [Maughan, 0075]. Since the “ceasing” of the claim happens “in response to a comparison… that indicates data drift”, the retraining disclosed by Maughan is in fact “in response to said ceasing” as required by the claim.)
wherein the method is performed by one or more computers. ([Maughan, 0031]: Maughan discloses that the methods disclosed therein “can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus… such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the” method disclosed therein [Maughan, 0031].)
As noted above, Maughan discloses drift being detected “if the drift detection module can tell the difference between the classes” [Maughan, 0065], though it does not distinctly disclose a method for determining whether there is a difference between the classes, and in particular, it does not distinctly disclose the use of a permutation test for this purpose. Also, while Maughan discloses retraining a model in response to drift detection [Maughan, 0075], it does not distinctly disclose the model being an anomaly detector whose retraining is unsupervised. In other words, Maughan does not distinctly disclose:
a) randomly selecting [a plurality of old network packets and a plurality of recent network packets]
including exponentially or super-exponentially increasing a respective size of at least one plurality of network packets selected from a group consisting of the plurality of old network packets and the plurality of recent network packets;
c) combining the plurality of old network packets and the plurality of recent network packets into a combined plurality of network packets;
d) permuting the labels of the combined plurality of network packets to generate a permuted plurality of network packets;
f) measuring: a first fitness score of the binary classifier by the binary classifier inferring, respectively from each network packet in the combined plurality of network packets tuples, the first label that indicates old network packets or the second label that indicates recent network packets, and a second fitness score of the binary classifier by the binary classifier inferring, respectively from each network packet in the permuted plurality of network packets, the first label that indicates old network packets or the second label that indicates recent network packets tuples;
ceasing the sequence of iterations in response to a comparison, in a last iteration of the sequence of iterations, of the first fitness score to the second fitness score that indicates data drift;
and unsupervised [retraining] an anomaly detector … and detecting, by the anomaly detector after said retraining, that a new network packet is anomalous;
Mukherjee is in the field of statistical analysis. It discusses a binary classifier [Mukherjee, section 3], which, in the combination, corresponds to the drift detection module performing binary classification as described in Maughan [Maughan, 0065]. The classifiers are implemented by training “linear Support Vector Machines” [Mukherjee, section 4], which is a supervised machine learning method as required by the claim. It also discusses observations x_k in R^n being paired with binary labels y_k for k = 1, …, l [Mukherjee, section 2 first paragraph], and in the combination, x_k of Mukherjee corresponds to the data in which drift is to be detected, and the binary labels y_k of Mukherjee correspond to the 0 and 1 labels of Maughan as described above [Maughan, 0065]. Mukherjee explicitly indicates that the test statistic T “can be estimated… using cross-validation” [Mukherjee, section 3], and cross-validation is a method which involves a plurality of iterations (cf. James as cited in the conclusion of a previous Office action). Moreover, Mukherjee discloses the use of a permutation test to tell whether there is a difference between the classes [Mukherjee, section 2]. In other words, Maughan in view of Mukherjee discloses:
a) randomly selecting [a plurality of old network packets and a plurality of recent network packets] ([Maughan, sections 2 and 4; Maughan, 0065]: As noted above, Mukherjee discloses a dataset {(x_k, y_k)}_{k = 1}^l where x_k are observations and y_k are binary labels [Mukherjee, section 2 first paragraph], and in the combination, the observations x_k are the data in which drift is to be detected and the binary labels y_k correspond to the labels 0 and 1 of Maughan [Maughan, 0065]. Mukherjee further discloses through their examples that the dataset are randomly selected from larger populations [Mukherjee, section 4]. For example, Mukherjee describes a dataset of observations from “50 patients diagnosed with dementia of the Alzheimer type and 50 normal controls of matched age” [Mukherjee, section 4.1 first paragraph], which means that the dataset is randomly selected from the larger populations of all people diagnosed with dementia of the Alzheimer type and of all normal controls. Similarly, Mukherjee also describes a dataset containing “48 samples of AML and 25 samples of ALL” [Mukherjee, section 4.1 paragraph beginning “Before proceeding”], which means that the dataset is randomly selected from the larger populations of all cases of AML (acute myeloid leukemia) and of all cases of ALL (acute lymphoblastic leukemia). In other words, the combination of Maughan and Mukherjee discloses that the “plurality of old network packets” and the “plurality of recent network packets” of the claim are “randomly select[ed” as recited by the claim.)
c) combining the plurality of old network packets and the plurality of recent network packets into a combined plurality of network packets; ([Mukherjee, section 2]: The full dataset {(x_k, y_k)}_{k = 1}^l maps to the “combined plurality” of the claim.)
d) permuting the labels of the combined plurality of network packets to generate a permuted plurality of network packets; ([Mukherjee, section 2]: Mukherjee discloses sampling a permutation π^m and using this to “permut[e] the labels” and obtain the dataset {(x_k, y_{π^m_k})}_{k = 1}^l [Mukherjee, section 2 page 4 paragraph beginning “Suppose”]. This {(x_k, y_{π^m_k})}_{k = 1}^l maps to the “permuted plurality” of the claim.)
f) measuring: a first fitness score of the binary classifier by the binary classifier inferring, respectively from each network packet in the combined plurality of network packets tuples, the first label that indicates old network packets or the second label that indicates recent network packets, ([Mukherjee, sections 2-3]: Mukherjee discloses “comput[ing] the test statistic for the actual labels t_0 = T(x_1, y_1, …, x_l, y_l)” [Mukherjee, section 2 page 4 paragraph beginning “Suppose”], where T is a an arbitrary statistic [Mukherjee, section 2 first paragraph centered equation], and then deriving a p-value p_0 [Mukherjee, section 2 page 4 paragraph beginning “Suppose”]. Either t_0 or p_0 can map to the “first fitness score” of the claim as recited.) and a second fitness score of the binary classifier by the binary classifier inferring, respectively from each network packet in the permuted plurality of network packets, the first label that indicates old network packets or the second label that indicates recent network packets tuples; ([Mukherjee, sections 2-3]: Mukherjee discloses “comput[ing] the statistic value for this permutation of labels t^m = T(x_1, y_{π^m_1}, …, x_l, y_{π^m_l})” [Mukherjee, section 2 page 4 paragraph beginning “Suppose”]. This step is repeated “M times (with index m = 1, …, M)” so, alternatively, the entire collection t^1, …, t^M can also be mapped to the “second fitness score” of the claim.)
a comparison, in a last iteration of the sequence of iterations, of the first fitness score to the second fitness score that indicates data drift; ([Mukherjee, section 2]: Mukherjee discloses using t^m to generate an “empirical cumulative distribution” hat{P}, computing a p-value p_0 for t_0 under hat{P}, and comparing p_0 to an “acceptable significance α” [Mukherjee, section 2 page 4 paragraph beginning “Suppose”]. This process falls under the broadest reasonable interpretation of “a comparison of the first fitness score to the second fitness score” as recited in the claim with the fitness scores being as mapped above. Since “the null hypothesis assumes that the two classes are indistinguishable” [Mukherjee, section 2 paragraph beginning “The procedure”], accepting (i.e., failing to reject) the null hypothesis using the method of Mukherjee becomes, in the combination with Maughan, an indication of data drift as recited in the claim.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the drift detection system of Maughan with the permutation test of Mukherjee because they are “useful in testing if the observed classification results are likely to be obtained by chance” [Mukherjee, section 3 paragraph beginning “To underscore”] and the combination would thus result in an effective drift detection system.
Maughan in view of Mukherjee might not distinctly disclose:
including exponentially or super-exponentially increasing a respective size of at least one plurality of network packets selected from a group consisting of the plurality of old network packets and the plurality of recent network packets;
ceasing the sequence of iterations in response to [a comparison… that indicates data drift]
and unsupervised [retraining] an anomaly detector … and detecting, by the anomaly detector after said retraining, that a new network packet is anomalous;
Aliferis is in the field of statistical analysis. In particular, Maughan in view of Mukherjee and Aliferis discloses:
including exponentially or super-exponentially increasing a respective size of at least one plurality of network packets selected from a group consisting of the plurality of old network packets and the plurality of recent network packets; ([Aliferis, 0090-0091]: Aliferis discloses a process of “generat[ing] random samples of increasing sample sizes. For example, for a starting point of sample size 1000 with 1000 repeats, this process would give 1000 randomly sampled data sets of sample size 100… Then, perform this repeated sampling for several increasing sample sizes (e.g., 100, 150, 200, …, total available sample)” [Aliferis, 0090]. These samples are then used to “estimate performance” of a model [Aliferis, 0091]. In the combination, the samples in the iterations disclosed by Maughan in view of Mukherjee and Wald are taken to have increasing sizes as disclosed by Aliferis. The examiner notes that any increase from one number a to another b falls under the broadest reasonable interpretation of an “exponential increase” since b can always be written as the ath power of some quantity (namely, b = (e^{ln(b)/a})^a), or, alternatively, b can always be written as some power of a (namely, b = a^{log_a(b)}). This implies that the increases described under the parent claim can be regarded as either “exponential” or as “super-exponential” (for example, an increase from 100 to 150 [Aliferis, 0090] is “exponential” by the logic presented here, but at the same time, the increase from 100 to 149 is also “exponential” by this logic, which means that the increase from 100 to 150 is “super-exponential”). Alternatively, the examiner notes that the sequence given by x_n = 100*2^n - 50*n^2 for each non-negative integer n is an exponential sequence (its dominant term is exponential) and elementary arithmetic (i.e., simply plugging in n = 0, 1, and 2) shows that the first three terms of this sequence are x_0 = 100, x_1 = 150, and x_2 = 200, i.e., the same as the first three terms in the example given in Aliferis. For yet another mapping of this limitation, the applicant is invited to consult Al-Omari as cited in the conclusion of a previous Office action.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the drift detection system disclosed by Maughan in view of Mukherjee with the statistical techniques of Aliferis because they allow for “determining sufficient sample size… taking into account the critical factors that affect the needed sample size [and] can be applied to practically any field where predictive modeling or causal modeling are desired” [Aliferis, abstract], so the system would be more robust overall.
Maughan in view of Mukherjee and Aliferis might not distinctly disclose:
ceasing the sequence of iterations in response to [a comparison… that indicates data drift]
and unsupervised [retraining] an anomaly detector … and detecting, by the anomaly detector after said retraining, that a new network packet is anomalous;
Wald is in the field of statistical analysis. Moreover, Maughan in view of Mukherjee, Aliferis, and Wald discloses:
ceasing the sequence of iterations in response to [a comparison… that indicates data drift] ([Wald, section A]: As noted above, Maughan in view of Mukherjee discloses a statistical method in which accepting a null hypothesis is an indication of data drift. Wald discloses a “sequential test of a statistical hypothesis” which entails repetition until a null hypothesis is either accepted or rejected (cf. “[i]f the first or second decision is made, the process is terminated”) [Wald, section A first paragraph]. In the combination, the statistical method of Maughan and Mukherjee is repeated to form a sequential test as in Wald. The combination thus discloses both a “sequence of iterations” having more than one iteration, as well as “ceasing the sequence of iterations in response to a comparison… that indicates data drift” as recited by the claim.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the drift detection system of Maughan in view of Mukherjee and Aliferis with sequential tests as disclosed by Wald because they have certain “optimum properties” [Wald, section A paragraph beginning “In this paper”], thereby resulting in a more robust statistical method.
Li is in the field of machine learning. Moreover, Maughan in view of Mukherjee, Aliferis, Wald, and Li discloses:
and unsupervised [retraining] an anomaly detector … and detecting, by the anomaly detector after said retraining, that a new network packet is anomalous; ([Li, abstract]: Li discloses a “method to detect anomalies in large datasets under a fully unsupervised setting” [Li, abstract]. In the combination, the model that is retrained in response to drift detection as in Maughan is taken to be an anomaly detector as in Li (and the data that is input into the anomaly detector of Li is the data described in Maughan), and the retraining of the anomaly detector is performed using the unsupervised method disclosed in Li. In other words, any of the data described in Maughan which is detected to be anomalous by the anomaly detector of Li after the unsupervised retraining disclosed by Maughan in view of Mukherjee and Li maps to the “new network packet” of the claim.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the drift detection system of Maughan in view of Mukherjee, Aliferis, and Wald with the unsupervised anomaly detector of Li because the method “outperforms state-of-the-art unsupervised techniques and is comparable to semi-supervised techniques in most cases” [Li, abstract], so the combination would couple the drift detection system with an effective and efficient anomaly detector.
Claim(s) 6 and 25 is/are rejected under 35 USC 103 as being unpatentable over Maughan in view of Mukherjee, Aliferis, Wald, and Li, further in view of Marco TABOGA (StatLect Glossary: Critical Value, published 2019; hereafter “Taboga”).
Claim 6
Maughan in view of Mukherjee, Aliferis, Wald, and Li discloses the elements of the parent claim(s). It also discloses:
[The method of Claim 1 wherein:] the first fitness score is based on a plurality of scores; ([Mukherjee, section 2]: As noted under the parent claim, either the quantity t_0 = T(x_1, y_1, …, x_l, y_l) where T is an arbitrary test statistic, or the p-value p_0 [Mukherjee, section 2] can be mapped to the “first fitness score” of the claim. Any quantities which are used in the calculation of these values can be mapped to the “plurality of scores” of the claim. For example, if t_0 is taken to be the “first fitness score” of the claim, then the input values x_1, y_1, …, x_l, y_l to the function T, or any intermediate values involved in the computation of T, can map to the “plurality of scores” of the claim as recited. If p_0 is taken to be the “first fitness score” of the claim, then again, any the input values x_1, y_1, …, x_l, y_l to the function T or any intermediate values involved in the computation of T can map to the “plurality of scores”, or, alternatively, the values t^1, …, t^M which are used to construct the empirical cumulative distribution hat{P} can map to the “plurality of scores” of the claim. In the combination with Taboga presented below, p_0 is mapped to the “first fitness score” and t^1, …, t^M to the “plurality of scores” of the claim.)
[said comparison of the first fitness score to the second fitness score comprises] a comparison of the second fitness score to a threshold score; ([Mukherjee, section 2]: As noted under the parent claim, the method disclosed by Mukherjee involves comparison against an “acceptable significance level α” [Mukherjee, section 2 page 4 paragraph beginning “Suppose”]. This significance level α maps to the “threshold score” of the claim. The process described in Mukherjee of comparing p_0 to α falls under the broadest reasonable interpretation of “a comparison of the second fitness score to a threshold score” since p_0 is computed in terms of t^m, i.e., in terms of the “second fitness scores” as mapped above. The examiner notes that, in the combination with Taboga presented below, the mapping of the “threshold score” is changed to a closely related value.)
Maughan in view of Mukherjee, Aliferis, Wald, and Li does not distinctly disclose:
[the method further comprises] calculating the threshold score based on the plurality of scores.
Taboga is in the field of statistical analysis. Moreover, Maughan in view of Mukherjee, Aliferis, Wald, Li, and Taboga discloses:
[the method further comprises] calculating the threshold score based on the plurality of scores. ([Taboga]: Taboga discusses the well-known method of computing a “critical value z” as the “inverse of the cdf z = F^{-1}(α)”, where F is the “cumulative distribution function (cdf) of the test statistic” and α is the “probability of rejecting the null hypothesis when it is true”, i.e., the significance level of the statistical test [Taboga]. In the combination, the cdf F of Taboga corresponds to the empirical cumulative distribution hat{P} of Mukherjee, and the α of Taboga corresponds to the α of Mukherjee, and the check p_0 ≤ α of Mukherjee is replaced with a check t_0 ≤ z. In other words, the critical value z maps to the “threshold score” of the claim: it is “based on the plurality of scores” t^1, …, t^M as mapped above, since it is computed as the inverse cdf applied to α, and the cdf hat{P} depends on t^1, …, t^M. The examiner notes that this is exactly the procedure described in the specification regarding the threshold score [0080-0082]; the values t^1, …, t^M are the “score_dist” of the specification and the critical value z is a “quantile of score_dist” as described in the specification.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the system for drift detection and correction as disclosed by Maughan in view of Mukherjee, Aliferis, Wald, and Li with the use of critical values as disclosed by Taboga because critical values provide a clear cutoff point in the distribution of the test statistic which applied consistently across analyses.
Claim 25 inherits limitations from claim 19 and recites additional limitations which are substantially similar to those recited by claim 6, so it is rejected by the same rationale.
Claim(s) 26 is/are rejected under 35 USC 103 as being unpatentable over Maughan in view of Mukherjee, Aliferis, Wald, and Li, further in view of Jingjie ZHU et al. (US20200235833A1, published 2020-01-23; hereafter “Zhu”).
Claim 26
Maughan in view of Mukherjee, Aliferis, Wald, and Li discloses the elements of the parent claims. While it discusses “disparate instructions stored in different locations” and operational data being “distributed over different locations” [Maughan, 0024-0025], it does not mention network switches. In other words, it might not distinctly disclose:
[The method of Claim 1 wherein:] the one or more computers comprises a network switch that contains said anomaly detector; the network switch does not contain the binary classifier.
Zhu is in the field of machine learning. Moreover, Maughan in view of Mukherjee, Aliferis, Wald, Li, and Zhu discloses:
[The method of Claim 1 wherein:] the one or more computers comprises a network switch that contains said anomaly detector; the network switch does not contain the binary classifier. ([Zhu, figure 1A-B]: Zhu discloses a distributed system having multiple terminal devices 104 [Zhu, figure 1A], each terminal device carrying a classification model 114 [Zhu, figure 1B]. It also discloses that a “terminal device 104 can be any suitable electronic network device, which may include… network switches” [Zhu, 0095]. In the combination, the binary classifier and anomaly detector disclosed by Maughan in view of Mukherjee and Li are distributed across different terminal devices as disclosed in Zhu, with the terminal device containing the anomaly detector mapping to the “network switch” of the claim.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to distribute the models of the drift detection system disclosed by Maughan in view of Mukherjee, Aliferis, Wald, and Li across multiple terminal devices as disclosed in Zhu because distributed systems are both more efficient and more fault-tolerant.
Claim(s) 27-28 is/are rejected under 35 USC 103 as being unpatentable over Maughan in view of Mukherjee, Aliferis, Wald, and Li, further in view of Trevor HASTIE et al. (The Elements of Machine Learning, published 2009; hereafter “Hastie”).
Claim 27
Maughan in view of Mukherjee, Aliferis, Wald, and Li discloses the elements of the parent claim(s). While it discusses “decision forests” [Maughan, 0108] and “forests of [decision] trees” [Maughan, 0160-0161], it does not distinctly disclose the “binary classifier” of the claim as mapped above being a random forest. In other words, it may be argued that Maughan in view of Mukherjee, Wald, and Li does not distinctly disclose:
[The method of Claim 1 wherein] the binary classifier is a random forest.
Hastie is in the field of machine learning. Moreover, Maughan in view of Mukherjee, Wald, Li, and Hastie discloses:
[The method of Claim 1 wherein] the binary classifier is a random forest. ([Hastie, chapter 15]: Hastie discusses random forests [Hastie, chapter 15 title], including an application of random forests for a binary classification task (namely, spam detection) [Hastie, figures 15.1-2]. In the combination, the binary classifier disclosed by Maughan in view of Mukherjee and Li is implemented as a random forest as in Hastie.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to use a random forest as disclosed by Hastie in the data drift detection system disclosed by Maughan in view of Mukherjee, Aliferis, Wald, and Li because “[o]n many problems the performance of random forests is very similar to boosting, and they are simpler to train and tune. As a consequence, random forests are popular” [Hastie, chapter 15 first paragraph].
Claim 28 inherits limitations from claim 19 and recites additional limitations which are substantially similar to those recited by claim 27, so it is rejected by the same rationale.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Shishir AGRAWAL whose telephone number is +1 703-756-1183. The examiner can normally be reached Monday through Thursday, 08:30-14:30 Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey SHMATOV can be reached on +1 571-270-3428. The fax phone number for the organization where this application or proceeding is assigned is +1 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at +1 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call +1 800-786-9199 (IN USA OR CANADA) or +1 571-272-1000.
/S.A./Examiner, Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights