DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on December 01, 2025 is/are in compliance with the provisions of 37 CFR 1.97 and has/have been considered by the examiner.
Response to Amendment
The Amendment filed February 03, 2026 has been entered. Claims 1, 5, 12 and 21 have been amended. Applicant’s amendments to claim 1 have overcome the claim objections previously set forth in the Office Action mailed November 03, 2025. The objection of claim 1 has been withdrawn.
Claim Objections
4. Claims 1, 2, 4, 5, 8, 12, 13, 15 and 23 are objected to because of the following informalities:
Claim 1 recites “wherein the verdicts being used to train the threat hunting model by…”. The examiner suggests amending the limitation to the following: “wherein the verdicts are being used to train the threat hunting model by…”.
In line 13, claim 1 recites “… weightings associated to each of the plurality of hunting functions”. The examiner suggests amending the limitation to the following: “…weightings associated with each of the plurality of hunting functions”.
In line 2, claim 2 recites “… comprises hunting function analytic logic….”. The examiner suggests amending the limitation to the following: “… comprises a hunting function analytic logic…”.
In claim 4, “event data …pertaining to network traffic or device” should read “event data …pertaining to network traffic or devices” or “event data …pertaining to network traffic or a network device”.
Claim 5 recites “agreement evaluation logic…”. The examiner suggests amending the limitation to the following: “an agreement evaluation logic…”. This suggestion also applies to claim 23.
Claim 5 recites “disagreement evaluation logic…”. The examiner suggests amending the limitation to the following: “a disagreement evaluation logic…”. This suggestion also applies to claim 23.
Claim 8 recites “coverage logic…”. The examiner suggests amending the limitation to the following: “a coverage logic…”.
In claim 12, in line 11, “… quality metrics that satisfies the first threshold”, should read “… quality metrics that satisfy the first threshold”.
In claim 13, in line 3, “…deployment as part of function evaluator; …a training dataset for ML model” should read “… deployment as part of the function evaluator; …a training dataset for the ML model”.
In claim 15, in line 4, “”… protected by ML model” should read “…protected by the ML model”.
Claim 23 recites “includes hunting function analytic logic that comprises…”. The examiner suggests amending the limitation to the following: “…includes a hunting function analytic logic that comprises: ”.
Claim Rejections - 35 USC § 112
5. The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
6. Claims 3 and 9 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 3 recites “(iii) performing analytics on the data to determine whether the training dataset is potentially malicious or benign,”. It is unclear what element the limitation “data” was referencing.
Claim 9 recites the limitation “the function evaluator". There is insufficient antecedent basis for this limitation in the claim. Claim depends from claim 1. However, claim 1 contains no earlier recitation or limitation of a function evaluator.
Response to Arguments
7. Applicant’s arguments, see pages 8-13, filed February 03, 2026, with respect to the rejections of claims 1, 2, 9, 12, 13, 16 and 17 under 35 U.S.C. § 102 have been considered but are moot in view of the new grounds of rejection. The claims (as amended) do not overcome the new ground of rejection made in view of newly found prior art references.
8. Applicant’s arguments, see pages 17-20, filed February 03, 2026, with respect to the rejection of claim 5 under 35 U.S.C. § 103 have been considered but are moot in view of the new grounds of rejection. The claim does not overcome the new ground of rejection made in view of newly found prior art references.
9. Applicant’s arguments, with respect to the rejections of claims 3, 4, 6-8, 10, 11, 14, 15 and 18-20 under 35 U.S.C. § 103 independent have been considered but are moot in view of the new grounds of rejection. The claims do not overcome the new ground of rejection made in view of newly found prior art references.
10. Applicant’s arguments, see pages 13-17, filed February 03, 2026, with respect to the rejections of claims 21-24 under 35 U.S.C. § 103 independent have been considered but are moot in view of the new grounds of rejection. The claim (as amended) do not overcome the new ground of rejection made in view of newly found prior art references.
Claim Rejections - 35 USC § 103
11. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
12. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
13. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
14. Claims 1, 2, 9, 21, 22 and 24 are rejected under 35 U.S.C. § 103 as being unpatentable over Conwell et al. (US 2022/0147815 A1), hereafter Conwell, in view of Wu (US 2022/0188517A1), hereafter Wu.
Noted that indicates what the cited art does not teach.
Regarding claim 1, Conwell teaches a cybersecurity threat hunting system, comprising: {Conwell [Para. 0019] “Systems for producing and using enhanced machine learning models and computer-implemented tools to investigate cybersecurity related data and threat intelligence data.”}
a model generation subsystem including a plurality of hunting functions configured to generate and train a threat hunting model, {Conwell [Para. 0020] “The EPSS (Enhanced Predictive Security System) uses a domain centric approach combined with advanced machine learning algorithms and a multi-level machine learning architecture that utilizes one or more subsets of the smaller models trained with different data, whose results are combined as input to an ensemble master classifier, which can be ultimately tuned and optimized for the type of data it is responsible for classifying. Hence each model subset acts as a set of “weak classifiers” for a particular type or collection of threat data. A combination of the results of each applicable subset of weak classifiers then is fed as input into the ensemble master classifier, which can be iteratively run with varying weights applied to the weak classifier subset outputs until a determined optimization value is reached.”} The ensemble master classifier corresponds to the threat hunting model, and the subsets of smaller model (e.g., weak classifier) correspond to the hunting functions.
wherein each hunting function of the plurality of hunting functions is configured to conduct analytics on features extracted from a training dataset to determine a verdict based on an analysis of the extracted feature defined in the hunting function, {Conwell [Para. 0026] “The feature class engines 111 are used to select and transform domain related data stored in repository 115 to actionable feature class vectors used as input into the weak classifiers.” [Para. 0051] “Each application comprises one or more collections (subsets) of models, which are trained using different training data but otherwise share the same machine learning algorithm, modeling tuning parameters, and feature vector values, which can be ultimately tuned and optimized for the type of data the model is responsible for classifying.” [Para. 0054] “Once the model subset 810 is built the model output from each of the models 811i (the “weak classifiers” ) is aggregated into model subset output 813. Each model 811i can output two values, which assist in forming the model subset output 813. Typically this score is a value pair (Cn, Sn), where the pair represents a pair of values (Boolean classification or a classification score value, an indicator of existence of a classification score) or a pair of values (likelihood/probability of classification, confidence in the likelihood of classification).”}
wherein the verdicts being used to train the threat hunting model by assigning a weighting to each hunting function; {Conwell [Para. 0051] “A combination of the results of each applicable subset of weak classifiers then is fed as input into an ensemble master classifier, which can be iteratively run with varying weights applied to the weak classifier subset outputs until a determined optimization value (e.g., a threshold, minimum, percentage, probability, etc.) is reached.” [Para. 0056] “The results of each model subset output 813, 822, and 832 are input into input vector 850. Each of these results is then initially weighted by some amount specified in weight vector 860 before being input into the ensemble classification engine 870.” [Para. 0057] “Master classifier 812 contains a third layer of machine learning, that is it includes a feedback loop 860-870, which iterates adjusting the weights 860 applied to the model subset outputs' input 850 until the classification result has been optimized, for example, using gradient descent boosting.”}
and a threat detection subsystem communicatively coupled to the model generation subsystem, {Conwell [Para. 0024] “The EPSS comprises one or more functional components /modules that work together to provide an architecture and a framework for building and deploying cybersecurity threat analysis application. The EPSS 100 may comprise one or more machine learning algorithms 110, feature class engines (for use with feature engineering) 111, tuning systems 112, ensemble classifier engines 113, and validation and testing engines 114.”} Conwell’s EPSS generates, trains, and deploys trained ML models to predict cyber threats.
the threat detection subsystem to receive the trained threat hunting model and conduct analytics on event data to determine an aggregate threat score associated with the event data based on the weightings associated to each of the plurality of hunting functions. {Conwell [Para. 0051] “The resultant ensemble master classifier can then be deployed as a cybersecurity threat analysis application and applied an unknown domain to predict whether the domain is “predictably malicious.” [Para. 0058] “The output of an optimized ensemble classifier is a final score such as final score 880. This output may be a single score Cn that represents the prediction, for example, in Boolean form or a score between 0 and 1.” [Para. 0059] “As shown in FIG. 8, a simpler ensemble classifier may be appropriate for the application, such as classifier 831 which may employed simple voting or weighted voting to achieve a final score 835.” [Para. 0057] “Master classifier 812 contains a third layer of machine learning, that is it includes a feedback loop 860-870, which iterates adjusting the weights 860 applied to the model subset outputs' input 850 until the classification result has been optimized, for example, using gradient descent boosting.”[Para. 0036] “In block 401, the system collects domain related data from Internet Infrastructure Data (IID), both gathered and derived, including for example, domain names, “whois” protocol information, IP addresses, DNS record data, passive DNS activity data, scraped HTML content,,…”} DS record data and passive DNS activity exemplify event data that are used for detecting malicious activities such as domain hijacking and botnets.
However, Conwell does not teach wherein the verdicts being used to train the threat hunting model by assigning a weighting to each hunting function, wherein the weighting is based on a level of agreement between the verdict of the hunting function and verdicts of other hunting functions of the plurality of hunting functions;
However, Wu teaches wherein the verdicts being used to train the threat hunting model by assigning a weighting to each hunting function, wherein the weighting is based on a level of agreement between the verdict of the hunting function and verdicts of other hunting functions of the plurality of hunting functions; {Wu [Para. 21] “Each of the real-time AI engine can be a light-weight machine learning model that can be trained and run in real time.” [Para. 0085] “The master AI engine can be a weighted ensemble based on each real-time AI model's performance on the gold data generated by other real-time AI engines. Since each real-time AI engine was learned from a human operator, the trained real-time AI engine may encode the operator's knowledge. The more agreement a real-time AI engine has with other real-time AI engines, the better the machine learning model of the real-time AI engine as reflected by a higher weight value associated with the real-time AI engine.”} See para. 85 for more details. Wu assigns weights to individual models based on their degree of consensus with other ensemble members.
Wu is analogous art because each of Conwell and Wu pertains to generating ensemble machine learning models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Conwell to include Wu’s teaching of the limitations of claim 1, listed above. Doing so would provide the advantages disclosed in paras. 92 and 93 of Wu, which “makes the final model more general, more robust, and more accurate” (Wu, para. 93).
Claim 2:
Regarding claim 2, Conwell and Wu teach the elements of claim 1 as stated above.
Conwell further discloses wherein the model generation subsystem comprises hunting function analytic logic configured to determine quality metrics associated with at least a first hunting function of the plurality of hunting functions and, when the quality metrics fail to satisfy a threshold, the model generation subsystem at least (i) alters the first hunting function deployed within the threat hunting model or (ii) substitutes the first hunting function for a second hunting function different than the first hunting function prior to providing a function evaluator including the plurality of hunting functions and the trained threat hunting model to the threat detection subsystem. {Conwell [Para. 0030] “Pipeline 200 illustrates how models are built and tuned for deployment as a cybersecurity threat analysis application. Portions of the pipeline 200 are looped and assessed (or reassessed) until the executed model 220 is capable of predicting a result 221 that is considered “acceptable” (e.g., correct according to some determined value, percentage of time, threshold, precision and/or recall statistical requirements, etc.). According to the pipeline 200, labeled (known) malicious data 201 along with labeled (known) neutral data 202 in the form of training data 203 along with model tuning parameters 205 and a certain machine learning algorithm 204 are input into a build process 210 to build a trained model instance 211 (a binary). This trained model instance 211 (i.e., trained model) is then run (shown as model execution 220) on labeled malicious and neutral test data 212 to generate a prediction/result 221. The resultant prediction 221 is input along with labeled malicious and neutral validation data 216 into a tuning system 215, which is used to determine the (potentially modified) model tuning parameters 205 to run in the next iteration of the pipeline (rebuilding the model instance 210 and executing the trained and tuned model 220) until the trained model 220 predicts an outcome (result) that is correct sufficient times and with sufficient accuracy to be considered acceptable (the validation data is used to validate the prediction of the test data as malicious or not). This loop continues until a prediction/result 221 is generated that is considered within acceptable characteristics as described above. When an acceptable trained model state is achieved, trained model instance 211 can be deployed in an application (model execution 220) with new (unlabeled data) domain data 214 to generate a prediction/result 230.”} Altering a ML model includes tuning it.
Claim 9:
Regarding claim 9, Conwell and Wu teach the elements of claim 1 as outlined above.
Conwell further discloses wherein the function evaluator includes a first hunt pack comprising a first plurality of the hunting functions and a second hunt pack comprising a second plurality of the hunting functions. {Conwell [Para. 0020] “The EPSS uses… a multi-level machine learning architecture that utilizes one or more subsets of the smaller models trained with different data, whose results are combined as input to an ensemble master classifier. Each subset of the smaller models includes multiple instances of a same model sharing a same machine learning algorithm, modeling tuning parameters, and feature vector values but trained using different trained data. Hence each model subset acts as a set of “weak classifiers” for a particular type or collection of threat data.”}
Claim 21:
Regarding claim 21, Conwell teaches a threat hunting engine comprising: {Conwell [Para. 19] “Systems for producing and using enhanced machine learning models and computer-implemented tools to investigate cybersecurity related data and threat intelligence data.”}
a function evaluator to analyze received event data and extract therefrom features of potential relevance to a determination of cybersecurity risks and to form one or more verdicts based on an analysis of the features defined in a corresponding hunting function, {Conwell [Para. 0020] “The EPSS (Enhanced Predictive Security System) uses… a multi-level machine learning architecture that utilizes one or more subsets of the smaller models trained with different data, whose results are combined as input to an ensemble master classifier… Hence each model subset acts as a set of “weak classifiers” for a particular type or collection of threat data.” [Para. 0026] “The feature class engines 111 are used to select and transform domain related data stored in repository 115 to actionable feature class vectors used as input into the weak classifiers.” [Para. 0054] “Once the model subset 810 is built the model output from each of the models 811i (the “weak classifiers”) is aggregated into model subset output 813. Each model 811i can output two values, which assist in forming the model subset output 813.”} Para. 36 describes examples of event data (e. g., passive DNS activity data, DNS record data) analyzed by the EPSS.
wherein the function evaluator including a set of one or more hunt packs, each of the one or more hunt packs including one or more hunting functions, {Conwell [Para. 0020] “The EPSS uses… a multi-level machine learning architecture that utilizes one or more subsets of the smaller models trained with different data, whose results are combined as input to an ensemble master classifier. Each subset of the smaller models includes multiple instances of a same model sharing a same machine learning algorithm, modeling tuning parameters, and feature vector values but trained using different trained data.”}
each of the one or more hunt packs and the one or more hunting functions comprising an executable program. {Conwell [Para. 0024] “The trained model library 117 stores definitions of each model subset for easy re-instantiation, including an indication of the machine learning algorithm used to create the model along with hyper parameters for tuning the model, and a description of the feature class information used to build an input feature vector associated with the model, an indication of a source for training data, and an indication of training data sampling parameters.”} Also see para. 0026 and 0030 in Conwell.
and a threat hunting Machine Learning (ML) model configured to analyze the one or more verdicts of the one or more hunting functions to form a threat score indicative of the quality or ability of the one or more hunting functions to identify cybersecurity risks, {Conwell [Para. 0056] “The model subset output from each of the model subsets is then configured to be fed into an ensemble master classifier for that particular application so that the predictions can be reduced to a single (final) score. For Application(j), model subset output 813, 822, and 832 are configured as input 850 to the ensemble master classifier 812. The results of each model subset output 813, 822, and 832 are input into input vector 850.” [Para. 0058] “The output of an optimized ensemble classifier is a final score such as final score 880.”}
wherein the threat hunting ML model utilizes a weighting for each of the one or more hunting functions to form the threat score, {Conwell [Para. 0056] “The results of each model subset output 813, 822, and 832 are input into input vector 850. Each of these results is then initially weighted by some amount specified in weight vector 860 before being input into the ensemble classification engine 870.” [Para. 0057] “Master classifier 812 contains a third layer of machine learning, that is it includes a feedback loop 860-870, which iterates adjusting the weights 860 applied to the model subset outputs' input 850 until the classification result has been optimized, for example, using gradient descent boosting” [Para. 0059] “A simpler ensemble classifier may be appropriate for the application, such as classifier 831 which may employed simple voting or weighted voting to achieve a final score 835.”}
However, Conwell does not teach wherein the weighting is based on a level of agreement between a verdict of the hunting function and verdicts of other hunting functions.
However, Wu teaches wherein the weighting is based on a level of agreement between a verdict of the hunting function and verdicts of other hunting functions. {Wu [Para. 21] “Each of the real-time AI engine can be a light-weight machine learning model that can be trained and run in real time.” [Para. 0085] “The master AI engine can be a weighted ensemble based on each real-time AI model's performance on the gold data generated by other real-time AI engines. Since each real-time AI engine was learned from a human operator, the trained real-time AI engine may encode the operator's knowledge. The more agreement a real-time AI engine has with other real-time AI engines, the better the machine learning model of the real-time AI engine as reflected by a higher weight value associated with the real-time AI engine.”} See para. 85 for more details. Wu assigns weights to individual models based on their degree of consensus with other ensemble members.
Wu is analogous art because each of Conwell and Wu pertains to generating ensemble machine learning models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Conwell to include Wu’s teaching of the limitations of claim 21, listed above. Doing so would provide the advantages disclosed in paras. 92 and 93 of Wu, which “makes the final model more general, more robust, and more accurate” (Wu, para. 93).
Claim 22:
Regarding claim 22, Conwell further teaches the threat hunting engine of claim 21 being deployed as part of a model generation subsystem of a cybersecurity threat hunting system, {Conwell [Para. 0024] “The EPSS comprises one or more functional components /modules that work together to provide an architecture and a framework for building and deploying cybersecurity threat analysis application. The EPSS 100 may comprise one or more machine learning algorithms 110, feature class engines (for use with feature engineering) 111, tuning systems 112, ensemble classifier engines 113, and validation and testing engines 114.”} Conwell’s EPSS generates, trains, and deploys trained ML models to predict cyber threats.
wherein the model generation subsystem being configured to (i) determine quality metrics for each hunting function of the one or more hunting functions and {Conwell [Para. 0030] “Pipeline 200 illustrates how models are built and tuned for deployment as a cybersecurity threat analysis application. Portions of the pipeline 200 are looped and assessed (or reassessed) until the executed model 220 is capable of predicting a result 221 that is considered “acceptable” (e.g., correct according to some determined value, percentage of time, threshold, precision and/or recall statistical requirements, etc.).”} See para. 0030 in Conwell.
(ii) generate and train the ML model on a training dataset using the verdicts to set parameters of the ML model to achieve a prescribed operating level based on the quality metrics for the one or more hunting functions. {Conwell [Para. 0020] “A combination of the results of each applicable subset of weak classifiers then is fed as input into the ensemble master classifier, which can be iteratively run with varying weights applied to the weak classifier subset outputs until a determined optimization value is reached.” [Para. 0056] “The results of each model subset output 813, 822, and 832 are input into input vector 850. Each of these results is then initially weighted by some amount specified in weight vector 860 before being input into the ensemble classification engine 870.” [Para. 0057] “Master classifier 812 contains a third layer of machine learning, that is it includes a feedback loop 860-870, which iterates adjusting the weights 860 applied to the model subset outputs' input 850 until the classification result has been optimized, for example, using gradient descent boosting.”}
Claim 24:
Regarding claim 24, Conwell further teaches wherein the threat detection subsystem is configured to form a threat score associated with event data based on analysis, by a trained ML model, of the event data. {Conwell [Para. 0051] “The resultant ensemble master classifier can then be deployed as a cybersecurity threat analysis application and applied an unknown domain to predict whether the domain is “predictably malicious.” [Para. 0058] “The output of an optimized ensemble classifier is a final score such as final score 880. This output may be a single score Cn that represents the prediction, for example, in Boolean form or a score between 0 and 1.” [Para. 0036] “In block 401, the system collects domain related data from Internet Infrastructure Data (IID), both gathered and derived, including for example, domain names, “whois” protocol information (e.g., administrator and ownership information), IP addresses, DNS record data, passive DNS activity data,… and/or other domain related data.”} DS record data and passive DNS activity are event data that are used for detecting malicious activities such as domain hijacking and botnets. Also see para. 0059 in Conwell.
15. Claims 3, 4 and 8 are rejected under 35 U.S.C. § 103 as being unpatentable over Conwell and Wu as applied to claims 1 and 2, and further in view of Gottin (US 2019/0303799 A1), hereafter Gottin.
Regarding claim 3, Conwell and Wu teach the elements of claim 1 as outlined above.
Conwell further teaches wherein each hunting function of the plurality of hunting functions undergoes training by at least (i) receiving the training dataset, {Conwell [Para. 30] “Labeled (known) malicious data 201 along with labeled (known) neutral data 202 in the form of training data 203 along with model tuning parameters 205 and a certain machine learning algorithm 204 are input into a build process 210 to build a trained model instance 211.”}
(ii) identifying the features included in the training dataset where the extracted features can differ between the plurality of hunting functions, {Conwell [Para. 0044] “Feature engineering is performed by the EPSS to determine what IID characteristics are desirable to be examined to select and encode data for each domain to be used as input to the various machine learning algorithms.” [Para. 53] “In order to build and train a new model subset such as subset 810, feature classes are selected from a feature class library and applied to sampled domain data which are then transformed into feature vectors.”}
(iii) performing analytics on the data to determine whether the training dataset is potentially malicious or benign, and (iv) generating output data including the verdict. {Conwell [Para. 0030] “This trained model instance 211 (i.e., trained model) is then run (shown as model execution 220) on labeled malicious and neutral test data 212 to generate a prediction/result 221. The resultant prediction 221 is input along with labeled malicious and neutral validation data 216 into a tuning system 215, which is used to determine the (potentially modified) model tuning parameters 205 to run in the next iteration of the pipeline until the trained model 220 predicts an outcome (result) that is correct sufficient times and with sufficient accuracy to be considered acceptable.”}
However, Conwell and Wu do not teach (ii) identifying the features included in the training dataset where the extracted features can differ between the plurality of hunting functions.
However, Gottin teaches (ii) identifying the features included in the training dataset where the extracted features can differ between the plurality of hunting functions. {Gottin [Para. 0033] “The exemplary contextual model agreement network generation process 100 comprises the following steps, in relation to historical training data.” [Para. 0034] “Exemplary contextual model agreement network generation process 100 defines appropriate feature extraction procedures for the domain. Thereafter, a set of predictive models is generated during step 120, each considering potentially different subsets of the features in the domain.”}
Gottin is analogous art because each of Conwell, Wu and Gottin pertains to generating ensemble machine learning models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Conwell and Wu to include Gottin’s teaching of the limitation of claim 3, listed above. Doing so would “provide improved methods for real-time anomaly detection” (Gottin, para. 0119).
Claim 4:
Regarding claim 4, Conwell and Gottin teach the elements of claim 3 as outlined above.
Conwell further teaches wherein the event data includes security-related event data stored in an event log or pertaining to network traffic or device. {Conwell [Para. 0036] “In block 401, the system collects domain related data from Internet Infrastructure Data (IID), both gathered and derived, including for example, domain names, “whois” protocol information, IP addresses, DNS record data, passive DNS activity data,.”} DNS record data and passive DNS activity data exemplify security-related event data pertaining to network traffic or device.
Claim 8:
Regarding claim 8, Conwell teaches the elements of claim 2 as stated above.
Conwell further teaches wherein the hunting function analytic logic further comprises: coverage logic configured to determine a coverage value being a measure of a capability of the first hunting function or a hunt pack inclusive of the first hunting function to classify events including in the event data, wherein the coverage value representing a number of events or a percentage of total events associated with the training dataset that the first hunting function has applied a classification instead of abstaining due to an inability of the first hunting function to render a decision based on data associated with the training dataset. {Conwell [Para. 0054] “Once the model subset 810 is built the model output from each of the models 811i (the “weak classifiers”) is aggregated into model subset output 813. Each model 811i can output two values, which assist in forming the model subset output 813. Typically this score is a value pair (Cn, Sn), where the pair represents a pair of values (Boolean classification or a classification score value, an indicator of existence of a classification score) or a pair of values (likelihood/probability of classification, confidence in the likelihood of classification). In the first case, Cn is a “0” or “1” value or a score (e.g., a value between 0-1) and Sn indicates whether the model was able to make the classification. Thus, a value of (0,1) or (0.1,1) may indicate that something is not malicious or not likely malicious, but a value of (0,0) indicates that no classification was reached.”} Weak classifiers may output a value of (0,0), indicating that a classification model fails to make a definitive prediction for a given input.
However, Conwell and Wu do not teach wherein the hunting function analytic logic further comprises: coverage logic configured to determine a coverage value being a measure of a capability of the first hunting function or a hunt pack inclusive of the first hunting function to classify events including in the event data, wherein the coverage value representing a number of events or a percentage of total events associated with the training dataset that the first hunting function has applied a classification instead of abstaining due to an inability of the first hunting function to render a decision based on data associated with the training dataset.
However, Gottin teaches wherein the hunting function analytic logic further comprises: coverage logic configured to determine a coverage value being a measure of a capability of the first hunting function or a hunt pack inclusive of the first hunting function to classify events including in the event data, wherein the coverage value representing a number of events or a percentage of total events associated with the training dataset that the first hunting function has applied a classification instead of abstaining due to an inability of the first hunting function to render a decision based on data associated with the training dataset; {Gottin [Para. 33] “The exemplary contextual model agreement network generation process 100 comprises the following steps, in relation to historical training data.” [Para. 0034] “Exemplary contextual model agreement network generation process 100 defines appropriate feature extraction procedures for the domain. Thereafter, a set of predictive models is generated during step 120, each considering potentially different subsets of the features in the domain.” [Para. 0035] “During step 130, a model agreement network 300 is generated, which computes an agreement metric for each pair of machine learning models in the ensemble.” [Para. 0036] “The model agreement network 300 is extended during step 140 with feature-dependent agreement metrics, representing the agreement between machine learning models for a plurality of subsets of features of the domain to generate the contextual model agreement network 800.”} Also see para. 0079 and 0083 in Gottin. Gottin teaches a pairwise agreement calculation function that computes an agreement metric for ML models in an ensemble.
Gottin is analogous art because each of Conwell, Wu and Gottin pertains to generating ensemble machine learning models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Conwell to include Gottin’s teaching of the elements of claim 8 identified above. Conwell’s ML models may output no classification (see para. 0054 in Conwell), which indicates that a classification model fails to make a definitive prediction for a given input. Gottin computes an agreement metric that computes an agreement metric for ML models. In this regard, one could use this combination to determine a coverage value recited by claim 8 identified above. Doing so would “provide improved methods for real-time anomaly detection” (Gottin, para. 0119).
16. Claims 5 and 23 are rejected under 35 U.S.C. § 103 as being unpatentable over Conwell and Wu as applied to claims 1, 2 and 21, and further in view of Ferreira et al. (US 2023/0110993 A1), hereafter Ferreira, and further in view of Zhang et al. (US 2022/0237504 A1), hereafter Zhang.
Regarding claim 5, Conwell teaches the elements of claim 2 as outlined above.
However, Conwell and Wu does no teach the limitations of claim 5.
However, Ferreira teaches wherein the hunting function analytic logic comprises: agreement evaluation logic configured to determine a first quality metric for at least the first hunting function when (i) the first hunting function assigns a first classification to the training dataset by issuance of a first verdict, and (ii) a first majority of other hunting functions of the plurality of hunting functions assigns a verdict identical to the first verdict to the training dataset; {Ferreira [Para. 26] “A method for calculating model agreement in a multi-sensor multi-model environment without requiring ground truth labeled data.” [Para. 0038] “In FIG. 4, the algorithm 400 may comprise obtaining a global sensor_set set of observations from sensor streams, feeding that set to the models, and counting the number of correct predictions that a particular pair of models achieves. Then, an agreement score may be computed that ranges between −1 and 1. A score of 0 means that the models agree and disagree, that is, with each other, an exactly equal number of times, positive values mean that the models agree more frequently than they disagree, and for negative values the converse is true.” [Para. 49] “FIG. 5 discloses a configuration showing the relative agreements between models in a domain. Edges with |Agreement|<w for a certain support range are pruned and not represented.” [Para. 0051] “Such embodiments may also intuitively can disregard models whose relative agreement is not representative enough.” [Para. 0054] “An example… comprise yielding a classification by a majority voting, that is, whichever classification was assigned most often by the models would be the classification assigned.” [Para. 0057] “In an example with |M|=3 models M0, M1 and M2, all possible configurations of G are represented as a current model agreement 700 in FIG. 7. In… FIG. 7, part (a) shows a circumstance in which all the models agree with each other—this is indicated by the solid lines connecting the models.”} Ferreira discloses identifying a current agreement between classifications generated by ML models. An aggregated agreement score is generated for each model, and that score is used to determine whether a model should be removed from the model agreement network. Thus, a model’s quality depends on how closely its output align with the rest of the ensemble.
Ferreira is analogous art because each of Conwell, Wu and Ferreira pertains to generating ensemble machine learning models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Conwell and Wu to include Ferreira’s teaching of the limitations of claim 5, listed above. Doing so would allow data to be “classified, and used by an ML model even if the data does not fall within any classifications with the domain of the ML model” (Ferreira, para. 22).
However, Ferreira also does not teach disagreement evaluation logic configured to determine a second quality metric for at least the first hunting function when (i) the first hunting function assigns a second classification to the training dataset by issuance of a second verdict, and (ii) a second majority of other hunting functions of the plurality of hunting functions assigns a verdict, different than the second verdict, to the training dataset.
However, Zhang teaches disagreement evaluation logic configured to determine a second quality metric for at least the first hunting function when (i) the first hunting function assigns a second classification to the training dataset by issuance of a second verdict, and (ii) a second majority of other hunting functions of the plurality of hunting functions assigns a verdict, different than the second verdict, to the training dataset. {Zhang [Para. 93] “A “Max Disagreement” is utilized by the maximum disagreement identification logic for selecting the query instance based on a particular outlier ML system from the ML systems 224 a-224 n that is most different in its label and/or its confidence in its label.” [Para. 0094] “Assume that ML system 224a, by evaluating the labeled data 204 and training instances of data 216, comes to the conclusion that this data describes a pump that is in proper working order (using the values described above), but is only 20% confident that this label (“This is a properly working pump”) is correct. Assume that all of the other ML systems 224 b-224 n generate a label “This is a faulty pump”, with a confidence level of 95% in their labels. Therefore, ML system 224 a is in the maximum disagreement from the other ML systems 224 a-224 n, and needs to be retrained.”} Zhang identifies disagreements between classifications generated by ML models. Models within the ensemble are retrained based on their level of disagreement with the rest of the group. Since maximum disagreement determines which models are retrained, model quality is measured by how much a model diverges from its peers.
Zhang is analogous art because each of Conwell, Wu, Ferreira and Zhang pertains to generating and training machine learning models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Conwell, WU and Ferreira to include Zhang’s teaching of the limitations of claim 5 identified above. Doing so “enables machine learning model to be trained faster” (Zhang, para. 47).
Claim 23:
Regarding claim 23, Conwell and Wu teach the elements of claim 21 as outlined above.
However, Conwell and Wu do not teach the elements of claim 23.
However, Ferreira teaches wherein the model generation subsystem includes hunting function analytic logic that comprises agreement evaluation logic configured to determine a first quality metric for at least a first hunting function when (i) the first hunting function assigns a first classification to the training dataset by issuance of a first verdict, and (ii) a first majority of hunting functions of the one or more hunting functions, different than the first hunting function, assigning the first verdict to the training dataset; {Ferreira [Para. 26] “A method for calculating model agreement in a multi-sensor multi-model environment without requiring ground truth labeled data.” [Para. 0038] “In FIG. 4, the algorithm 400 may comprise obtaining a global sensor_set set of observations from sensor streams, feeding that set to the models, and counting the number of correct predictions that a particular pair of models achieves. Then, an agreement score may be computed that ranges between −1 and 1. A score of 0 means that the models agree and disagree, that is, with each other, an exactly equal number of times, positive values mean that the models agree more frequently than they disagree, and for negative values the converse is true.” [Para. 49] “FIG. 5 discloses a configuration showing the relative agreements between models in a domain. Edges with |Agreement|<w for a certain support range are pruned and not represented.” [Para. 0051] “Such embodiments may also intuitively can disregard models whose relative agreement is not representative enough.” [Para. 0054] “An example… comprise yielding a classification by a majority voting, that is, whichever classification was assigned most often by the models would be the classification assigned.” [Para. 0057] “In an example with |M|=3 models M0, M1 and M2, all possible configurations of G are represented as a current model agreement 700 in FIG. 7. In… FIG. 7, part (a) shows a circumstance in which all the models agree with each other—this is indicated by the solid lines connecting the models.”} Ferreira discloses identifying a current agreement between classifications generated by ML models. An aggregated agreement score is generated for each model, and that score is used to determine whether a model should be removed from the model agreement network. Thus, a model’s quality depends on how closely its output align with the rest of the ensemble.
Ferreira is analogous art because each of Conwell, Wu and Ferreira pertains to generating ensemble machine learning models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Conwell and Wu to include Ferreira’s teaching of the limitations of claim 23, listed above. Doing so would allow data to be “classified, and used by an ML model even if the data does not fall within any classifications with the domain of the ML model” (Ferreira, para. 22).
However, Ferreira also does not teach disagreement evaluation logic configured to determine a second quality metric for at least the first hunting function when (i) the first hunting function assigns a second classification to the training dataset by issuance of a second verdict, and (ii) a second majority of hunting functions of the one or more hunting functions, different than the first hunting function, assigning a different second verdict to the training dataset.
However, Zhang teaches disagreement evaluation logic configured to determine a second quality metric for at least the first hunting function when (i) the first hunting function assigns a second classification to the training dataset by issuance of a second verdict, and (ii) a second majority of hunting functions of the one or more hunting functions, different than the first hunting function, assigning a different second verdict to the training dataset. {Zhang [Para. 93] “A “Max Disagreement” is utilized by the maximum disagreement identification logic for selecting the query instance based on a particular outlier ML system from the ML systems 224a -224n that is most different in its label and/or its confidence in its label.” [Para. 0094] “Assume that ML system 224a, by evaluating the labeled data 204 and training instances of data 216, comes to the conclusion that this data describes a pump that is in proper working order (using the values described above), but is only 20% confident that this label (“This is a properly working pump”) is correct. Assume that all of the other ML systems 224 b-224n generate a label “This is a faulty pump”, with a confidence level of 95% in their labels. ML system 224 a is in the maximum disagreement from the other ML systems 224a -224n, and needs to be retrained.”} Zhang identifies disagreements between classifications generated by ML models. Models within the ensemble are retrained based on their level of disagreement with the rest of the group. Since maximum disagreement determines which models are retrained, model quality is measured by how much a model diverges from its peers.
Zhang is analogous art because each of Conwell, Wu, Ferreira and Zhang pertains to generating and training machine learning models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Conwell, Wu and Ferreira to include Zhang’s teaching of the limitations of claim 23, listed above. Doing so “enables machine learning model to be trained faster” (Zhang, para. 47).
17. Claims 6 and 7 are rejected under 35 U.S.C. § 103 as being unpatentable over Conwell, Wu, Ferreira and Zhang as applied to claims 1 and 5, and further in view of Almukaynizi et al. (US 2022/0004630 A1), hereafter Almukaynizi.
Regarding claim 6, Ferreira and Zhang teach the elements of claim 5 as stated.
However, Conwell, Wu, Ferreira and Zhang do not teach the limitations of claim 6.
However, Almukaynizi teaches wherein the agreement evaluation logic determines the first quality metric operating as a true positive metric representing the model generation subsystem assigning a malicious classification to the training dataset and the first majority of other hunting functions of the plurality of hunting functions assigns the malicious classification. {Almukaynizi [Para. 0051] “A multi-model ensemble process 400 is shown for predicting cyber threats to a given technology.” [Para. 0054] “Process 500 for dynamic model retraining to predict cyber threats to a given technology and associated vulnerabilities is shown.” [Para. 0055] “System 100 may be configured to extract and filter new attack data (Step 502). System 100 may identify previous predictions (Step 504) by comparing new attack information with the previous prediction for the same technology and/or vulnerability. System 100 may compare performance metrics (Step 506) such as, precision, recall, true positive rate, and/or false positive rate. System 100 may determine whether a model should be retrained based on, for example, a threshold established during model training (Step 510).”} Almukaynizi trains ensemble models to predict cyber threats and calculates performance metrics such as true positive rate.
Almukaynizi is analogous art because each of Conwell, Wu, Ferreira, Zhang and Almukaynizi pertains to generating and training machine learning models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Conwell, Wu, Ferreira and Zhang to include Almukaynizi’s teaching of the limitations of claim 6, listed above. Doing so would “increase the number of correct predictions, decrease the number incorrect predictions, and/or decrease processing time” (Almukaynizi, para. 0038).
Claim 7:
Regarding claim 7, Almukaynizi teaches the elements of claim 6 as stated above.
However, the combination of Conwell, Wu, Ferreira, Zhang and Almukaynizi teaches wherein the disagreement evaluation logic determines the second quality metric operating as a false positive metric representing the model generation subsystem assigning a malicious classification to the training dataset and the second majority of other hunting functions of the plurality of hunting functions assigns a non-malicious classification. {Almukaynizi [Para. 0054] “Process 500 for dynamic model retraining to predict cyber threats to a given technology and associated vulnerabilities is shown.” [Para. 0055] “System 100 may be configured to extract and filter new attack data (Step 502). System 100 may identify previous predictions (Step 504) by comparing new attack information with the previous prediction for the same technology and/or vulnerability. System 100 may compare performance metrics (Step 506) such as, precision, recall, true positive rate, and/or false positive rate. System 100 may determine whether a model should be retrained based on, for example, a threshold established during model training (Step 510).”} Almukaynizi trains ensemble models to predict cyber threats and calculates performance metrics such as false positive rate.
Almukaynizi is analogous art because each of Conwell, Wu, Ferreira, Zhang and Almukaynizi pertains to generating and training machine learning models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Conwell, Wu, Ferreira and Zhang to include Almukaynizi’s teaching of the limitations of claim 7, listed above. Doing so would “increase the number of correct predictions, decrease the number incorrect predictions” (Almukaynizi, para. 0038).
18. Claim 10 is rejected under 35 U.S.C. § 103 as being unpatentable over Conwell and Wu as applied to claims 1 and 9, and further in view Manthey et al. (US 11,494,285 B1), hereafter Manthey.
Regarding claim 10, Conwell teaches the elements of claim 9 as outlined above.
However, Conwell and Wu do not teach the limitations of claim 10.
However, Manthey teaches wherein the first hunt pack constitutes a public hunt pack corresponding to a publicly available software module obtained from a public data store operating as a software marketplace. {Manthey [Col. 6, line 41-52] “Another service 110 referred to a software catalog service 118 (or marketplace) provides a digital catalog filled with software listings from a variety of software vendors that makes it easy for users to find, test, buy, and deploy software that can be run in the provider network 100. The software catalog service 118 may include static analysis tools 138 that can be obtained/licensed by a user and utilized to analyze a codebase.” [Col. 13, line 35-42] ‘The plurality of identifiers of static analysis tools are presented with one or more license indicators, wherein each of the one or more license indicators indicates that a corresponding static analysis tool: is an open source application; requires an associated license known to be held to the user; or requires an associated license known to be available to the user via a separate software catalog service.”} Also see col. 9, line 21-34. Static analysis tools are listed on a software catalog, and they may be open source, or publicly available. These static analysis tools correspond to the public hunt packs.
Manthey is analogous art because each of Conwell, Wu and Manthey pertains to using machine learning models to perform predictive tasks. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Conwell and Wu to include Manthey’s teaching of a public hunt pack that corresponds to a publicly available software module obtained from a public data store. Doing so would allow the system to customize recommendations of threat hunting tools according to the user’s preferences (Manthey, col. 2 line 27-42).
19. Claim 11 is rejected under 35 U.S.C. § 103 as being unpatentable over Conwell and Wu as applied to claims 1 and 9, and further in view of Yumer (US 10,104,097 B1), hereafter Yumer.
Regarding claim 11, Conwell teaches the elements of claim 9 as outlined above.
However, Conwell and Wu do not teach the limitations of claim 11.
However, Yumer teaches wherein the first hunt pack constitutes a private hunt pack corresponding to a proprietary software module configured to allow an enterprise to concentrate a hunting for cybersecurity threats based on direct knowledge of a computing environment or threat landscape confronting the enterprise. {Yumer [Col. 6, line 47-55] “risk factors of an organization include, without limitation, a sector of the organization, a size of the organization, an age of the organization,…” [Col. 6 line 65–col. 7 line 1] “Risk factor module 104 may identify risk factors 208 by first identifying the targets of previous targeted attacks 210. For example, risk factor module 104 may access or analyze a database that stores information identifying individuals and/or organizations of individuals that received targeted attacks.” [Col. 10, line 4-11] “At step 308 one or more of the systems described herein may adjust a security policy assigned to the candidate target of the targeted malware attack based on the calculated degree of association between the candidate risk factor and the previous targeted malware attacks.”} Also see col. 4, line 50-53. Yumer’s system detects targeted malware targeting an enterprise based on threat landscape an enterprise/organization faces. Risk factor module 104 is part of server 206, and server 206 provides services for customers (see col. 14, line 40-46). Risk factor module 104 corresponds to the private hunt pack.
Yumer is analogous art because each of Conwell, Wu and Yumer pertains to implementing measures to safeguard data. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Conwell and Wu to include Yumer’s teaching of a private hunt pack that allows an enterprise to concentrate a hunting for cybersecurity threats based on direct knowledge of a computing environment or threat landscape confronting the enterprise. Doing so would “prevent malicious attacks that are specifically directed to targets based on profiles of the targets” (Yumer, col. 12 line 65-col. 13 line 2).
20. Claims 12, 13 and 16-18 are rejected under 35 U.S.C. § 103 as being unpatentable over Conwell et al. (US 2022/0147815 A1), hereafter Conwell, in view of Ferreira et al. (US 2023/0110993 A1), hereafter Ferreira,
Noted that indicates what the cited art does not teach.
Regarding claim 12, Conwell teaches a computerized method, comprising: {Conwell [Para. 0019] “Enhanced computer-and network-based methods… for producing and using enhanced machine learning models and computer-implemented tools to investigate cybersecurity related data and threat intelligence data.”}
during a training of a Machine Learning (ML) model to operate in conjunction with a function evaluator on a training dataset, generating quality metrics for each hunting function of a plurality of hunting functions included in a hunt pack deployed as part of the function evaluator, {Conwell [Para. 0030] “Pipeline 200 illustrates how models are built and tuned for deployment as a cybersecurity threat analysis application. Portions of the pipeline 200 are looped and assessed (or reassessed) until the executed model 220 is capable of predicting a result 221 that is considered “acceptable” (e.g., correct according to some determined value, percentage of time, threshold, precision and/or recall statistical requirements, etc.). According to the pipeline 200, labeled (known) malicious data 201 along with labeled (known) neutral data 202 in the form of training data 203 along with model tuning parameters 205 and a certain (e.g., determined, selected, designated, etc.) machine learning algorithm 204 are input into a build process 210 to build a trained model instance 211 (a binary). This trained model instance 211 (i.e., trained model) is then run (shown as model execution 220) on labeled malicious and neutral test data 212 to generate a prediction/result 221.”}
altering the hunt pack in response to a first hunting function of the plurality of hunting functions having a quality metric that fails to satisfy at least a first threshold; {Conwell [Para. 0030] “The resultant prediction 221 is input along with labeled malicious and neutral validation data 216 into a tuning system 215, which is used to determine the (potentially modified) model tuning parameters 205 to run in the next iteration of the pipeline (rebuilding the model instance 210 and executing the trained and tuned model 220) until the trained model 220 predicts an outcome (result) that is correct sufficient times and with sufficient accuracy to be considered acceptable (the validation data is used to validate the prediction of the test data as malicious or not). This loop continues until a prediction/result 221 that is considered within acceptable characteristics as described above.”} Altering a ML model includes tuning it.
and providing the function evaluator and the ML model to a threat detection subsystem to conduct analytics on data being monitored for cybersecurity threats in response to each hunting function associated with the hunt pack including quality metrics that satisfies the first threshold. {Conwell [Para. 0051] “Each application comprises one or more collections (subsets) of models, which are trained using different training data but otherwise share the same machine learning algorithm, modeling tuning parameters, and feature vector values, which can be ultimately tuned and optimized for the type of data the model is responsible for classifying. A combination of the results of each applicable subset of weak classifiers then is fed as input into an ensemble master classifier, which can be iteratively run with varying weights applied to the weak classifier subset outputs until a determined optimization value (e.g., a threshold, minimum, percentage, probability, etc.) is reached. The resultant ensemble master classifier can then be deployed as a cybersecurity threat analysis application and applied an unknown domain to predict whether the domain is “predictably malicious.” [Para. 0036] “The system collects domain related data from Internet Infrastructure Data (IID), both gathered and derived, including for example, domain names, “whois” protocol information, IP addresses, DNS record data, passive DNS activity data, scraped HTML content, TLS certificate information, and/or other domain related data.”} DS record data and passive DNS activity exemplify data that are monitored for cybersecurity threats.
However, Conwell does not teach wherein the quality metrics are based on a level of agreement between the verdict of the hunting function and verdicts of other hunting functions of the plurality of hunting functions.
However, Ferreria teaches wherein the quality metrics are based on a level of agreement between the verdict of the hunting function and verdicts of other hunting functions of the plurality of hunting functions; {Ferreira [Para. 26] “A method for calculating model agreement in a multi-sensor multi-model environment without requiring ground truth labeled data.” [Para. 0038] “In FIG. 4, the algorithm 400 may comprise obtaining a global sensor_set set of observations from sensor streams, feeding that set to the models, and counting the number of correct predictions that a particular pair of models achieves. Then, an agreement score may be computed that ranges between−1 and 1. A score of 0 means that the models agree and disagree, that is, with each other, an exactly equal number of times, positive values mean that the models agree more frequently than they disagree, and for negative values the converse is true.” [Para. 49] “FIG. 5 discloses a configuration showing the relative agreements between models in a domain. Edges with |Agreement|<w for a certain support range are pruned and not represented.” [Para. 0051] “Such embodiments may also intuitively can disregard models whose relative agreement is not representative enough.” [Para. 0057] “In an example with |M|=3 models M0, M1 and M2, all possible configurations of G are represented as a current model agreement 700 in FIG. 7. In… FIG. 7, part (a) shows a circumstance in which all the models agree with each other - this is indicated by the solid lines connecting the models.”} Ferreira discloses identifying a current agreement between classifications generated by ML models. An aggregated agreement score is generated for each model, and that score is used to determine whether a model should be removed from the model agreement network. Thus, a model’s quality depends on how closely its output align with the rest of the ensemble.
Ferreira is analogous art because each of Conwell and Ferreira pertains to generating ensemble machine learning models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Conwell to include Ferreira’s teaching of the elements of claim 12 identified above. Doing so would allow data to be “classified, and used by an ML model even if the data does not fall within any classifications with the domain of the ML model” (Ferreira, para. 22).
Claim 13:
Regarding claim 13, Conwell and Ferreira teach the elements of claim 12 as stated.
Conwell further teaches wherein prior to generating the quality metrics for each hunting function, the method further comprising: selecting the hunt pack for deployment as part of function evaluator; {Conwell [Para. 0030] “Labeled (known) malicious data 201 along with labeled (known) neutral data 202 in the form of training data 203 along with model tuning parameters 205 and a certain machine learning algorithm 204 are input into a build process 210 to build a trained model instance 211… When an acceptable trained model state is achieved, trained model instance 211 can be deployed in an application (model execution 220) with new (unlabeled data) domain data 214 to generate a prediction/result 230.”}
and selecting a training dataset for ML model. {Conwell [Para. 0032] “Labeled (known) malicious and neutral data 310 is input into data sampling process, which is tuned using sampling parameters 302, to generate different types of sampled data, including labeled test data 212, labeled training data 203, and labeled validation data 216. This labeled data can then be incorporated into a machine learning pipeline such as machine learning pipeline 200.”}
Claim 16:
Regarding claim 16, Conwell and Ferreira teach the elements of claim 12 as stated.
Conwell further teaches wherein each of the hunting functions extracts one or more features from the training dataset and forms one or more verdicts corresponding to the one or more features, {Conwell [Para. 0026] “The feature class engines 111 are used to select and transform domain related data stored in repository 115 to actionable feature class vectors used as input into the weak classifiers.” [Para. 0051] “Each application comprises one or more collections (subsets) of models, which are trained using different training data but otherwise share the same machine learning algorithm, modeling tuning parameters, and feature vector values. A combination of the results of each applicable subset of weak classifiers then is fed as input into an ensemble master classifier.”} Also see para. 0053 and 0054. .
the one or more features and the associated verdicts being used in training the ML model. {Conwell [Para. 0051] “A combination of the results of each applicable subset of weak classifiers then is fed as input into an ensemble master classifier, which can be iteratively run with varying weights applied to the weak classifier subset outputs until a determined optimization value is reached.” [Para. 0057] “Of note, master classifier 812 contains a third layer of machine learning, that is it includes a feedback loop 860-870, which iterates adjusting the weights 860 applied to the model subset outputs' input 850 until the classification result has been optimized, for example, using gradient descent boosting.”}
Claim 17:
Regarding claim 17, Conwell and Ferreira teach the elements of claim 12 as stated.
Conwell further teaches wherein the ML model analyses the one or more features provided by the function evaluator using machine learning techniques to produce a threat score. {Conwell [Para. 56] “The model subset output from each of the model subsets is then configured to be fed into an ensemble master classifier for that particular application so that the predictions can be reduced to a single (final) score. For Application(j), model subset output 813, 822 and 832 are configured as input 850 to the ensemble master classifier 812. The results of each model subset output 813, 822, and 832 are input into input vector 850.” [Para. 58] “The output of an optimized ensemble classifier is a final score such as final score 880.” [Para. 59] “A simpler ensemble classifier may be appropriate for the application, such as classifier 831 which may employed simple voting or weighted voting to achieve a final score 835.” }
Claim 18:
Regarding claim 18, Conwell and Ferreira teach the elements of claim 12 as stated.
However, Conwell does not teach the limitations of claim 18.
However, Ferreira teaches wherein the quality metrics include a first quality metric that identifies a level of agreement between verdicts on one or more of the features by a hunting function of the plurality of hunting functions and verdicts on the one or more features by one or more hunting functions of the plurality of hunting functions other than the hunting function. {Ferreira [Para. 26] “A method for calculating model agreement in a multi-sensor multi-model environment without requiring ground truth labeled data.” [Para. 0038] “In FIG. 4, the algorithm 400 may comprise obtaining a global sensor_set set of observations from sensor streams, feeding that set to the models, and counting the number of correct predictions that a particular pair of models achieves. Then, an agreement score may be computed that ranges between −1 and 1. A score of 0 means that the models agree and disagree, that is, with each other, an exactly equal number of times, positive values mean that the models agree more frequently than they disagree, and for negative values the converse is true.” [Para. 49] “FIG. 5 discloses a configuration showing the relative agreements between models in a domain. Edges with |Agreement|<w for a certain support range are pruned and not represented.” [Para. 0051] “Such embodiments may also intuitively can disregard models whose relative agreement is not representative enough.” [Para. 0057] “In an example with |M|=3 models M0, M1 and M2, all possible configurations of G are represented as a current model agreement 700 in FIG. 7. In… FIG. 7, part (a) shows a circumstance in which all the models agree with each other—this is indicated by the solid lines connecting the models.”} Ferreira discloses identifying a current agreement between classifications generated by ML models. An aggregated agreement score is generated for each model, and that score is used to determine whether a model should be removed from the model agreement network. Thus, a model’s quality depends on how closely its output align with the rest of the ensemble.
Ferreira is analogous art because each of Conwell and Ferreira pertains to generating ensemble machine learning models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Conwell and Wu to include Ferreira’s teaching of the limitations of claim 18, listed above. Doing so would allow data to be “classified, and used by an ML model even if the data does not fall within any classifications with the domain of the ML model” (Ferreira, para. 22).
21. Claim 14 is rejected under 35 U.S.C. § 103 as being unpatentable over Conwell and Ferreira as applied to claims 12 and 13, and further in view Manthey et al. (US 11,494,285 B1), hereafter Manthey.
Regarding claim 14, Conwell teaches the elements of claim 13 as outlined above.
However, Conwell and Ferreira do not teach the limitations of claim 14.
However, Manthey teaches wherein the selecting of the hunt pack comprises selecting a public hunt pack corresponding to a publicly available software module obtained from a public data store operating as a software marketplace. {Manthey [Col. 6, line 41-52] “Another service 110 referred to a software catalog service 118 (or marketplace) provides a digital catalog filled with software listings from a variety of software vendors… The software catalog service 118 may include static analysis tools 138 that can be obtained/licensed by a user and utilized to analyze a codebase.” [Col. 13, line 35-42] “The plurality of identifiers of static analysis tools are presented with one or more license indicators, wherein each of the one or more license indicators indicates that a corresponding static analysis tool: is an open source application; requires an associated license known to be held to the user; or requires an associated license known to be available to the user via a separate software catalog service.”} Also see col. 9, line 21-34. Static analysis tools are listed on a software catalog, and they may be open source, or publicly available.
Manthey is analogous art because each of Conwell, Ferreira and Manthey pertains to using machine learning models to perform predictive tasks. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Conwell and Ferreira to include Manthey’s teaching of selecting a public hunt pack that corresponds to a publicly available software module obtained from a public data store. Doing so would allow the system to customize recommendations of threat hunting tools according to the user's preferences (Manthey, col. 2 line 27-42).
22. Claim 15 is rejected under 35 U.S.C. § 103 as being unpatentable over Conwell and Ferreira as applied to claims 12 and 13, and further in view of Yumer (US 10,104,097), hereafter Yumer.
Regarding claim 15, Conwell teaches the elements of claim 13 as outlined above.
However, Conwell and Ferreira do not teach the limitations of claim 15.
However, Yumer teaches wherein the first hunt pack constitutes a private hunt pack corresponding to a proprietary software module configured to allow an enterprise to concentrate a hunting for cybersecurity threats based on direct knowledge of a computing environment or threat landscape confronting the enterprise. {Yumer [Col. 6, line 47-55] “risk factors of an organization include, without limitation, a sector of the organization, a size of the organization, an age of the organization,…” [Col. 6 line 65–col. 7 line 1] “Risk factor module 104 may identify risk factors 208 by first identifying the targets of previous targeted attacks 210. For example, risk factor module 104 may access or analyze a database that stores information identifying individuals and/or organizations of individuals that received targeted attacks.” [Col. 10, line 4-11] “Returning to FIG. 3, at step 308 one or more of the systems described herein may adjust a security policy assigned to the candidate target of the targeted malware attack based on the calculated degree of association between the candidate risk factor and the previous targeted malware attacks.”} Also see col. 4, line 50-53. Yumer detects targeted malware targeting an enterprise based on threat landscape an enterprise/organization faces. Risk factor module 104 is part of server 206, and server 206 provides services for customers (see col. 14, line 40-46). Risk factor module 104 corresponds to the private hunt pack.
Yumer is analogous art because each of Conwell, Ferreira and Yumer pertains to implementing measures to safeguard data. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Conwell and Ferreira to include Yumer’s teaching of limitations of claim 15, listed above. Doing so would “prevent malicious attacks that are specifically directed to targets based on profiles of the targets” (Yumer, col. 12 line 65-col. 13 line 2).
23. Claim 19 is rejected under 35 U.S.C. § 103 as being unpatentable over Conwell and Ferreira as applied to claim 12, and further in view of Zhang et al. (US 2022/0237504 A1), hereafter Zhang.
Regarding claim 19, Conwell and Ferreria teaches the elements of claim 12 as stated.
However, Conwell and Ferreria do not teach the limitations of claim 19.
However, Zhang teaches wherein the quality metrics include a second quality metric that identifies a level of disagreement between verdicts on one or more features by the hunting function of the plurality of hunting functions and verdicts on the one or more features by other hunting functions of the plurality of hunting functions. {Zhang [Para. 93] “A “Max Disagreement” is utilized by the maximum disagreement identification logic for selecting the query instance based on a particular outlier ML system from the ML systems 224a -224n that is most different in its label and/or its confidence in its label.” [Para. 0094] “Assume that ML system 224a, by evaluating the labeled data 204 and training instances of data 216, comes to the conclusion that this data describes a pump that is in proper working order (using the values described above), but is only 20% confident that this label (“This is a properly working pump”) is correct. Assume that all of the other ML systems 224 b-224n generate a label “This is a faulty pump”, with a confidence level of 95% in their labels. ML system 224 a is in the maximum disagreement from the other ML systems 224a -224n, and needs to be retrained.”} Zhang identifies disagreements between classifications generated by ML models. Models within the ensemble are retrained based on their level of disagreement with the rest of the group. Since maximum disagreement determines which models are retrained, model quality is measured by how much a model diverges from its peers.
Zhang is analogous art because each of Conwell, Ferreira and Zhang pertains to generating and training machine learning models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Conwell and Ferreira to include Zhang’s teaching of the limitations of claim 19, listed above. Doing so “enables machine learning model to be trained faster” (Zhang, para. 47).
24. Claim 20 is rejected under 35 U.S.C. § 103 as being unpatentable over Conwell and Ferreira as applied to claim 12, and further in view of Gottin (US 2019/0303799 A1), hereafter Gottin.
Regarding claim 20, Conwell teaches the elements of claim 12 as stated above.
Conwell further teaches wherein during the training of the threat hunting model, generating a coverage value, the coverage value representing a number of events or a percentage of total events included in the training dataset to which the hunting function has applied a classification instead of abstaining from applying the classification due to an inability of the hunting function to render a decision based on event data associated with the training dataset. {Conwell [Para. 0054] “Once the model subset 810 is built the model output from each of the models 811i (the “weak classifiers”) is aggregated into model subset output 813. In the example shown, each model 811i can output two values, which assist in forming the model subset output 813. Typically this score is a value pair (Cn, Sn), where the pair represents a pair of values (Boolean classification or a classification score value, an indicator of existence of a classification score) or a pair of values (likelihood/probability of classification, confidence in the likelihood of classification). In the first case, Cn is a “0” or “1” value or a score (e.g., a value between 0-1) and Sn indicates whether the model was able to make the classification. Thus, a value of (0,1) or (0.1,1) may indicate that something is not malicious or not likely malicious, but a value of (0,0) indicates that no classification was reached.”} Weak classifiers may output a value of (0,0), indicating that a classification model fails to make a definitive prediction for a given input.
However, Conwell and Ferreira do not teach wherein during the training of the threat hunting model, generating a coverage value, the coverage value representing a number of events or a percentage of total events included in the training dataset to which the hunting function has applied a classification instead of abstaining from applying the classification due to an inability of the hunting function to render a decision based on event data associated with the training dataset.
However, Gottin teaches wherein during the training of the threat hunting model, generating a coverage value, the coverage value representing a number of events or a percentage of total events included in the training dataset to which the hunting function has applied a classification instead of abstaining from applying the classification due to an inability of the hunting function to render a decision based on event data associated with the training dataset. {Gottin [Para. 0033] “The exemplary contextual model agreement network generation process 100 comprises the following steps, in relation to historical training data.” [Para. 34] “The exemplary contextual model agreement network generation process 100 defines appropriate feature extraction procedures for the domain. Thereafter, a set of predictive models is generated during step 120, each considering potentially different subsets of the features in the domain.” [Para. 0035] “During step 130, a model agreement network 300 is generated, which computes an agreement metric for each pair of machine learning models in the ensemble.” [Para. 0036] “The model agreement network 300 is extended during step 140 with feature-dependent agreement metrics, representing the agreement between machine learning models for a plurality of subsets of features of the domain to generate the contextual model agreement network 800.”} Also see para. 0079 and 0083 in Gottin. Gottin teaches a pairwise agreement calculation function that computes an agreement metric for ML models in an ensemble.
Gottin is analogous art because each of Conwell, Ferreira and Gottin pertains to generating ensemble machine learning models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Conwell to include Gottin’s teaching of the elements of claim 20 identified above. Conwell’s ML models may output no classification (see para. 54 in Conwell), which indicates that a classification model fails to make a definitive prediction for a given input. Gottin computes an agreement metric that computes an agreement metric for ML models. In this regard, one could use this combination to determine a coverage value recited by claim 20 identified above. Doing so would “provide improved methods for real-time anomaly detection” (Gottin, para. 0119).
Conclusion
25. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Verma et al. (US 2023/0281518 A1) discloses a system and a method for selecting training data for machine learning models. In one embodiment, the system discards a model with accuracy prediction that does not meet a threshold quality according to an agreement with other models. See para. 0048.
26. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
27. Any inquiry concerning this communication or earlier communications from the examiner should be directed to BIN QING ZHENG whose telephone number is (703)756-1535. The examiner can normally be reached on M-F 9:30 am -5:30 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Philip J. Chea can be reached on 571-272-3951. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BIN QING ZHENG/
Examiner, Art Unit 2499
/PHILIP J CHEA/Supervisory Patent Examiner, Art Unit 2499
Prosecution Insights