DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on April 23, 2026 has been entered.
Response to Amendment/Arguments
1. Applicant’s arguments filed on April 23, 206 regarding the rejection under 35 U.S.C. 101 have been fully considered but are not persuasive.
Applicant contends that the Office incorrectly characterized the claims as being directed to a mental process. Applicant asserts that the claims are instead directed to improving performance of an asset attribution model based on low attribution accuracy data slices and corresponding data slicing rules. Applicant further discusses the general Alice/Mayo two-part test and states that a proper “directed to” analysis should result in a determination that the claims are directed to eligible subject matter.
Applicant’s discussion of the general Alice/Mayo framework is noted. However, Applicant’s recitation of the general legal standard does not show error in the Office’s analysis. The Office applied the Alice/Mayo framework to the actual claim language and identified the claimed determinations, evaluations, comparisons, and selections that recite mental processes.
In particular, the claim recites determining accuracies of attributions, determine whether an attribution fails or satisfies an accuracy criterion, and selecting metadata fields of a data slicing rule. These limitations are based on observation, comparison, evaluation, selection, and judgement. MPEP 2106.04(a)(2)(III) states that the “mental processes” abstract grouping is defined as “concepts performed in the human mind,” and that “examples of mental processes include observations, evaluations, judgments, and opinions.” Thus, the Office properly characterized the recited determinations and selections as falling within the mental process grouping of abstract ideas.
Applicant’s characterization of the claims as improving performance of an asset attribution model does not change the nature of the recited determinations and selections. The claim language still relies on evaluating attribution results, comparing those results to know attributions, selecting metadata/feature information, and determining whether the model satisfies the desired accuracy requirement.
Applicant further argues that the claims are directed to improving attribution accuracy of an attribution model using data slices for which the attribution accuracy fails an accuracy criterion. Applicant points to claim language reciting feature engineering, retraining an asset attribution model, and updating model architecture based on the feature engineering. Applicant also argues that a person cannot mentally improve the accuracy of an asset attribution model, even with pen and paper.
This argument is not persuasive. The Office is not asserting that a person mentally executes or retrains the asset attribution model. Rather, the rejection identifies the claimed determinations, evaluations, comparisons, and selections as mental processes. A person could review attribution results for assets in a data slice, compare the model’s predicted attributions to known attributions, determine whether the accuracy satisfies a criterion, and select metadata fields or feature information based on the observed performance. These acts are based on observation, comparison, evaluation, selection, and judgement.
Applicant’s reliance on “feature engineering” and “updating architecture” also does not establish a technological improvement. The claims do not recite a specific technological architecture or a particular technological change to the internal operation of the asset attribution model. The claims broadly recite selecting metadata fields, adding the selected metadata fields as additional features, configuring an input component to process the additional features, and retraining the model. The claims do not specify what particular features are generated, how the metadata fields are transformed into features, what specific processing algorithm is used, or how the model architecture is technically changed beyond accepting additional feature information.
Rather, the claims recite using model accuracy results to determine what feature information to add or adjust, and then instructing generic model components to process that information and retrain until the desired accuracy criterion is satisfied. This amounts to generic feature selection and model configuration based on desired model performance, not a specific improvement to computer functionality of model architecture.
Applicant’s reliance on the Specification’s discussion of degenerate model performance, underperforming data slices, feature engineering, retraining, and reducing false positive attributions is also not persuasive. The claim language, not merely the stated purpose in the Specification, must reflect the asserted technological improvement. Here, the asserted improvement is based on the abstract idea itself: evaluating attribution accuracy, selecting metadata/feature information, and using that information to improve the desired attribution result. Reciting asset metadata, data slices, a feature preprocessor, updating architecture, and retraining does not transform the abstract evaluation and selection process into a technological improvement.
Accordingly, Applicant’s arguments have been fully considered but are not persuasive. For the reasons set forth above, claims 1, 3-4, 6-9, 12-13, 15, 18-19, and 21-28 remain directed to an abstract idea and do not recite additional elements that integrate the abstract idea into a practical application or amount to significantly more than the abstract idea. Therefore, the rejections of claims 1, 3-4, 6-9, 12-13, 15, 18-19, and 21-28 under 35 U.S.C. 101 is maintained.
2. Applicant’s arguments filed on April 23, 2026 regarding the rejection under 35 U.S.C. 103 has been fully considered but are not persuasive.
Applicant’s arguments on pages 18-19 regarding claim 1 have been fully considered but are not persuasive. Applicant contends that Raz and Kraning, alone or in combination, do not teach or suggest amended claim 1. In particular, Applicant argues that Raz and Kraning do not teach or suggest selecting one or more metadata fields of a first of the data slicing rules and updating a feature preprocessor of the asset attribution model. Applicant further argues that Raz merely teaches evaluating performance of a predictor on predefined data slices and applying generalized mitigation actions when performance fails a threshold, while Kraning merely teaches collecting and processing network asset metadata.
This argument is not persuasive. Raz teaches evaluating machine learning model performance on data slices and mitigating underperformance when the predictor performs poorly on a data slice. Raz further teaches various ways of mitigating poor performance of a machine learning model, including feature engineering, changing features, adding features, removing features, retraining the model, and changing model architecture. Thus, Raz does not merely teach an unrelated or generic mitigation step. Raz expressly teaches feature engineering as a mitigation for poor model performance on a data slice.
Applicant’s characterization of Raz as using predefined data slices also does not distinguish claim 1. Claim 1 likewise recites data slices generated according to data slicing rules. Thus, the claimed data slices are rule-defined before evaluation in the same general manner that the Applicant characterizes Raz as using predefined data slices. Claim 1 further recites determining that attribution accuracy on a data slice fails an accuracy criterion, which corresponds to threshold-based performance evaluation.
Applicant also argues that Raz does not disclose a relationship between slice definitions and feature generation. This argument is not persuasive. Raz teaches that each data instance may be associated with “metadata values in a metadata space” (Raz, paragraph [0060]). Raz further teaches that data slices may be defined by constraints on feature values or meta values. Thus, Raz’s “meta values” reasonably refer to metadata values associated with the data instances. Raz also teaches that poor performance on a data slice may be mitigated through feature engineering, including adding, changing, or removing features. Therefore, when a data slice defined by feature values or meta values underperforms, those feature values or meta values identify the data characteristics associated with underperformance. Using those corresponding data characteristics as additional feature information is a predictable application of Raz’s feature engineering mitigation.
Moreover, claim 1 does not require a particular algorithm that “reuses” metadata values in a specific way or transforms those metadata values using a particular technique. Claim 1 broadly recites selecting metadata fields of the data slicing rule and adding the selected metadata fields as additional features. Raz teaches that data slices may be defined by constraints on feature values or meta values, and also teaches adding, changing, or removing features as mitigation when a predictor underperforms on a data slice.
Kraning supplies the asset attribution context and the asset metadata. Metadata is a type of data, and the claimed metadata fields are data fields associated with assets. Therefore, applying Raz’s data-slice evaluation and feature-engineering mitigation framework to Kraning’s asset metadata would have resulted in selecting metadata information corresponding to the underperforming data slice and adding that metadata information as additional feature input for the asset attribution model.
Applicant’s argument that Raz teaches feature engineering only at a high level is also not persuasive because claim 1 is likewise broad. Claim 1 does not recite a particular feature construction technique, a particular encoding, a particular preprocessing algorithm, or a specific transformation of metadata fields into features. Rather, claim 1 broadly recites selecting metadata fields, adding selected metadata fields as additional features, updating a feature preprocessor, configuring an input component to process additional features, and retraining. Applicant cannot require the prior art to disclose more specificity than the claim itself requires.
Accordingly, Raz teaches data-slice performance evaluation, determining underperformance based on a performance threshold, mitigating underperformance through feature engineering by changing, adding, or removing features, retraining a model, and changing model architecture. Kraning supplies the asset metadata and asset attribution context. The combination therefore teaches or at least suggests the claimed selecting of metadata fields, updating the feature preprocessor to add those fields as additional features, updating the model architecture to process the additional features, and retraining the model until the accuracy criterion is satisfied.
Applicant’s arguments regarding claim 21 have been fully considered but are not persuasive. Applicant contends that Raz does not teach or suggest determining a quantity of features to add based on the accuracy of the asset attribution model on the first data slice, wherein the quantity is higher for lower accuracy and lower for higher accuracy. Applicant further argues that Raz merely discloses triggering a mitigation action when performance falls below a threshold and does not disclose determining a quantity of features based on a difference between attribution accuracy and an accuracy criterion.
This argument is not persuasive. The rejection does not rely merely on Raz’s disclosure of triggering mitigation. Raz teaches determining a performance measurement for a predictor over a data slice, wherein the performance measurement may be based on accuracy, F1-score, success ratio, or the like. Raz further teaches comparing the performance measurement to a threshold and performing mitigation when the performance measurement is below a threshold. Thus, Raz teaches determining how poorly the model performs on a data slice relative to a threshold.
Raz also teaches that the mitigation may include feature engineering to change a feature, add a feature, remove a feature, or the like. Therefore, Raz teaches adjusting features as a corrective action for poor data slice performance. Under the broadest reasonable interpretation, the claimed difference between the attribution accuracy and the accuracy criterion corresponds to the degree of underperformance of the model relative to the required threshold. A lower accuracy relative to the criterion indicates a greater performance deficiency, while higher accuracy relative to the criterion indicates a smaller performance deficiency.
It would have been obvious to adjust the amount of corrective feature engineering based on the degree of underperformance. When a model performs further below an accuracy criterion, more corrective feature information would have been used to address the greater deficiency. When the model performs closer to the criterion, fewer additional features would have been needed. Claim 21 does not recite a specific formula, algorithm, or technical mechanism for determining the number of additional features. Rather, claim 21 broadly recites determining the quantity based on accuracy or the difference between the attribution accuracy and the accuracy criterion.
Kraning supplies the asset attribution context and the asset metadata. Accordingly, Raz’s accuracy-based data slice mitigation and feature engineering, when applied to Kraning’s asset attribution environment, teaches or at least suggests determining the number of metadata based features to add based on the degree of underperformance of the asset attribution model on the data slice.
Applicant’s assertion that claim 1 is allowable has been fully considered but is not persuasive for the reasons discussed above. As set forth above, Raz in view of Kraning teaches or at least suggests the limitations of claim 1, including evaluating model performance on data slices, determining underperformance based on an accuracy/performance threshold, mitigating the underperformance through feature engineering by adding, changing, or removing features, updating the model architecture, retraining the model, and applying those teachings in Kraning’s asset attribution context using asset metadata.
Applicant does not present separate arguments for dependent claims 3, 4, and 6-8 apart from the arguments presented for claim 1. Therefore, because the arguments regarding claim 1 are not persuasive, the arguments regarding dependent claims 3, 4, and 6-8 are likewise not persuasive. Claim 21 has been separately addressed above. Accordingly, the rejection of claim 1, 3, 4, 6-8, and 21 under 35 U.S.C. 103 is maintained.
Applicant’s arguments regarding claims 9 and 15 have been fully considered but are not persuasive. Applicant asserts that claims 9 and 15 are amended with subject matter substantially similar to the subject matter incorporated in claim 1 and that claims 9 and 15, along with their respective dependent claims, are allowable for the same reasons presented for claim 1.
This argument is not persuasive for the reasons discussed above with respect to claim 1. Claims 9 and 15 recite subject matter substantially similar to claim 1, and Applicant has not presented separate arguments for patentability of claims 9 and 15 apart from the arguments presented for claim 1. As discussed above, Raz in view of Kraning teaches or at least suggests the disputed limitations, including data slice performance evaluation, determining underperformance based on an accuracy/performance threshold, mitigating underperformance through feature engineering by adding, changing, or removing features, updating model architecture, retraining the model, and applying those teachings in Kraning’s asset attribution context using asset metadata.
Applicant has not presented separate arguments for the respective dependent claims of claims 9 and 15. Therefore, because Applicant’s arguments regarding claim 1 are not persuasive, the arguments regarding claims 9 and 15 and their respective dependent claims are likewise not persuasive. Accordingly, the rejection of claims 9, 12-13, 15, 18-19, and 22-28 under 35 U.S.C. 103 is maintained.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1,3-4,6-9,12-13,15,18-19 and 21-28 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
101 Subject Matter Eligibility Analysis
Step 1: Claims 1,3-4,6-9,12-13,15,18-19 and 21-28 are within the four statutory categories (a process, machine, manufacture or composition of matter).
Claims 1, 3-5, 7-8, and 21, 24, and 25 are directed to a method consisting of a series of steps, meaning that it is directed to the statutory category of process. Claims 9, 12-13, 15, 18-19, 22, 23, and 26-28 are directed to computer-readable mediums and processors which are machines.
Step 2A Prong One, Step 2A Prong Two, and Step 2B Analysis:
Step 2A Prong One asks if the claim recites a judicial exception (abstract idea, law of nature, or natural phenomenon). If the claim recites a judicial exception, analysis proceeds to Step 2A Prong Two, which asks if the claim recites additional elements that integrate the abstract idea into a practical application. If the claim does not integrate the judicial exception, analysis proceeds to Step 2B, which asks if the claim amounts to significantly more than the judicial exception. If the claim does not amount to significantly more than the judicial exception, the claim is not eligible subject matter under 35 U.S.C. 101.
None of the claims represent an improvement to technology.
Regarding claim 1, the following claim elements are abstract ideas:
determining accuracies of attributions of assets to organizations by an asset attribution model across a plurality of data slices generated according to data slicing rules, wherein each data slice comprises asset metadata of a plurality of assets (This is an abstract idea of a mental process. A person could review a subset of asset records, compare the model’s attribution to the known organization for each asset, and use judgement to determine whether the asset metadata, such as an IP address, domain, or other asset information, is connected to the organization identified by the model. The person could repeat this for different subsets and determine the accuracy for each subset. This type of observation, comparison, and judgement can be practically performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
based on a determination that attribution accuracy of the asset attribution model on a first of the plurality of data slices fails an accuracy criterion, engineering features for asset attribution…until the accuracy criterion is satisfied (This is an abstract idea of a mental process. After determining that the attribution accuracy does not satisfy a criterion, a person could review the attribution results for the particular subset of asset records and use observation and judgement to decide what feature information should be added, removed, or adjusted based on the asset metadata. For example, the person could decide whether IP address information, domain information, certificate information, location, information, or ownership information should be used or changed until the attribution results satisfied the desired accuracy requirement. This type of observation, comparison, and judgement can practically be performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
wherein engineering features and retraining until the accuracy criterion is satisfied comprises, selecting one or more metadata fields of a first of the data slicing rules (This is an abstract idea of a mental process. A person could review a data slicing rule and use observation and judgement to decide which metadata fields are relevant to that rule, such as domain, name, URL, country code, language, registrar, certificate information, or organization-related information. For example, if the rule is directed to assets associated with email providers or a particular region, a person could select the domain name field, region field, or language field as the metadata fields relevant to applying that rule. The type of observation, selection, and judgement can practically be performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
determining whether the retrained asset attribution model satisfies the accuracy criterion (This is an abstract idea of a mental process. A person could review attribution results of the retrained model, compare the model’s attributions to known organizations, and use judgement to determine whether the accuracy meets the required criterion. This type of observation, comparison, and judgement can practically be performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
to predict attribution of one or more assets to an organization (This is an abstract idea of a mental process. A person could review information about an asset such as an IP address, domain name, certificate information, location information, or ownership information, and use observation and judgement to determine whether the asset is connected to a particular organization. This type of observation, comparison, and judgement can be practically performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
The following claim elements are additional elements which, taken alone or in combination with the other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
asset attribution model (This a high-level recitation of generic computer components for performing the abstract idea. See MPEP 2106.05(f).)
retraining the asset attribution model according to the feature engineering (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).)
updating a feature preprocessor of the asset attribution model to add the selected one or more metadata fields as additional features for preprocessing by the feature preprocessor (The step of “updating a feature preprocessor” is merely an instruction to apply the abstract idea using a generic model component. Adding selected metadata fields as additional features is no more than generic preprocessing/configuring of data for use by a model and amounts to insignificant extra-solution activity.)
updating architecture of the asset attribution model, wherein updating architecture of the asset attribution model comprises, configuring an input component of the asset attribution model to process the additional features based on updating of the feature preprocessor (The step of “updating architecture” is merely an instruction to apply the abstract idea using a generic model component. Configuring an input component to process additional features is no more than generic data preprocessing/configuring for use by a model and amounts to insignificant extra solution activity.)
after updating the feature preprocessor and the architecture, retraining the asset attribution model (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).)
deploying the asset attribution model with the updated architecture (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).)
Regarding claim 3, the rejection of claim 1 is incorporated herein. Further, claim 3 recites the following additional elements, which taken alone or in combination with other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein the data slicing rules comprise rules for at least one of assertion testing-based, regression testing-based, location-based, and organization-based metadata indicated by the metadata fields (This limitation amounts to insignificant extra-solution activity because it merely identifies types of data slicing rules or metadata categories use with the abstract idea.)
Regarding claim 4, the rejection of claim 1 is incorporated herein. Further, claim 4 recites the following abstract ideas:
wherein selecting the one or more metadata fields comprises querying a repository for assets of the first data slice, wherein the query is generated based, at least in part, on logic for metadata of the plurality of assets expressed by the first data slicing rule (This is an abstract idea of a mental process. A person could select certain metadata fields based on what the person is looking for in the asset records. For example, if a person is looking for assets associated with a domain, location, or organization, the person could select metadata fields such as domain name, country code, location, certificate information, or organization-related information. This type of observation, selection, and judgement can practically be performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas. The step of “querying a repository” is merely a well-understood, routine, and conventional data retrieval operation and does not add a meaningful limitation beyond the abstract ideas.)
Regarding claim 6, the rejection of claim 1 is incorporated herein. Further, claim 6 recites the following abstract ideas:
wherein the accuracy criterion comprises a determination of whether the asset attribution model correctly predicts a threshold number of assets of the first of the plurality of data slices according to known attributions of the assets in the first data slice to one or more organizations (This is an abstract idea of a mental process. A person could review assets in the first data slice, compare the model’s predicted organization for each asset to the known organization for that asset, and use judgement to decide whether enough of the predictions are correct to satisfy the required threshold. This type of observation, comparison, and judgement can practically be performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
Regarding claim 7, the rejection of claim 1 is incorporated herein. Further, claim 7 recites the following additional elements, which taken alone or in combination with other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein updating architecture of the asset attribution model further comprises, updating at least one of a type of the asset attribution model, parameters of the asset attribution model, hyperparameters of the asset attribution model, and a training method for the asset attribution model (This limitation amounts to insignificant extra-solution activity.)
Regarding claim 8, the rejection of claim 1 is incorporated herein. Further, claim 8 recites the following abstract ideas:
selecting one or more additional data slices from the plurality of data slices based, at least in part, on the data slicing rules (This is an abstract idea of a mental process. A person could review the different subsets of asset records and select the data slices that meet the rule requirement being used, such as a rule directed to domain information, location information, organization-related information, or other asset metadata. This type of observation, selection, and judgement can practically be performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
based on a determination that the asset attribution model satisfies the accuracy criterion for the one or more additional data slices (This is an abstract idea of a mental process. A person could review the model’s attribution results for the additional data slices, compare the predicted organizations to known organizations, and use judgement to determine whether the model satisfies the required accuracy criterion for those data slices. This type of observation, comparison, and judgement can practically be performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
The following claim elements are additional elements which, taken alone or in combination with the other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
deploying the asset attribution model for asset attribution (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).)
Regarding claim 9, the following claim elements are abstract ideas:
determine accuracies of attributions of assets to organizations by an asset attribution model across a plurality of data slices generated according to data slicing rules, wherein each data slice comprises asset metadata of a plurality of assets (This is an abstract idea of a mental process. A person could review a subset of asset records, compare the model’s attribution to the known organization for each asset, and use judgement to determine whether the asset metadata, such as an IP address, domain, or other asset information, is connected to the organization identified by the model. The person could repeat this for different subsets and determine the accuracy for each subset. This type of observation, comparison, and judgement can be practically performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
based on a determination that attribution accuracy of the asset attribution model on a first of the plurality of data slices fails an accuracy criterion, engineer features for asset attribution…until the accuracy criterion is satisfied (This is an abstract idea of a mental process. After determining that the attribution accuracy does not satisfy a criterion, a person could review the attribution results for the particular subset of asset records and use observation and judgement to decide what feature information should be added, removed, or adjusted based on the asset metadata. For example, the person could decide whether IP address information, domain information, certificate information, location, information, or ownership information should be used or changed until the attribution results satisfied the desired accuracy requirement. This type of observation, comparison, and judgement can practically be performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
wherein the instructions to engineer features and retrain until the accuracy criterion is satisfied comprises instructions to, select one or more metadata fields of a first of the data slicing rules corresponding to the first data slice (This is an abstract idea of a mental process. A person could review a data slicing rule and use observation and judgement to decide which metadata fields are relevant to that rule, such as domain, name, URL, country code, language, registrar, certificate information, or organization-related information. For example, if the rule is directed to assets associated with email providers or a particular region, a person could select the domain name field, region field, or language field as the metadata fields relevant to applying that rule. The type of observation, selection, and judgement can practically be performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
determine whether the retrained asset attribution model satisfies the accuracy criterion (This is an abstract idea of a mental process. A person could review attribution results of the retrained model, compare the model’s attributions to known organizations, and use judgement to determine whether the accuracy meets the required criterion. This type of observation, comparison, and judgement can practically be performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
to predict attribution of one or more assets to an organization (This is an abstract idea of a mental process. A person could review information about an asset such as an IP address, domain name, certificate information, location information, or ownership information, and use observation and judgement to determine whether the asset is connected to a particular organization. This type of observation, comparison, and judgement can be practically performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
The following claim elements are additional elements which, taken alone or in combination with the other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
A non-transitory computer-readable medium having program code stored thereon (This a high-level recitation of generic computer components for performing the abstract idea. See MPEP 2106.05(f).)
asset attribution model (This a high-level recitation of generic computer components for performing the abstract idea. See MPEP 2106.05(f).)
retrain the asset attribution model according to the feature engineering (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).)
update a feature preprocessor of the asset attribution model to add the selected one or more metadata fields as additional features for preprocessing by the feature preprocessor (The step of “updating a feature preprocessor” is merely an instruction to apply the abstract idea using a generic model component. Adding selected metadata fields as additional features is no more than generic preprocessing/configuring of data for use by a model and amounts to insignificant extra-solution activity.)
update architecture of the asset attribution model, wherein the instructions to update architecture of the asset attribution model comprise instructions to configure an input component of the asset attribution model to process the additional features based on the update of the feature preprocessor (The step of “updating architecture” is merely an instruction to apply the abstract idea using a generic model component. Configuring an input component to process additional features is no more than generic data preprocessing/configuring for use by a model and amounts to insignificant extra solution activity.)
after the update of the feature preprocessor and the architecture, retrain the asset attribution model (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).)
deploy the retrained asset attribution model with the updated architecture (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).)
Regarding claim 12, the rejection of claim 9 is incorporated herein. Further, claim 12 recites the following abstract ideas:
wherein the accuracy criterion comprises a determination that a number of correct predictions by the asset attribution model on metadata for the first of the plurality of data slices is above a threshold number of correct predictions, wherein correct predictions are according to known attributed organizations for assets in the first data slice (This is an abstract idea of a mental process. A person could review the assets in the first data slice, compare the model’s predicted organization for each asset to the known organization for that asset, and use judgement to determine whether enough predictions are correct to be above the required threshold. This type of observation, comparison, and judgement can practically be performed in the human mind and therefore falls within the mental process grouping of abstract ideas.)
Regarding claim 13, the rejection of claim 9 is incorporated herein. Further, claim 13 recites the following additional elements, which taken alone or in combination with other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein the instructions to update the architecture of the asset attribution model comprises instructions to update at least one of a type of the asset attribution model, parameters of the asset attribution model, hyperparameters of the asset attribution model, and a training method for the asset attribution model based, at least in part, on the asset attribution model failing the accuracy criterion (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).)
Regarding claim 15, the following claim elements are abstract ideas:
determine attributions accuracies of attributions of assets to organizations by an asset attribution model across a plurality of data slices generated according to data slicing rules, wherein each data slice comprises asset metadata of a plurality of assets (This is an abstract idea of a mental process. A person could review a subset of asset records, compare the model’s attribution to the known organization for each asset, and use judgement to determine whether the asset metadata, such as an IP address, domain, or other asset information, is connected to the organization identified by the model. The person could repeat this for different subsets and determine the accuracy for each subset. This type of observation, comparison, and judgement can be practically performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
based on a determination that attribution accuracy of the asset attribution model on a first of the plurality of data slices fails an accuracy criterion, engineer features for asset attribution…until the accuracy criterion is satisfied (This is an abstract idea of a mental process. After determining that the attribution accuracy does not satisfy a criterion, a person could review the attribution results for the particular subset of asset records and use observation and judgement to decide what feature information should be added, removed, or adjusted based on the asset metadata. For example, the person could decide whether IP address information, domain information, certificate information, location, information, or ownership information should be used or changed until the attribution results satisfied the desired accuracy requirement. This type of observation, comparison, and judgement can practically be performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
wherein the instructions to engineer features and retrain until the accuracy criterion is satisfied comprises instructions executable by the processor to cause the apparatus to select one or more metadata fields of a first of the data slicing rules corresponding to the first data slice (This is an abstract idea of a mental process. A person could review a data slicing rule and use observation and judgement to decide which metadata fields are relevant to that rule, such as domain, name, URL, country code, language, registrar, certificate information, or organization-related information. For example, if the rule is directed to assets associated with email providers or a particular region, a person could select the domain name field, region field, or language field as the metadata fields relevant to applying that rule. The type of observation, selection, and judgement can practically be performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
determine whether the retrained asset attribution model satisfies the accuracy criterion (This is an abstract idea of a mental process. A person could review attribution results of the retrained model, compare the model’s attributions to known organizations, and use judgement to determine whether the accuracy meets the required criterion. This type of observation, comparison, and judgement can practically be performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
to predict attribution to an organization (This is an abstract idea of a mental process. A person could review information about an asset such as an IP address, domain name, certificate information, location information, or ownership information, and use observation and judgement to determine whether the asset is connected to a particular organization. This type of observation, comparison, and judgement can be practically performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
The following claim elements are additional elements which, taken alone or in combination with the other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
a processor; and a computer-readable medium having instructions stored thereon (This a high-level recitation of generic computer components for performing the abstract idea. See MPEP 2106.05(f).)
asset attribution model (This a high-level recitation of generic computer components for performing the abstract idea. See MPEP 2106.05(f).)
retrain the asset attribution model according to the feature engineering (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).)
update a feature preprocessor of the asset attribution model to add the selected one or more metadata fields as additional features for preprocessing by the feature preprocessor (The step of “updating a feature preprocessor” is merely an instruction to apply the abstract idea using a generic model component. Adding selected metadata fields as additional features is no more than generic preprocessing/configuring of data for use by a model and amounts to insignificant extra-solution activity.)
update architecture of the asset attribution model, wherein the instructions to update architecture of the asset attribution model comprise instructions executable by the processor to cause the apparatus to configure an input component of the asset attribution model to process the additional features based on the update of the feature preprocessor (The step of “updating architecture” is merely an instruction to apply the abstract idea using a generic model component. Configuring an input component to process additional features is no more than generic data preprocessing/configuring for use by a model and amounts to insignificant extra solution activity.)
after the update of the feature preprocessor and the architecture, retrain the asset attribution model (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).)
deploy the retrained asset attribution model with the updated architecture (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).)
Regarding claim 18, the rejection of claim 15 is incorporated herein. Further, claim 18 recites the following abstract ideas:
determination that accuracy of the asset attribution model for at least the first data slice fails an accuracy criterion (This is an abstract idea of a mental process. A person could review the model’s attribution results for the data slice, compare the predicted organizations to known organizations, and use judgement to determine that the model does not satisfy the required accuracy criterion. This type of observation, comparison, and judgement can practically be performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
The following claim elements are additional elements which, taken alone or in combination with the other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
to update at least one type of the asset attribution model, parameters of the asset attribution model, hyperparameters of the asset attribution model, and a training method for the asset attribution model (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).)
Regarding claim 19, the rejection of claim 15 is incorporated herein. Further, claim 19 recites the following additional elements, which taken alone or in combination with other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein the data slicing rules comprises rules for obtaining at least one of assertion testing-based data slices, regression testing-based data slices, location-based data slices, and organization-based data slices (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).)
Regarding claim 21, the rejection of claim 1 is incorporated herein. Further, claim 21 recites the following abstract ideas:
determining a number of the one or more features based, at least in part, on the accuracy of the asset attribution model on the first subset of the plurality of assets, wherein the number of the one or more features is determined to be higher for a lower accuracy of the asset attribution model and lower for a higher accuracy of the asset attribution model (This is an abstract idea of a mental process. A person could review the model’s accuracy for the first data slice and use judgement to decide how many additional feature information should be added. For example, if the model performs poorly on the data slice, the person could decide to add more metadata fields, and if the model performs better, the person could decide to add few metadata fields. The person could then select the metadata fields based on that determined amount. The type of observation, comparison, selection, and judgement can practically be performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
The following claim elements are additional elements which, taken alone or in combination with the other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein engineering the one or more features further (This is merely an instruction to apply the abstract idea and does provide a meaningful limitation.)
Regarding claim 22, the rejection of claim 9 is incorporated herein. Further, claim 22 recites the following abstract ideas:
determine a number of the one or more additional features to be generated corresponding to the first data slice based, at least in part, on a difference between the attribution accuracy of the first data slice and the accuracy criterion, and wherein the number of the one or more additional features is determined to be higher for a lower attribution accuracy relative to the accuracy criterion (This is an abstract idea of a “mental process.” A person could review how the model performed on the first data slice, compare the attribution accuracy to the required accuracy criterion, and use judgement to decide how much additional feature information should be added. For example, if the model performs far below the required accuracy, the person could decide to add more features, and if the model performs closer to the required accuracy, the person could decide to add fewer features. This type of observation, comparison, and judgement can practically be performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.).
The following claim elements are additional elements which, taken alone or in combination with the other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein the instructions to select one or more metadata fields of the first data slicing rule comprise instructions to (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).)
Regarding claim 23, the rejection of claim 15 is incorporated herein. Further, claim 22 recites the following abstract ideas:
determine a number of the one or more additional features to be generated corresponding to the first data slice based, at least in part, on a difference between attribution accuracy of the first data slice and the accuracy criterion and wherein the number of the one or more additional features is determined to be higher for a lower attribution accuracy relative to the accuracy criterion and lower for a higher attribution accuracy relative to the accuracy (This is an abstract idea of a “mental process.” A person could review how far the model’s attribution accuracy is from the required accuracy criterion and use judgement to decide how much additional feature information should be added. For example, if the model performs far below the required accuracy, the person could decide to add more features, and if the model performs closer to the required accuracy, the person could decide to add fewer features. This type of observation, comparison, and judgement can practically be performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.)
The following claim elements are additional elements which, taken alone or in combination with the other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein the instructions to select one or more metadata fields of the first data slicing rule comprise instructions executable by the processor to cause the apparatus to (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).)
Regarding claim 24, the rejection of claim 1 is incorporated herein. Further, claim 24 recites the following abstract ideas:
wherein the first data slicing rule defines the first of the plurality of data slices based on one or more metadata fields such that the first of the plurality of data slices comprises assets for which the asset attribution model produces incorrect attributions of the assets to organizations (This is an abstract idea of a mental process. A person could review assets that the model attributed to the wrong organizations, observe the metadata fields for those assets, and use judgement to define a rule that selects assets that have similar metadata. For example, the person could look at fields such as domain name, URL, location, certificate information, or organization-related information and create a rule to group assets where the model is producing incorrect attributions. This type of observation, comparison, rule creation, and judgment can be practically performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.).
Regarding claim 25, the rejection of claim 24 is incorporated herein. Further, claim 25 recites the following additional elements, which taken alone or in combination with other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein the incorrect attributions comprise false positive attributions of assets to organizations, and wherein engineering features for asset attribution and retraining the asset attribution model reduces the false positive attributions for the first of the plurality of data slices (This limitation amounts to insignificant extra-solution activity because it merely identifies the type of incorrect attribution as a false positive and states the intended result of reducing those false positives. The step of “retraining” the asset attribution model is merely an instruction to apply the abstract idea using a generic model and does not add a meaningful limitation.).
Regarding claim 26, the rejection of claim 9 is incorporated herein. Further, claim 26 recites the following abstract ideas:
to identify the first data slicing rule such that the first of the plurality of data slices comprises assets for which the asset attribution model produces incorrect attributions of the assets to organizations based on one or more metadata fields referenced by the first data slicing rule (This is an abstract idea of a mental process. A person could review assets that the model attributed to the wrong organizations, compare the asset metadata for those assets, and use judgement to identify a data slicing rule that captures the assets with the incorrect attributions. For example, a person could identify a rule based on metadata fields such as domain name, URL, location, certification information, or organization-related information where the model is making the wrong attributions. This type of observation, comparison, rule identification, and judgement can practically be performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.).
Regarding claim 27, the rejection of claim 26 is incorporated herein. Further, claim 27 recites the following additional elements, which taken alone or in combination with other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein the instructions to engineer features and retrain the asset attribution model further comprise instructions to reduce incorrect attributions of the assets to organizations by the asset attribution model for the first of the plurality of data slices (This limitation amounts to insignificant extra-solution activity because it merely states the intended result of reducing incorrect attributions for the first data slice. The limitation does not add a meaningful limitation beyond instructing the model to apply the feature engineering to improve attribution results.).
Regarding claim 28, the rejection of claim 15 is incorporated herein. Further, claim 28 recites the following additional elements, which taken alone or in combination with other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein the instructions to engineer features further comprise instructions executable by the processor to cause the apparatus to identify the first data slicing rule such that the first of the plurality of data slices comprises assets for which the asset attribution model produces incorrect attributions of the assets to organizations based on one or more metadata fields referenced by the first data slicing rule (This is an abstract idea of a mental process. A person could review assets that the model attributed to the wrong organizations, compare the asset metadata for those assets, and use judgement to identify a data slicing rule that captures the assets with the incorrect attributions. For example, a person could identify a rule based on metadata fields such as domain name, URL, location, certification information, or organization-related information where the model is making the wrong attributions. This type of observation, comparison, rule identification, and judgement can practically be performed in the human mind or with pen and paper and therefore falls within the mental process grouping of abstract ideas.).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1,3-4,6-9,12-13,15,18-19 and 21-28 are rejected under the 35 U.S.C. 103 as being unpatentable over Raz et al., (Pub. No.: US 20210319354 A1 (File: 2020)) in view of Kraning et al., (Pub. No.: US 20210105304 A1 (Published: 04/08/2021)).
Regarding claim 1, Raz in view of Kraning teaches the following limitations:
determining accuracies of attributions of assets to organizations by an asset attribution model across a plurality of data slices generated according to data slicing rules, wherein each data slice comprises asset metadata of a plurality of assets (Raz, paragraph [0030] “the data slice may be determined by obtaining a definition of data slice and applying the definition to identify the instances that are included in the data slice.” [0035] “a performance measurement of the predictor over each data slice may be computed… the performance measurement may measure how well the predictor predicts the actual label. In some exemplary embodiments, for each data instance of the dataset that is in the data slice, the predictor may be utilized to predict a label. The predicted label may be compared with the actual label, to determine whether the prediction is correct or not. The performance measurement may be computed based on the number of instances, based on the number of instances for which a correct prediction was provided, or the like. In some exemplary embodiments, the performance measurement may be based on, for example, F1 score, Accuracy, R-squared, RSME, or the like. [0060] “ Each data instance may comprise features values in a feature space….Each data instance may be associated with at least one metadata value.” [0062] “Data slices may be defined by constraints on the features values, on the meta values, or the like.” Kraning, [Abstract] “Response data is received from one or more network systems connected to the computer network and processed to identify one or more network assets associated with an entity such as an enterprise organization.” [0075] “Asset data may include the asset itself (e.g., a registration or a digital certificate), an asset identifier (e.g., an IP address, a domain name, etc.), an entity identifier (e.g., a name) associated with a related entity (e.g., an individual) that is responsible for the network asset, a resource identifier (e.g., a cloud instance UID) associated with the network asset, credentials needed to access the network asset, and any other such data or metadata associated with a network asset.” [0126] “machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Raz teaches determining accuracies across data slices by computing a performance measurement for each data slice, where the predictor’s label predictions are compared with actual labels and the performance measurement may be based on accuracy. Raz also teaches that the data slices are generated using data slice definitions/constraints, including constraints on meta values. Kraning supplies the asset-attribution context by teaching machine-learning-based identification of network assets associated with entities/organizations, where the network assets have associated asset data/metadata. In the combination, Raz’s predictor is applied to Kraning’s network-asset attribution context, such that the predictor corresponds to the claimed asset attribution model, the data instances correspond to network assets, the predicted label corresponds to the entity/organization associated with the network asset, and Kraning’s network-asset metadata corresponds to the claimed asset metadata. Accordingly, Raz in view of Kraning teaches the limitation.)
based on a determination that attribution accuracy of the asset attribution model on a first of the plurality of data slices fails an accuracy criterion, engineering features for asset attribution and retraining the asset attribution model according to the feature engineering until the accuracy criterion is satisfied (Raz, paragraph [0017] “ In some exemplary embodiments, the mitigating action may comprise obtaining an additional dataset and retraining the predictor therewith…Additionally or alternatively, the mitigating action may comprise feature engineering in order to change a feature, add a feature, remove a feature, or the like.” [0035] “In some exemplary embodiments, different data slices may have substantially different performance measurements…in case that the performance measurement of a data slice is below a second threshold the performance measurement of the data slice may be a value that is configured to cause a reduction in performance measurement.” [0036] “Additionally or alternatively, the success ratio of the predictor over slice C may be 70% which may be below the success threshold level. As a result, the performance measurement of the predictor over slice C may be −1. “ [0045] “In some exemplary embodiments, the mitigating action may comprise re-training the predictor (174). Re-training the predictor may comprise obtaining another dataset to be used for training the predictor.” [0047] “It is noted that after the mitigation action is performed, the predictor may be re-assessed (Steps 140-160). In case, the performance measurement of the predictor after the mitigating action is implemented is above the threshold, the predictor may be utilized” Kraning paragraph [0126] “ In some embodiments, machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Raz teaches determining that a first data slice fails an accuracy criterion by disclosing different data slices may have different performance measurements and that a data slice may fall below a threshold, such as a success ratio threshold. Raz further teaches performing mitigation when the predictor underperforms, where the mitigation may include feature engineering to change, add, or remove features and retraining the predictor. Raz also teaches reassessing the predictor after mitigation and utilizing the predictor when the performance measurement is above a threshold. Kraning supplies the asset-attribution content by teaching machine-learning-based identification of network assets associated with an entity. Accordingly, the combination, Raz’s predictor corresponds to the asset attribution model, and Raz’s feature engineering/retraining in response to a failed slice accuracy threshold corresponds to engineering features for asset attribution and retraining the asset attribution model until the accuracy criterion is satisfied.),
wherein engineering features and retraining until the accuracy criterion is satisfied comprises, selecting one or more metadata fields of a first of the data slicing rules and updating a feature preprocessor of the asset attribution model to add the selected one or more metadata fields as additional features for preprocessing by the feature preprocessor (Raz, paragraph [0017] “The feature values may be utilized as an input for the machine learning model, as an input for the predictor, or the like…Additionally or alternatively, the mitigating action may comprise changing the architecture of the model used to train the predictor, such as modifying an architecture of a network-based model, modifying the number of layers, the number of nodes in a layer, or the like…Additionally or alternatively, the mitigating action may comprise feature engineering in order to change a feature, add a feature, remove a feature, or the like.” [0031] “ On Step 134, a constraint may be obtained. In some exemplary embodiments, the constraint may represent a definition of the data slice… Applying the constraint may comprise identifying at least one data instance for which the constraint is held, e.g., the one or more feature values of the at least one data instance are in line with the constraint. The at least one identified data instance may be a member of the data slice.” [0061] “Predictor 212 may be trained based on feature values in a feature space. Additionally or alternatively, metadata values may be excluded from an input provided to the predictor.” [0062] “Data slices may be defined by constraints on the features values, on the meta values, or the like.” Kraning, paragraph [0075] “” Asset data may include the asset itself (e.g., a registration or a digital certificate), an asset identifier (e.g., an IP address, a domain name, etc.), an entity identifier (e.g., a name) associated with a related entity (e.g., an individual) that is responsible for the network asset, a resource identifier (e.g., a cloud instance UID) associated with the network asset, credentials needed to access the network asset, and any other such data or metadata associated with a network asset.” [0126] “ In some embodiments, machine learning may be applied to identify network assets associated with an entity.” – Raz teaches that a data slice may be defined by a constraint, including constraints on meta values, and applying the constraint identifies the data instances in the slice. Raz further teaches that metadata values may be excluded from the predictor input, while feature engineering may add feature and feature values may be used as input to the predictor. Thus, when a data slice defined by metadata/meta-value constraints underperforms, it would have been obvious to select the corresponding metadata information from the data slice rule and add it as additional input features through feature preprocessing. Kraning supplies the asset-attribution context by teaching asset metadata associated with network assets and machine learning based identification of network assets associated with an entity. Accordingly, Raz in view of Kraning teaches the limitation.);
updating architecture of the asset attribution model, wherein updating architecture of the asset attribution model comprises, configuring an input component of the asset attribution model to process the additional features based on updating of the feature preprocessor (Raz, [0017] “The feature values may be utilized as an input for the machine learning model, as an input for the predictor, or the like…Additionally or alternatively, the mitigating action may comprise feature engineering in order to change a feature, add a feature, remove a feature, or the like.” [0046] “In some exemplary embodiments, the predictor may be trained based on a machine learning model. In that embodiment, the mitigating action may comprise changing the network architecture, the algorithm utilized to train the network, or the like (176). In some exemplary embodiments, layers may be added to the ANN, a layer may be removed from the ANN, a node may be added to the ANN, connectivity between nodes or layers may be modified, or the like.” [0061] “Predictor 212 may be configured to provide a predicted label for an input such as a data instance…Predictor 212 may be trained based on feature values in a feature space.” Kraning, paragraph [0126] “In some embodiments, machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Raz teaches that feature values may be used as input to the predictor and that feature engineering may add features. Raz further teaches changing the network architecture, including adding or removing layers or nodes and modifying connectivity. Thus, when additional features are adding through feature engineering, it would have been obvious to update the architecture of the model so that the input side of the predictor can receive and process the added feature values. Kraning supplies the asset attribution context by teaching machine learning based identification of network assets associated with an entity. Accordingly, Raz in view of Kraning teaches updating architecture of the asset attribution model, including configuring an input component of the asset attribution model to process the additional features based on updating of the feature preprocessor.);
after updating the feature preprocessor and the architecture, retraining the asset attribution model and determining whether the retrained asset attribution model satisfies the accuracy criterion (Raz, paragraph [0017] “Additionally or alternatively, the mitigating action may comprise feature engineering in order to change a feature, add a feature, remove a feature, or the like.” [0041] “ In some exemplary embodiments, the threshold may be a confidence level threshold, a success rate threshold, or the like.” [0045] “In some exemplary embodiments, the mitigating action may comprise re-training the predictor (174). Re-training the predictor may comprise obtaining another dataset to be used for training the predictor. In some exemplary embodiments, the other dataset may include instances in data slices in which the predictor performs below par.” [0046] “In some exemplary embodiments, the predictor may be trained based on a machine learning model. In that embodiment, the mitigating action may comprise changing the network architecture, the algorithm utilized to train the network, or the like” [0147] “It is noted that after the mitigation action is performed, the predictor may be re-assessed (Steps 140-160). In case, the performance measurement of the predictor after the mitigating action is implemented is above the threshold, the predictor may be utilized” [0126] “In some embodiments, machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Raz teaches retraining the predictor after mitigation, including retraining using another dataset that may include instances from data slices where the predictor performs below par. Raz also teaches changing the network architecture as a mitigation action and reassessing the predictor after the mitigation action to determine whether the performance measurement is above the threshold. Kraning supplies the asset-attribution context by teaching machine-learning-based identification of network assets associated with an entity. Accordingly, in the combination, Raz’s retraining and reassessment after mitigation correspond to retraining an asset attribution model after updating the feature preprocessor and architecture and determining whether the retrained asset attribution model satisfies an accuracy criterion.); and
deploying the asset attribution model with the updated architecture to predict attribution of one or more assets to an organization (Raz, paragraph [0042] “On Step 165, as the performance measurement of the predictor is above a threshold, the predictor may be utilized. In some exemplary embodiments, the predictor may be utilized in order to provide a predicted label for a data instance that is not comprised by the dataset. In some exemplary embodiments, the predictor may be deployed in the field, may be provided as part of an update of a software utilizing the predictor, or the like.” [0046] “the mitigating action may comprise changing the network architecture, the algorithm utilized to train the network, or the like” [0047] “It is noted that after the mitigation action is performed, the predictor may be re-assessed (Steps 140-160). In case, the performance measurement of the predictor after the mitigating action is implemented is above the threshold, the predictor may be utilized” Kraning, paragraph [0126] “ machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Raz teaches changing the network architecture as a mitigation action, reassessing the predictor after the mitigation action, and utilizing/deploying the predictor when the performance measurement is above the threshold. Raz further teaches that the deployed predictor provides a predicted label for a data instance. Kraning supplies the asset-attribution context by teaching machine learning based identification of network assets associated with an entity. Accordingly, in the combination, deploying Raz’s predictor with the changed architecture to provide predicted labels in Kraning’s asset-attribution context corresponds to deploying the asset attribution model with the updated architecture to predict attribution of one or more assets to an organization.)
Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having a combination of Raz and Kraning before them, to incorporate the use of subsets of network assets with known organizational ownership, as taught by Kraning, into the slice-based model evaluation system of Raz. One would have been motivated to make such a combination in order to evaluate how accurately a machine learning model predicts asset ownership by an organization within specific data segments, and to use these evaluation results to guide improvements to model architecture. This would allow more precise and reliable attribution of assets to organizations by identifying and addressing model weaknesses on targeted slices of asset metadata.
Regarding claim 3, Raz in view of Kraning, as outlined above, all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1, mutatis mutandis. Raz in view of Kraning further teaches:
wherein the data slicing rules comprise rules for at least one of assertion testing-based, regression testing-based, location-based, and organization-based metadata indicated by the metadata fields (Raz, paragraph [0030] “the data slice may be determined by obtaining a definition of data slice and applying the definition to identify the instances that are included in the data slice.” [0031] “On Step 134, a constraint may be obtained. In some exemplary embodiments, the constraint may represent a definition of the data slice…Applying the constraint may comprise identifying at least one data instance for which the constraint is held, e.g., the one or more feature values of the at least one data instance are in line with the constraint. The at least one identified data instance may be a member of the data slice.” [0062] “Data slices may be defined by constraints on the features values, on the meta values, or the like.” Kraning, [Abstract] “Response data is received from one or more network systems connected to the computer network and processed to identify one or more network assets associated with an entity such as an enterprise organization.” [0126] “machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” [0133] “asset data may include the asset itself (e.g., a registration or a digital certificate), an asset identifier (e.g., an IP address, a domain name, etc.), an entity identifier (e.g., a name, email address, etc.) associated with a related entity (e.g., an individual) that is responsible for the network asset, a resource identifier (e.g., a cloud instance UID) associated with the network asset, credentials needed to access the network asset, and any other such data or metadata associated with a network asset.” – Raz teaches the claimed data slicing rules by disclosing data slice definitions/constraints that are applied to identify data instances included in a data slice. Raz further teaches that the data slices may be defined by constraints on meta values, which corresponds to using metadata fields to define/select the data slices. Kraning teaches organization-related asset metadata by disclosing network assets associated with an entity, such as an enterprise organization, and by disclosing asset data including an entity identifier and metadata associated with a network asset. Kraning also teaches applying machine learning to identify network assets associated with an entity. Thus, Kraning’s entity identifier and organization-related metadata corresponds to the claimed organization-based metadata indicated by the metadata fields, while Raz supplies the rule/constraint-based data slicing framework.)
Regarding claim 4, Raz in view of Kraning, as outlined above, all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1, mutatis mutandis. Raz in view of Kraning further teaches:
wherein selecting the one or more metadata fields comprises querying a repository for assets of the first data slice, wherein the query is generated based, at least in part, on logic for metadata of the plurality of assets expressed by the first data slicing rule (Raz, paragraph [0030] “ In some exemplary embodiments, the data slice may be determined by obtaining a definition of data slice and applying the definition to identify the instances that are included in the data slice.” [0031] “ On Step 134, a constraint may be obtained. In some exemplary embodiments, the constraint may represent a definition of the data slice. The constraint may be a constraint on one or more feature values in the features space. On Step 138, the constraint may be applied on the dataset. Applying the constraint may comprise identifying at least one data instance for which the constraint is held, e.g., the one or more feature values of the at least one data instance are in line with the constraint. The at least one identified data instance may be a member of the data slice.” [0062] “Data slices may be defined by constraints on the features values, on the meta values, or the like.” Kraning, paragraph [0102] “The asset database 416 may store asset data associated with various network assets identified as part of the network mapping process including various derivative properties associated with the assets.” [0128] “step 620 may include querying automatically an external registration database of a registration authority (e.g., a WHOIS lookup search). Identifying information regarding a related entity (e.g., a name, account identifier, email address, etc.) that registered the domain name may be returned in response to the registration database.” [0133] “ asset data may include the asset itself (e.g., a registration or a digital certificate), an asset identifier (e.g., an IP address, a domain name, etc.), an entity identifier (e.g., a name, email address, etc.) associated with a related entity (e.g., an individual) that is responsible for the network asset, a resource identifier (e.g., a cloud instance UID) associated with the network asset, credentials needed to access the network asset, and any other such data or metadata associated with a network asset.” – Raz teaches obtaining a data slice definition/constraint and applying the constraint to a dataset to identify data instances that satisfy the constraint and belong to the data slice. Raz further teaches that data slices may be defined by constraints on meta values. Thus, Raz’s meta value constraint corresponds to logic for metadata expressed by the first data slicing rule. Kraning supplies the asset-repository context by teaching an asset database storing asset data associated with network assets, including metadata associated with the network assets. Kraning further teaches querying a registration database to obtain identifying information regarding a related entity associated with a network asset. Accordingly, in the combination, Raz’s application of a metadata-based constraint to identify data instances in a data slice is applied to Kraning’s stored network-asset metadata, such that the identified data instances correspond to assets of the first data slice. Therefore, Raz in view of Kraning teaches querying a repository for assets of the first data slice using a query generated based metadata logic expressed by the first data slicing rule.).
Regarding claim 6, Raz in view of Kraning, as outlined above, all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1, mutatis mutandis. Raz in view of Kraning further teaches:
wherein the accuracy criterion comprises a determination of whether the asset attribution model correctly predicts a threshold number of assets of the first of the plurality of data slices according to known attributions of the assets in the first data slice to one or more organizations (Raz, paragraph [0035] “In some exemplary embodiments, the performance measurement may measure how well the predictor predicts the actual label. In some exemplary embodiments, for each data instance of the dataset that is in the data slice, the predictor may be utilized to predict a label. The predicted label may be compared with the actual label, to determine whether the prediction is correct or not. The performance measurement may be computed based on the number of instances, based on the number of instances for which a correct prediction was provided, or the like. In some exemplary embodiments, the performance measurement may be based on, for example, F1 score, Accuracy, R-squared, RSME, or the like.” [0036] “the performance measurement is based on the size of the slice and on the percentage of correct predictions in the data slice (success ratio)…Additionally or alternatively, the success ratio of the predictor over slice C may be 70% which may be below the success threshold level. As a result, the performance measurement of the predictor over slice C may be −1.” [0041] “In some exemplary embodiments, the threshold may be a confidence level threshold, a success rate threshold, or the like.” [0075] “ In some exemplary embodiments, P(s) may refer to a performance measurement, such Accuracy, F1-Score, or the like, of a predictor over the data slice s…In some exemplary embodiments, MinP may refer to a minimal threshold on the performance measurement.” Kraning, [Abstract] “Response data is received from one or more network systems connected to the computer network and processed to identify one or more network assets associated with an entity such as an enterprise organization.” [0126] “In some embodiments, machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Raz teaches an accuracy criterion by disclosing that, for data instances in a data slice, the predictor predicts labels and the predicted labels are compared with actual labels to determine whether the predictions are correct. Raz further teaches that the performance measurement may be based on the number of correct predictions, accuracy, percentage of correct predictions, and a success rate threshold. Kraning supplies the asset-attribution context by teaching machine learning based identification of network assets associated with an entity, such as an enterprise organization. In the combination, Raz’s data instances correspond to Kraning’s network assets, Raz’s actual labels correspond to know organization attributions for assets, and Raz’s determination of correct predictions against a threshold corresponds to determining whether the asset attribution model correctly predicts a threshold number of assets in the first data slice according to known attributions to one or more organizations.).
Regarding claim 7, Raz in view of Kraning, as outlined above, all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1, mutatis mutandis. Raz in view of Kraning further teaches:
wherein updating architecture of the asset attribution model further comprises, updating at least one of a type of the asset attribution model, parameters of the asset attribution model, hyperparameters of the asset attribution model, and a training method for the asset attribution model (Raz, paragraph [0017] “In some exemplary embodiments, the predictor may be trained based on a machine learning model such as an Artificial Neural Network (ANN), a Deep Neural Network (DNN), Ordinary Least Squares Regression, Logistic Regression, Support Vector Machines, or the like… Additionally or alternatively, the mitigating action may comprise changing the architecture of the model used to train the predictor, such as modifying an architecture of a network-based model, modifying the number of layers, the number of nodes in a layer, or the like. Additionally or alternatively, the mitigating action may comprise changing the model utilized by the predictor.” [0019] “ As an example for a mitigating action comprising changing the model used to train the predictor, the predictor may utilize a ANN, and may have been trained using Gradient descent. The mitigating action may comprise changing the architecture of the ANN by adding a layer to the ANN, adding a node to a layer comprised by the ANN, removing a layer from the ANN, removing a node from a layer comprised by the ANN, modifying connectivity between nodes in the ANN, or the like. Additionally or alternatively, the mitigating action may comprise re-training the ANN by utilizing a different algorithm than Gradient descent, such as for example, Newton's method, Conjugate gradient, Levenberg-Marquardt algorithm, or the like.” [0046] “In that embodiment, the mitigating action may comprise changing the network architecture, the algorithm utilized to train the network, or the like (176). In some exemplary embodiments, layers may be added to the ANN, a layer may be removed from the ANN, a node may be added to the ANN, connectivity between nodes or layers may be modified, or the like. Additionally or alternatively, the action may comprise changing the machine learning algorithm to a different machine learning algorithm. As an example, the predictor may be trained based on a decision tree classifier. The mitigating action may comprise changing the machine learning algorithm into a random forest classifier and retraining the predictor” Kraning, paragraph [0126] “In some embodiments, machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Raz teaches updating the model type by disclosing different machine learning model types and changing the model or machine learning algorithm used by the predictor. Raz teaches updating parameters/hyperparameters by disclosing changes to the network architecture, including modifying the number of layers, number of nodes, and connectivity between nodes or layers. Raz also teaches updating the training method by disclosing retraining using a different training algorithm, such as changing from Gradient descent to Newton’s method, Conjugate gradient, or Levenberg-Marquardt. Kraning supplies the asset-attribution context by teaching machine-learning based identification of network assets associated with an entity. Accordingly, in the combination, Raz’s updates to the predictor/model correspond to updating the type, parameters, hyperparameters, or training method of the asset attribution model.).
Regarding claim 8, Raz in view of Kraning, as outlined above, all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1, mutatis mutandis. Raz in view of Kraning further teaches:
selecting one or more additional data slices from the plurality of data slices based, at least in part, on the data slicing rules (Raz, paragraph [0030] “ In some exemplary embodiments, the data slice may be determined by obtaining a definition of data slice and applying the definition to identify the instances that are included in the data slice.” [0031] “On Step 134, a constraint may be obtained. In some exemplary embodiments, the constraint may represent a definition of the data slice…Applying the constraint may comprise identifying at least one data instance for which the constraint is held, e.g., the one or more feature values of the at least one data instance are in line with the constraint. The at least one identified data instance may be a member of the data slice.” [0049] “On Step 135, a slice for analysis may be determined. In some exemplary embodiments, the slice for analysis may be a data slice of the dataset…The domain expert may select the slice for analysis from the plurality of data slices.” [0062] “Data slices may be defined by constraints on the features values, on the meta values, or the like.” [0063] “In some exemplary embodiments, Slices Determinator 240 may be configured to determine a data slice based on the definition of data slice obtained by Slices Definitions Obtainer 230. In some exemplary embodiments, Slices Determinator 240 may be configured to apply a constraint on the dataset in order to identify data instances that are members of a data slice. “ – Raz teaches selecting data slices from a plurality of data slices by disclosing that a slice for analysis may be selected from the plurality of data slices. Raz further teaches that each data slice may be determined using a data slice definition/constraint, including constraints on feature values or meta values, and that applying the constraint identifies the data instances that are members of the data slice. Thus, Raz’s selection of data slices using data slice definition/constraints corresponds to selecting one or more additional data slices from the plurality of data slices based on the data slicing rule.); and
based on a determination that the asset attribution model satisfies the accuracy criterion for the one or more additional data slices, deploying the asset attribution model for asset attribution (Raz, paragraph [0035] “On Step 140, a performance measurement of the predictor over each data slice may be computed…The performance measurement may be computed based on the number of instances, based on the number of instances for which a correct prediction was provided, or the like. In some exemplary embodiments, the performance measurement may be based on, for example, F1 score, Accuracy, R-squared, RSME, or the like.” [0040] “ On Step 160, it may be determined whether the performance measurement of the predictor over the dataset is below a threshold or above the threshold. In case that the performance measurement of the predictor over the dataset is above the threshold, Step 165 may be performed.” [0042] “On Step 165, as the performance measurement of the predictor is above a threshold, the predictor may be utilized. In some exemplary embodiments, the predictor may be utilized in order to provide a predicted label for a data instance that is not comprised by the dataset. In some exemplary embodiments, the predictor may be deployed in the field, may be provided as part of an update of a software utilizing the predictor, or the like.” [0047] “ It is noted that after the mitigation action is performed, the predictor may be re-assessed (Steps 140-160). In case, the performance measurement of the predictor after the mitigating action is implemented is above the threshold, the predictor may be utilized (165).” Kraning, paragraph [0126] “In some embodiments, machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Raz teaches computing a performance measurement, including accuracy, over each data slice and determining whether the performance measurement is above a threshold. Raz further teaches utilizing/deploying the predictor when the performance measurement is above the threshold, including after reassessment following mitigation. Kraning supplies the asset attribution context by teaching a machine learning based identification of network assets associated with an entity. Accordingly, in the combination, Raz’s deployment of the predictor after satisfying the threshold corresponds to deploying the asset attribution model for asset attribution based on determining that the model satisfies the accuracy criterion for one or more additional data slices.)
Regarding claim 9, Raz teaches the following limitations:
A non-transitory, computer-readable medium having program code stored thereon, the program code comprising instructions (Raz, paragraph [0006] “a computer program product comprising a non-transitory computer readable storage medium retaining program instructions, which program instructions when read by a processor, cause the processor to perform”)
determine attribution accuracies of attributions of assets to organizations by an asset attribution model across a plurality of data slices generated according to data slicing rules, wherein each data slice comprises asset metadata of a plurality of assets (Raz, paragraph [0030] “the data slice may be determined by obtaining a definition of data slice and applying the definition to identify the instances that are included in the data slice.” [0035] “a performance measurement of the predictor over each data slice may be computed… the performance measurement may measure how well the predictor predicts the actual label. In some exemplary embodiments, for each data instance of the dataset that is in the data slice, the predictor may be utilized to predict a label. The predicted label may be compared with the actual label, to determine whether the prediction is correct or not. The performance measurement may be computed based on the number of instances, based on the number of instances for which a correct prediction was provided, or the like. In some exemplary embodiments, the performance measurement may be based on, for example, F1 score, Accuracy, R-squared, RSME, or the like. [0060] “ Each data instance may comprise features values in a feature space….Each data instance may be associated with at least one metadata value.” [0062] “Data slices may be defined by constraints on the features values, on the meta values, or the like.” Kraning, [Abstract] “Response data is received from one or more network systems connected to the computer network and processed to identify one or more network assets associated with an entity such as an enterprise organization.” [0075] “Asset data may include the asset itself (e.g., a registration or a digital certificate), an asset identifier (e.g., an IP address, a domain name, etc.), an entity identifier (e.g., a name) associated with a related entity (e.g., an individual) that is responsible for the network asset, a resource identifier (e.g., a cloud instance UID) associated with the network asset, credentials needed to access the network asset, and any other such data or metadata associated with a network asset.” [0126] “machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Raz teaches determining accuracies across data slices by computing a performance measurement for each data slice, where the predictor’s label predictions are compared with actual labels and the performance measurement may be based on accuracy. Raz also teaches that the data slices are generated using data slice definitions/constraints, including constraints on meta values. Kraning supplies the asset-attribution context by teaching machine-learning-based identification of network assets associated with entities/organizations, where the network assets have associated asset data/metadata. In the combination, Raz’s predictor is applied to Kraning’s network-asset attribution context, such that the predictor corresponds to the claimed asset attribution model, the data instances correspond to network assets, the predicted label corresponds to the entity/organization associated with the network asset, and Kraning’s network-asset metadata corresponds to the claimed asset metadata. Accordingly, Raz in view of Kraning teaches the limitation.)
based on a determination that attribution accuracy of the asset attribution model on a first of the plurality of data slices fails an accuracy criterion, engineer features for asset attribution and retrain the asset attribution model according to the feature engineering until the accuracy criterion is satisfied (Raz, paragraph [0017] “ In some exemplary embodiments, the mitigating action may comprise obtaining an additional dataset and retraining the predictor therewith…Additionally or alternatively, the mitigating action may comprise feature engineering in order to change a feature, add a feature, remove a feature, or the like.” [0035] “In some exemplary embodiments, different data slices may have substantially different performance measurements…in case that the performance measurement of a data slice is below a second threshold the performance measurement of the data slice may be a value that is configured to cause a reduction in performance measurement.” [0036] “Additionally or alternatively, the success ratio of the predictor over slice C may be 70% which may be below the success threshold level. As a result, the performance measurement of the predictor over slice C may be −1. “ [0045] “In some exemplary embodiments, the mitigating action may comprise re-training the predictor (174). Re-training the predictor may comprise obtaining another dataset to be used for training the predictor.” [0047] “It is noted that after the mitigation action is performed, the predictor may be re-assessed (Steps 140-160). In case, the performance measurement of the predictor after the mitigating action is implemented is above the threshold, the predictor may be utilized” Kraning paragraph [0126] “ In some embodiments, machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Raz teaches determining that a first data slice fails an accuracy criterion by disclosing different data slices may have different performance measurements and that a data slice may fall below a threshold, such as a success ratio threshold. Raz further teaches performing mitigation when the predictor underperforms, where the mitigation may include feature engineering to change, add, or remove features and retraining the predictor. Raz also teaches reassessing the predictor after mitigation and utilizing the predictor when the performance measurement is above a threshold. Kraning supplies the asset-attribution content by teaching machine-learning-based identification of network assets associated with an entity. Accordingly, the combination, Raz’s predictor corresponds to the asset attribution model, and Raz’s feature engineering/retraining in response to a failed slice accuracy threshold corresponds to engineering features for asset attribution and retraining the asset attribution model until the accuracy criterion is satisfied.),
wherein the instructions to engineer features and retrain until the accuracy criterion is satisfied comprises instructions to, select one or more metadata fields of a first data slicing rule corresponding to the first data slice and update a feature preprocessor of the asset attribution model to add the selected one or more metadata fields as additional features for preprocessing by the feature preprocessor (Raz, paragraph [0017] “The feature values may be utilized as an input for the machine learning model, as an input for the predictor, or the like…Additionally or alternatively, the mitigating action may comprise changing the architecture of the model used to train the predictor, such as modifying an architecture of a network-based model, modifying the number of layers, the number of nodes in a layer, or the like…Additionally or alternatively, the mitigating action may comprise feature engineering in order to change a feature, add a feature, remove a feature, or the like.” [0031] “ On Step 134, a constraint may be obtained. In some exemplary embodiments, the constraint may represent a definition of the data slice… Applying the constraint may comprise identifying at least one data instance for which the constraint is held, e.g., the one or more feature values of the at least one data instance are in line with the constraint. The at least one identified data instance may be a member of the data slice.” [0061] “Predictor 212 may be trained based on feature values in a feature space. Additionally or alternatively, metadata values may be excluded from an input provided to the predictor.” [0062] “Data slices may be defined by constraints on the features values, on the meta values, or the like.” Kraning, paragraph [0075] “” Asset data may include the asset itself (e.g., a registration or a digital certificate), an asset identifier (e.g., an IP address, a domain name, etc.), an entity identifier (e.g., a name) associated with a related entity (e.g., an individual) that is responsible for the network asset, a resource identifier (e.g., a cloud instance UID) associated with the network asset, credentials needed to access the network asset, and any other such data or metadata associated with a network asset.” [0126] “ In some embodiments, machine learning may be applied to identify network assets associated with an entity.” – Raz teaches that a data slice may be defined by a constraint, including constraints on meta values, and applying the constraint identifies the data instances in the slice. Raz further teaches that metadata values may be excluded from the predictor input, while feature engineering may add feature and feature values may be used as input to the predictor. Thus, when a data slice defined by metadata/meta-value constraints underperforms, it would have been obvious to select the corresponding metadata information from the data slice rule and add it as additional input features through feature preprocessing. Kraning supplies the asset-attribution context by teaching asset metadata associated with network assets and machine learning based identification of network assets associated with an entity. Accordingly, Raz in view of Kraning teaches the limitation.);
update architecture of the asset attribution model, wherein the instructions to update architecture of the asset attribution model comprises instructions to configure an input component of the asset attribution model to process the additional features based on update of the feature preprocessor (Raz, [0017] “The feature values may be utilized as an input for the machine learning model, as an input for the predictor, or the like…Additionally or alternatively, the mitigating action may comprise feature engineering in order to change a feature, add a feature, remove a feature, or the like.” [0046] “In some exemplary embodiments, the predictor may be trained based on a machine learning model. In that embodiment, the mitigating action may comprise changing the network architecture, the algorithm utilized to train the network, or the like (176). In some exemplary embodiments, layers may be added to the ANN, a layer may be removed from the ANN, a node may be added to the ANN, connectivity between nodes or layers may be modified, or the like.” [0061] “Predictor 212 may be configured to provide a predicted label for an input such as a data instance…Predictor 212 may be trained based on feature values in a feature space.” Kraning, paragraph [0126] “In some embodiments, machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Raz teaches that feature values may be used as input to the predictor and that feature engineering may add features. Raz further teaches changing the network architecture, including adding or removing layers or nodes and modifying connectivity. Thus, when additional features are adding through feature engineering, it would have been obvious to update the architecture of the model so that the input side of the predictor can receive and process the added feature values. Kraning supplies the asset attribution context by teaching machine learning based identification of network assets associated with an entity. Accordingly, Raz in view of Kraning teaches updating architecture of the asset attribution model, including configuring an input component of the asset attribution model to process the additional features based on updating of the feature preprocessor.);
after the update of the feature preprocessor and the architecture, retrain the asset attribution model and determine whether the retrained asset attribution model satisfies the accuracy criterion (Raz, paragraph [0017] “Additionally or alternatively, the mitigating action may comprise feature engineering in order to change a feature, add a feature, remove a feature, or the like.” [0041] “ In some exemplary embodiments, the threshold may be a confidence level threshold, a success rate threshold, or the like.” [0045] “In some exemplary embodiments, the mitigating action may comprise re-training the predictor (174). Re-training the predictor may comprise obtaining another dataset to be used for training the predictor. In some exemplary embodiments, the other dataset may include instances in data slices in which the predictor performs below par.” [0046] “In some exemplary embodiments, the predictor may be trained based on a machine learning model. In that embodiment, the mitigating action may comprise changing the network architecture, the algorithm utilized to train the network, or the like” [0147] “It is noted that after the mitigation action is performed, the predictor may be re-assessed (Steps 140-160). In case, the performance measurement of the predictor after the mitigating action is implemented is above the threshold, the predictor may be utilized” [0126] “In some embodiments, machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Raz teaches retraining the predictor after mitigation, including retraining using another dataset that may include instances from data slices where the predictor performs below par. Raz also teaches changing the network architecture as a mitigation action and reassessing the predictor after the mitigation action to determine whether the performance measurement is above the threshold. Kraning supplies the asset-attribution context by teaching machine-learning-based identification of network assets associated with an entity. Accordingly, in the combination, Raz’s retraining and reassessment after mitigation correspond to retraining an asset attribution model after updating the feature preprocessor and architecture and determining whether the retrained asset attribution model satisfies an accuracy criterion.); and
deploy the retrained asset attribution model with the updated architecture to predict attribution of one or more assets to an organization (Raz, paragraph [0042] “On Step 165, as the performance measurement of the predictor is above a threshold, the predictor may be utilized. In some exemplary embodiments, the predictor may be utilized in order to provide a predicted label for a data instance that is not comprised by the dataset. In some exemplary embodiments, the predictor may be deployed in the field, may be provided as part of an update of a software utilizing the predictor, or the like.” [0046] “the mitigating action may comprise changing the network architecture, the algorithm utilized to train the network, or the like” [0047] “It is noted that after the mitigation action is performed, the predictor may be re-assessed (Steps 140-160). In case, the performance measurement of the predictor after the mitigating action is implemented is above the threshold, the predictor may be utilized” Kraning, paragraph [0126] “ machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Raz teaches changing the network architecture as a mitigation action, reassessing the predictor after the mitigation action, and utilizing/deploying the predictor when the performance measurement is above the threshold. Raz further teaches that the deployed predictor provides a predicted label for a data instance. Kraning supplies the asset-attribution context by teaching machine learning based identification of network assets associated with an entity. Accordingly, in the combination, deploying Raz’s predictor with the changed architecture to provide predicted labels in Kraning’s asset-attribution context corresponds to deploying the asset attribution model with the updated architecture to predict attribution of one or more assets to an organization.)
Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having a combination of Raz and Kraning before them, to incorporate the use of subsets of network assets with known organizational ownership, as taught by Kraning, into the slice-based model evaluation system of Raz. One would have been motivated to make such a combination in order to evaluate how accurately a machine learning model predicts asset ownership by an organization within specific data segments, and to use these evaluation results to guide improvements to model architecture. This would allow more precise and reliable attribution of assets to organizations by identifying and addressing model weaknesses on targeted slices of asset metadata.
Regarding claim 12, Raz in view of Kraning, as outlined above, all the elements of claim 9, therefore is rejected for the same reasons as those presented for claim 9, mutatis mutandis. Raz in view of Kraning further teaches:
wherein the accuracy criterion comprises a determination that a number of correct predictions by the asset attribution model on metadata for the first of the plurality of data slices is above a threshold number of correct predictions, wherein correct predictions are according to known attributed organizations for assets in the first data slice (Raz, paragraph [0035] “In some exemplary embodiments, the performance measurement may measure how well the predictor predicts the actual label. In some exemplary embodiments, for each data instance of the dataset that is in the data slice, the predictor may be utilized to predict a label. The predicted label may be compared with the actual label, to determine whether the prediction is correct or not. The performance measurement may be computed based on the number of instances, based on the number of instances for which a correct prediction was provided, or the like. In some exemplary embodiments, the performance measurement may be based on, for example, F1 score, Accuracy, R-squared, RSME, or the like.” [0036] “the performance measurement is based on the size of the slice and on the percentage of correct predictions in the data slice (success ratio)…Additionally or alternatively, the success ratio of the predictor over slice C may be 70% which may be below the success threshold level. As a result, the performance measurement of the predictor over slice C may be −1.” [0041] “In some exemplary embodiments, the threshold may be a confidence level threshold, a success rate threshold, or the like.” [0075] “ In some exemplary embodiments, P(s) may refer to a performance measurement, such Accuracy, F1-Score, or the like, of a predictor over the data slice s…In some exemplary embodiments, MinP may refer to a minimal threshold on the performance measurement.” Kraning, [Abstract] “Response data is received from one or more network systems connected to the computer network and processed to identify one or more network assets associated with an entity such as an enterprise organization.” [0126] “In some embodiments, machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” [0133] “asset data may include the asset itself (e.g., a registration or a digital certificate), an asset identifier (e.g., an IP address, a domain name, etc.), an entity identifier (e.g., a name, email address, etc.) associated with a related entity (e.g., an individual) that is responsible for the network asset, a resource identifier (e.g., a cloud instance UID) associated with the network asset, credentials needed to access the network asset, and any other such data or metadata associated with a network asset.” – Raz teaches an accuracy criterion by disclosing that, for data instances of a data slice, the predictor predicts labels and the predicted labels are compared with actual labels to determine whether the predictions are correct. Raz further teaches that the performance measurement may be based on the number of correct predictions, accuracy, percentage of correct predictions, and a success rate threshold. Kraning supplies the asset attribution context by teaching machine learning based identification of network assets associated with an entity, such as an enterprise organization, and asset data including metadata associated with network assets. In the combination, Raz’s data instances correspond to Kraning’s network assets, Raz’s actual labels correspond to known attributed organizations for the assets, and Raz’s determination of correct predictions above a threshold corresponds to determining that a number of correct predictions by the asset attribution model on metadata for the first data slice is above a threshold number of correct predictions.).
Regarding claim 13, Raz in view of Kraning, as outlined above, all the elements of claim 9, therefore is rejected for the same reasons as those presented for claim 9, mutatis mutandis. Raz in view of Kraning further teaches:
wherein the instructions to update the architecture of the asset attribution model comprises instructions to update at least one of a type of the asset attribution model, parameters of the asset attribution model, hyperparameters of the asset attribution model, and a training method for the asset attribution model based, at least in part, on the asset attribution model failing the accuracy criterion (Raz, paragraph [0019] “Additionally or alternatively, the mitigating action may comprise re-training the ANN by utilizing a different algorithm than Gradient descent, such as for example, Newton's method, Conjugate gradient, Levenberg-Marquardt algorithm, or the like.” [0035] “In some exemplary embodiments, the performance measurement may be based on, for example, F1 score, Accuracy, R-squared, RSME, or the like. In some exemplary embodiments, different data slices may have substantially different performance measurements.” [0036] “ Additionally or alternatively, the success ratio of the predictor over slice C may be 70% which may be below the success threshold level. As a result, the performance measurement of the predictor over slice C may be −1.” [0041] “In some exemplary embodiments, the threshold may be a confidence level threshold, a success rate threshold, or the like.” [0043] “On Step 170, as the performance measurement of the predictor over the dataset is below a threshold, a mitigating action may be performed.” [0046] “In some exemplary embodiments, the predictor may be trained based on a machine learning model. In that embodiment, the mitigating action may comprise changing the network architecture, the algorithm utilized to train the network, or the like (176). In some exemplary embodiments, layers may be added to the ANN, a layer may be removed from the ANN, a node may be added to the ANN, connectivity between nodes or layers may be modified, or the like. Additionally or alternatively, the action may comprise changing the machine learning algorithm to a different machine learning algorithm…The mitigating action may comprise changing the machine learning algorithm into a random forest classifier and retraining the predictor” Kraning, paragraph [0126] “In some embodiments, machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Raz teaches that the performance measurement may be based on accuracy and that a success ratio below a success threshold corresponds to underperformance. Raz further teaches performing a mitigating action when the predictor’s performance measurement is below a threshold, which corresponds to updating the model based on failing the accuracy criterion. Raz also teaches that the mitigating action may include changing the network architecture, changing the machine learning algorithm, modifying layers, nodes, or connectivity, and retraining using a different training algorithm. These teachings correspond to updating a type of the model, parameters/hyperparameters of the model, or a training method for the model. Kraning supplies the asset attribution context by teaching machine learning based identification of network assets associated with an entity. Accordingly, Raz in view of Kraning teaches the limitation.).
Regarding claim 15, Raz teaches the following limitations:
An apparatus comprising: a processor and a computer-readable medium having instructions stored thereon that are executable by the processor to cause the apparatus to (Raz, paragraph [0006]” computer program product comprising a non-transitory computer readable storage medium retaining program instructions, which program instructions when read by a processor, cause the processor to perform”)
determine attribution accuracies of attributions of assets to organizations by an asset attribution model across a plurality of data slices generated according to data slicing rules, wherein each data slice comprises asset metadata of a plurality of assets (Raz, paragraph [0030] “the data slice may be determined by obtaining a definition of data slice and applying the definition to identify the instances that are included in the data slice.” [0035] “a performance measurement of the predictor over each data slice may be computed… the performance measurement may measure how well the predictor predicts the actual label. In some exemplary embodiments, for each data instance of the dataset that is in the data slice, the predictor may be utilized to predict a label. The predicted label may be compared with the actual label, to determine whether the prediction is correct or not. The performance measurement may be computed based on the number of instances, based on the number of instances for which a correct prediction was provided, or the like. In some exemplary embodiments, the performance measurement may be based on, for example, F1 score, Accuracy, R-squared, RSME, or the like. [0060] “ Each data instance may comprise features values in a feature space….Each data instance may be associated with at least one metadata value.” [0062] “Data slices may be defined by constraints on the features values, on the meta values, or the like.” Kraning, [Abstract] “Response data is received from one or more network systems connected to the computer network and processed to identify one or more network assets associated with an entity such as an enterprise organization.” [0075] “Asset data may include the asset itself (e.g., a registration or a digital certificate), an asset identifier (e.g., an IP address, a domain name, etc.), an entity identifier (e.g., a name) associated with a related entity (e.g., an individual) that is responsible for the network asset, a resource identifier (e.g., a cloud instance UID) associated with the network asset, credentials needed to access the network asset, and any other such data or metadata associated with a network asset.” [0126] “machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Raz teaches determining accuracies across data slices by computing a performance measurement for each data slice, where the predictor’s label predictions are compared with actual labels and the performance measurement may be based on accuracy. Raz also teaches that the data slices are generated using data slice definitions/constraints, including constraints on meta values. Kraning supplies the asset-attribution context by teaching machine-learning-based identification of network assets associated with entities/organizations, where the network assets have associated asset data/metadata. In the combination, Raz’s predictor is applied to Kraning’s network-asset attribution context, such that the predictor corresponds to the claimed asset attribution model, the data instances correspond to network assets, the predicted label corresponds to the entity/organization associated with the network asset, and Kraning’s network-asset metadata corresponds to the claimed asset metadata. Accordingly, Raz in view of Kraning teaches the limitation.)
based on a determination that attribution accuracy of the asset attribution model on a first of the plurality of data slices fails an accuracy criterion, engineer features for asset attribution and retrain the asset attribution model according to the feature engineering until the accuracy criterion is satisfied (Raz, paragraph [0017] “ In some exemplary embodiments, the mitigating action may comprise obtaining an additional dataset and retraining the predictor therewith…Additionally or alternatively, the mitigating action may comprise feature engineering in order to change a feature, add a feature, remove a feature, or the like.” [0035] “In some exemplary embodiments, different data slices may have substantially different performance measurements…in case that the performance measurement of a data slice is below a second threshold the performance measurement of the data slice may be a value that is configured to cause a reduction in performance measurement.” [0036] “Additionally or alternatively, the success ratio of the predictor over slice C may be 70% which may be below the success threshold level. As a result, the performance measurement of the predictor over slice C may be −1. “ [0045] “In some exemplary embodiments, the mitigating action may comprise re-training the predictor (174). Re-training the predictor may comprise obtaining another dataset to be used for training the predictor.” [0047] “It is noted that after the mitigation action is performed, the predictor may be re-assessed (Steps 140-160). In case, the performance measurement of the predictor after the mitigating action is implemented is above the threshold, the predictor may be utilized” Kraning paragraph [0126] “ In some embodiments, machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Raz teaches determining that a first data slice fails an accuracy criterion by disclosing different data slices may have different performance measurements and that a data slice may fall below a threshold, such as a success ratio threshold. Raz further teaches performing mitigation when the predictor underperforms, where the mitigation may include feature engineering to change, add, or remove features and retraining the predictor. Raz also teaches reassessing the predictor after mitigation and utilizing the predictor when the performance measurement is above a threshold. Kraning supplies the asset-attribution content by teaching machine-learning-based identification of network assets associated with an entity. Accordingly, the combination, Raz’s predictor corresponds to the asset attribution model, and Raz’s feature engineering/retraining in response to a failed slice accuracy threshold corresponds to engineering features for asset attribution and retraining the asset attribution model until the accuracy criterion is satisfied.),
wherein the instructions to engineer features and retrain until the accuracy criterion is satisfied comprises instructions to, select one or more metadata fields of a first data slicing rule and update a feature preprocessor of the asset attribution model to add the selected one or more metadata fields as additional features for preprocessing by the feature preprocessor (Raz, paragraph [0017] “The feature values may be utilized as an input for the machine learning model, as an input for the predictor, or the like…Additionally or alternatively, the mitigating action may comprise changing the architecture of the model used to train the predictor, such as modifying an architecture of a network-based model, modifying the number of layers, the number of nodes in a layer, or the like…Additionally or alternatively, the mitigating action may comprise feature engineering in order to change a feature, add a feature, remove a feature, or the like.” [0031] “ On Step 134, a constraint may be obtained. In some exemplary embodiments, the constraint may represent a definition of the data slice… Applying the constraint may comprise identifying at least one data instance for which the constraint is held, e.g., the one or more feature values of the at least one data instance are in line with the constraint. The at least one identified data instance may be a member of the data slice.” [0061] “Predictor 212 may be trained based on feature values in a feature space. Additionally or alternatively, metadata values may be excluded from an input provided to the predictor.” [0062] “Data slices may be defined by constraints on the features values, on the meta values, or the like.” Kraning, paragraph [0075] “” Asset data may include the asset itself (e.g., a registration or a digital certificate), an asset identifier (e.g., an IP address, a domain name, etc.), an entity identifier (e.g., a name) associated with a related entity (e.g., an individual) that is responsible for the network asset, a resource identifier (e.g., a cloud instance UID) associated with the network asset, credentials needed to access the network asset, and any other such data or metadata associated with a network asset.” [0126] “ In some embodiments, machine learning may be applied to identify network assets associated with an entity.” – Raz teaches that a data slice may be defined by a constraint, including constraints on meta values, and applying the constraint identifies the data instances in the slice. Raz further teaches that metadata values may be excluded from the predictor input, while feature engineering may add feature and feature values may be used as input to the predictor. Thus, when a data slice defined by metadata/meta-value constraints underperforms, it would have been obvious to select the corresponding metadata information from the data slice rule and add it as additional input features through feature preprocessing. Kraning supplies the asset-attribution context by teaching asset metadata associated with network assets and machine learning based identification of network assets associated with an entity. Accordingly, Raz in view of Kraning teaches the limitation.);
update architecture of the asset attribution model, wherein the instructions to update architecture of the asset attribution model comprises instructions executable by the processor to cause the apparatus to configure an internal component of the asset attribution model to process the additional features based on update of the feature preprocessor (Raz, [0017] “The feature values may be utilized as an input for the machine learning model, as an input for the predictor, or the like…Additionally or alternatively, the mitigating action may comprise feature engineering in order to change a feature, add a feature, remove a feature, or the like.” [0046] “In some exemplary embodiments, the predictor may be trained based on a machine learning model. In that embodiment, the mitigating action may comprise changing the network architecture, the algorithm utilized to train the network, or the like (176). In some exemplary embodiments, layers may be added to the ANN, a layer may be removed from the ANN, a node may be added to the ANN, connectivity between nodes or layers may be modified, or the like.” [0061] “Predictor 212 may be configured to provide a predicted label for an input such as a data instance…Predictor 212 may be trained based on feature values in a feature space.” Kraning, paragraph [0126] “In some embodiments, machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Raz teaches that feature values may be used as input to the predictor and that feature engineering may add features. Raz further teaches changing the network architecture, including adding or removing layers or nodes and modifying connectivity. Thus, when additional features are adding through feature engineering, it would have been obvious to update the architecture of the model so that the input side of the predictor can receive and process the added feature values. Kraning supplies the asset attribution context by teaching machine learning based identification of network assets associated with an entity. Accordingly, Raz in view of Kraning teaches updating architecture of the asset attribution model, including configuring an input component of the asset attribution model to process the additional features based on updating of the feature preprocessor.);
after the update of the feature preprocessor and the architecture, retrain the asset attribution model and determine whether the retrained asset attribution model satisfies the accuracy criterion (Raz, paragraph [0017] “Additionally or alternatively, the mitigating action may comprise feature engineering in order to change a feature, add a feature, remove a feature, or the like.” [0041] “ In some exemplary embodiments, the threshold may be a confidence level threshold, a success rate threshold, or the like.” [0045] “In some exemplary embodiments, the mitigating action may comprise re-training the predictor (174). Re-training the predictor may comprise obtaining another dataset to be used for training the predictor. In some exemplary embodiments, the other dataset may include instances in data slices in which the predictor performs below par.” [0046] “In some exemplary embodiments, the predictor may be trained based on a machine learning model. In that embodiment, the mitigating action may comprise changing the network architecture, the algorithm utilized to train the network, or the like” [0147] “It is noted that after the mitigation action is performed, the predictor may be re-assessed (Steps 140-160). In case, the performance measurement of the predictor after the mitigating action is implemented is above the threshold, the predictor may be utilized” [0126] “In some embodiments, machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Raz teaches retraining the predictor after mitigation, including retraining using another dataset that may include instances from data slices where the predictor performs below par. Raz also teaches changing the network architecture as a mitigation action and reassessing the predictor after the mitigation action to determine whether the performance measurement is above the threshold. Kraning supplies the asset-attribution context by teaching machine-learning-based identification of network assets associated with an entity. Accordingly, in the combination, Raz’s retraining and reassessment after mitigation correspond to retraining an asset attribution model after updating the feature preprocessor and architecture and determining whether the retrained asset attribution model satisfies an accuracy criterion.); and
deploy the asset attribution model with the updated architecture to predict attribution to an organization (Raz, paragraph [0042] “On Step 165, as the performance measurement of the predictor is above a threshold, the predictor may be utilized. In some exemplary embodiments, the predictor may be utilized in order to provide a predicted label for a data instance that is not comprised by the dataset. In some exemplary embodiments, the predictor may be deployed in the field, may be provided as part of an update of a software utilizing the predictor, or the like.” [0046] “the mitigating action may comprise changing the network architecture, the algorithm utilized to train the network, or the like” [0047] “It is noted that after the mitigation action is performed, the predictor may be re-assessed (Steps 140-160). In case, the performance measurement of the predictor after the mitigating action is implemented is above the threshold, the predictor may be utilized” Kraning, paragraph [0126] “ machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Raz teaches changing the network architecture as a mitigation action, reassessing the predictor after the mitigation action, and utilizing/deploying the predictor when the performance measurement is above the threshold. Raz further teaches that the deployed predictor provides a predicted label for a data instance. Kraning supplies the asset-attribution context by teaching machine learning based identification of network assets associated with an entity. Accordingly, in the combination, deploying Raz’s predictor with the changed architecture to provide predicted labels in Kraning’s asset-attribution context corresponds to deploying the asset attribution model with the updated architecture to predict attribution to an organization.)
Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having a combination of Raz and Kraning before them, to incorporate the use of subsets of network assets with known organizational ownership, as taught by Kraning, into the slice-based model evaluation system of Raz. One would have been motivated to make such a combination in order to evaluate how accurately a machine learning model predicts asset ownership by an organization within specific data segments, and to use these evaluation results to guide improvements to model architecture. This would allow more precise and reliable attribution of assets to organizations by identifying and addressing model weaknesses on targeted slices of asset metadata.
Regarding claim 18, Raz in view of Kraning, as outlined above, all the elements of claim 15, therefore is rejected for the same reasons as those presented for claim 15, mutatis mutandis. Raz in view of Kraning further teaches:
to update at least one type of the asset attribution model, parameters of the asset attribution model, hyperparameters of the asset attribution model, and a training method for the asset attribution model based, at least in part, on the determination that accuracy of the asset attribution model for at least the first data slice fails an accuracy criterion (Raz, paragraph [0017] “In some exemplary embodiments, the predictor may be trained based on a machine learning model such as an Artificial Neural Network (ANN), a Deep Neural Network (DNN), Ordinary Least Squares Regression, Logistic Regression, Support Vector Machines, or the like… Additionally or alternatively, the mitigating action may comprise changing the model utilized by the predictor.” [0019] “Additionally or alternatively, the mitigating action may comprise re-training the ANN by utilizing a different algorithm than Gradient descent, such as for example, Newton's method, Conjugate gradient, Levenberg-Marquardt algorithm, or the like.” [0035] “In some exemplary embodiments, the performance measurement may be based on, for example, F1 score, Accuracy, R-squared, RSME, or the like. In some exemplary embodiments, different data slices may have substantially different performance measurements…In some exemplary embodiments, in case that the number of instances comprised by a data slice is below a first threshold or in case that the performance measurement of a data slice is below a second threshold the performance measurement of the data slice may be a value that is configured to cause a reduction in performance measurement.” [0036] “ Additionally or alternatively, the success ratio of the predictor over slice C may be 70% which may be below the success threshold level. As a result, the performance measurement of the predictor over slice C may be −1.” [0041] “In some exemplary embodiments, the threshold may be a confidence level threshold, a success rate threshold, or the like.” [0043] “On Step 170, as the performance measurement of the predictor over the dataset is below a threshold, a mitigating action may be performed.” [0046] “In some exemplary embodiments, the predictor may be trained based on a machine learning model. In that embodiment, the mitigating action may comprise changing the network architecture, the algorithm utilized to train the network, or the like (176). In some exemplary embodiments, layers may be added to the ANN, a layer may be removed from the ANN, a node may be added to the ANN, connectivity between nodes or layers may be modified, or the like. Additionally or alternatively, the action may comprise changing the machine learning algorithm to a different machine learning algorithm…The mitigating action may comprise changing the machine learning algorithm into a random forest classifier and retraining the predictor” Kraning, paragraph [0126] “In some embodiments, machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Raz teaches that different data slices may have different performance measurements and that the performance measurement of a data slice may fall below a threshold, such as where the success ratio for slice C is below a success threshold. Raz’s disclosure of accuracy/performance below a threshold corresponds to determining the accuracy for at least the first data slice fails an accuracy criterion. Raz further teaches performing mitigating action based on underperformance, where the mitigating action may include changing the model, changing the network architecture, modifying layers, nodes, or connectivity, changing the machine learning algorithm, and retraining using a different training algorithm. These teachings correspond to updating a type of the model, parameters/hyperparameters of the model, or a training method for the model. Kraning supplies the asset attribution context by teaching machine learning based identification of network assets associated with an entity. Accordingly, Raz in view of Kraning teaches the limitation.).
Regarding claim 19, Raz in view of Kraning, as outlined above, all the elements of claim 15, therefore is rejected for the same reasons as those presented for claim 15, mutatis mutandis. Raz in view of Kraning further teaches:
wherein the data slicing rules comprises rules for obtaining at least one of assertion testing-based data slices, regression testing-based data slices, location-based data slices, and organization-based data slices (Raz, paragraph [0030] “the data slice may be determined by obtaining a definition of data slice and applying the definition to identify the instances that are included in the data slice.” [0031] “On Step 134, a constraint may be obtained. In some exemplary embodiments, the constraint may represent a definition of the data slice…Applying the constraint may comprise identifying at least one data instance for which the constraint is held, e.g., the one or more feature values of the at least one data instance are in line with the constraint. The at least one identified data instance may be a member of the data slice.” [0062] “Data slices may be defined by constraints on the features values, on the meta values, or the like.” Kraning, [Abstract] “Response data is received from one or more network systems connected to the computer network and processed to identify one or more network assets associated with an entity such as an enterprise organization.” [0126] “machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” [0133] “asset data may include the asset itself (e.g., a registration or a digital certificate), an asset identifier (e.g., an IP address, a domain name, etc.), an entity identifier (e.g., a name, email address, etc.) associated with a related entity (e.g., an individual) that is responsible for the network asset, a resource identifier (e.g., a cloud instance UID) associated with the network asset, credentials needed to access the network asset, and any other such data or metadata associated with a network asset.” – Raz teaches data slicing rules by disclosing data slice definitions/constraints that are applied to identify data instances included in a data slice. Raz further teaches that data slices may be defined by constraints on meta values. Kraning teaches organization-related asset information by disclosing network asset associated with an entity, such as an enterprise organization, and asset data including an entity identifier and metadata associated with a network asset. Thus, when Raz’s constraint based data slicing is applied to Kraning’s organization-related asset data, the resulting data slice corresponds to an organization based data slice. Accordingly, Raz in view of Kraning teaches the limitation.)
Regarding claim 21, Raz in view of Kraning, as outlined above, all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1, mutatis mutandis. Raz in view of Kraning further teaches:
wherein engineering the one or more features further comprises determining a number of the one or more features based, at least in part, on the accuracy of the asset attribution model on the first subset of the plurality of assets, wherein the number of the one or more features is determined to be higher for a lower accuracy of the asset attribution model and lower for a higher accuracy of the asset attribution model (Raz, paragraph [0017] “The feature values may be utilized as an input for the machine learning model, as an input for the predictor, or the like…Additionally or alternatively, the mitigating action may comprise feature engineering in order to change a feature, add a feature, remove a feature, or the like.” [0035] “In some exemplary embodiments, the performance measurement may be based on, for example, F1 score, Accuracy, R-squared, RSME, or the like..” [0036] “ Additionally or alternatively, the success ratio of the predictor over slice C may be 70% which may be below the success threshold level. As a result, the performance measurement of the predictor over slice C may be −1.” [0041] “In some exemplary embodiments, the threshold may be a confidence level threshold, a success rate threshold, or the like.” [0043] “On Step 170, as the performance measurement of the predictor over the dataset is below a threshold, a mitigating action may be performed.” [0062] “Data slices may be defined by constraints on the features values, on the meta values, or the like. “ [0075] “In some exemplary embodiments, P(s) may refer to a performance measurement, such Accuracy, F1-Score, or the like, of a predictor over the data slice s…MinP may refer to a minimal threshold on the performance measurement. “ Kraning, paragraph [0126] “ machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.”[0133] “asset data may include the asset itself (e.g., a registration or a digital certificate), an asset identifier (e.g., an IP address, a domain name, etc.), an entity identifier (e.g., a name, email address, etc.) associated with a related entity (e.g., an individual) that is responsible for the network asset, a resource identifier (e.g., a cloud instance UID) associated with the network asset, credentials needed to access the network asset, and any other such data or metadata associated with a network asset.” – Under the broadest reasonable interpretation, the claimed difference between the attribution accuracy and the accuracy criterion corresponds to how accurate the model is on the data slice relative to the required threshold. Raz teaches measuring data-slice performance using accuracy, F1-score, or success ratio, comparing the performance to a threshold, and performing mitigation when performance is below a threshold. Raz further teaches that the mitigation may include feature engineering to add, change, or remove features, thereby adjusting the number of features used by the model. Kraning supplies the asset attribution context by teaching machine learning based identification of network assets associated with an entity and asset data including metadata associated with network assets. In the combination, Raz’s accuracy based data slice mitigation is applied to Kraning’s asset attribution environment. Since lower accuracy relative to the criterion indicates greater underperformance, it would have been obvious to add a higher number of corrective metadata based features for lower attribution accuracy and a lower number of features for higher attribution accuracy.).
Regarding claim 22, Raz in view of Kraning, as outlined above, all the elements of claim 9, therefore is rejected for the same reasons as those presented for claim 9, mutatis mutandis. Raz in view of Kraning further teaches:
wherein the instructions to select one or more metadata fields of the first data slicing rule comprise instructions to determine a number of the one or more additional features to be generated corresponding to the first data slice based, at least in part, on a difference between the attribution accuracy of the first data slice and the accuracy criterion, and wherein the number of the one or more additional features is determined to be higher for a lower attribution accuracy relative to the accuracy criterion (Raz, paragraph [0017] “The feature values may be utilized as an input for the machine learning model, as an input for the predictor, or the like… Additionally or alternatively, the mitigating action may comprise feature engineering in order to change a feature, add a feature, remove a feature, or the like.” [0035] “In some exemplary embodiments, the performance measurement may be based on, for example, F1 score, Accuracy, R-squared, RSME, or the like.” [0036] “ Additionally or alternatively, the success ratio of the predictor over slice C may be 70% which may be below the success threshold level. As a result, the performance measurement of the predictor over slice C may be −1.” [0041] “ In some exemplary embodiments, the threshold may be a confidence level threshold, a success rate threshold, or the like.” [0043] “On Step 170, as the performance measurement of the predictor over the dataset is below a threshold, a mitigating action may be performed.” [0062] “ Data slices may be defined by constraints on the features values, on the meta values, or the like.” [0075] “In some exemplary embodiments, P(s) may refer to a performance measurement, such Accuracy, F1-Score, or the like, of a predictor over the data slice s… In some exemplary embodiments, MinP may refer to a minimal threshold on the performance measurement.” Kraning, [0126] “In some embodiments, machine learning may be applied to identify network assets associated with an entity.” [0133] “ asset data may include the asset itself (e.g., a registration or a digital certificate), an asset identifier (e.g., an IP address, a domain name, etc.), an entity identifier (e.g., a name, email address, etc.) associated with a related entity (e.g., an individual) that is responsible for the network asset, a resource identifier (e.g., a cloud instance UID) associated with the network asset, credentials needed to access the network asset, and any other such data or metadata associated with a network asset.” – Under the broadest reasonable interpretation, the claimed difference between the attribution accuracy and the accuracy criterion corresponds to how accurate the model is on the data slice relative to the required threshold. Raz teaches measuring data slice performance using accuracy, F1-score, or success ratio, comparing the performance to a threshold, and performing mitigation when performance is below the threshold. Raz further teaches that the mitigation may include feature engineering to add, change, or remove features, thereby adjusting the number of features used by the model. Kraning supplies the asset attribution context by teaching machine learning based identification of network assets associated with an entity and asset data including metadata associated with network assets. In the combination, Raz’s accuracy based data slice mitigation is applied to Kraning’s asset attribution environment. Since lower accuracy relative to the criterion indicates greater underperformance, it would have been obvious to add a higher number of corrective metadata based features for lower attribution accuracy and a lower number of features for higher attribution accuracy.).
Regarding claim 23, Raz in view of Kraning, as outlined above, all the elements of claim 15, therefore is rejected for the same reasons as those presented for claim 15, mutatis mutandis. Raz in view of Kraning further teaches:
wherein the instructions to select one or more metadata fields of the first data slicing rule comprise instructions executable by the processor to cause the apparatus to determine a number of the one or more additional features to be generated corresponding to the first data slice based, at least in part, on a difference between attribution accuracy of the first data slice and the accuracy criterion and wherein the number of the one or more additional features is determined to be higher for a lower attribution accuracy relative to the accuracy criterion and lower for a higher attribution accuracy relative to the accuracy criterion (Raz, paragraph [0017] “The feature values may be utilized as an input for the machine learning model, as an input for the predictor, or the like… Additionally or alternatively, the mitigating action may comprise feature engineering in order to change a feature, add a feature, remove a feature, or the like.” [0035] “In some exemplary embodiments, the performance measurement may be based on, for example, F1 score, Accuracy, R-squared, RSME, or the like.” [0036] “ Additionally or alternatively, the success ratio of the predictor over slice C may be 70% which may be below the success threshold level. As a result, the performance measurement of the predictor over slice C may be −1.” [0041] “ In some exemplary embodiments, the threshold may be a confidence level threshold, a success rate threshold, or the like.” [0043] “On Step 170, as the performance measurement of the predictor over the dataset is below a threshold, a mitigating action may be performed.” [0062] “ Data slices may be defined by constraints on the features values, on the meta values, or the like.” [0075] “In some exemplary embodiments, P(s) may refer to a performance measurement, such Accuracy, F1-Score, or the like, of a predictor over the data slice s… In some exemplary embodiments, MinP may refer to a minimal threshold on the performance measurement.” Kraning, [0126] “In some embodiments, machine learning may be applied to identify network assets associated with an entity.” [0133] “ asset data may include the asset itself (e.g., a registration or a digital certificate), an asset identifier (e.g., an IP address, a domain name, etc.), an entity identifier (e.g., a name, email address, etc.) associated with a related entity (e.g., an individual) that is responsible for the network asset, a resource identifier (e.g., a cloud instance UID) associated with the network asset, credentials needed to access the network asset, and any other such data or metadata associated with a network asset.” – Under the broadest reasonable interpretation, the claimed difference between the attribution accuracy and the accuracy criterion corresponds to how accurate the model is on the data slice relative to the required threshold. Raz teaches measuring data slice performance using accuracy, F1-score, or success ratio, comparing the performance to a threshold, and performing mitigation when performance is below the threshold. Raz further teaches that the mitigation may include feature engineering to add, change, or remove features, thereby adjusting the number of features used by the model. Kraning supplies the asset attribution context by teaching machine learning based identification of network assets associated with an entity and asset data including metadata associated with network assets. In the combination, Raz’s accuracy based data slice mitigation is applied to Kraning’s asset attribution environment. Since lower accuracy relative to the criterion indicates greater underperformance, it would have been obvious to add a higher number of corrective metadata based features for lower attribution accuracy and a lower number of features for higher attribution accuracy.).
Regarding claim 24, Raz in view of Kraning, as outlined above, all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1 mutatis mutandis. Raz in view of Kraning further teaches:
wherein the first data slicing rule defines the first of the plurality of data slices based on one or more metadata fields such that the first of the plurality of data slices comprises assets for which the asset attribution model produces incorrect attributions of the assets to organizations (Raz, paragraph [0030] “ In some exemplary embodiments, the data slice may be determined by obtaining a definition of data slice and applying the definition to identify the instances that are included in the data slice.” [0031] “On Step 134, a constraint may be obtained. In some exemplary embodiments, the constraint may represent a definition of the data slice… Applying the constraint may comprise identifying at least one data instance for which the constraint is held, e.g., the one or more feature values of the at least one data instance are in line with the constraint. The at least one identified data instance may be a member of the data slice.” [0035] “ In some exemplary embodiments, for each data instance of the dataset that is in the data slice, the predictor may be utilized to predict a label. The predicted label may be compared with the actual label, to determine whether the prediction is correct or not.” [0036] “Additionally or alternatively, the success ratio of the predictor over slice C may be 70% which may be below the success threshold level.” Kraning, [0126] “In some embodiments, machine learning may be applied to identify network assets associated with an entity.” [0133] “ asset data may include the asset itself (e.g., a registration or a digital certificate), an asset identifier (e.g., an IP address, a domain name, etc.), an entity identifier (e.g., a name, email address, etc.) associated with a related entity (e.g., an individual) that is responsible for the network asset, a resource identifier (e.g., a cloud instance UID) associated with the network asset, credentials needed to access the network asset, and any other such data or metadata associated with a network asset.” – under the broadest reasonable interpretation, the limitation requires the first data slicing rule to define a metadata based data slice, where the resulting data slice includes assets for which the model produces incorrect organization attributions. Raz teaches defining data slices using constraints, including constraints on meta values, and applying the constraint to identify data instances included in the data slice. Raz further teaches applying the predictor to data instances in the data slice and comparing predicted labels with actual labels to determine whether the predictions are correct or not. Raz’s example of slice C having a 70% success ratio indicates that the data slice includes incorrectly predicted instances. Kraning supplies the asset attribution context by teaching machine learning based identification of network assets associated with an entity and asset metadata associated with network assets. Accordingly, in the combination, Raz’s metadata defined data slice corresponds to the first data slice, Kraning’s network assets correspond to the claimed assets, and Raz’s incorrect predicted labels correspond to incorrect attributions of assets to organizations.).
Regarding claim 25, Raz in view of Kraning, as outlined above, all the elements of claim 24, therefore is rejected for the same reasons as those presented for claim 24 mutatis mutandis. Raz in view of Kraning further teaches:
wherein the incorrect attributions comprise false positive attributions of assets to organizations, and wherein engineering features for asset attribution and retraining the asset attribution model reduces the false positive attributions for the first of the plurality of data slices (Raz, paragraph [0035] “In some exemplary embodiments, for each data instance of the dataset that is in the data slice, the predictor may be utilized to predict a label. The predicted label may be compared with the actual label, to determine whether the prediction is correct or not…In some exemplary embodiments, the performance measurement may be based on, for example, F1 score, Accuracy, R-squared, RSME, or the like. [0036] “Additionally or alternatively, the success ratio of the predictor over slice C may be 70% which may be below the success threshold level.” [0017] “Additionally or alternatively, the mitigating action may comprise feature engineering in order to change a feature, add a feature, remove a feature, or the like.” [0045] “In some exemplary embodiments, the mitigating action may comprise re-training the predictor” [0047] “It is noted that after the mitigation action is performed, the predictor may be re-assessed (Steps 140-160). In case, the performance measurement of the predictor after the mitigating action is implemented is above the threshold, the predictor may be utilized (165).” Kraning, paragraph [0126] “ In some embodiments, machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Under the broadest reasonable interpretation, a false positive attribution is an incorrect positive prediction that an asset is associated with an organization. Raz teaches applying a predictor to data instances in a data slice and comparing predicted labels to actual labels to determine whether the predictions are correct or incorrect. Raz further teaches that the performance may be measured using accuracy of F1 score, that a data slice may fall below a success threshold, and that mitigation may include feature engineering and retraining followed by reassessment. Kraning supplies the asset attribution context by teaching machine learning based identification of network assets associated with an entity. Accordingly, in the combination, Raz’s incorrect predicted labels correspond to incorrect asset attributions, including false positive attributions, and Raz’s feature engineering/retraining mitigation reduces such incorrect attributions by improving the predictor’s performance for the underperforming data slice.).
Regarding claim 26, Raz in view of Kraning, as outlined above, all the elements of claim 9, therefore is rejected for the same reasons as those presented for claim 9 mutatis mutandis. Raz in view of Kraning further teaches:
wherein the instructions to engineer features further comprise instructions to identify the first data slicing rule such that the first of the plurality of data slices comprises assets for which the asset attribution model produces incorrect attributions of the assets to organizations based on one or more metadata fields referenced by the first data slicing rule (Raz, paragraph [0017] “Additionally or alternatively, the mitigating action may comprise feature engineering in order to change a feature, add a feature, remove a feature, or the like.” [0030] “In some exemplary embodiments, the data slice may be determined by obtaining a definition of data slice and applying the definition to identify the instances that are included in the data slice.” [0031] “On Step 134, a constraint may be obtained. In some exemplary embodiments, the constraint may represent a definition of the data slice… Applying the constraint may comprise identifying at least one data instance for which the constraint is held, e.g., the one or more feature values of the at least one data instance are in line with the constraint. The at least one identified data instance may be a member of the data slice.” [0062] “Data slices may be defined by constraints on the features values, on the meta values, or the like.” [0035] “In some exemplary embodiments, for each data instance of the dataset that is in the data slice, the predictor may be utilized to predict a label. The predicted label may be compared with the actual label, to determine whether the prediction is correct or not. “ [0036] “Additionally or alternatively, the success ratio of the predictor over slice C may be 70% which may be below the success threshold level.” Kraning, paragraph [0077] “a network asset can exist in various states from the point of view of a given entity such as an enterprise organization. “ [0126] “ In some embodiments, machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” [0133] “asset data may include the asset itself (e.g., a registration or a digital certificate), an asset identifier (e.g., an IP address, a domain name, etc.), an entity identifier (e.g., a name, email address, etc.) associated with a related entity (e.g., an individual) that is responsible for the network asset, a resource identifier (e.g., a cloud instance UID) associated with the network asset, credentials needed to access the network asset, and any other such data or metadata associated with a network asset.” – Under the broadest reasonable interpretation, the claimed data slice rule corresponds to a definition or constraint used to identify data instances included in a data slice. Raz teaches obtaining a data slice definition/constraint and applying the constraint to identify data instances that satisfy the constraint. Raz further teaches that data slices may be defined by constraints on meta values. Raz also teaches applying the predictor to data instances in the data slice and comparing predicted labels with actual labels to determine whether the predictions are correct or incorrect. Kraning supplies the asset attribution context by teaching machine learning based identification of network assets associated with an entity and asset data including metadata associated with network assets. In the combination, Raz’s meta value constraint corresponds to the claimed first data slicing rule referencing metadata fields. Kraning’s network assets correspond to the claimed assets, and Raz’s incorrect predicted labels correspond to incorrect attributions of assets to organizations. Accordingly, Raz in view of Kraning teaches identifying the first data slicing rule such that the first data slice comprises assets for which the asset attribution model produces incorrect attributions based on metadata fields referenced by the rule.).
Regarding claim 27, Raz in view of Kraning, as outlined above, all the elements of claim 26, therefore is rejected for the same reasons as those presented for claim 26 mutatis mutandis. Raz in view of Kraning further teaches:
wherein the instructions to engineer features and retrain the asset attribution model further comprise instructions to reduce incorrect attributions of the assets to organizations by the asset attribution model for the first of the plurality of data slices (Raz, paragraph [0017] “Additionally or alternatively, the mitigating action may comprise feature engineering in order to change a feature, add a feature, remove a feature, or the like.” [0035] “ In some exemplary embodiments, for each data instance of the dataset that is in the data slice, the predictor may be utilized to predict a label. The predicted label may be compared with the actual label, to determine whether the prediction is correct or not.” [0036] “Additionally or alternatively, the success ratio of the predictor over slice C may be 70% which may be below the success threshold level.” [0045] “In some exemplary embodiments, the mitigating action may comprise re-training the predictor” [0047] “It is noted that after the mitigation action is performed, the predictor may be re-assessed (Steps 140-160). In case, the performance measurement of the predictor after the mitigating action is implemented is above the threshold, the predictor may be utilized (165)” [0077] “a network asset can exist in various states from the point of view of a given entity such as an enterprise organization.” [0126] “In some embodiments, machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” – Under the broadest reasonable interpretation, reducing incorrect attributions means improving the model’s attribution performance for the first data slice so that fewer asset-to-organization predictions are incorrect. Raz teaches applying a predictor to data instances in a data slice, comparing predicted labels with actual labels to determine whether predictions are correct or incorrect, and identifying a data slice with below-threshold performance. Raz further teaches performing mitigation, including feature engineering and retraining, and reassessing the predictor after mitigation to determine whether the performance measurement is above the threshold. Kraning supplies the asset attribution context by teaching machine learning based identification of network assets with an entity. According, in the combination, Raz’s feature engineering and retraining to improve below threshold slice performance corresponds to reducing incorrect attributions of assets to organizations for the first data slice.).
Regarding claim 28, Raz in view of Kraning, as outlined above, all the elements of claim 15, therefore is rejected for the same reasons as those presented for claim 15 mutatis mutandis. Raz in view of Kraning further teaches:
wherein the instructions to engineer features further comprise instructions executable by the processor to cause the apparatus to identify the first data slicing rule such that the first of the plurality of data slices comprises assets for which the asset attribution model produces incorrect attributions of the assets to organizations based on one or more metadata fields referenced by the first data slicing rule (Raz, paragraph [0017] “Additionally or alternatively, the mitigating action may comprise feature engineering in order to change a feature, add a feature, remove a feature, or the like.” [0030] “In some exemplary embodiments, the data slice may be determined by obtaining a definition of data slice and applying the definition to identify the instances that are included in the data slice.” [0031] “On Step 134, a constraint may be obtained. In some exemplary embodiments, the constraint may represent a definition of the data slice… Applying the constraint may comprise identifying at least one data instance for which the constraint is held, e.g., the one or more feature values of the at least one data instance are in line with the constraint. The at least one identified data instance may be a member of the data slice.” [0062] “Data slices may be defined by constraints on the features values, on the meta values, or the like.” [0035] “In some exemplary embodiments, for each data instance of the dataset that is in the data slice, the predictor may be utilized to predict a label. The predicted label may be compared with the actual label, to determine whether the prediction is correct or not. “ [0036] “Additionally or alternatively, the success ratio of the predictor over slice C may be 70% which may be below the success threshold level.” Kraning, paragraph [0077] “a network asset can exist in various states from the point of view of a given entity such as an enterprise organization. “ [0126] “ In some embodiments, machine learning may be applied to identify network assets associated with an entity. For example, data included in responses from multiple network systems can be input into and processed using one or more machine learning models to identify information included in the responses that is indicative of network assets associated with the entity.” [0133] “asset data may include the asset itself (e.g., a registration or a digital certificate), an asset identifier (e.g., an IP address, a domain name, etc.), an entity identifier (e.g., a name, email address, etc.) associated with a related entity (e.g., an individual) that is responsible for the network asset, a resource identifier (e.g., a cloud instance UID) associated with the network asset, credentials needed to access the network asset, and any other such data or metadata associated with a network asset.” – Under the broadest reasonable interpretation, the claimed data slice rule corresponds to a definition or constraint used to identify data instances included in a data slice. Raz teaches obtaining a data slice definition/constraint and applying the constraint to identify data instances that satisfy the constraint. Raz further teaches that data slices may be defined by constraints on meta values. Raz also teaches applying the predictor to data instances in the data slice and comparing predicted labels with actual labels to determine whether the predictions are correct or incorrect. Kraning supplies the asset attribution context by teaching machine learning based identification of network assets associated with an entity and asset data including metadata associated with network assets. In the combination, Raz’s meta value constraint corresponds to the claimed first data slicing rule referencing metadata fields. Kraning’s network assets correspond to the claimed assets, and Raz’s incorrect predicted labels correspond to incorrect attributions of assets to organizations. Accordingly, Raz in view of Kraning teaches identifying the first data slicing rule such that the first data slice comprises assets for which the asset attribution model produces incorrect attributions based on metadata fields referenced by the rule.).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Daravanh Phakousonh whose telephone number is (571)272-6324. The examiner can normally be reached Mon - Thurs 7 AM - 5 PM, Every other Friday 7 AM - 4PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at 571-272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Daravanh Phakousonh/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights