Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This action is in response to the application filed 27 February 2026. Claims 1, 2, 11, 12, 16 and 17 are amended. Claims 1-20 are pending and have been examined.
Response to Arguments
Applicant's arguments, see pages 9-11, filed 27 February 2026, with respect to the rejections of Claims 1-20 under 35 U.S.C. 101 have been fully considered but they are not persuasive.
APPLICANT'S ARGUMENT: Applicant argues (page 9, paragraph 4) that "the claimed subject matter of the present case is similarly concerned with improving the performance a machine learning classification model by updating the training dataset used to train the model by considering data samples that were previously rejected from classification."
EXAMINER'S RESPONSE: Examiner respectfully disagrees that amended Claim 1 recites eligible subject matter. As indicated in the 35 U.S.C. 101 rejection below, amended Claim 1 recites several mental process steps. The additional elements of amended Claim 1 do pertain to training a machine learning classification model by means of an updated training dataset, but the additional elements do not appear as currently recited to integrate the application or provide significantly more.
Examiner further notes that amended Claim 1 does not recite "data samples that were previously rejected from classification."
Applicant' s arguments, see pages 11-13, filed 27 February 2026, with respect to the rejections of Claims 1-20 under 35 U.S.C. 103 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
APPLICANT'S ARGUMENT: Applicant argues (page 11, paragraph 3) that "the combination does not teach or suggest, at least, these features of [amended] claim 1."
Applicant argues (page 12, paragraph 4) that "Applicant disagrees that Sadaghiani's disclosure of ATO risk score constitutes the selecting of claim 1."
Applicant argues (page 13, paragraph 3) that "Johnston appears to be operating the under the assumption that the 'data instances' in its 'labeled data reservoir' are already labeled. As such, Johnston appears to be silent on the manner in which these data instances were selected for classification. Accordingly, Johnston does not disclose ... [features] as recited in claim 1."
EXAMINER'S RESPONSE: Examiner notes that Applicant's arguments are moot. Amended independent Claims 1 and 11 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Kozodoi, et al., "Shallow Self-learning for Reject Inference in Credit Scoring." Amended independent Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Kozodoi in view of Dai.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1
Claim 1 is ineligible.
Step 1
Claim 1 recites a method, and thus the claimed process falls within a statutory category of invention.
Step 2A Prong 1
The claim recites to generate a plurality of model scores for the plurality of unlabeled data samples, wherein, for a given one of the plurality of unlabeled data samples, a corresponding model score indicates a probability that the given unlabeled data sample belongs to one of the first and second categories, which is a mental process. The claim recites selecting ... a subset of the plurality of unlabeled data samples for classification into one of the first and second categories, wherein the subset of unlabeled data samples are selected based on the unlabeled data samples in the subset having model scores exceeding a particular threshold value, which is a mental process. The claim recites classifying ... the selected subset of the unlabeled data samples into one of the first and second categories, which is a mental process. The claim recites identifying a plurality of attributes of the selected subset of unlabeled data samples that contributed to the model scores exceeding the particular threshold value, which is a mental process and/or a mathematical calculation. The claim recites based on the plurality of attributes, generating new labeled data samples from the subset, wherein the new labeled data samples include a first new labeled data sample indicating that a first unlabeled data sample in the selected subset belongs to the first category, which is a mental process. The claim recites updating ... the training dataset to include the new labeled data samples, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2
The additional element accessing, by a computer system, a machine learning classification model trained using a training dataset amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering"). The additional element a training dataset that includes: a first set of labeled data samples belonging to a first category does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The claim recites a training dataset that includes: ... a second set of labeled data samples belonging to a second category does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element processing, by the computer system, a plurality of unlabeled data samples using the machine learning classification model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by the computer system invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element training, by the computer system, the machine learning classification model based on the updated training dataset invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element to generate an updated machine learning classification model does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment").
Step 2B
The additional element accessing, by a computer system, a machine learning classification model trained using a training dataset is well-understood, routine, conventional activity (see MPEP 2106.05(d), "storing and retrieving information in memory"). The additional element a training dataset that includes: a first set of labeled data samples belonging to a first category does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The claim recites a training dataset that includes: ... a second set of labeled data samples belonging to a second category does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element processing, by the computer system, a plurality of unlabeled data samples using the machine learning classification model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by the computer system invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element training, by the computer system, the machine learning classification model based on the updated training dataset invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element to generate an updated machine learning classification model does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 2
Claim 2 is ineligible.
Step 1
Regarding Claim 2, the rejection of Claim 1 is incorporated.
Step 2A Prong 1
The claim recites selecting, from the plurality of attributes, a set of most-common attributes of the selected subset of unlabeled data samples that contributed to the model scores exceeding the particular threshold value, which is a mental process. The claim recites applying a policy rule to the selected subset of unlabeled data samples to identify the first unlabeled data sample, wherein the policy rule is based on at least one of the set of most-common attributes of the selected subset of unlabeled data samples, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 3
Claim 3 is ineligible.
Step 1
Regarding Claim 3, the rejection of Claim 2 is incorporated.
Step 2A Prong 1
The claim recites prior to classifying the first unlabeled data sample, verifying, by the computer system, the policy rule using the training dataset, which is a mental process. The claim recites wherein the verifying includes determining an accuracy of the policy rule using the first set of labeled data samples that belong to the first category, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 4
Claim 4 is ineligible.
Step 1
Regarding Claim 4, the rejection of Claim 3 is incorporated.
Step 2A Prong 1
The claim recites wherein generating the new labeled data samples includes assigning a particular label, to the first unlabeled data sample, that corresponds to the accuracy of the policy rule, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 5
Claim 5 is ineligible.
Step 1
Regarding Claim 5, the rejection of Claim 2 is incorporated.
Step 2A Prong 1
The claim recites generating ... the policy rule based on a combination of one or more of the set of most-common attributes of the subset of unlabeled data samples, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The additional element by the computer system invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 6
Claim 6 is ineligible.
Step 1
Regarding Claim 6, the rejection of Claim 1 is incorporated.
Step 2A Prong 1
Claim 6 recites the abstract ideas recited by parent Claim 1.
Step 2A Prong 2, Step 2B
The additional element wherein the training dataset corresponds to a plurality of active user accounts with a server system does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element wherein the plurality of unlabeled data samples correspond to a plurality of dormant user accounts with the server system does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment").
The claim lacks additional elements that integrate it into a practical application or provide
significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 7
Claim 7 is ineligible.
Step 1
Regarding Claim 7, the rejection of Claim 6 is incorporated.
Step 2A Prong 1
The claim recites identifying ... a first dormant user account corresponding to the first unlabeled data sample, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The additional element by the computer system invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element a training dataset that includes: a first set of labeled data samples belonging to a first category (as recited by Claim 1), wherein the first category corresponds to malicious user accounts and the second category corresponds to non-malicious user accounts does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element performing, by the computer system, one or more risk-mitigation operations for the first dormant user account invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 8
Claim 8 is ineligible.
Step 1
Regarding Claim 8, the rejection of Claim 7 is incorporated.
Step 2A Prong 1
Claim 8 recites the abstract ideas recited by parent Claim 7.
Step 2A Prong 2, Step 2B
The additional element performing, by the computer system, one or more risk-mitigation operations for the first dormant user account (as recited by Claim 7), wherein the one or more risk-mitigation operations include disabling particular functionality of the first dormant user account pending performance of multifactor authentication by a user of the first dormant user account invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 9
Claim 9 is ineligible.
Step 1
Regarding Claim 9, the rejection of Claim 2 is incorporated.
Step 2A Prong 1
Claim 9 recites the abstract ideas recited by parent Claim 2.
Step 2A Prong 2, Step 2B
The additional element wherein the set of most-common attributes includes a number of user accounts registered using a given IP address does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 10
Claim 10 is ineligible.
Step 1
Regarding Claim 10, the rejection of Claim 1 is incorporated.
Step 2A Prong 1
The claim recites classifying ... a new unlabeled data sample as belonging to the first category, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The additional element by the computer system ... using the updated machine learning classification model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 11
Claim 11 is ineligible.
Step 1
Claim 11 recites a non-transitory, computer-readable medium having instructions stored thereon that are executable by a computer system to perform operations, and thus the claimed process falls within a statutory category of invention.
Step 2A Prong 1
The claim recites to generate a plurality of model scores for the plurality of unlabeled data samples, wherein, for a given one of the plurality of unlabeled data samples, a corresponding model score indicates a probability that the given unlabeled data sample belongs to one of the first and second categories, which is a mental process. The claim recites selecting for classification into one of the first and second categories a subset of the plurality of unlabeled data samples based on the unlabeled data samples in the subset having model scores exceeding a particular threshold value, which is a mental process. The claim recites classifying the selected subset of the unlabeled data samples into one of the first and second categories, which is a mental process. The claim recites identifying a plurality of attributes of the set of unlabeled data samples that contributed to the model scores exceeding the particular threshold value, which is a mental process and/or a mathematical calculation. The claim recites based on the plurality of attributes, generating new labeled data samples from the subset, wherein the new labeled data samples include a first new labeled data sample indicating that a first unlabeled data sample in the selected subset belongs to the first category, which is a mental process. The claim recites updating the training dataset to include the new labeled data samples, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2
The additional element accessing a machine learning model trained using a training dataset amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering"). The additional element a training dataset that includes: a first set of labeled data samples belonging to a first category does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The claim recites a training dataset that includes: ... a second set of labeled data samples belonging to a second category does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element processing a plurality of unlabeled data samples using the machine learning model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it").
Step 2B
The additional element accessing, by a computer system, a machine learning classification model trained using a training dataset is well-understood, routine, conventional activity (see MPEP 2106.05(d), "storing and retrieving information in memory"). The additional element a training dataset that includes: a first set of labeled data samples belonging to a first category does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The claim recites a training dataset that includes: ... a second set of labeled data samples belonging to a second category does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element processing, by the computer system, a plurality of unlabeled data samples using the machine learning classification model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Claim 12, dependent on Claim 11, incorporates the rejection of Claim 11. Claim 12 incorporates substantively all the limitations of Claim 2 and is rejected under the same rationale.
Regarding Claim 13
Claim 13 is ineligible.
Step 1
Regarding Claim 13, the rejection of Claim 12 is incorporated.
Step 2A Prong 1
The claim recites prior to classifying the first unlabeled data sample, verifying the policy rule using the training dataset, which is a mental process. The claim recites wherein the verifying includes determining an accuracy of the policy rule using the first set of labeled data samples that belong to the first category, which is a mental process. The claim recites wherein generating the new labeled data sample includes assigning a particular label, to the first unlabeled data sample, that corresponds to the accuracy of the policy rule, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 14
Claim 14 is ineligible.
Step 1
Regarding Claim 14, the rejection of Claim 11 is incorporated.
Step 2A Prong 1
The claim recites identifying a first dormant user account corresponding to the first unlabeled data sample, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The additional element a training dataset that includes: a first set of labeled data samples belonging to a first category (as recited by Claim 11), wherein the plurality of unlabeled data samples correspond to a plurality of dormant user accounts, and wherein the operations further comprise does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element performing one or more risk-mitigation operations for the first dormant user account invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Claim 15, dependent on Claim 11, incorporates the rejection of Claim 11. Claim 15 incorporates substantively all the limitations of Claim 10 and is rejected under the same rationale.
Regarding Claim 16
Claim 16 is ineligible.
Step 1
Claim 16 recites a method, and thus the claimed process falls within a statutory category of invention.
Step 2A Prong 1
The claim recites to generate a plurality of model scores, wherein the plurality of unlabeled data samples correspond to a plurality of dormant user accounts with the server system, which is a mental process. The claim recites selecting ... a subset of the plurality of unlabeled data samples for classification in response to the unlabeled data samples in the subset having model scores in a particular range, which is a mental process. The claim recites classifying ... unlabeled data samples in the selected subset as one of the malicious user accounts or the non-malicious user accounts, which is a mental process. The claim recites accessing a plurality of attributes of the subset of unlabeled data samples that contributed to the corresponding model scores, which is a mental process and/or a mathematical calculation. The claim recites identifying a first unlabeled data sample in the subset using a policy rule that is based on at least one of the plurality of attributes, which is a mental process. The claim recites generating ... a new labeled data sample that labels the first unlabeled data sample as one of the malicious user accounts, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2
The additional element training, by a computer system, a classification model using a training dataset invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element a training dataset that corresponds to a plurality of active user accounts with a server system does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element the training dataset includes: a first set of labeled data samples corresponding to malicious user accounts does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The claim recites a training dataset that includes: ... a second set of labeled data samples corresponding to non-malicious user accounts does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element processing, by the computer system, a plurality of unlabeled data samples using the classification model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element accessing a plurality of attributes of the subset of unlabeled data samples that contributed to the corresponding model scores amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering"). The additional element retraining, by the computer system, the classification model using an expanded training dataset that includes the new labeled data sample invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it").
Step 2B
The additional element training, by a computer system, a classification model using a training dataset invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element a training dataset that corresponds to a plurality of active user accounts with a server system does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element the training dataset includes: a first set of labeled data samples corresponding to malicious user accounts does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The claim recites a training dataset that includes: ... a second set of labeled data samples corresponding to non-malicious user accounts does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element processing, by the computer system, a plurality of unlabeled data samples using the classification model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element accessing a plurality of attributes of the subset of unlabeled data samples that contributed to the corresponding model scores is well-understood, routine, conventional activity (see MPEP 2106.05(d), "storing and retrieving information in memory").The additional element retraining, by the computer system, the classification model using an expanded training dataset that includes the new labeled data sample invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 17
Claim 17 is ineligible.
Step 1
Regarding Claim 17, the rejection of Claim 16 is incorporated.
Step 2A Prong 1
The claim recites identifying, from the plurality of attributes, a set of most-common attributes of the set of unlabeled data samples that contributed to the corresponding model scores, wherein the at least one attribute is included in the set of most-common attributes, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The additional element training, by a computer system, a classification model using a training dataset (as recited by Claim 16) wherein the classification model is a binary classification model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 18
Claim 18 is ineligible.
Step 1
Regarding Claim 18, the rejection of Claim 17 is incorporated.
Step 2A Prong 1
The claim recites prior to classifying the first unlabeled data sample, verifying, by the computer system, the policy rule using the training dataset, which is a mental process. The claim recites wherein the verifying includes determining an accuracy of the policy rule using the first set of labeled data samples, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Claim 19, dependent on Claim 18, incorporates the rejection of Claim 18. Claim 19 incorporates substantively all the limitations of Claim 4 and is rejected under the same rationale.
Regarding Claim 20
Claim 20 is ineligible.
Step 1
Regarding Claim 20, the rejection of Claim 16 is incorporated.
Step 2A Prong 1
The claim recites identifying a first dormant user account corresponding to the first unlabeled data sample, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The additional element training, by a computer system, a classification model using a training dataset (as recited by Claim 16) wherein the computer system is included in the server system invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element performing one or more risk-mitigation operations for the first dormant user account invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 10, 11, and 15 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Kozodoi, et al., "Shallow Self-learning for Reject Inference in Credit Scoring" (hereinafter "Kozodoi").
Regarding Claim 1, Kozodoi teaches:
a method (Kozodoi, p. 2, 1 Introduction: "we introduce a novel self-learning framework for reject inference in credit scoring applications. Our framework includes two different probabilistic classifiers for the training and labeling stages" and p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference"), comprising:
... by a computer system ... (Kozodoi, p. 5, 3.1 Self-Learning for Reject Inference: "on the first iteration, we compute the corresponding score values
c
g
and
c
b
for the selected
α
%
and
α
θ
%
probability percentiles" and p. 8, 4.1 Data Description: "The empirical experiments are based on a real-world credit scoring data set on consumer micro-loans .... The data set contains 2,410 features describing the applicants, their behavior and loan characteristics. ... The data consist of 59,593 loan applications," where Kozodoi's computing of score values using the experimental data set reasonably suggests a computer system) accessing ... a machine learning classification model (Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," line 4, "Use
f
x
to predict PD [probability of default] for all unlabeled examples in
X
*
," where Kozodoi's f(x) is referred to as a classifier in line 3, "Train L1 classifier
f
x
") trained using a training dataset that includes: a first set of labeled data samples belonging to a first category; and a second set of labeled data samples belonging to a second category (Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," line 3, "Train L1 classifier
f
x
with penalty parameter
λ
using all data in
X
a
," where p. 4, 3.1 Self-Learning for Reject Inference: "In reject inference, we are given a set of n examples
x
1
,
…
,
x
n
∈
R
k
, where
k
is the number of features. Set
X
consists of
l
accepted clients
x
1
a
,
…
,
x
l
a
∈
X
a
with corresponding labels
y
1
a
,
…
,
y
l
a
∈
{
g
o
o
d
,
b
a
d
}
," where Kozodoi's good and bad correspond to the instant label categories);
processing, by the computer system, a plurality of unlabeled data samples using the machine learning classification model (Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," line 4, "Use
f
x
to predict PD [probability of default] for all unlabeled examples in
X
*
," where Kozodoi's
X
*
is unlabeled per line 2: "Set
X
*
=
X
r
" and p. 4, 3.1 Self-Learning for Reject Inference: "Set
X
consists of ... m rejected examples
x
l
r
,
…
,
x
m
r
∈
X
r
, whose labels are unknown") to generate a plurality of model scores for the plurality of unlabeled data samples (Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," lines 5-6: "Derive
c
g
such that
P
f
x
i
*
∈
X
*
<
c
g
=
,
α
is a pre-defined percentile threshold" and "Derive
c
b
such that
P
f
x
i
*
∈
X
*
<
c
g
=
α
θ
,
θ
is the imbalance parameter," where the unlabeled samples
x
i
*
are scored by
f
x
and compared to threshold scores, as in p. 5, 3.1 Self-Learning for Reject Inference: "We suggest using the combined approach: on the first iteration, we compute the corresponding score values
c
g
and
c
b
for the selected
α
%
and
α
θ
%
probability percentiles"), wherein, for a given one of the plurality of unlabeled data samples, a corresponding model score indicates a probability that the given unlabeled data sample belongs to one of the first and second categories (Kozodoi, p. 5, 3.1 Self-Learning for Reject Inference: "We suggest using the combined approach: on the first iteration, we compute the corresponding score values
c
g
and
c
b
for the selected
α
%
and
α
θ
%
probability percentiles," where p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," line 9 appends a sample over the threshold to the good/bad labeled dataset);
selecting, by computer system, a subset of the plurality of unlabeled data samples for classification into one of the first and second categories (Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," lines 8-9: "Remove examples in
X
*
from
X
r
" and "Append examples in
X
*
to
X
a
"), wherein the subset of unlabeled data samples are selected based on the unlabeled data samples in the subset having model scores exceeding a particular threshold value (Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," line 7: "Select a subset of examples
X
*
⊂
X
r
such that
f
x
i
*
∈
X
*
<
c
g
or
f
x
i
*
∈
X
*
>
c
b
");
classifying, by the computer system, the selected subset of the unlabeled data samples into one of the first and second categories (Kozodoi, p. 5, 3.1 Self-Learning for Reject Inference: "On each labeling iteration, we only select the top
α
%
of the good loans and top
α
θ
%
of the bad loans among rejects for labeling. Keeping only the top-ranked instances ensures that we append rejects with high confidence in the assigned labels, reducing the potential amount of noise"), including by:
identifying a plurality of attributes of the selected subset of unlabeled data samples that contributed to the model scores exceeding the particular threshold value (Kozodoi, p. 4, 3.1 Self-Learning for Reject Inference: "we propose using a weak learner for labeling rejects because of its ability to produce better-calibrated predictions [23]. In this paper, we rely on L1-regularized logistic regression (L1) to label rejects. ¶ ... [A]dding regularization to logistic regression is important as we are dealing with high-dimensional data with noisy features" and p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," line 3: "Train L1 classifier
f
x
with penalty parameter
λ
using all data in
X
a
," where Kozodoi's regularization for noisy data corresponds to the instant identifying attributes, where Kozoi's features correspond to the instant attributes, as in p. 4, 3.1 Self-Learning for Reject Inference: "In reject inference, we are given a set of
n
examples
x
1
,
…
,
x
n
∈
R
k
, where
k
is the number of features");
based on the plurality of attributes, generating new labeled data samples from the subset, wherein the new labeled data samples include a first new labeled data sample (Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," lines 8-9: "Remove examples in
X
*
from
X
r
" and "Append examples in
X
*
to
X
a
", where Kozodoi's
X
a
are labeled samples, and where the instant specification indicates a step of labelling as a step of generation, as in [0051]: " generating the new labeled data sample includes assigning the first unlabeled data sample with a label") indicating that a first unlabeled data sample in the selected subset belongs to the first category (Kozodoi, p. 5, 3.1 Self-Learning for Reject Inference: "On each labeling iteration, we only select the top
α
%
of the good loans and top
α
θ
%
of the bad loans among rejects for labeling"); and
updating, by the computer system, the training dataset to include the new labeled data samples (Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," lines 8-9: "Remove examples in
X
*
from
X
r
" and "Append examples in
X
*
to
X
a
"); and
training, by the computer system, the machine learning classification model based on the updated training dataset to generate an updated machine learning classification model (Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," line 3, "Train L1 classifier
f
x
... using all data in
X
a
," where on subsequent iterations
X
a
is the labeled dataset updated on line 9).
Regarding Claim 11, Kozodoi teaches:
A non-transitory, computer-readable medium having instructions stored thereon that are executable by a computer system (Kozodoi, p. 5, 3.1 Self-Learning for Reject Inference: "on the first iteration, we compute the corresponding score values
c
g
and
c
b
for the selected
α
%
and
α
θ
%
probability percentiles" and p. 8, 4.1 Data Description: "The empirical experiments are based on a real-world credit scoring data set on consumer micro-loans .... The data set contains 2,410 features describing the applicants, their behavior and loan characteristics. ... The data consist of 59,593 loan applications," where Kozodoi's computing of score values using the experimental data set reasonably suggests a computer system, and where a non-transitory computer readable medium is inherent in storing a data set for computation) to perform operations comprising precisely those steps recited by the method of Claim 1. Claim 11 is rejected under the same rationale as Claim 1.
Regarding Claim 10, the rejection of Claim 1 is incorporated. Kozodoi teaches:
classifying, by the computer system, a new unlabeled data sample as belonging to the first category using the updated machine learning classification model (Kozodoi, p. 5, 3.1 Self-Learning for Reject Inference: "On each labeling iteration, we only select the top
α
%
of the good loans and top
α
θ
%
of the bad loans among rejects for labeling. Keeping only the top-ranked instances ensures that we append rejects with high confidence in the assigned labels, reducing the potential amount of noise," where a sample filtered for labeling on a subsequent iteration corresponds to the instant new sample).
Regarding Claim 15, the rejection of Claim 11 is incorporated. Kozodoi teaches:
training, by the computer system, the machine learning classification model based on the updated training dataset to generate an updated machine learning classification model (Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," line 3, "Train L1 classifier
f
x
... using all data in
X
a
," where on subsequent iterations
X
a
is the labeled dataset updated on line 9); and
classifying, by the computer system, a new unlabeled data sample as belonging to the first category using the updated machine learning classification model (Kozodoi, p. 5, 3.1 Self-Learning for Reject Inference: "On each labeling iteration, we only select the top
α
%
of the good loans and top
α
θ
%
of the bad loans among rejects for labeling. Keeping only the top-ranked instances ensures that we append rejects with high confidence in the assigned labels, reducing the potential amount of noise," where a sample filtered for labeling on a subsequent iteration corresponds to the instant new sample).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 2, 5, 6, 12, 16, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Kozodoi, et al., "Shallow Self-learning for Reject Inference in Credit Scoring" (hereinafter "Kozodoi ") in view of Dai, et al., "SSL Malicious Traffic Detection Based On Multi-view Features" (hereinafter "Dai").
Regarding Claim 16, Kozodoi teaches:
A method (Kozodoi, p. 2, 1 Introduction: "we introduce a novel self-learning framework for reject inference in credit scoring applications. Our framework includes two different probabilistic classifiers for the training and labeling stages" and p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference"), comprising:
training, by a computer system, a classification model using a training dataset ..., wherein the training dataset includes: a first set of labeled data samples ...; and a second set of labeled data samples ... (Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," line 3, "Train L1 classifier
f
x
with penalty parameter
λ
using all data in
X
a
," where p. 4, 3.1 Self-Learning for Reject Inference: "In reject inference, we are given a set of n examples
x
1
,
…
,
x
n
∈
R
k
, where
k
is the number of features. Set
X
consists of
l
accepted clients
x
1
a
,
…
,
x
l
a
∈
X
a
with corresponding labels
y
1
a
,
…
,
y
l
a
∈
{
g
o
o
d
,
b
a
d
}
," where Kozodoi's good and bad correspond to the instant label categories);
processing, by the computer system, a plurality of unlabeled data samples using the classification model to generate a plurality of model scores ... (Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," line 4, "Use
f
x
to predict PD [probability of default] for all unlabeled examples in
X
*
," where Kozodoi's
X
*
is unlabeled per line 2: "Set
X
*
=
X
r
" and p. 4, 3.1 Self-Learning for Reject Inference: "Set
X
consists of ... m rejected examples
x
l
r
,
…
,
x
m
r
∈
X
r
, whose labels are unknown");
selecting, by computer system, a subset of the plurality of unlabeled data samples for classification (Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," lines 8-9: "Remove examples in
X
*
from
X
r
" and "Append examples in
X
*
to
X
a
") in response to the unlabeled data samples in the subset having model scores in a particular range (Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," line 7: "Select a subset of examples
X
*
⊂
X
r
such that
f
x
i
*
∈
X
*
<
c
g
or
f
x
i
*
∈
X
*
>
c
b
");
classifying, by the computer system, unlabeled data samples in the selected subset (Kozodoi, p. 5, 3.1 Self-Learning for Reject Inference: "On each labeling iteration, we only select the top
α
%
of the good loans and top
α
θ
%
of the bad loans among rejects for labeling. Keeping only the top-ranked instances ensures that we append rejects with high confidence in the assigned labels, reducing the potential amount of noise) ... including by:
accessing a plurality of attributes of the subset of unlabeled data samples that contributed to the corresponding model scores (Kozodoi, p. 4, 3.1 Self-Learning for Reject Inference: "we propose using a weak learner for labeling rejects because of its ability to produce better-calibrated predictions [23]. In this paper, we rely on L1-regularized logistic regression (L1) to label rejects. ¶ ... [A]dding regularization to logistic regression is important as we are dealing with high-dimensional data with noisy features" and p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," line 3: "Train L1 classifier
f
x
with penalty parameter
λ
using all data in
X
a
," where Kozodoi's regularization for noisy data corresponds to the instant identifying attributes, where Kozoi's features correspond to the instant attributes, as in p. 4, 3.1 Self-Learning for Reject Inference: "In reject inference, we are given a set of
n
examples
x
1
,
…
,
x
n
∈
R
k
, where
k
is the number of features"); and
identifying a first unlabeled data sample in the subset using a policy rule that is based on at least one of the plurality of attributes (Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," line 7, assuming a subsequent iteration of self-learning: "Select a subset of examples
X
*
⊂
X
r
such that
f
x
i
*
∈
X
*
<
c
g
or
f
x
i
*
∈
X
*
>
c
b
," where Kozodoi's use of
c
g
and
c
b
thresholds corresponds to the instant application of a policy rule, as in p. 5, 3.1 Self-Learning for Reject Inference: "On each labeling iteration, we only select the top
α
%
of the good loans and top
α
θ
%
of the bad loans among rejects for labeling. Keeping only the top-ranked instances ensures that we append rejects with high confidence in the assigned labels, reducing the potential amount of noise)");
generating, by the computer system, a new labeled data sample that labels the first unlabeled data sample ... (Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," lines 8-9: "Remove examples in
X
*
from
X
r
" and "Append examples in
X
*
to
X
a
", where Kozodoi's
X
a
are labeled samples, and where the instant specification indicates a step of labelling as a step of generation, as in [0051]: " generating the new labeled data sample includes assigning the first unlabeled data sample with a label"); and
retraining, by the computer system, the classification model using an expanded training dataset that includes the new labeled data sample (Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," line 3, "Train L1 classifier
f
x
... using all data in
X
a
," where on subsequent iterations
X
a
is the labeled dataset updated on line 9).
Kozodoi teaches a training dataset that that includes a first set of labeled data samples and a second set of labeled data samples, classifying unlabeled samples, and generating new labeled samples.
Kozodoi does not teach a training dataset that corresponds to a plurality of active user accounts with a server system, wherein the training dataset includes: a first set of labeled data samples corresponding to malicious user accounts; and a second set of labeled data samples corresponding to non-malicious user accounts; the plurality of unlabeled data samples correspond to a plurality of dormant user accounts with the server system; classifying, by the computer system, unlabeled data samples in the selected subset as one of the malicious user accounts or the non-malicious user accounts; generating, by the computer system, a new labeled data sample that labels the first unlabeled data sample as one of the malicious user accounts.
However, Dai teaches:
a training dataset that corresponds to a plurality of active user accounts with a server system, wherein the training dataset includes: a first set of labeled data samples corresponding to malicious user accounts; and a second set of labeled data samples corresponding to non-malicious user accounts (Dai, p. 44, 4.1 Dataset Processing and Feature Extraction: "After the fusion analysis of the datasets, we generate 11,607 normal four-tuples and 12,894 malicious four-tuples. We use 80% of the four-tuples as a training set and 20% as a validation set. In addition, we generate another new 1812 normal four tuples and 979 malicious four tuples as a test set, which is not contained in the training set, to verify the generalization ability of the model");
the plurality of unlabeled data samples correspond to a plurality of dormant user accounts with the server system (Dai, p. 43, 4.1 Dataset Processing and Feature Extraction: "The Dataset used in this paper is the CTU Malware Dataset.... The labels include botnet phase, attack, normal, and background. ... We use 80% of the four-tuples as a training set and 20% as a validation set. In addition, we generate another new 1812 normal four tuples and 979 malicious four tuples as a test set, which is not contained in the training set, to verify the generalization ability of the model," where Dai's botnet includes zombies and corresponds the instant dormant user account, as in p. 42, 3.2 Multi-view Feature Extraction: "In botnets, the C&C server typically uses software or programs to control zombies, so traffic between the C&C server and zombies typically exhibits machine behavior that is significantly different from normal daily traffic")
classifying, by the computer system, unlabeled data samples in the selected subset as one of the malicious user accounts or the non-malicious user accounts (Dai, p. 43, 4.1 Dataset Processing and Feature Extraction: "We use 80% of the four-tuples as a training set and 20% as a validation set. In addition, we generate another new 1812 normal four tuples and 979 malicious four tuples as a test set, which is not contained in the training set, to verify the generalization ability of the model," where Dai's normal and malicious test data corresponds the instant non-malicious and malicious accounts, respectively),
generating, by the computer system, a new labeled data sample that labels the first unlabeled data sample as one of the malicious user accounts (Dai, p. 44, 4.2 Indicators and Experimental Settings: "Experiment 3 compares the detection effects of the four machine algorithms on new SSL malicious traffic for testing the generalization ability of the models" and p. 45, 4.3.3 New SSL malicious traffic detection (Experiment 3): "We use a malicious dataset and a normal dataset from the CTU Malware Dataset as the test datasets, which are not in the training set and validation set. Table 6 shows the detection effect of the four models on the new SSL malicious traffic," where Dai's test-time detection corresponds to the instant new labeled sample).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Kozodoi regarding a training dataset that that includes a first set of labeled data samples and a second set of labeled data samples, classifying unlabeled samples, and generating new labeled samples with those of Dai regarding a training dataset that corresponds to a plurality of active user accounts with a server system, wherein the training dataset includes: a first set of labeled data samples corresponding to malicious user accounts; and a second set of labeled data samples corresponding to non-malicious user accounts; the plurality of unlabeled data samples correspond to a plurality of dormant user accounts with the server system; classifying, by the computer system, unlabeled data samples in the selected subset as one of the malicious user accounts or the non-malicious user accounts; generating, by the computer system, a new labeled data sample that labels the first unlabeled data sample as one of the malicious user accounts.
The motivation to do so would be to facilitate collecting communication statistics reflective of zombie activity and identify the activity (Dai, p. 40, Abstract: "Our method comprehensively extracts features from multiple views, including flow statistics" and p. 42, 1. Features from the flow statistics: "In botnets, the C&C server typically uses software or programs to control zombies, so traffic between the C&C server and zombies typically exhibits machine behavior that is significantly different from normal daily traffic. ... Above all, we extracted 17-dimensional flow statistics features from multiple perspectives, as shown in Table 1").
Regarding Claim 2, the rejection of Claim 1 is incorporated. Kozodoi teaches:
selecting, from the plurality of attributes, a set of ... attributes of the selected subset of unlabeled data samples (Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference, line 3: "Train L1 classifier
f
x
with penalty parameter
λ
using all data in
X
a
," where Kozodoi's application of penalty parameter
λ
corresponds to the instant selection by a computer system, and p. 4, 3.1 Self-Learning for Reject Inference: "we propose using a weak learner for labeling rejects because of its ability to produce better-calibrated predictions [23]. In this paper, we rely on L1-regularized logistic regression (L1) to label rejects. ¶ ... [A]dding regularization to logistic regression is important as we are dealing with high-dimensional data with noisy features," and where Kozoi's features correspond to the instant attributes, as in p. 4, 3.1 Self-Learning for Reject Inference: "In reject inference, we are given a set of
n
examples
x
1
,
…
,
x
n
∈
R
k
, where
k
is the number of features") that contributed to the model scores exceeding the particular threshold value (Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference, line 4, "Use
f
x
to predict PD [probability of default] for all unlabeled examples in
X
*
" and line 7, where
f
x
is compared to thresholds); and
applying a policy rule to the selected subset of unlabeled data samples to identify the first unlabeled data sample, wherein the policy rule is based on at least one of the set of ... attributes of the selected subset of unlabeled data samples (Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," line 7, assuming a subsequent iteration of self-learning: "Select a subset of examples
X
*
⊂
X
r
such that
f
x
i
*
∈
X
*
<
c
g
or
f
x
i
*
∈
X
*
>
c
b
," where Kozodoi's use of
c
g
and
c
b
thresholds corresponds to the instant application of a policy rule, as in p. 5, 3.1 Self-Learning for Reject Inference: "On each labeling iteration, we only select the top
α
%
of the good loans and top
α
θ
%
of the bad loans among rejects for labeling. Keeping only the top-ranked instances ensures that we append rejects with high confidence in the assigned labels, reducing the potential amount of noise)").
Kozodoi teaches selecting a set of attributes of unlabeled data samples that contributed to the model scores exceeding the particular threshold value and applying a policy rule to the selected subset of unlabeled data samples.
Kozodoi does not teach selecting ... a set of most-common attributes of the selected subset of unlabeled data samples and applying a policy rule ..., wherein the policy rule is based on at least one of the set of most-common attributes.
However, Dai teaches:
selecting ... a set of most-common attributes of the selected subset of unlabeled data samples (Dai, p. 43, Table 2 the features of the SSL handshake field, listing selected features for distinguishing normal and malicious traffic, and p. 43, 2. Features from the SSL handshake fields: "Anderson et al. [8] found that most enterprise traffics use the 512-bit ECDHE_RSA public key, while malwares often use the 2048-bit DHE_RSA public key. The value of the extension field in the normal software is diversified, and some malicious clients always only use 0x000d as the signature_algorithms extension field. ... So the values of some handshake fields in the malware traffic are obviously different from the normal. As shown in Table 2, we extracted the 7-dimensional features from the handshake fields," where Dai's feature selection according to certificate information corresponds to the instant most-common attributes)
applying a policy rule ..., wherein the policy rule is based on at least one of the set of most-common attributes (Dai, p. 43, 4.1 Dataset Processing and Feature Extraction: "The Dataset used in this paper is the CTU Malware Dataset .... There are now more than 400 sub datasets, consisting of botnets and normal networks. ¶ The dataset provides parsed log files ... including conn.log, ssl.log, x509.log, etc. ... The x509.log records the specific contents of the x509 certificate, such as ... the signature algorithm and the SAN. ¶ ... [W]e take the four-tuple ID as the unique identification, and combine the above three types of information as the analysis unit of feature extraction. ¶ After the fusion analysis of the datasets, we generate 11,607 normal four-tuples and 12,894 malicious four-tuples. We use 80% of the four-tuples as a training set," where Dai's use of selected certificate features corresponds to the instant application of the policy rule).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Kozodoi regarding selecting a set of attributes of unlabeled data samples that contributed to the model scores exceeding the particular threshold value and applying a policy rule to the selected subset of unlabeled data samples with those of Dai regarding selecting a set of most-common attributes of the selected subset of unlabeled data samples and applying a policy rule based on at least one of the set of most-common attributes.
The motivation to do so would be to facilitate training of a model with greater generalization ability (Dai p. 43, 3.3 Feature Selection: "In order to improve the performance of the model, we use the Mutual Information algorithm to eliminate the irrelevant or redundant features. Feature selection can reduce features and improve the generalization ability of the model").
Claim 12 incorporates substantively all limitations of Claim 2 in non-transitory computer-readable medium form and is rejected under the same rationale.
Regarding Claim 5, the rejection of Claim 2 is incorporated.
Dai further teaches:
generating, by the computer system, the policy rule based on a combination of one or more of the set of most-common attributes of the subset of unlabeled data samples (Dai, p. 43, 3.3 Feature Selection: "In order to improve the performance of the model, we use the Mutual Information algorithm to eliminate the irrelevant or redundant features. ... Mutual information is used to measure the association between two variables. ... We can get the mutual information between the feature 𝑥𝑖 and the traffic category y which has the value of malicious or normal using Eq. (1), and the calculation result is shown in Figure 2. Most features contain useful information and play a positive role in category determination," where Dai's mutual information calculation corresponds to the instant combination of attributes).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Kozodoi/Dai combination regarding applying a policy rule to the unlabeled data to identify the first unlabeled data sample with the further teachings of Dai regarding generating the new labeled data sample with the further teachings of Dai regarding generating the policy rule based on a combination of one or more of the set of most-common attributes of the subset of unlabeled data samples.
The motivation to do so would be to facilitate training of a model with greater generalization ability (Dai, p. 43, 3.3 Feature Selection: "In order to improve the performance of the model, we use the Mutual Information algorithm to eliminate the irrelevant or redundant features. Feature selection can reduce features and improve the generalization ability of the model").
Regarding Claim 6, the rejection of Claim 1 is incorporated.
Kozodoi teaches a training dataset that includes a first set of labeled data samples belonging to a first category and a second set of labeled data samples belonging to a second category.
Kozodoi does not teach wherein the training dataset corresponds to a plurality of active user accounts with a server system, and wherein the plurality of unlabeled data samples correspond to a plurality of dormant user accounts with the server system.
However, Dai teaches:
wherein the training dataset corresponds to a plurality of active user accounts with a server system (Dai, p. 41, 1. Introduction: "We design a feature extraction method for SSL protocol. We analyze the macro flow statistics and key SSL contents between clients and servers without decrypting the SSL traffic" and p. 43, 4.1 Dataset Processing and Feature Extraction: "The Dataset used in this paper is the CTU Malware Dataset [17], which comes from the Malware Capture Facility Project of the Czech Technical University ATG Group [18]. This group gets traffics by executing real malware for a long time, and each piece of traffic is labeled manually. The labels include botnet phase, attack, normal, and background. There are now more than 400 sub datasets, consisting of botnets and normal networks. ¶ ... After the fusion analysis of the datasets, we generate 11,607 normal four-tuples and 12,894 malicious four-tuples. We use 80% of the four-tuples as a training set," where Dai's normal four-tuples reasonably suggests to the instant active user account training data), and wherein the plurality of unlabeled data samples correspond to a plurality of dormant user accounts with the server system (Dai, p. 43, 4.1 Dataset Processing and Feature Extraction: "The Dataset used in this paper is the CTU Malware Dataset [17], which comes from the Malware Capture Facility Project of the Czech Technical University ATG Group [18]. ... The labels include botnet phase, attack, normal, and background. ... [W]e generate another new 1812 normal four tuples and 979 malicious four tuples as a test set, which is not contained in the training set," where Dai's botnet includes zombies and corresponds the instant dormant user account, as in p. 42, 3.2 Multi-view Feature Extraction: "In botnets, the C&C server typically uses software or programs to control zombies, so traffic between the C&C server and zombies typically exhibits machine behavior that is significantly different from normal daily traffic").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Kozodoi regarding a training dataset that includes a first set of labeled data samples belonging to a first category and a second set of labeled data samples belonging to a second category with those of Dai regarding wherein the training dataset corresponds to a plurality of active user accounts with a server system, and wherein the plurality of unlabeled data samples correspond to a plurality of dormant user accounts with the server system.
The motivation to do so would be to facilitate collecting communication statistics distinguishing zombie from normal activity(Dai, p. 40, Abstract: "Our method comprehensively extracts features from multiple views, including flow statistics" and p. 42, 1. Features from the flow statistics: "In botnets, the C&C server typically uses software or programs to control zombies, so traffic between the C&C server and zombies typically exhibits machine behavior that is significantly different from normal daily traffic. ... Above all, we extracted 17-dimensional flow statistics features from multiple perspectives, as shown in Table 1").
Regarding Claim 17, the rejection of Claim 16 is incorporated. The Kozodoi/Dai combination teaches:
wherein the classification model is a binary classification model (Kozodoi, p. 4, 3.1 Self-Learning for Reject Inference: "In reject inference, we are given a set of n examples
x
1
,
…
,
x
n
∈
R
k
, where
k
is the number of features. Set
X
consists of
l
accepted clients
x
1
a
,
…
,
x
l
a
∈
X
a
with corresponding labels
y
1
a
,
…
,
y
l
a
∈
{
g
o
o
d
,
b
a
d
}
").
Dai further teaches:
wherein the classifying the first unlabeled data sample further includes: identifying, from the plurality of attributes, a set of most-common attributes of the subset of unlabeled data samples that contributed to the corresponding model scores (Dai, p. 43, Table 2 the features of the SSL handshake field, listing selected features for distinguishing normal and malicious traffic, and p. 43, 2. Features from the SSL handshake fields: "Anderson et al. [8] found that most enterprise traffics use the 512-bit ECDHE_RSA public key, while malwares often use the 2048-bit DHE_RSA public key. The value of the extension field in the normal software is diversified, and some malicious clients always only use 0x000d as the signature_algorithms extension field. ... So the values of some handshake fields in the malware traffic are obviously different from the normal. As shown in Table 2, we extracted the 7-dimensional features from the handshake fields," where Dai's feature selection according to certificate information corresponds to the instant most-common attributes), wherein the at least one attribute is included in the set of most-common attributes (Dai, p. 43, 4.1 Dataset Processing and Feature Extraction: "The Dataset used in this paper is the CTU Malware Dataset .... There are now more than 400 sub datasets, consisting of botnets and normal networks. ¶ The dataset provides parsed log files ... including conn.log, ssl.log, x509.log, etc. ... The x509.log records the specific contents of the x509 certificate, such as ... the signature algorithm and the SAN. ¶ ... [W]e take the four-tuple ID as the unique identification, and combine the above three types of information as the analysis unit of feature extraction. ¶ After the fusion analysis of the datasets, we generate 11,607 normal four-tuples and 12,894 malicious four-tuples. We use 80% of the four-tuples as a training set," where Dai's use of selected certificate features corresponds to the instant application of the policy rule).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Kozodoi regarding selecting a set of attributes of unlabeled data samples that contributed to the model scores exceeding the particular threshold value and applying a policy rule to the selected subset of unlabeled data samples with those of Dai regarding selecting a set of most-common attributes of the selected subset of unlabeled data samples and applying a policy rule based on at least one of the set of most-common attributes.
The motivation to do so would be to facilitate training of a model with greater generalization ability (Dai p. 43, 3.3 Feature Selection: "In order to improve the performance of the model, we use the Mutual Information algorithm to eliminate the irrelevant or redundant features. Feature selection can reduce features and improve the generalization ability of the model").
Claims 3, 4, 13, 18, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Kozodoi, et al., "Shallow Self-learning for Reject Inference in Credit Scoring" (hereinafter "Kozodoi ") in view of Dai, et al., "SSL Malicious Traffic Detection Based On Multi-view Features" (hereinafter "Dai") in view of Stange, et al., "An Adaptive Algorithm for Rule Learning: Case Study and Preliminary Results" (hereinafter "Stange").
Regarding Claim 3, the rejection of Claim 2 is incorporated. Dai further teaches:
prior to classifying the first unlabeled data sample, verifying, by the computer system, the policy rule using the training dataset (Dai, p. 42, 3.1 Overview: "The features from flow statistics, SSL handshake fields, and certificates complement each other. They can retain enough original information. Therefore, we first extract features from SSL traffic based on these three perspectives" and p. 43, 3.3 Feature Selection: "In order to improve the performance of the model, we use the Mutual Information algorithm to eliminate the irrelevant or redundant features. ... We can get the mutual information between the feature 𝑥𝑖 and the traffic category y which has the value of malicious or normal using Eq. (1), and the calculation result is shown in Figure 2. Most features contain useful information and play a positive role in category determination," where Dai's calculating mutual information corresponds to the instant verifying).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Kozodoi/Dai combination regarding applying a policy rule to the unlabeled data to identify the first unlabeled data sample with the further teachings of Dai regarding prior to classifying the first unlabeled data sample, verifying, by the computer system, the policy rule using the training dataset.
The motivation to do so would be to facilitate training of a model with greater generalization ability (Dai, p. 43, 3.3 Feature Selection: "In order to improve the performance of the model, we use the Mutual Information algorithm to eliminate the irrelevant or redundant features. Feature selection can reduce features and improve the generalization ability of the model").
The Kozodoi/Dai combination does not teach wherein the verifying includes determining an accuracy of the policy rule using the first set of labeled data samples that belong to the first category.
However, Stange teaches:
wherein the verifying includes determining an accuracy of the policy rule (Stange, p. 47, 3 An adaptivity-based rule learning technique: "The resulting classifier can contain three types of rules: (a) rules with 100% accuracy (higher rank), (b) rules with good accuracy (lower rank), i.e. < 100% and > 50%, and (c) rules with poor accuracy (mean rank), i.e. <= 50%. When searching for an applicable rule, the algorithm goes over the rules in a topdown fashion starting with the primary rules (higher rank) until reaching rules with a poor rank") using the first set of labeled data samples that belong to the first category (Stange, p. 45, 3 An adaptivity-based rule learning technique: "We propose a modification to this subroutine, such that it now returns different rules that cover both positive and negative examples (we have also renamed such subroutine to LEARN-RULES). We also treat the set of examples differently, such that Learn-rules is invoked on all available training examples" where Stange's Algorithm 4 is LEARN-RULES).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Kozodoi/Dai combination regarding prior to classifying the first unlabeled data sample, verifying, by the computer system, the policy rule using the training dataset with those of Stange regarding wherein the verifying includes determining an accuracy of the policy rule using the first set of labeled data samples that belong to the first category.
The motivation to do so would be to facilitate an adaptive learning technique that provides adding rules with favorable rule coverage (Stange, p. 52, 5 Final remarks: "Our approach is based on modifications of conventional sequential covering strategies in order to adapt acquired knowledge during the learning process. Studies indicate that adaptive techniques confer a more adequate rule set fitting").
Regarding Claim 4, the rejection of Claim 3 is incorporated.
The Kozodoi/Dai/Stange combination has been shown to teach:
generating the new labeled data samples (as recited in the rejection of Claim 1, Kozodoi, p. 6, Algorithm 2, "Shallow Self-Learning for Reject Inference," lines 8-9: "Remove examples in
X
*
from
X
r
" and "Append examples in
X
*
to
X
a
", where Kozodoi's
X
a
are labeled samples, and where the instant specification indicates a step of labelling as a step of generation, as in [0051]: " generating the new labeled data sample includes assigning the first unlabeled data sample with a label").
Stange further teaches:
assigning a particular label, to the first unlabeled data sample, that corresponds to the accuracy of the policy rule (Stange, p. 47, 3 An adaptivity-based rule learning technique: "The resulting classifier can contain three types of rules: (a) rules with 100% accuracy (higher rank), (b) rules with good accuracy (lower rank), i.e. < 100% and > 50%, and (c) rules with poor accuracy (mean rank), i.e. <= 50%. When searching for an applicable rule, the algorithm goes over the rules in a topdown fashion starting with the primary rules (higher rank) until reaching rules with a poor rank," where Stange's rules are selected and applied according to a ranking of descending order of accuracy).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Kozodoi/Dai/Stange combination regarding generating the new labeled data samples with the further teachings of Stange regarding assigning a particular label, to the first unlabeled data sample, that corresponds to the accuracy of the policy rule.
The motivation to do so would be to facilitate implementing a classification procedure that favors rule accuracy over rule coverage (Stange, p. 52, 4 An illustrative example: "The algorithm mainly focus on maximizing the rule accuracy, even when the discovered rules covers one sample from the training data").
Claim 13 incorporates substantively all limitations of Claims 3 and 4 in non-transitory computer-readable medium form and is rejected under the same rationales.
Claim 19 incorporates substantively all limitations of Claim 4 in method form and is rejected under the same rationale.
Regarding Claim 18, the rejection of Claim 17 is incorporated. Dai further teaches:
prior to classifying the first unlabeled data sample, verifying, by the computer system, the policy rule using the training dataset (Dai, p. 42, 3.1 Overview: "The features from flow statistics, SSL handshake fields, and certificates complement each other. They can retain enough original information. Therefore, we first extract features from SSL traffic based on these three perspectives" and p. 43, 3.3 Feature Selection: "In order to improve the performance of the model, we use the Mutual Information algorithm to eliminate the irrelevant or redundant features. ... We can get the mutual information between the feature 𝑥𝑖 and the traffic category y which has the value of malicious or normal using Eq. (1), and the calculation result is shown in Figure 2. Most features contain useful information and play a positive role in category determination," where Dai's calculating mutual information corresponds to the instant verifying).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Kozodoi/Dai combination regarding applying a policy rule to the unlabeled data to identify the first unlabeled data sample with the further teachings of Dai regarding prior to classifying the first unlabeled data sample, verifying, by the computer system, the policy rule using the training dataset.
The motivation to do so would be to facilitate training of a model with greater generalization ability (Dai, p. 43, 3.3 Feature Selection: "In order to improve the performance of the model, we use the Mutual Information algorithm to eliminate the irrelevant or redundant features. Feature selection can reduce features and improve the generalization ability of the model").
The Kozodoi/Dai combination does not teach wherein the verifying includes determining an accuracy of the policy rule using the first set of labeled data samples that belong to the first category.
However, Stange teaches:
wherein the verifying includes determining an accuracy of the policy rule (Stange, p. 47, 3 An adaptivity-based rule learning technique: "The resulting classifier can contain three types of rules: (a) rules with 100% accuracy (higher rank), (b) rules with good accuracy (lower rank), i.e. < 100% and > 50%, and (c) rules with poor accuracy (mean rank), i.e. <= 50%. When searching for an applicable rule, the algorithm goes over the rules in a topdown fashion starting with the primary rules (higher rank) until reaching rules with a poor rank") using the first set of labeled data samples (Stange, p. 45, 3 An adaptivity-based rule learning technique: "We propose a modification to this subroutine, such that it now returns different rules that cover both positive and negative examples (we have also renamed such subroutine to LEARN-RULES). We also treat the set of examples differently, such that Learn-rules is invoked on all available training examples" where Stange's Algorithm 4 is LEARN-RULES).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Kozodoi/Dai combination regarding prior to classifying the first unlabeled data sample, verifying, by the computer system, the policy rule using the training dataset with those of Stange regarding wherein the verifying includes determining an accuracy of the policy rule using the first set of labeled data samples.
The motivation to do so would be to facilitate an adaptive learning technique that provides adding rules with favorable rule coverage (Stange, p. 52, 5 Final remarks: "Our approach is based on modifications of conventional sequential covering strategies in order to adapt acquired knowledge during the learning process. Studies indicate that adaptive techniques confer a more adequate rule set fitting").
Claims 7, 8, 14, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Kozodoi, et al., "Shallow Self-learning for Reject Inference in Credit Scoring" (hereinafter "Kozodoi ") in view of Dai, et al., "SSL Malicious Traffic Detection Based On Multi-view Features" (hereinafter "Dai") in view of Sadaghiani, et al. (US 2019/0108334 A1, hereinafter "Sadaghiani").
Regarding Claim 7, the rejection of Claim 6 is incorporated. Dai further teaches:
wherein the first category corresponds to malicious user accounts and the second category corresponds to non-malicious user accounts (Dai, p. 43, 4.1 Dataset Processing and Feature Extraction: "The Dataset used in this paper is the CTU Malware Dataset .... The labels include botnet phase, attack, normal, and background. There are now more than 400 sub datasets, consisting of botnets and normal networks. ¶ ... After the fusion analysis of the datasets, we generate 11,607 normal four-tuples and 12,894 malicious four-tuples. We use 80% of the four-tuples as a training set," where Dai's training data reasonably suggests malicious and non-malicious user accounts as the first and second categories, respectively), wherein the method further comprises:
identifying, by the computer system, a first dormant user account corresponding to the first unlabeled data sample (Dai, Table 6, "the detection result of new SSL malicious traffic," showing identification of zombie/dotnet accounts with the given performance statistics, and p. 45, 4.3.3 New SSL malicious traffic detection (Experiment 3): "We use a malicious dataset and a normal dataset from the CTU Malware Dataset as the test datasets, which are not in the training set and validation set. Table 6 shows the detection effect of the four models on the new SSL malicious traffic. ... Putting all the detection indicators together, the XGBoost model is the best, especially accuracy is 74.95% and precision is 69.66%, which proves that this method is effective in detecting new SSL malicious traffic").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Kozodoi/Dai combination regarding a first set of labeled data samples belonging to a first category and a second set of labeled data samples belonging to a second category with the further teachings of Dai regarding wherein the first category corresponds to malicious user accounts and the second category corresponds to non-malicious user accounts and identifying a first dormant user account corresponding to the first unlabeled data sample.
The motivation to do so would be to facilitate collecting communication statistics distinguishing zombie from normal activity that supports threat detection (Dai, p. 40, Abstract: "Our method comprehensively extracts features from multiple views, including flow statistics" and p. 42, 1. Features from the flow statistics: "In botnets, the C&C server typically uses software or programs to control zombies, so traffic between the C&C server and zombies typically exhibits machine behavior that is significantly different from normal daily traffic. ... Above all, we extracted 17-dimensional flow statistics features from multiple perspectives, as shown in Table 1").
The Kozodoi/Dai combination teaches wherein the first category corresponds to malicious user accounts and the second category corresponds to non-malicious user accounts and identifying, by the computer system, a first dormant user account corresponding to the first unlabeled data sample.
The Kozodoi/Dai combination does not explicitly teach performing, by the computer system, one or more risk-mitigation operations for the first dormant user account.
However, Sadaghiani teaches:
performing ... one or more risk-mitigation operations for the ... dormant user account (Sadaghiani, [0076]: "the intelligent API may enable a user or service provider to specifically select an ... ATO risk assessment and perform one or more ATO risk mitigation processes," where Sadaghiani's risk assessment may be undertaken on a suspended account corresponding to the instant dormant account, as in [0020]: "the embodiments of the present application function to collect account activity data associated with an account or a suspected hacked account to determine a likelihood that the account or suspected hacked account may be appropriated or accessed by a malicious actor").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Kozodoi/Dai combination regarding wherein the first category corresponds to malicious user accounts and the second category corresponds to non-malicious user accounts and identifying a first dormant user account corresponding to the first unlabeled data sample with those of Sadaghiani regarding performing one or more risk-mitigation operations for the dormant user account.
The motivation to do so would be to facilitate detection of malicious activity and restore a taken-over account (Sadaghiani, [0020]: "The embodiments of the present application, however, enable a detection of a misappropriation of a legitimate (digital) account of a user by a malicious actor. ... The likelihood determination (e.g., account takeover score or probability) by the embodiments of the present application may function to trigger one or more threat mitigation protocols ( e.g., account restriction or lockdown, password reset, alerts to account owner, and the like) that enables the user and/or owner of the account to regain lawful control of a compromised account").
Regarding Claim 8, the rejection of Claim 7 is incorporated. Sadaghiani further teaches:
wherein the one or more risk-mitigation operations include disabling particular functionality of the first dormant user account pending performance of multifactor authentication by a user of the first dormant user account (Sadaghiani, [0076]: "the intelligent API may enable a user or service provider to specifically select an interactive session and/or ATO risk assessment and perform one or more ATO risk mitigation processes (e.g., disable an active interactive session with an adverse ATO risk assessment, perform user verification to ensure that the user of the interactive session is the legitimate owner (e.g., via two-factor authentication, or other identity/authority verification processes)," where Sadaghiani's disabling an interactive session corresponds to the instant disabling particular functionality).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Kozodoi/Dai/Sadaghiani combination regarding performing one or more risk-mitigation operations for a dormant user account with the further teachings of Sadaghiani regarding wherein the one or more risk-mitigation operations include disabling particular functionality of the first dormant user account pending performance of multifactor authentication by a user of the first dormant user account.
The motivation to do so would be to facilitate detection of malicious activity and restore a taken-over account (Sadaghiani, [0020]: "The embodiments of the present application, however, enable a detection of a misappropriation of a legitimate (digital) account of a user by a malicious actor. ... The likelihood determination (e.g., account takeover score or probability) by the embodiments of the present application may function to trigger one or more threat mitigation protocols ( e.g., account restriction or lockdown, password reset, alerts to account owner, and the like) that enables the user and/or owner of the account to regain lawful control of a compromised account").
Regarding Claim 14, the rejection of Claim 11 is incorporated.
Kozodoi teaches processing a plurality of unlabeled data samples.
Kozodoi does not teach wherein the plurality of unlabeled data samples correspond to a plurality of dormant user accounts with the server system and identifying a first dormant user account corresponding to the first unlabeled data sample.
However, Dai teaches:
wherein the plurality of unlabeled data samples correspond to a plurality of dormant user accounts (Dai, p. 43, 4.1 Dataset Processing and Feature Extraction: "The Dataset used in this paper is the CTU Malware Dataset [17], which comes from the Malware Capture Facility Project of the Czech Technical University ATG Group [18]. ... The labels include botnet phase, attack, normal, and background. ... [W]e generate another new 1812 normal four tuples and 979 malicious four tuples as a test set, which is not contained in the training set," where Dai's botnet includes zombies and corresponds the instant dormant user account, as in p. 42, 3.2 Multi-view Feature Extraction: "In botnets, the C&C server typically uses software or programs to control zombies, so traffic between the C&C server and zombies typically exhibits machine behavior that is significantly different from normal daily traffic"), and wherein the operations further comprise:
identifying a first dormant user account corresponding to the first unlabeled data sample (Dai, Table 6, "the detection result of new SSL malicious traffic," showing identification of zombie/dotnet accounts with the given performance statistics, and p. 45, 4.3.3 New SSL malicious traffic detection (Experiment 3): "We use a malicious dataset and a normal dataset from the CTU Malware Dataset as the test datasets, which are not in the training set and validation set. Table 6 shows the detection effect of the four models on the new SSL malicious traffic. ... Putting all the detection indicators together, the XGBoost model is the best, especially accuracy is 74.95% and precision is 69.66%, which proves that this method is effective in detecting new SSL malicious traffic").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Kozodoi regarding processing a plurality of unlabeled data samples with those of Dai regarding wherein the plurality of unlabeled data samples correspond to a plurality of dormant user accounts with the server system and wherein the operations further comprise identifying a first dormant user account corresponding to the first unlabeled data sample.
The motivation to do so would be to facilitate collecting communication statistics distinguishing zombie from normal activity(Dai, p. 40, Abstract: "Our method comprehensively extracts features from multiple views, including flow statistics" and p. 42, 1. Features from the flow statistics: "In botnets, the C&C server typically uses software or programs to control zombies, so traffic between the C&C server and zombies typically exhibits machine behavior that is significantly different from normal daily traffic. ... Above all, we extracted 17-dimensional flow statistics features from multiple perspectives, as shown in Table 1").
The Kozodoi/Dai combination teaches performing operations identifying a first dormant user account corresponding to the first unlabeled data sample.
The Kozodoi/Dai combination does not explicitly teach performing one or more risk-mitigation operations for the ... dormant user account.
However, Sadaghiani teaches:
performing one or more risk-mitigation operations for the ... dormant user account (Sadaghiani, [0076]: "the intelligent API may enable a user or service provider to specifically select an ... ATO risk assessment and perform one or more ATO risk mitigation processes," where Sadaghiani's risk assessment may be undertaken on a suspended account corresponding to the instant dormant account, as in [0020]: "the embodiments of the present application function to collect account activity data associated with an account or a suspected hacked account to determine a likelihood that the account or suspected hacked account may be appropriated or accessed by a malicious actor").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Kozodoi/Dai combination regarding performing operations identifying a first dormant user account corresponding to the first unlabeled data sample with those of Sadaghiani regarding performing one or more risk-mitigation operations for the dormant user account.
The motivation to do so would be to facilitate detection of malicious activity and restore a taken-over account (Sadaghiani, [0020]: "The embodiments of the present application, however, enable a detection of a misappropriation of a legitimate (digital) account of a user by a malicious actor. ... The likelihood determination (e.g., account takeover score or probability) by the embodiments of the present application may function to trigger one or more threat mitigation protocols ( e.g., account restriction or lockdown, password reset, alerts to account owner, and the like) that enables the user and/or owner of the account to regain lawful control of a compromised account").
Regarding Claim 20, the rejection of Claim 16 is incorporated. Dai further teaches:
wherein the method further comprises: identifying a first dormant user account corresponding to the first unlabeled data sample (Dai, Table 6, "the detection result of new SSL malicious traffic," showing identification of zombie/dotnet accounts with the given performance statistics, and p. 45, 4.3.3 New SSL malicious traffic detection (Experiment 3): "We use a malicious dataset and a normal dataset from the CTU Malware Dataset as the test datasets, which are not in the training set and validation set. Table 6 shows the detection effect of the four models on the new SSL malicious traffic. ... Putting all the detection indicators together, the XGBoost model is the best, especially accuracy is 74.95% and precision is 69.66%, which proves that this method is effective in detecting new SSL malicious traffic").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Kozodoi/Dai combination regarding a first set of labeled data samples belonging to a first category and a second set of labeled data samples belonging to a second category with the further teachings of Dai regarding wherein the first category corresponds to malicious user accounts and the second category corresponds to non-malicious user accounts and identifying a first dormant user account corresponding to the first unlabeled data sample.
The motivation to do so would be to facilitate collecting communication statistics distinguishing zombie from normal activity that supports threat detection (Dai, p. 40, Abstract: "Our method comprehensively extracts features from multiple views, including flow statistics" and p. 42, 1. Features from the flow statistics: "In botnets, the C&C server typically uses software or programs to control zombies, so traffic between the C&C server and zombies typically exhibits machine behavior that is significantly different from normal daily traffic. ... Above all, we extracted 17-dimensional flow statistics features from multiple perspectives, as shown in Table 1").
The Kozodoi/Dai combination teaches wherein the first category corresponds to malicious user accounts and the second category corresponds to non-malicious user accounts and identifying, by the computer system, a first dormant user account corresponding to the first unlabeled data sample.
The Kozodoi/Dai combination does not explicitly teach performing, by the computer system, one or more risk-mitigation operations for the first dormant user account.
However, Sadaghiani teaches:
wherein the computer system is included in the server system (Sadaghiani, [0017]: "The digital threat score may be exposed via a score application program interface (API) that may function to interact with various endpoints of the digital threat mitigation platform. Specifically, the score API may function to interact with one or more web computing servers that implement the ensembles of machine learning models used to predict a likelihood of digital fraud and/or digital abuse")
performing ... one or more risk-mitigation operations for the ... dormant user account (Sadaghiani, [0076]: "the intelligent API may enable a user or service provider to specifically select an ... ATO risk assessment and perform one or more ATO risk mitigation processes," where Sadaghiani's risk assessment may be undertaken on a suspended account corresponding to the instant dormant account, as in [0020]: "the embodiments of the present application function to collect account activity data associated with an account or a suspected hacked account to determine a likelihood that the account or suspected hacked account may be appropriated or accessed by a malicious actor").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Kozodoi/Dai combination regarding wherein the first category corresponds to malicious user accounts and the second category corresponds to non-malicious user accounts and identifying a first dormant user account corresponding to the first unlabeled data sample with those of Sadaghiani regarding wherein the computer system is included in the server system and performing one or more risk-mitigation operations for the dormant user account.
The motivation to do so would be to facilitate detection of malicious activity and restore a taken-over account (Sadaghiani, [0020]: "The embodiments of the present application, however, enable a detection of a misappropriation of a legitimate (digital) account of a user by a malicious actor. ... The likelihood determination (e.g., account takeover score or probability) by the embodiments of the present application may function to trigger one or more threat mitigation protocols ( e.g., account restriction or lockdown, password reset, alerts to account owner, and the like) that enables the user and/or owner of the account to regain lawful control of a compromised account").
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Kozodoi, et al., "Shallow Self-learning for Reject Inference in Credit Scoring" (hereinafter "Kozodoi ") in view of Dai, et al., "SSL Malicious Traffic Detection Based On Multi-view Features" (hereinafter "Dai") in view of Leddy, et al. (US 2020/0067861 A1, hereinafter "Leddy").
Regarding Claim 9, the rejection of Claim 2 is incorporated.
The Kozodoi/Dai combination teaches selecting a set of most-common attributes of the selected subset of unlabeled data samples that contributed to the model scores exceeding the particular threshold value.
The Kozodoi/Dai combination does not explicitly teach wherein the set of most-common attributes includes a number of user accounts registered using a given IP address.
However, Leddy teaches:
wherein the set of most-common attributes includes a number of user accounts registered using a given IP address (Leddy, [0935]: "Based on the security decisions, it is determined whether to place Alice on a blacklist, so that all traffic from her is Filtered and scored for all users. If Alice is added to the blacklist, when a new account Alice' is investigated, and is found to correspond to Alice (e.g., same IP, same computer, etc.), then Alice' is also placed on the blacklist, assuming Alice' exhibits some minimum threshold level of scammess").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Kozodoi/Dai combination regarding wherein the set of most-common attributes includes a number of user accounts registered using a given IP address with those of Leddy regarding wherein the set of most-common attributes includes a number of user accounts registered using a given IP address.
The motivation to do so would be to facilitate inferring relationships between potentially malicious actors with greater confidence (Leddy, [0807]-[0809]: "When the source of messages is a known scammer, the message content is unambiguously scam. Thus, all their message content can be used for deriving new scam Filters, as described above; When available, additional information is gathered about the scammers including, but not limited to: Their IP addresses" and [0932]: "If a new user Fred contacts Cindy/Dave and there is some linkage to Alice via available information like domain, IP address, message content, then in can be determined that there is high probability that Fred is affiliated with Alice. In this case, Fred can be linked to Alice and treated as Alice'").
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT N DAY whose telephone number is (703)756-1519. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/R.N.D./Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights