DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
2. A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on February 11, 2026 has been entered.
Response to Amendment
3. Claims 1, 7, 11, 15, 18 and 19 have been amended. Claims 1-20 are presented for examination.
Response to Arguments
4. Applicant’s arguments, see pages 9-12, February 11, 2026, with respect to claims 1-10 and 15-18 have been considered but are moot in view of the new grounds of rejection. The claims do not overcome the new ground of rejection made in view of newly found prior art references.
5. Applicant’s arguments, see pages 9-12, February 11, 2026, with respect to claims 11-14 and 19-20 have been considered but are moot in view of the new grounds of rejection. The claims do not overcome the new ground of rejection made in view of newly found prior art references.
Claim Rejections - 35 USC § 103
6. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
7. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
8. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
9. Claims 1, 2, 4-7, 8, 15, 17 and 18 are rejected under 35 U.S.C. § 103 as being unpatentable over Wright et al. (US 2023/0351009 A1), hereafter Wright, in view of Nguyen et al. (US 2020/0159916 A1), hereafter Nguyen.
Noted that indicates what the cited art does not teach.
Regarding claim 1, Wright teaches a method comprising: {Wright [Para. 5] “A method training an AI engine for real-time monitoring of false positives during user registration.”}
receiving, by a computer device, a decision validation model for updating a decision from a prediction model of a security agent of the computing device, {Wright [Para. 0004] “The system… receive a first set of data from at least one AI security model and a second set of data from a false positive database…. causes the processor to correlate the first and second sets of data and create a training dataset and a test dataset from the correlated data….causes the processor to use the training dataset to train a false positive identification model to identify false positives, and use the test dataset to evaluate the false positive identification model” [Para. 0006] “Applying the AI engine includes: receiving user information and extracting data from the user information; applying the at least one AI security model to the extracted data and receiving a result from the at least one AI security model. The result may be a negative or a positive. Upon receiving a negative result, user registration is allowed. Upon receiving a positive result, the false negative identification model is applied. The false negative identification model will provide a result, either a false or a true result. If a false result is obtained the user registration is allowed. If the true result is obtained, the user registration is blocked.” } Wright’s system applies an AI engine to user registration for identifying false positives during user registration. The AI engine includes an AI security model and a false positive identification model (see para. 0005). The AI security model predicts whether a user registration is fraudulent, and therefore is a prediction model. The false positive identification model determines whether the result from the AI security model is a true positive result or false positive result. Therefore, the false positive identification model is a decision validation model for the AI security model. Additionally, AI engine corresponds to the security agent.
following the decision from the prediction model, invoking, by the security agent, a function of the security agent to utilize the decision validation model and, as inputs to the decision validation model, an input vector from the prediction model and a decision value representing the decision from the prediction model; {Wright [Para. 0092] “At block 805, the system may receive user information from a user attempting to register a user account. Upon receiving the user information, the system may, at block 807, extract data from the user information.“ [Para. 0093] “At block 810 the system will apply the AI security model, as described above.” [Para. 0094] “After applying the at least one AI security model, the system will receive a result from the AI security model. The result will either be a positive result (fraudulent) or a negative result (not fraudulent). If the system receives a negative result from the AI security model, at block 820, then the system will allow the user registration, block 845. However, if the system receives a positive result from the AI security model, block 815, the system will apply the false positive identification model, block 825. The false positive identification model will provide the system a result, the result from the false positive identification model may be either true (this was a true positive result from the AI security model), or a false (this was a false positive result from the AI security model).”} As disclosed in Wright, after an AI security model outputs a positive result indicating that a user registration is fraudulent, the system subsequently applies a false positive identification model to the user registration to determine whether the result from the AI security model is a true positive result or a false positive result.
and outputting, by the security agent, a decision value from the decision validation model, the decision value from the decision validation model being different from the decision value from the prediction model. {Wright [Para. 0094] “After applying the at least one AI security model, the system will receive a result from the AI security model. The result will either be a positive result (fraudulent) or a negative result (not fraudulent). However, if the system receives a positive result from the AI security model, block 815, the system will apply the false positive identification model, block 825. The false positive identification model will provide the system a result, the result from the false positive identification model may be either true (this was a true positive result from the AI security model), or a false (this was a false positive result from the AI security model). Upon receiving a false result from the false positive identification model, block 830, the system will allow the user registration, block 845. If the system receives a true result from the false positive identification model, block 835, the system will block the user registration.”} An AI security model outputs a positive result, indicating that a user registration is fraudulent. Subsequently, a false positive identification model outputs a false result, indicating the result from the AI security model is a false positive result. That is, the user registration is not fraudulent. The false positive identification model provides its result to the system. The AI engine includes the AI security model and the false positive identification model (see para. 0005). Accordingly, a decision value from the false positive identification model was outputted by the AI engine and provided to the system.
However, Wright does not teach the decision validation model and the prediction model being non-executable data; following the decision from the prediction model, invoking, by the security agent, a function of the security agent to utilize the decision validation model and, as inputs to the decision validation model, an input vector from the prediction model and a decision value representing the decision from the prediction model.
However, Nguyen teaches the decision validation model and the prediction model being non-executable data. {Nguyen [Para. 0201] "The classification subsystem 246 can be configured to use first classification model 1004 and second classification model 1102 together.” [Para. 0069] “Data stored on CRM (computer-readable media) 214 can include at least one computational model (CM) 220, which can represent CM(s) 114, FIG. 1. CM 220 can be stored as data (e.g., parameters); as code (e.g., for testing branch points in a decision tree); or as a combination of data and code.” [Para. 0038] “The CMS (computational model) 114 can include parameters governing or affecting the output of the CM 114 for a particular input.”} As indicated in Nguyen, a computational model (e.g., CM 220) maybe stored as data (e.g., parameters), code, or a combination of both. Code is executable data, and data (e.g., parameters) is non-executable data. Therefore, a machine learning model may be stored as non-executable data. Nguyen discloses two classification models (e.g., a prediction model and a decision validation model). Thus, both the decision validation model and the prediction model are stored as non-executable data.
following the decision from the prediction model, invoking, by the security agent, a function of the security agent to utilize the decision validation model and, as inputs to the decision validation model, an input vector from the prediction model and a decision value representing the decision from the prediction model; {Nguyen [Para. 0202] “At 1104, the classification subsystem 246 can determine a first classification 1106 of the event 808 by providing the at least two term representations 804 as input to the first classification model 1004.” [Para. 0203] “First classification model 1004 includes a neural network.”[Para. 0204] “At 1108, the classification subsystem 246 can determine a second classification 1110 of the event 808 by providing the at least two term representations 804 as input to the second classification model 1102.” [Para. 0205] “At 1112, the classification subsystem 246 or other components of operation module 230 can determine the indication 830 that the event 808 is associated with a computer-security violation. Operation 1112 can be performed in response to at least one of the first classification 1106 or the second classification 1110 indicating that the event is associated with a computer-security violation.” [Para. 0191] “Each of the at least two term representations 804 comprises a respective feature vector 1002.”} Nguyen discloses a classification subsystem that uses two classification models for generating classifications indicating whether an event (e.g., event 808) is a security violation. Both classification models utilize term representations 804 as model inputs. Term representations 804 are feature vectors. The second classification model takes an input vector from the first classification model to generate a classification indicating whether event 808 is a malicious event. The first classification model corresponds to the prediction model, and the second classification model corresponds to the decision validation model.
Nguyen is analogous art because each of Wright and Nguyen pertains to implementing machine learning models to generate security-related classifications. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wright to include Nguyen’s teaching of the limitations of claim 1, listed above. Doing so would “greatly increase the ease of implementing security for a computing device and may prevent greater damage from occurring due to a violation” (Nguyen, para. 0024).
Claim 2:
Regarding claim 2, Wright and Nguyen teach the elements of claim 1 as stated.
Wright further teaches wherein the decision of the prediction model represents a false positive result or a false negative result. {Wright [Para. 0094] “After applying the at least one AI security model, the system will receive a result from the AI security model. The result will either be a positive result (fraudulent) or a negative result (not fraudulent). If the system receives a negative result from the AI security model, at block 820, then the system will allow the user registration, block 845. However, if the system receives a positive result from the AI security model, block 815, the system will apply the false positive identification model, block 825. The false positive identification model will provide the system a result, the result from the false positive identification model may be either true (this was a true positive result from the AI security model), or a false (this was a false positive result from the AI security model).”} An AI security model outputs a positive result, indicating that a user registration is fraudulent. Subsequently, a false positive identification model determines that the result of the AI security model is a false positive result.
Claim 4:
Regarding claim 4, Wright and Nguyen teach the elements of claim 1 as stated.
Wright further teaches wherein the decision validation model changes the decision from prediction model result given the input vector and the decision value. {Wright [Para. 0094] “After applying the at least one AI security model, the system will receive a result from the AI security model. If the system receives a negative result from the AI security model, at block 820, then the system will allow the user registration, block 845. However, if the system receives a positive result from the AI security model, block 815, the system will apply the false positive identification model, block 825. The false positive identification model will provide the system a result, the result from the false positive identification model may be either true (this was a true positive result from the AI security model), or a false (this was a false positive result from the AI security model).”} An AI security model outputs a positive result, indicating that a user registration is fraudulent. Subsequently, a false positive identification model determines that the decision of the AI security model is a false positive result, which indicates that the user registration is not fraudulent. That is, the false positive identification model changes the decision from the AI security model.
Claim 5:
Regarding claim 5, Wright and Nguyen teach the elements of claim 1 as stated above.
Wright further teaches wherein the difference between the decision value of the prediction model and the decision value of the decision validation model is a difference in decision confidence or a different prediction. {Wright [Para. 0094] “After applying the at least one AI security model, the system will receive a result from the AI security model. If the system receives a negative result from the AI security model, at block 820, then the system will allow the user registration, block 845. However, if the system receives a positive result from the AI security model, block 815, the system will apply the false positive identification model, block 825. The false positive identification model will provide the system a result, the result from the false positive identification model may be either true (this was a true positive result from the AI security model), or a false (this was a false positive result from the AO security model). Upon receiving a false result from the false positive identification model, block 830, the system will allow the user registration, block 845.”} As disclosed in Wright, an AI security model outputs a positive result, indicating that a user registration is fraudulent. Subsequently, a false positive identification model determines that the decision of the AI security model is a false positive result, which indicates that the user registration is not fraudulent. These two models generate different predictions. The difference between the decision value of the AI security model and the decision value of the false positive identification model is a different prediction.
Claim 6:
Regarding claim 6, Wright and Nguyen teach the elements of claim 1 as stated.
Wright further teaches further comprising sending, by the security agent, the decision value of the prediction model and the decision value of the decision validation model to a security service. {Wright [Para. 0094] “After applying the at least one AI security model, the system will receive a result from the AI security model. The result will either be a positive result (fraudulent) or a negative result (not fraudulent). If the system receives a negative result from the AI security model, at block 820, then the system will allow the user registration, block 845. However, if the system receives a positive result from the AI security model, block 815, the system will apply the false positive identification model, block 825. The false positive identification model will provide the system a result, the result from the false positive identification model may be either true (this was a true positive result from the AI security model), or a false (this was a false positive result from the AI security model). Upon receiving a false result from the false positive identification model, block 830, the system will allow the user registration, block 845.”} Wright’s system identifies false positive when a user registers for an account (see para. 0029). The system applies an AI engine to user registration. The AI engine includes an AI security model and a false identification model (see para. 0091). The system allows or blocks user registrations based on results received from the AI engine. The system also trains the AI engine to identify false positive during user registration (see para. 0087). Therefore, Wright’s system implements a security service. The AI engine corresponds to the security agent. The system receives results from the AI security model and false positive identification model. Therefore, Wright teaches an AI engine that sends decisions values of the AI security model and the false positive identification model to a system that implements a security service.
Claim 7:
Regarding claim 7, Wright and Nguyen teach the elements of claim 1 as stated.
Wright further teaches further comprising: following a second decision from the prediction model based on a second input vector, invoking, by the security agent, the function of the security agent to utilize the decision validation model and, as inputs to the decision validation model, the second input vector from the prediction model and a second decision value from the prediction model; {Wright [Para. 0092] “At block 805, the system may receive user information from a user attempting to register a user account. Upon receiving the user information, the system may, at block 807, extract data from the user information.” [Para. 0093] “At block 810 the system will apply the AI security model.” [Para. 0094] “After applying the at least one AI security model, the system will receive a result from the AI security model. The result will either be a positive result (fraudulent) or a negative result (not fraudulent). If the system receives a negative result from the AI security model, at block 820, then the system will allow the user registration, block 845. However, if the system receives a positive result from the AI security model, block 815, the system will apply the false positive identification model, block 825. The false positive identification model will provide the system a result, the result from the false positive identification model may be either true (this was a true positive result from the AI security model), or a false (this was a false positive result from the AI security model).”} After an AI security model outputs a positive result indicating that a user registration is fraudulent, the system applies a false positive identification model to the user registration to determine whether the result from the AI security model is a true positive result or a false positive result.
and outputting, by the security agent, a second decision value from the decision validation model, the decision value from the decision validation model being the same as the second decision value from the prediction model. {Wright [Para. 0094] “After applying the at least one AI security model, the system will receive a result from the AI security model. The result will either be a positive result (fraudulent) or a negative result (not fraudulent). However, if the system receives a positive result from the AI security model, block 815, the system will apply the false positive identification model, block 825. The false positive identification model will provide the system a result, the result from the false positive identification model may be either true (this was a true positive result from the AI security model), or a false (this was a false positive result from the AI security model). Upon receiving a false result from the false positive identification model, block 830, the system will allow the user registration, block 845. If the system receives a true result from the false positive identification model, block 835, the system will block the user registration.”} An AI security model outputs a positive result, indicating that a user registration is fraudulent. Subsequently, a false positive identification model outputs a true result, indicating the result from the AI security model is true positive result. That is, the user registration is fraudulent. The false positive identification model provides its result to the system and the system allows the user registration. The AI engine includes the AI security model and the false positive identification model (see para. 0005). Accordingly, a decision value from the false positive identification model was outputted by the AI engine and provided to the system.
However, Wright does not teach following a second decision from the prediction model based on a second input vector, invoking, by the security agent, a function of the security agent to utilize the decision validation model and, as inputs to the decision validation model, the second input vector from the prediction model and a second decision value from the prediction model.
However, Nguyen teaches following a second decision from the prediction model based on a second input vector, invoking, by the security agent, a function of the security agent to utilize the decision validation model and, as inputs to the decision validation model, the second input vector from the prediction model and a second decision value from the prediction model; {Nguyen [Para. 0201] "The classification subsystem 246 can be configured to use first classification model 1004 and second classification model 1102 together.” [Para. 0202] “At 1104, the classification subsystem 246 can determine a first classification 1106 of the event 808 by providing the at least two term representations 804 as input to the first classification model 1004.” [Para. 0204] “At 1108, the classification subsystem 246 can determine a second classification 1110 of the event 808 by providing the at least two term representations 804 as input to the second classification model 1102.” [Para. 0205] “At 1112, the classification subsystem 246 or other components of operation module 230 can determine the indication 830 that the event 808 is associated with a computer-security violation. Operation 1112 can be performed in response to at least one of the first classification 1106 or the second classification 1110 indicating that the event is associated with a computer-security violation.” [Para. 0191] “Each of the at least two term representations 804 comprises a respective feature vector 1002.”} Nguyen’s classification subsystem uses two classification models for determining whether an event (e.g., event 808) is a security violation. Both classification models utilize term representations 804 as model inputs to generate classifications indicating whether event 808 is a security violation. Term representations 804 are feature vectors. The second classification model takes an input vector from the first classification model to generate a classification indicating whether event 808 is a malicious event.
Nguyen is analogous art because each of Wright and Nguyen pertains to implementing machine learning models to generate security-related classifications. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wright to include Nguyen’s teaching of the limitations of claim 7, listed above. Doing so would “greatly increase the ease of implementing security for a computing device” (Nguyen, para. 0024).
Claim 8:
Regarding claim 8, Wright and Nguyen teach the elements of claim 1 as stated.
Wright further teaches following a decision from a second prediction model of the security agent, determining that there is no decision validation model for the second prediction model. {Wright [Para. 0094] “After applying the at least one AI security model, the system will receive a result from the AI security model. The result will either be a positive result (fraudulent) or a negative result (not fraudulent). If the system receives a negative result from the AI security model, at block 820, then the system will allow the user registration, block 845. However, if the system receives a positive result from the AI security model, block 815, the system will apply the false positive identification model, block 825. The false positive identification model will provide the system a result, the result from the false positive identification model may be either true (this was a true positive result from the AI security model), or a false (this was a false positive result from the AI security model).”} Wright teaches applying more than one AI security model to predict fraudulent user registration. When an AI security model outputs a negative result, indicating that a user registration is not fraudulent. The system does not apply the false positive identification model. In this regard, when the system receives a negative result from any AI security model, the system determines that there is no false positive identification model for this particular AI security model and therefore, does not apply the false positive identification model. The AI security model corresponds to the second prediction model, and the false positive identification model corresponds to the decision validation model.
Claims 15, 17 and 18:
Regarding claims 15, 17 and 18, the claims are directed to a computing device that implements the operations recited by claims 1, 5 and 6. Therefore, the rejection applied to claims 1, 5 and 6 also applies to claims 15, 17 and 18. Claims 1, 5 and 6 are rejected under the same rationale as claims 15, 17 and 18.
Claim 15 further recites a computing device comprising: a processor; and programming instructions configured to be operated by the processor to implement a security agent to perform operations including: the operations of claims 1, 5 and 6. {Wright [Para. 0007] “A system for eliminating false positives during user registration. The system consists of at least one processor,…and a memory device storing executable code causing the processor to: receive user information from a user attempting to register a new account, extract data from the user information, and apply an AI security model to the extracted data.” [Para. 0004] “Based on the evaluation, the executable code may cause the processor to apply an AI engine to user registration, where the AI engine includes an AI security model and the false positive identification model.”}
10. Claims 3, 9, 10 and 16 are rejected under 35 U.S.C. § 103 as being unpatentable over Wright and Nguyen as applied to claims 1 and 15, and further in view of Vashisht et al. (US 11, 227, 047 B1), hereafter Vashisht.
Regarding claim 3, Wright and Nguyen teach the elements of claim 1 as outlined above.
However, Wright does not teach wherein the decision validation model includes weights and activation function identifiers for a generic neural network of the security agent and decision confidence thresholds for assigning decision confidences to decisions of the decision validation model.
However, Nguyen teaches wherein the decision validation model includes weights and activation function identifiers for a generic neural network of the security agent and decision confidence thresholds for assigning decision confidences to decisions of the decision validation model. {Nguyen [Para. 0038] “The CMS (computational model) 114 can include parameters governing or affecting the output of the CM 114 for a particular input. Parameters can include, but are not limited to, e.g., per-neuron, per-input weight or bias values, activation-function selections, neuron weights, edge weights, tree-node weights, or other data values.”
[Para. 0037] “CM(s) 114 for classifying events or other use cases noted herein, the CM(s) 114 may include, multilayer perceptrons (MLPs), neural networks (NNs), convolutional NNs (CNNS), deep neural networks (DNNs).” [Para. 0152] “The NN can produce other outputs, in some examples. The NN can include a softmax, threshold, clipping, clamping, or other operation to output(s) from neuron outputs of the output layer.”}
Nguyen is analogous art because each of Wright and Nguyen pertains to implementing machine learning models to generate security-related classifications. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wright to include Nguyen’s teaching of the limitations of claim 3, listed above. Doing so would “greatly increase the ease of implementing security for a computing device” (Nguyen, para. 0024).
However, Nguyen does not teach decision confidence thresholds for assigning decision confidences to decisions of the decision validation model.
However, Vashisht teaches decision confidence thresholds for assigning decision confidences to decisions of the decision validation model. {Vashisht [Col. 6, line 12-20] “The cybersecurity detection engine 105 determines that a digital resource is malicious when such classification (e.g., malicious) is produced with a confidence value that is over a predetermined threshold value, and thus, can activate an alert, execute quarantine process.”}
Vashisht is analogous art because each of Wright, Nguyen and Vashisht pertains to implementing machine learning models to generate security-related classifications. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wright and Nguyen to include Vashisht’s teaching of decision confidence thresholds for assigning decision confidences to decisions of a decision model. Doing so would ensure “that an optimal cyber threat detection system is continuously operating in a compute device” (Vashisht, Col 3, line 61-63).
Claim 9:
Regarding claim 9, Wright and Nguyen teach the elements of claim 1 as stated.
Wright further teaches further comprising: checking for decision validation model after assigning a decision confidence to the decision of the prediction model; or checking for the decision validation model after outputting the decision of the prediction model but before assigning the decision confidence to the decision of the prediction model; {Wright [Para. 0094] “After applying the at least one AI security model, the system will receive a result from the AI security model. The result will either be a positive result (fraudulent) or a negative result (not fraudulent). If the system receives a negative result from the AI security model, at block 820, then the system will allow the user registration, block 845. However, if the system receives a positive result from the AI security model, block 815, the system will apply the false positive identification model, block 825. The false positive identification model will provide the system a result, the result from the false positive identification model may be either true (this was a true positive result from the AI security model), or a false (this was a false positive result from the AI security model).”} As disclosed in Wright, if an AI security model outputs a positive result, indicating that a user registration is fraudulent, the system will execute a false positive identification model. The system only applies the false positive identification model if the AI security model outputs a positive result. Accordingly, Wright’s system checks for the false positive identification model after outputting the decision of the AI security model. The AI security model corresponds to the prediction model, and the false positive identification model corresponds to the decision validation model.
However, Wright and Nguyen do not teach checking for decision validation model after assigning a decision confidence to the decision of the prediction model; or checking for the decision validation model after outputting the decision of the prediction model but before assigning the decision confidence to the decision of the prediction model.
However, Vashisht teaches checking for decision validation model after assigning a decision confidence to the decision of the prediction model; {Vashisht [Col. 6, line 12-20] “The cybersecurity detection engine 105 determines that a digital resource is malicious when such classification (e.g., malicious) is produced with a confidence value that is over a predetermined threshold value, and thus, can activate an alert, execute quarantine process, and/or take other action to protect against the digital resource classified as malicious or determined to be part of a cyberattack.” [Col. 6, line 26-57] “When the cybersecurity detection engine 105 produces a seemingly FN classification value or a FP classification value for a given digital resource, the digital resource is labeled with the produced classification value and sent to the CDS 123 for further analysis through one or more machine learning models implemented in CDS 123. When the cybersecurity detection engine 105 determines that compute device 101 or other compute device operatively coupled to compute device 101 exhibits abnormal characteristics or behavior, the cybersecurity detection engine 105 can label the digital resource as suspicious and collect data flows and/or performance data of the abnormal performance of the compute device along with digital resources suspected to have caused the anomaly and sends these data to CDS 123 for further analysis through one or more machine learning models implemented in CDS 123. The “suspicious” label can be applied, in some cases, regardless of the confidence level, or in other cases, where the confidence level is below the threshold for a maliciousness classification and above the threshold for a benign (or not malicious) classification.”} Cybersecurity detection engine 105 implements machine learning models that classify digital resources as malicious or not malicious. Vashisht teaches assigning a confidence value to the classifications. When the cybersecurity detection engine 105 generates FN classification value, a FP classification value or a suspicious label for a given digital resource, the digital resource is sent to the CDS 123 for further analysis through one or more machine learning models implemented in CDS 123. In this regard, ML models implemented in CDS 123 that are used to further analyze the given digital resource sent by the cybersecurity detection engine 105 corresponds to the decision validation model. Accordingly, Vashisht teaches checking for checking for decision validation model after assigning a decision confidence to the decision of the prediction model.
Vashisht is analogous art because each of Wright, Vashisht and Nguyen pertains to implementing machine learning models to generate security-related classifications. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wright and Nguyen to include Vashisht’s teaching of checking for decision validation model after assigning a decision confidence to a decision of the prediction model. Doing so would ensure “that an optimal cyber threat detection system is continuously operating in a compute device” (Vashisht, Col 3, line 61-63).
Claim 10:
Regarding claim 10, Wright and Nguyen teach the elements of claim 1 as stated.
Wright further teaches receiving an updated decision validation model; {Wright
[Para. 0004] “The processor to use the training dataset to train a false positive identification model to identify false positives, and use the test dataset to evaluate the false positive identification model. Based on the evaluation,…the processor to alter and retrain the false positive identification model. Based on the evaluation, …the processor to apply an AI engine to user registration, where the AI engine includes an AI security model and the false positive identification model.”} Wright teaches training and retraining a false positive identification model to generate an updated false positive identification model.
However, Wright and Nguyen do not teach in response to receiving the updated decision validation model, overwriting the decision validation model with the updated decision validation model or using the updated decision validation model in place of the decision validation model.
However, Vashisht teaches in response to receiving the updated decision validation model, overwriting the decision validation model with the updated decision validation model or using the updated decision validation model in place of the decision validation model. {Vashisht [Col. 12, line 39-49] “Monitor 142 can periodically produce performance metrics for the trained machine learning models (e.g., AUROC, as discussed above). In some instances, when the monitor 142 determines that another machine learning model different from the machine learning model currently deployed in active mode exhibits a superior performance by, for example, achieving a significantly better classification accuracy than the currently deployed machine learning model, the CDS 123 replaces the currently deployed machine learning model with the machine learning model that has outperformed it.” [Col. 14, line 8-20] “At 403, the CDS 123 builds new machine learning models, and/or tunes, retrains, revalidates, and/or retests trained machine learning models 143 (shown in FIG. 2) with the new training datasets. The machine learning models are evaluated based on one or more performance metrics, at 405. The CDS 123 selects the machine learning model(s) that exhibit superior performance based on the one or more performance metrics, at 407, and replaces a previously deployed, active machine learning model with the selected machine learning model(s), at 409.”} Vashisht teaches retraining ML models to generate updated ML models. Vashisht’s CDS (cybersecurity detection system) 123 replaces the currently deployed, active machine learning model used by the cybersecurity detection engine 105 implemented in compute device 101 with another updated machine learning model.
Vashisht is analogous art because each of Vashisht, Wright and Nguyen pertains to implementing machine learning models to generate security-related classifications. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wright and Nguyen to include Vashisht’s teaching of using an updated decision model in place of another currently deployed decision model in response to receiving the updated decision model. Doing so would ensure “that an optimal cyber threat detection system is continuously operating in a compute device” (Vashisht, Col 3, line 61-63).
Claim 16:
Regarding claim 16, the claim is directed to a computing device that implements the operations recited by claim 3. Therefore, the rejection applied to claim 3 also applies to claim 16. Claim 16 is rejected under the same rationale as claim 3.
11. Claims 11, 12, 14, 19 and 20 are rejected under 35 U.S.C. § 103 as being unpatentable over Wright et al. (US 2023/0351009 A1), hereafter Wright, in view of Williams et al. (US 2015/0254555 A1), hereafter Williams, and further in view of Nguyen et al. (US 2020/0159916 A1), hereafter Nguyen.
Noted that indicates what the cited art does not teach.
Regarding claim 11, Wright teaches a system comprising: one or more processors; and programming instructions configured to be operated by the processors to perform operations including: {Wright [Para. 0004] “A system for training an AI engine for real-time monitoring of false positives during user registration. The system consists of at least one processor,…and a memory device storing executable code causing the processor to: receive a first set of data from at least one AI security model and a second set of data from a false positive database.”}
determining that a decision of a prediction model represents a false positive prediction or a false negative prediction; {Wright [Para. 0094] “After applying the at least one AI security model, the system will receive a result from the AI security model. The result will either be a positive result (fraudulent) or a negative result (not fraudulent). If the system receives a negative result from the AI security model, at block 820, then the system will allow the user registration, block 845. However, if the system receives a positive result from the AI security model, block 815, the system will apply the false positive identification model, block 825. The false positive identification model will provide the system a result, the result from the false positive identification model may be either true (this was a true positive result from the AI security model), or a false (this was a false positive result from the AI security model).”} An AI security model outputs a positive result, indicating that a user registration is fraudulent. Subsequently, a false positive identification model determines that the result of the AI security model is a false positive result. That is, the decision of the AI security model represents a false positive result. The AI security model corresponds to the prediction mode.
in response to the determining, training a decision validation model for updating the decision of the prediction model based at least on an input vector associated with the decision of the prediction model and a decision value associated with the decision of the prediction model such that a decision value output for the decision validation model is not the false positive prediction or the false negative prediction; {Wright [Para. 0087] “At block 705, the system receives a first set of data from at least one AI security model. The data received from the at least one security model includes input data into the at least one AI security model, output data from the at least one AI security model. Input data includes data indicative of user registration. In some embodiments, output data from the at least one AI security model includes a classification of the input data.”[Para. 0088] “At block 710, the system receives a second set of data from a false positive database. At block 715, the system correlates the first set of data with the second set of data.” [Para. 0089] “At block 720, the system creates a training dataset and a test dataset from the correlated data.” [Para. 0090] At block 725, the system will use the training dataset to train a false positive identification model to identify false positives. The identification false positives may include labelling or classifying the positive data as false. In some embodiments, the false positive identification model includes a neural network.” [Para. 0075] “The machine learning program may include one or more support vector machines…The support vector machine(s) may be trained similarly to the training of neural networks, e.g., by providing a known input vector (including values for the input variables) and a known output classification. The support vector machine is trained by selecting the support vectors and/or a portion of the input vectors that maximize the determined margin.”} Wright teaches training a false positive identification model using input data associated with a decision of an AI security model and output data from the AI security model. The fp identification model may be a neural network or a support vector machine, which may be trained based on a known input vector and a known output classification. In this regard, an input vector associated with the decision of an AI security model is used to train the false positive identification model. Wright’s system applies the trained false positive identification model to user registration to identify whether a prediction from an AI security model is a false positive result or a true positive result (see paras. 92-94). The AI security model corresponds to the prediction model, and the false positive identification model corresponds to the decision validation model.
and providing the decision validation model to a security agent on a client device to be utilized by a function of the security agent in association with the prediction model. {Wright [Para. 0091] “Training the false positive identification model follows the steps illustrated in flow chart 600 of FIG. 6. This includes the steps represented at block 730, using the test dataset to evaluate the false positive identification model. Once desired evaluation results are received the system will proceed to block 732, where an AI engine will be applied to user registration. The AI engine may comprise an AI security model and the false positive identification model.”}
After the false positive identification model is trained, the system will apply the AI engine to user registration. As disclosed in paras. 0092-0094, the AI engine utilizes the false positive identification model to identify false positive results generated by the AI security model. The AI engine corresponds to the security agent.
However, Wright does not teach the prediction model being non-executable data; in response to the determining, training a decision validation model for updating the decision of the prediction model based at least on an input vector associated with the decision of the prediction model and a decision value associated with the decision of the prediction model such that a decision value output for the decision validation model is not the false positive prediction or the false negative prediction; the decision validation model being non-executable data;
However, Williams teaches in response to the determining, training a decision validation model for updating the decision of the prediction model based at least on an input vector associated with the decision of the prediction model and a decision value associated with the decision of the prediction such that that a decision value output for the decision validation model is not the false positive prediction or the false negative prediction; {Williams [Para. 0113] “FIG. 7 shows a flowchart for process 700 for interactive tuning of a classification system. After a start block, at decision block 702, if the number of classification errors are determined to exceed a defined threshold, control may flow to block 704;” [Para. 0114] “Errors may include false positives, label conflicts, missed classifications, or the like.” [Para. 0115] “At block 704, a user may modify and/or tune one or more classifiers and/or create new classifiers based on the errors and/or signals of the model.” [Para. 0116] “Also, a fast learning (FL) model may be trained using the tuned and/or new classifier.” [Para. 0118] “The FL model may be retrained based on the data and/or network information associated with the classification errors made by the deep learning model. Also, the modified classifier may be associated with some or all of the data that may be associated with one or more of errors counted in decision block 702.” [Para. 0120] “At block 708, optionally, the provided source data may be classified using the fast learning (FL) model.” [Para. 0123] “At block 712, the FL model classification result may be employed. Since the FL model produced a classification results that has a higher values confidence level that the classification result produced by the DLNN model, the data may be classified based on the FL model rather than the DLNN model.”} Williams teaches training and retraining a fast learning (FL) model if the number of classification errors (e.g., false positives) generated by a DLNN model exceeds a threshold. The FL model is trained and retrained based on classification errors made by the DLNN model. The FL model corresponds to the decision validation model, and DLNN model corresponds to the prediction model. Accordingly, the training of the FL model is triggered in response to the determining of false positives generated by the DLNN model.
Williams is analogous art because each of Wright and Williams pertains to deploying machine learning models to classify data and identify misclassifications. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wright to include Williams’ teaching of the limitations of claim 11, listed above. Doing so would enable a system “to continuously improves its scoring predictions” (Williams, para. 0168).
However, Williams also does not teach the prediction model being non-executable data; the decision validation model being non-executable data.
However, Nguyen teaches the prediction model being non-executable data; the decision validation model being non-executable data. {Nguyen [Para. 0201] "The classification subsystem 246 can be configured to use first classification model 1004 and second classification model 1102 together.” [Para. 0069] “Data stored on CRM (computer-readable media) 214 can include at least one computational model (CM) 220, which can represent CM(s) 114, FIG. 1. CM 220 can be stored as data (e.g., parameters); as code (e.g., for testing branch points in a decision tree); or as a combination of data and code.” [Para. 0038] “The CMS (computational model) 114 can include parameters governing or affecting the output of the CM 114 for a particular input.”} As indicated in Nguyen, a computational model (e.g., CM 220) maybe stored as data (e.g., parameters), code, or a combination of both. Code is executable data, and data (e.g., parameters) is non-executable data. Therefore, a machine learning model may be stored as non-executable data. Nguyen discloses two classification models (e.g., a prediction model and a decision validation model). Thus, both the decision validation model and the prediction model are stored as non-executable data. As indicated in paras. 0202-0205 and 0191, both Nguyen’s classification models uses input vector from the first classification as model inputs.
Nguyen is analogous art because each of Wright, Williams and Nguyen pertains to implementing machine learning models to generate security related classifications. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wright and Nguyen to include Nguyen’s teaching of the limitations of claim 11, listed above. Doing so would “greatly increase the ease of implementing security for a computing device and may prevent greater damage from occurring due to a violation” (Nguyen, para. 0024).
Claim 12:
Regarding claim 12, Wright, Williams and Nguyen teach the elements of claim 11 as outlined above.
Wright further teaches wherein the training comprises training the decision validation model to correct for multiple false positive predictions or multiple false negative predictions of the prediction model. {Wright [Para. 0090] “At block 725, the system will use the training dataset to train a false positive identification model to identify false positives. The identification false positives may include labelling or classifying the positive data as false.”} Also see para. 0004. Wright teaches training a FP identification model to identify fp results generated by an AI security model (see paras. 0092-0094). Therefore, the FP identification model is trained to correct for multiple false positive predictions of the AI security model. Wright also teaches training an AI security model where the training is triggered by a false positive rate (see para. 0095).
Claim 14:
Regarding claim 14, Wright, Williams and Nguyen teach the elements of claim 11 as outlined above.
Wright further teaches wherein the operations further include: receiving a decision value from the decision validation model and a decision value from the prediction model; and using the decision value from the decision validation model as a result for the prediction model. {Wright [Para. 0094] “After applying the at least one AI security model, the system will receive a result from the AI security model. The result will either be a positive result (fraudulent) or a negative result (not fraudulent). If the system receives a negative result from the AI security model, at block 820, then the system will allow the user registration, block 845. However, if the system receives a positive result from the AI security model, block 815, the system will apply the false positive identification model, block 825. The false positive identification model will provide the system a result, the result from the false positive identification model may be either true (this was a true positive result from the AI security model), or a false (this was a false positive result from the AI security model). Upon receiving a false result from the false positive identification model, block 830, the system will allow the user registration, block 845. If the system receives a true result from the false positive identification model, block 835, the system will block the user registration.”} Wright’s system receives a positive result from an AI security model and a subsequent false positive result from a false positive identification model. The decision value from the AI security model indicates a user registration is fraudulent, and the decision value from the FP identification model indicates that the user registration is not fraudulent. Based on the decision value from the FP identification model, the system allows the user registration. That is, the system uses the decision value from the false positive identification model.
Claims 19 and 20:
Regarding claims 19 and 20, the claims are directed to a method comprising the operations recited by claims 11 and 14. The rejection applied to claims 11 and 14 also applies to claims 19 and 20. Claims 19 and 20 are rejected under the same rationale as claims 11 and 14.
Claim 19 further recites a computer-implemented method comprising: the operations recited by claim 11. Claim 19 further recites a security service that implements the operations recited by claim 11. {Wright [Para. 0005] “A method training an AI engine for real-time monitoring of false positives during user registration is presented. The method includes: receiving a first set of data from at least one AI security model and a second set of data from a false positive database. The method further includes correlating the first and second sets of data and generating a training dataset and a test dataset. The method then uses the training dataset to train a false positive identification model and uses the test dataset to evaluate the false positive identification model, prior to applying an AI engine to user registration, where the AI engine includes both an AI security model and a false positive identification model.”} Another embodiment of Wright’s system is a method. Wright’s system identifies false positive when a user registers for an account (see para. 0029). The system applies an AI engine to user registration, and allows or blocks user registrations based on results generated by the AI engine. The system also trains the FP identification model to identify false positive during user registration (see para. 0087). Therefore, Wright’s system implements a security service that determines a decision of an AI security model is a false positive result, trains the FP identification model, and provides the trained FP identification model to the AI engine. The AI engine corresponds to the security agent.
12. Claims 13 is rejected under 35 U.S.C. § 103 as being unpatentable over Wright, Williams, and Nguyen as applied to claim 11, and further in view of Vashisht et al. (US 11,227,047 B1), hereafter Vashisht.
Regarding claim 103, Wright, Williams and Nguyen teach the elements of claim 11 as stated above.
Wright further teaches updating the decision validation model; and providing the updated decision validation model to the security agent on the client device to be used in place of the decision validation model; {Wright [Para. 0004] “The processor to use the training dataset to train a false positive identification model to identify false positives, and use the test dataset to evaluate the false positive identification model. Based on the evaluation,… the processor to alter and retrain the false positive identification model. Based on the evaluation,… the processor to apply an AI engine to user registration, where the AI engine includes an AI security model and the false positive identification model.”} Wright teaches training and retraining a false positive identification model to update the false positive identification model. After training, the system will apply the AI engine, including the updated false positive identification model, to user registration. The AI engine corresponds to the security agent. The FP identification model corresponds to the decision validation model.
However, Wright, Williams and Nguyen do not teach providing the updated decision validation model to the security agent on the client device to be used in place of the decision validation model.
However, Vashisht teaches providing the updated decision validation model to the security agent on the client device to be used in place of the decision validation model. {Vashisht [Col. 12, line 39-49] “Monitor 142 can periodically produce performance metrics for the trained machine learning models. When the monitor 142 determines that another machine learning model different from the machine learning model currently deployed in active mode exhibits a superior performance by, for example, achieving a significantly better classification accuracy than the currently deployed machine learning model, the CDS 123 replaces the currently deployed machine learning model with the machine learning model that has outperformed it.” [Col. 14, line 8-20] “At 403, the CDS 123 builds new machine learning models, and/or tunes, retrains, revalidates, and/or retests trained machine learning models 143 (shown in FIG. 2) with the new training datasets. The machine learning models are evaluated based on one or more performance metrics, at 405. The CDS 123 selects the machine learning model(s) that exhibit superior performance based on the one or more performance metrics, at 407, and replaces a previously deployed, active machine learning model with the selected machine learning model(s), at 409.”} Vashisht teaches retraining ML models to generate updated ML models. Vashisht’s CDS 123 replaces the currently deployed, active machine learning model with another updated machine learning model.
Vashisht is analogous art because each of Vashisht, Wright, Nguyen and Williams pertains to implementing machine learning models to generate security related classifications. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wright, Williams and Nguyen to include Vashisht’s teaching of providing an updated decision model to a security agent on a client device to be used in place of a decision model. Doing so would ensure “that an optimal cyber threat detection system is continuously operating in a compute device” (Vashisht, Col 3, line 61-63).
Conclusion
13. Any inquiry concerning this communication or earlier communications from the examiner should be directed to BIN QING ZHENG whose telephone number is (703)756-1535. The examiner can normally be reached on M-F 9:30 am -5:30 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Philip J. Chea can be reached on 571-272-3951. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BIN QING ZHENG/
Examiner, Art Unit 2499