Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment/Arguments
1. Applicant’s amendments to independent claim 12 overcome the rejection under 35 U.S.C. 101.
2. Applicant’s arguments filed on June 11, 2026, regarding the rejection under 35 U.S.C. 103 have been fully considered but are not persuasive.
Applicant argues that amended claim 12 includes a number of limitations allegedly not disclosed by Nasr-Azadani, including the recited network data analytics functions, trusted second apparatus functionality, retrieval and decryption of the machine learning model, model probing, and subsequent anomaly-detection actions. This argument is not persuasive. The present rejection does not rely on Nasr-Azadani alone to disclose every limitation of claim 12. Applicant is directed to the rejection, which identifies the particular teachings of Nasr-Azadani, Chen, Whatley, and Baldwin relied upon for each limitation of claim 12. Nasr-Azadani provides the production side execution and monitoring framework, including evaluation of the production machine learning model, detection of suspicious or adversarial behavior, transmission of information to a separate on-demand pipeline for further model inspection, and responsive action based on the results of the inspection. Chen, Whatley, and Baldwin provide the additional claimed implementation details, including the particular resource and network monitoring, communication networks analytics and model probing functionality, and encrypted model protection and controlled decryption, as specifically mapped in the rejection.
Applicant further characterizes Nasr-Azadani as merely detecting adversarial “live input data” rather than monitoring behavior of the machine learning model itself. This distinction is not persuasive. Nasr-Azadani evaluates incoming data while the production machine learning model is operating and uses a detected suspicious condition to initiate further inspection and evaluation. Thus, Nasr-Azadani’s use of live input is consistent with an operational monitoring environment rather than distinguishable from it. Moreover, the rejection does not rely on Nasr-Azadani alone for the particular categories of monitored behavior recited in claim 12. As set forth in the rejection, Chen provides the additional resource consumption and network communications monitoring, Baldwin provides the expected behavior and encrypted model functionality, and Whatley provides the explanation and model inspection functionality.
Applicant argues that Nasr-Azadani individually does not disclose the claimed network data analytics functionality, ADRF/NRF functionality, encrypted-model retrieval and decryption, or probing of decrypted model internals. These arguments again addresses Nasr-Azadani individually rather than the combined teachings forming the basis of the present rejection. Applicant is directed to the rejection for the specific reference teachings and explanations corresponding to each of these limitations. Obviousness does not require the primary reference, standing alone, to disclose every limitation where the rejection relies upon the collective teachings of the applied references. The rejection relies on Nasr-Azadani’s on-demand inspection and data storage functionality together with Whatley’s network analytics and probing functionality and Baldwin’s encrypted model and controlled decryption functionality.
To the extent Applicant’s remarks are directed specifically to Verma, those arguments are moot, as Verma is no longer relied upon in the present rejection. No further response to the asserted deficiencies of Verma is necessary.
Applicant further characterizes Nasr-Azadani as addressing a different problem from amended claim 12, asserting that Nasr-Azadani concerns adversarial live input data and model retraining or correction, whereas claim 12 concerns determining whether the machine learning model itself is anomalous or malicious. The distinction is not persuasive. Applicant’s own Specification directly relates these concepts. The Specification provides that detection of abnormal behavior of the machine learning model may initiate detection of an adversarial attack and further describes determining whether the machine learning model has been attacked or is “behaving maliciously or not.” See Specification paragraphs [0082] - [0085]. The Specification further provides that detection of either abnormal behavior or an adversarial attack may result in a request to the second apparatus for further anomaly detection. See Specification paragraphs [0086] – [0087]. The second apparatus thereafter obtains a decrypted version of the machine learning model to determine whether the model is malicious. See Specification paragraphs [0088] - [0090]. Thus, Applicant’s own Specification relates abnormal behavior, adversarial attack detection, and further determination of whether the machine learning model is malicious as part of the same machine learning security process.
Applicant’s own assertion of impermissible hindsight is likewise unpersuasive. The present rejection relies on the complementary teachings of Nasr-Azadani, Chen, Whatley, and Baldwin, with the particular teachings relied upon for each limitation identified in the rejection. The reason for combining the references arises from their complementary machine learning security functions rather than from Applicant’s disclosure. Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having a combination of Nasr-Azadani, Chen, Whatley, and Baldwin before them, to incorporate Chen’s monitoring and resource usage and network activity, Whatley’s network analytics and probing techniques, and Baldwin’s encrypted model protection and trusted decryption techniques into the machine learning model monitoring and on-demand inspection system of Nasr-Azadani. One would have been motivated to make such a combination in order to more securely monitor and evaluate a deployed machine learning model for abnormal or potentially adversarial behavior while protecting the model from unauthorized access. This would allow anomalous behavior to be identified using multiple sources of runtime information, permit further inspection of the model in a trusted environment, and improve the security and reliability of the deployed machine learning model against potential attacks.
Accordingly, Applicant’s arguments do not overcome the rejection of claim 12 under 35 U.S.C. 103. Applicant is directed to the rejection for the complete mapping and explanation of each limitation of claim 12.
Claim Objections
Claim 28 is objected to because it appears to contain a typographical error. Claim 28 recites “The second apparatus of Claim 28,” thereby improperly referring to itself. Correction is required to identify the proper claim from which claim 28 depends. For the purposes of the rejection set forth, claim 28 is treated as depending from claim 27.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 12 and 21-27 are rejected under the 35 U.S.C. 103 as being unpatentable over Nasr-Azadani et al., (Pub. No.: US 20210224425 A1 (Filed: 2021)) in view of Whatley (Pub. No.: US 20230033680 A1 (Filed: July 2022)) Chen et al., (Pub. No.: US 20170024660 A1 (Filed: 2015)) further in view of Baldwin (Pub. No.: US 20230409756 A1 (Filed: July 2020)).
Regarding claim 12, Nasr-Azadani in view of Whatley teaches the following limitations:
A second apparatus in a communications network, the second trusted by an owner of the machine learning model or by an operator, the second apparatus comprising: a network data analytics function in the communication network; at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the second apparatus at least to perform (Nasr-Azadani, paragraph [0023] “The results returned from the production model are made available via API for download by users with access to the system 100. This may also be implemented as part of the model manager.” [0046] “The system circuitry 404 may implement any desired functionality of the machine learning model production system and its various components. As just one example, the system circuitry 404 may include one or more instruction processor 418 and memory 420.” Whatley, paragraph [0205] “a time-based algorithm could be used for network monitoring, problem analysis and optimization when analyzing data covering an extended time span. Such an algorithm could also be used as part of real-time operational analytics for communications networks. In an example embodiment, an implementation of the algorithm could be executed periodically or continuously, and can feed an operational analytics system configured for deriving the evolution of possible causes and/or correlations in network performance over time.” – Under the broadest reasonable interpretation, being trusted by an owner/operator encompasses having authorized access to the system, as taught by Nasr-Azadani. A network data analytics function is a function that analyzes network-related data to generate analytics concerning operation or performance of the communications network; Whatley’s real-time analytics system performs such network monitoring and performance analysis.):
However, Nasr-Azadani in view of Whatley does not teach but Nasr-Azadani in view of Whatley further in view of Baldwin teaches:
receiving, from a first apparatus comprising a network data analytics function analytics logical function that executes an encrypted version of the machine learning model as a black-box container, a request for anomaly detection on the machine learning model (Nasr-Azadani, paragraph [0014] “the DE 104 may determine that the incoming data sample is adversarial and may submit an API call to the on-demand pipeline 150 for further inspection of the potentially adversarial data by an Inspection Engine (IE) 122 of the ARC 120 and for further correction of issues pertaining to either the detected adversarial data sample or the machine learning model.” [0025] “the on-demand pipeline 150 handles further inspection and correction of issues with the input data or machine learning model” Whatley, paragraph [0205] “Such an algorithm could also be used as part of real-time operational analytics for communications networks. In an example embodiment, an implementation of the algorithm could be executed periodically or continuously, and can feed an operational analytics system configured for deriving the evolution of possible causes and/or correlations in network performance over time.” [0057] “FIG. 1 is a simplified block diagram showing components of a system 100 for ML-based automatic performance and fault analysis, in accordance with example embodiments. As show, system 100 includes a Data Processor 106, a Database 108, an ML model 110” [0067] “Still referring to FIG. 1, when the processing unit is running over a virtualization layer, the components 106, 108, 110, 112, 114 may run inside one or multiple virtual machine or container instances.” Baldwin, paragraph [0162] “The service provider may encrypt the ML model 114 (or at least part of the ML model 114) and associated data and send it to the control module 804 on the end-point—passing through an untrusted OS (which cannot read the data it handles).” – Nasr-Azadani teaches a first, online production-side system that executes the production machine learning model, through the DE 104, transmits an API request to a separate on-demand pipeline 150 for further inspection of an issue pertaining to the machine learning model. Thus, the online production-side system corresponds to the first apparatus, while the on-demand pipeline 150/ARC 120 corresponds to the second apparatus receiving the anomaly-detection request. Whatley further teaches a communications network analytics function and execution of its ML model in a container instance. Baldwin teaches encrypting the ML model and protecting it from access by untrusted software. Under BRI, the combined teachings correspond to the first apparatus comprising a network analytics logical function that executes an encrypted machine learning model as a black-box container and transmits a request for anomaly detection to the second apparatus.),
wherein the request is transmitted by the first apparatus after the first apparatus detects abnormal behavior of the machine learning model during execution of the encrypted version of the machine learning model by comparing monitored behavior information of the machine learning model with expected behavior information received together with the encrypted version of the machine learning model (Nasr-Azadani, paragraph [0022] “ The CE component 112 may be designed to compare the prediction output 111 of the main model 110 against an ensemble of proxy models trained using different architectures and/or training data… these proxy models do not need to be as accurate as the main machine leaning model. They only need to be sufficient for verifying and auditing a consistent behavior for the main machine learning model.” [0022] “The comparison between the prediction of these proxy models and the main models may be conducted in 306 using various algorithms involving, for example, various comparison thresholds. In some implementations, any inconsistency among the results may be considered as indication of adversarial attack. In some other implementations, adversarial attack determination may be trigger only when the inconsistency level among the prediction results reaches certain level of inconsistency threshold. Once the CE component 112 determines that the input data contains an adversarial attack, it informs the escalator 108 for issuing an alert to the users, other components of the production system 100, and external systems via API.” [0023] “The DE results can be forwarded to any stages in the online pipeline 140 that may be added after the production model execution. This data can be used for evaluating the expected result (from the Detection Engine) versus the actual result (what the model returns, e.g., mis-prediction).” Baldwin, paragraph [0053] “ In some examples, the ML model 114 may be encrypted according a root of trust identity associated with the computing device 102.” [0054] “ the ML model 114 and the additional information may form a ‘model package’ as created by the controller or owner of the ML model 114. In some examples, the additional information may be referred to as a ‘contract’, ‘model contract’, ‘model specification’, ‘model execution specification’, ‘a condition’, ‘model execution condition’…” [0055] “ the additional information comprises… A ‘test procedure indicator’, for example, comprising a set of descriptions for any auxiliary processing that goes along with the main data pipeline such as a series of tests to check that data remains within a valid range.” – Nasr-Azadani teaches that the first apparatus evaluates the executing ML model by comparing its monitored runtime result with expected or reference results, where an inconsistency indicates abnormal or adversarial behavior, and thereafter initiates an API-based escalation. Under BRI, the model’s actual runtime result constitutes monitored behavior information. Baldwin further teaches receiving an encrypted ML model together with a model package containing execution specifications, conditions, and test criteria defining expected operation. Thus, the combination teaches detecting abnormal behavior by comparing monitored behavior of the executing encrypted model with expected behavior information received together with the encrypted model, followed by transmission of the request.),
the request at least comprising behavior information of the machine learning model during execution of the machine learning model on the first apparatus (Nasr-Azadani, paragraph [0014] “ the DE 104 may determine that the incoming data sample is adversarial and may submit an API call to the on-demand pipeline 150 for further inspection…For that purpose, the DE 104 may collect various data and pass such data to a data store and data management engine 124 of the ARC 120 for use by the IE 122.” [0023] “The DE results can be forwarded to any stages in the online pipeline 140 that may be added after the production model execution. This data can be used for evaluating the expected result (from the Detection Engine) versus the actual result (what the model returns, e.g., mis-prediction).” – Nasr-Azadani teaches that the first, production-side apparatus sends an API request to the on-demand inspection pipeline and provides data collected for use in that inspection. The forwarded information includes results associated with execution of the production ML model, including model’s actual returned result. Under BRI, such runtime model information constitutes behavior information of the machine learning model during execution.)
However, Nasr-Azadani in view of Watley further in view of Baldwin does not teach but Nasr-Azadani in view of Whatley further in view of Baldwin further in view of Chen teaches:
wherein the behavior information comprises: monitored resource consumption behavior indicating whether the machine learning model consumed processing resources when no request for execution of the machine learning model was sent to the machine learning model (Nasr-Azadani, paragraph [0023] “ In the online pipeline described above, the main model in production 110 is responsible for processing safe live input data (as determined by the DE 104) and returns the prediction results… The DE results can be forwarded to any stages in the online pipeline 140 that may be added after the production model execution.” Chen, [0069] “The behavior observer module 202 may also monitor the activities of the computing device by monitoring the system resource usage, which may include monitoring the number of forks, memory access operations, number of files open, etc.” [0100] “The specific execution state in which certain tasks/activities are performed in the computing device may be a strong indicator of whether a behavior or activity merits additional or closer scrutiny, monitoring and/or analysis. As such, in the various aspects, the device processor may be configured to use information identifying the actual execution states in which certain tasks/activities are performed to focus its behavioral monitoring and analysis operations, and better determine whether an activity is a critical activity and/or whether the activity is non-benign.” [0101] “the device processor may be configured to associate the activities/tasks performed by a software application with the execution states in which those activities/tasks were performed… As an example, the device processor may generate a behavior vector that includes a “location_background” data field whose value identifies the number or rate that the software application accessed location information when it was operating in a background state.” – Nasr-Azadani teaches execution of the production ML model on the first apparatus. Chen teaches monitoring system resource usage and associating monitored software activity with the execution state in which the activity occurs, including a background state. Applying Chen’s resource and execution state monitoring to Nasr-Azadani’s production model provides monitoring of whether the model consumes processing resources outside an affirmative model-execution activity, corresponding under BRI to resource consumption when no request for execution of the model was sent.);
monitored network communication behavior identifying one or more host addresses or port numbers used by the machine learning model during the execution on the first apparatus (Nasr-Azadani, paragraph [0023] “In the online pipeline described above, the main model in production 110 is responsible for processing safe live input data (as determined by the DE 104) and returns the prediction results.” Chen, paragraph [0068] “The behavior observer module 202 may also monitor the activities of the computing device by monitoring data network activity, which may include types of connections, protocols, port numbers, server/client that the device is connected to, the number of connections, volume or frequency of communications, etc.” – Nasr-Azadani teaches execution of the production machine learning model on the first, online production-side apparatus. Chen teaches monitoring network activity of executing software, including the port numbers and server/client connections used during operation. Applying Chen’s network activity monitoring to Nasr-Azadani’s executing production model provides monitored network communication behavior identifying port numbers used by the machine learning model during execution on the first apparatus.);
at least one monitored model output provided by the machine learning model for at least one model input; and an explanation of a decision by the first apparatus that the abnormal behavior occurred (Nasr-Azadani, paragraph [0022] “In some implementations, any inconsistency among the results may be considered as indication of adversarial attack… Once the CE component 112 determines that the input data contains an adversarial attack, it informs the escalator 108 for issuing an alert to the users, other components of the production system 100, and external systems via API.” [0023] “In the online pipeline described above, the main model in production 110 is responsible for processing safe live input data (as determined by the DE 104) and returns the prediction results…This data can be used for evaluating the expected result (from the Detection Engine) versus the actual result (what the model returns, e.g., mis-prediction).” Whatley, paragraph [0015] “The conventional use of SHAP is for model interpretability, to understand and explain why a model is making specific predictions…” [0096] “Analysis may be performed of problematic data samples by producing a SHAP explainer from the learned model, using it to provide explanations of data in problematic samples.” – Nasr-Azadani provides the overall first apparatus production system and teaches receiving model input, monitoring the resulting model output, and determining abnormal/adversarial behavior from the model results. Whatley further teaches generating explanations identifying why model results are associated with a problematic outcome. Incorporating Whatley’s explanation functionality into Nasr-Azadani’s detection framework provides an explanation supporting the first apparatus’s decision that abnormal behavior occurred.);
obtaining a decrypted version of the machine learning model by: retrieving, from an analytics data repository function, an encrypted version of the machine learning model (Nasr-Azadani, paragraph [0025] “The ARC 120 may further include a data store and data management (DM) component 124 for facilitating the IE 122 and the COE 126 to perform their functionalities.” Baldwin, paragraph [0053] “ In some examples, the OS 116 comprises ‘fetch model’ instructions 116a to download at least part of the ML model 114 from the cloud 104. Thus, where a reference is made herein to receiving the ML model 114 from the cloud, this may refer to receiving part of or the entire ML model 114. In some examples, the ML model 114 may be encrypted according a root of trust identity associated with the computing device 102.” [0069] “The cloud 104 may be trusted by the third party entity that owns the ML model 114 and at least part of the ML model 114 may be stored in and accessible from the cloud 104. The cloud 104 may store the additional information and/or implement cryptographic controls for ensuring the integrity of the ML model 114 and/or the additional information.” – Nasr-Azadani provides the overall system and teaches that the on-demand inspection apparatus includes a data store and data-management component used to facilitate inspection and model evaluation. Under BRI, this component corresponds to an analytics data repository function. Baldwin further teaches storing an ML model in a repository and retrieving the model therefrom, with the retrieved model being encrypted. Incorporating Baldwin’s encrypted model storage and retrieval into Nasr-Azadani’s analytics data store provides retrieval of an encrypted version of the machine learning model from the analytics data repository function.);
retrieving, from a network repository function operating as an authorization server for the machine learning model, an encryption key for the machine learning model; and decrypting the encrypted version of the machine learning model with the encryption key to obtain the decrypted version of the machine learning model (Whatley, paragraph [0167] “ In the example fingerprint graphed in FIG. 7A, pinning functionality to specific cores in the CPU layer of the system allows us to know which cores are running the Network Repository Function (NRF) Cores group (i.e., those cores running the 5G service)” Baldwin, paragraph [0159] “In response to receiving an indication that a computing device 102 under control of the control module 804 complies with a third party policy (e.g., the ‘additional information’ such as a model contract) associated with the machine learning model 114, the control module 804 is to release the information to a component of the computing device 102.” [0184] “In some examples, the instructions 1012 to load the machine learning model to the computing device 102 comprise instructions to release a private portion of the key pair to enable the computing device 102 to decrypt the encrypted version of the machine learning model and load the machine learning model to the computing device 102.” – Whatley teaches a Network Repository Function within the communications network. Baldwin teaches conditionally releasing protected information associated with an ML model after determining compliance with a policy governing the model, and specifically releasing key material that enables decryption of the encrypted ML model. Incorporating Baldwin’s policy-controlled key-release functionality into Whatley’s Network Repository Function provides an NRF operating as an authorization server for the machine learning model. The second apparatus obtains the released encryption key from that NRF and uses the key to decrypt the encrypted version of the machine learning model.);
detecting anomaly of the machine learning model by applying one or more probes to model internals of the decrypted version of the machine learning model and analyzing the decrypted version of the machine learning model against the behavior information comprising the monitored resource consumption behavior, the monitored network communication behavior, the at least one monitored model output, and the explanation received from the first apparatus (Nasr-Azadani, paragraph [0025] “] As shown in FIG. 1, the on-demand pipeline 150 handles further inspection and correction of issues with the input data or machine learning model by model retraining via the ARC 120. The inspection functionalities may be performed by the inspection engine (IE) 122 of FIG. 1 while the correction functionalities may be performed by the correction engine (COE) 126.” [0023] “In the online pipeline described above, the main model in production 110 is responsible for processing safe live input data (as determined by the DE 104) and returns the prediction results…This data can be used for evaluating the expected result (from the Detection Engine) versus the actual result (what the model returns, e.g., mis-prediction).” Chen, paragraph [0068] “The behavior observer module 202 may also monitor the activities of the computing device by monitoring data network activity, which may include types of connections, protocols, port numbers, server/client that the device is connected to, the number of connections, volume or frequency of communications, etc.” [0069] “The behavior observer module 202 may also monitor the activities of the computing device by monitoring the system resource usage, which may include monitoring the number of forks, memory access operations, number of files open, etc.” Whatley, paragraph [0014] “Once the representation is formed, the quality of which can be determined by the model prediction performance on samples of unseen data, the strategy involves probing the model to gauge the importance of different input features or elements (e.g., a continuous or discrete value, setting, or category identifier for a feature) on specific outcomes given a problematic context” [0015] “The SHAP technique computes Shapley values for the marginal expectation or conditional expectations for feature values in the context of a specific example by analyzing a machine learning model. This approach provides fair contributions of each feature-value pair to the model prediction.” [0096] “Analysis may be performed of problematic data samples by producing a SHAP explainer from the learned model, using it to provide explanations of data in problematic samples. By framing specific questions, where samples of interest are compared against a representative baseline, control sample, divergence between the relative importance of different features can be quantified to highlight specific problems in the telecom data for the problematic sample.” – Nasr-Azadani provides the overall system in which the second, on-demand apparatus performs further inspection of the machine learning model. The behavior information used in that inspection is supplied by the previously combined teachings: Chen provides monitored resource-consumption and network communication behavior, Nasr-Azadani provides monitored model output, and Whatley provides explanatory information associated with problematic behavior. Whatley further teaches probing a machine learning model and analyzing the model using SHAP to determine the contributions associated with problematic outcomes. Accordingly, after the model is decrypted as set forth in the preceding limitation, applying Whatley’s probing and model-analysis techniques into Nasr-Azadani’s second apparatus inspection system provides analysis of the decrypted model against the received monitored behavior information to detect an anomaly.);
transmitting, to the first apparatus and based on the detecting, a response at least indicating a positive detection or a negative detection of anomaly of the machine learning model (Nasr-Azadani, paragraph [0032] “If a data sample is suspected to be adversarial as determined by the DE 104, the ME 130 would function to evaluate the model's robustness against the detected adversarial attack. If the model is determined to be robust to the detected attack, no retraining is necessary… If the production machine learning model 110 is not robust to the detected attack, then the ME 130 may generate an output that triggers the model retrainer (MR) 128 for model retraining. Additionally, the ME 130 may further return output that informs the DE 104 to prevent future data that matches the profile of the detected adversarial data from reaching the model in production.” – Nasr-Azadani teaches that the model evaluator of the second, on-demand apparatus evaluates the production machine learning model and reaches alternative results depending on whether the model satisfies the evaluation, and further returns output from the model evaluator to the DE 104 of the first apparatus. In the combined system, where the preceding limitation uses the second apparatus to detect whether the decrypted machine learning model is anomalous, this return mechanism provides the first apparatus with the result of that detection. Thus, the returned response indicates either a positive detection or a negative detection of anomaly of the machine learning model.);
when the response indicates the positive detection of anomaly of the machine learning model, transmitting, to the first apparatus, a recommendation to discard the machine learning model (Nasr-Azadani, paragraph [0032] “ If the production machine learning model 110 is not robust to the detected attack, then the ME 130 may generate an output that triggers the model retrainer (MR) 128 for model retraining. Additionally, the ME 130 may further return output that informs the DE 104 to prevent future data that matches the profile of the detected adversarial data from reaching the model in production.” [0033] “As described above, when a new unique adversarial attack is discovered by the DE 104 or the CE 112, the production system 100 should deploy two things from ARC 120: retrained robust model to replace the main production model 110 in the online pipeline 140 and updates to DE 104 for detection method/algorithm for detecting the identified attack in the future.” – Nasr-Azadani teaches that, when the model is determined not to be robust to a detected attack, the second-side model evaluator returns output to the DE 104 and the ARC 120 supplies a retrained model to replace the existing production model 110. Under BRI, identifying the existing model for replacement with a corrected model corresponds to recommending that the existing machine learning model be discarded following a positive detection of a problem with the model.); and
when the response indicates the positive detection of anomaly of the machine learning model, transmitting, to a third apparatus corresponding to a producer of the machine learning model, an indication of the positive detection of anomaly of the machine learning model (Nasr-Azadani, paragraph [0014] “As shown by 107 of FIG. 1, the DE 104 may alternatively determine that the incoming data sample is adversarial and returns an alert via the escalator 108. This alert may be sent via API and may be received by another component of the system 100 or an external system.” Baldwin, paragraph [0038] “In the example of FIG. 1, the system 100 comprises a computing device 102 communicatively coupled to a cloud 104 (e.g., operated by a service provider or, in some examples, operated by an untrusted entity) via a network connection 106 (e.g., wired or wireless).” [0054] “In some examples, the ML model 114 and the additional information may form a ‘model package’ as created by the controller or owner of the ML model 114.” – Nasr-Azadani teaches transmitting an indication of a detected abnormal/adversarial condition to an external system via an API. Baldwin further teaches a separate network apparatus, such as cloud 104 operated by the service provider, where the service provider is the controller or owner associated with the ML model. Under BRI, Baldwin’s model provider/controller corresponds to a producer of the machine learning model. Incorporating Baldwin’s model-provider apparatus as a recipient of Nasr-Azadani’s detection notification provides transmission of the positive anomaly indication to a third apparatus corresponding to the producer of the machine learning model.).
Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having a combination of Nasr-Azadani, Chen, Whatley, and Baldwin before them, to incorporate Chen’s monitoring of resource usage and network activity, Whatley’s network analytics and model probing techniques, and Baldwin’s encrypted model protection and trusted decryption techniques into the machine learning model monitoring and on-demand inspection system of Nasr-Azadani. One would have been motivated to make such a combination to more securely monitor and evaluate a deployed machine learning model for abnormal or potentially adversarial behavior while protecting the model from unauthorized access. This would allow anomalous behavior to be identified using multiple sources of runtime information, permit further inspection of the model in a trusted environment, and improve the security and reliability of a deployed machine learning model against potential attacks.
Regarding claim 21, Nasr-Azadani in view of Whatley further in view of Baldwin further in view of Chen teaches all the elements of claim 12, therefore is rejected for the same reasons as those presented for claim 12. Nasr-Azadani in view of Whatley further in view of Baldwin further in view of Chen further teaches:
wherein the request further comprises an abnormal trust score indicating a confidence level of the first apparatus that the abnormal behavior of the machine learning model occurred during execution of the encrypted version of the machine learning model (Nasr-Azadani, paragraph [0014] “the DE 104 may determine that the incoming data sample is adversarial and may submit an API call to the on-demand pipeline 150 for further inspection of the potentially adversarial data by an Inspection Engine (IE) 122 of the ARC 120… For that purpose, the DE 104 may collect various data and pass such data to a data store and data management engine 124 of the ARC 120 for use by the IE 122.” [0024] “ The on-demand pipeline 150 may be triggered by the generalizable detection engine 104 which, as described above, may be agnostic to the model in production and detects incoming data sample as adversarial above a given confidence threshold.” – As established with respect to claim 12, the combined system provides the first apparatus executing the encrypted version of the machine learning model and detecting abnormal behavior during that execution. Nasr-Azadani further teaches making an abnormal/adversarial determination according to a confidence threshold and transmitting detection information to the on-demand apparatus for further inspection. Incorporating Nasr-Azadani’s confidence-based determination into the abnormal behavior detection of claim 21 provides an abnormal trust score indicating the first apparatus’s confidence that the detected abnormal behavior occurred during execution of the encrypted machine learning model, with the score included with the detection information in the request.).
Regarding claim 22, Nasr-Azadani in view of Whatley further in view of Baldwin further in view of Chen teaches all the elements of claim 21, therefore is rejected for the same reasons as those presented for claim 21. Nasr-Azadani in view of Whatley further in view of Baldwin further in view of Chen further teaches:
wherein the abnormal behavior detected by the first apparatus is determined by the first apparatus only after a mismatch between the monitored behavior information and the expected behavior information occurs for a threshold number of times or lasts for a threshold period of time (Chen, paragraph [0094] “ Boosted decision stumps are one level decision trees that have exactly one node (and thus one test question or test condition) and a weight value, and thus are well suited for use in a binary classification of data/behaviors… For example, if the question/condition tested by a boosted decision stump is “is the frequency of Short Message Service (SMS) transmissions less than x per minute,” applying a value of “3” to the boosted decision stump will result in either a “yes” answer (for “less than 3” SMS transmissions) or a “no” answer (for “3 or more” SMS transmissions).” [0101] “As an example, the device processor may generate a behavior vector that includes a “location_background” data field whose value identifies the number or rate that the software application accessed location information when it was operating in a background state… Generating the behavior vector in this manner also allows the system to aggregate information (e.g., frequency or rate) over time.” – As established in claim 12, the combined system compares monitored behavior information with expected behavior information to determine abnormal behavior of the machine learning model. Chen further teaches aggregating the number or rate of occurrences of monitored behavior over time and determining behavior according to whether the number of occurrences satisfies a specified threshold within a period of time. Applying Chen’s teaching to the mismatch determination of claim 12 provides determining abnormal behavior only after the monitored mismatch has occurred a threshold number of times.).
Regarding claim 23, Nasr-Azadani in view of Whatley further in view of Baldwin further in view of Chen teaches all the elements of claim 22, therefore is rejected for the same reasons as those presented for claim 22. Nasr-Azadani in view of Whatley further in view of Baldwin further in view of Chen further teaches:
wherein the expected behavior information is accessible to the second apparatus and indicates at least one of: an expected resource consumption behavior of the machine learning model, an expected network communication behavior of the machine learning model, an expected type of a model output of the machine learning model, an expected value range of the model output, or an expected explanation type of the machine learning model (Nasr-Azadani, paragraph [0025] “The ARC 120 may further include a data store and data management (DM) component 124 for facilitating the IE 122 and the COE 126 to perform their functionalities. The ARC 120 may inspect the output of the detection engine 104…” Baldwin, paragraph [0054] “ In some examples the ML model 114 (or at least part of the ML model 114) downloaded from the cloud 104 may be accompanied by additional information in order to support third party entity (e.g., service provider) control over the implementation of the ML model 114 … In some examples, the ML model 114 and the additional information may form a ‘model package’ as created by the controller or owner of the ML model 114.” [0058] “In some examples, the additional information (e.g., a model contract) may define the expected data flows (e.g., data sources, transformation paths and security properties) that are acceptable to the third party entity in use of the ML model 114 by the computing device 102.” – Nasr-Azadani provides the overall second, on-demand apparatus and a data store used by the inspection and model-evaluation components in performing further inspection. Baldwin further teaches providing an ML model together with additional model-package information that defines expected data flows for use of the model. Under BRI, expected data flows define expected network communication behavior of the machine learning model. Incorporating Baldwin’s model package information into Nasr-Azadani’s second apparatus inspection system makes the expected behavior information accessible to the second apparatus for use in evaluating the machine learning model.).
Regarding claim 24, Nasr-Azadani in view of Whatley further in view of Baldwin further in view of Chen teaches all the elements of claim 23, therefore is rejected for the same reasons as those presented for claim 23. Nasr-Azadani in view of Whatley further in view of Baldwin further in view of Chen further teaches:
wherein detecting the anomaly comprises determining, using the decrypted version of the machine learning model and the expected behavior information, that the monitored behavior information includes at least one of: resource consumption by the machine learning model when no request for execution was sent to the machine learning model, network communication using a host address or port number outside the expected network communication behavior, a model output having a type different from the expected type, a model output having a value outside the expected value range, or an explanation having a type different from the expected explanation type (Nasr-Azadani, paragraph [0023] “The DE results can be forwarded to any stages in the online pipeline 140 that may be added after the production model execution. This data can be used for evaluating the expected result (from the Detection Engine) versus the actual result (what the model returns, e.g., mis-prediction).” Baldwin, paragraph [0055] “Examples of specifications for such components include:…(5) A ‘test procedure indicator’, for example, comprising a set of descriptions for any auxiliary processing that goes along with the main data pipeline such as a series of tests to check that data remains within a valid range.” [0056] “This additional information may specify how the computing device 102 is to be set up and/or how to operate a data processing pipeline for executing the ML model 114. In other similar words, the additional information may provide a way for the third party entity (e.g., service provider or owner of the ML model 114) to define how the ML model 114 is to be executed by the computing device 102.” [0060] “In some examples, the additional information may comprise a test property (e.g., a test to be performed on model load and/or acceptable performance thresholds resulting from such a test)…In some examples, the test policy may comprise an input, associated output and acceptance criteria to determine whether the output is sufficiently in line with what is expected for the given input.” – As established with respect to claims 12 and 13, the combined system provides a second apparatus with the decrypted version of the machine learning model and expected behavior information. Nasr-Azadani teaches evaluating the actual result returned by the machine learning model against an expected result. Baldwin further teaches that the expected model-execution information includes valid-range testing, acceptable performance thresholds, and acceptance criteria for determining whether an associated model output is sufficiently in line with the expected output. Under BRI, Baldwin’s acceptable performance thresholds and output acceptance criteria define an expected value range for the model output. Accordingly, using the decrypted model and the expected behavior information to determine that the monitored model output fails those criteria corresponds to determining that the model output has a value outside the expected value range.).
Regarding claim 25, Nasr-Azadani in view of Whatley further in view of Baldwin further in view of Chen teaches all the elements of claim 24, therefore is rejected for the same reasons as those presented for claim 24. Nasr-Azadani in view of Whatley further in view of Baldwin further in view of Chen further teaches:
wherein retrieving the encryption key comprises transmitting, to the network repository function, a key request including model identity information of the machine learning model, the model identity information having been registered with the network repository function by a third apparatus corresponding to a producer of the machine learning model (Whatley, paragraph [0167] “ In the example fingerprint graphed in FIG. 7A, pinning functionality to specific cores in the CPU layer of the system allows us to know which cores are running the Network Repository Function (NRF) Cores group (i.e., those cores running the 5G service)…” Baldwin, paragraph [0054] “ In some examples, the ML model 114 and the additional information may form a ‘model package’ as created by the controller or owner of the ML model 114.” [0055] “In some examples, the additional information comprises a set of descriptions of the data pipeline for the ML model 114 and the associated hash of the model 114…” [0103] “When a service is running in the cloud 104, it may be straightforward for the service provider who created the ML model 114 to ensure that the correct data pipeline and ML model 114 is used.” [0223] “In some examples, the model provider can then send an encrypted model to the computing device 102 where the encryption key K.sub.enc and a nonce, nonce, is encrypted with the pk.sub.bind.” – As established with respect to claim 12, Whatley provides the Network Repository Function within the communication-network arrangement. Baldwin further teaches that a producer or owner of the machine learning model provides model-specific information including an associated hash of the model and also provides cryptographic information associated with the encrypted model. Under BRI, the model hash constitutes model identity information. Incorporating Baldwin’s model specific identification and encryption-key information into Whatley’s NRF-based arrangement provides registering the producer-supplied model identify information with the NRF and subsequently including the model identity information in a key request transmitted to the NRF to identify the corresponding encryption key.).
Regarding claim 26, Nasr-Azadani in view of Whatley further in view of Baldwin further in view of Chen teaches all the elements of claim 25, therefore is rejected for the same reasons as those presented for claim 25. Nasr-Azadani in view of Whatley further in view of Baldwin further in view of Chen further teaches:
wherein retrieving the encrypted version of the machine learning model comprises downloading, from the analytics data repository function, the encrypted version of the machine learning model that was previously stored in the analytics data repository function together with the expected behavior information by a third apparatus corresponding to a producer of the machine learning model (Nasr-Azadani, paragraph [0025] “The ARC 120 may further include a data store and data management (DM) component 124 for facilitating the IE 122 and the COE 126 to perform their functionalities.” Baldwin, paragraph [0053] “In some examples, the OS 116 comprises ‘fetch model’ instructions 116a to download at least part of the ML model 114 from the cloud 104…In some examples, the ML model 114 may be encrypted according a root of trust identity associated with the computing device 102.” [0054] “In some examples the ML model 114 (or at least part of the ML model 114) downloaded from the cloud 104 may be accompanied by additional information in order to support third party entity (e.g., service provider) control over the implementation of the ML model 114…In some examples, the ML model 114 and the additional information may form a ‘model package’ as created by the controller or owner of the ML model 114.” [0069] “The cloud 104 may be trusted by the third party entity that owns the ML model 114 and at least part of the ML model 114 may be stored in and accessible from the cloud 104. The cloud 104 may store the additional information and/or implement cryptographic controls for ensuring the integrity of the ML model 114 and/or the additional information. The cloud 104 may be under the control of the third party entity or at least be trusted by the third party entity.” [0103] “When a service is running in the cloud 104, it may be straightforward for the service provider who created the ML model 114 to ensure that the correct data pipeline and ML model 114 is used.” – Nasr-Azadani provides the overall system and a data store used by its inspection and model-evaluation components, which under BRI corresponds to the claim analytics data repository function. Baldwin further teaches storing a ML model and associated additional information in a cloud repository controlled or trusted by the owner of the model, downloading the encrypted model from that repository, and receiving the model together with respect to claim 23, Baldwin’s additional information corresponds to the expected behavior information. Baldwin further identifies the service provider as the entity that created the ML model. Incorporating Baldwin’s model-package storage and retrieval into Nasr-Azadani’s analytics data repository therefore provides downloading from the analytics data repository function the encrypted model previously stored together with the expected behavior information by a third apparatus corresponding to the producer of the machine learning model.).
Regarding claim 27, Nasr-Azadani in view of Whatley further in view of Baldwin further in view of Chen teaches all the elements of claim 26, therefore is rejected for the same reasons as those presented for claim 26. Nasr-Azadani in view of Whatley further in view of Baldwin further in view of Chen further teaches:
wherein the third apparatus corresponding to the producer of the machine learning model comprises a network data analytics function model training logical function that trained the machine learning model (Whatley, paragraph [0016] “The method may include: obtaining a set of computer-readable training data records that each characterize operation of a communication network… using at least a portion of the set of training data records to train a machine learning (ML) model of network performance to predict expected performance characteristics given the plurality of operational features in the training data records as input and the one or more observed performance characteristics as ground truths…” [0205] “ Example embodiments of PFA techniques and PFA systems may also be extended to be able to provide real-time analysis of communication networks as they operate… Such an algorithm could also be used as part of real-time operational analytics for communications networks.” Baldwin, paragraph [0103] “When a service is running in the cloud 104, it may be straightforward for the service provider who created the ML model 114 to ensure that the correct data pipeline and ML model 114 is used.” – As established with respect to claim 26, Baldwin provides the third apparatus corresponding to the producer of the machine learning model. Whatley further teaches a communication network analytics system that trains a ML model of network performance using data characterizing operation of the communications network. Under BRI, the logical functionality within the network analytics system that performs this model training corresponds to the claimed network data analytics function model training logical function. Incorporating Whatley’s network analytics model training functionality into Baldwin’s model producing apparatus therefore provides the third apparatus comprising a network data analytics function model training logical function that trained the machine learning model.).
Claim 28 is rejected under the 35 U.S.C. 103 as being unpatentable over Nasr-Azadani et al., (Pub. No.: US 20210224425 A1 (Filed: 2021)) in view of Whatley (Pub. No.: US 20230033680 A1 (Filed: July 2022)) Chen et al., (Pub. No.: US 20170024660 A1 (Filed: 2015)) further in view of Baldwin (Pub. No.: US 20230409756 A1 (Filed: July 2020)) further in view of Ramanathan et al., (NPL: “BLAG: Improving Accuracy of Blacklists” (Published: 2020)).
Regarding claim 28, Nasr-Azadani in view of Whatley further in view of Baldwin further in view of Chen teaches all the elements of claim 27, therefore is rejected for the same reasons as those presented for claim 27. Nasr-Azadani in view of Whatley further in view of Baldwin further in view of Chen further teaches:
wherein the second apparatus is further caused to perform, in accordance with the positive detection of the anomaly of the machine learning model, adding a vendor of the machine learning model into a blacklist (Ramanathan, [Introduction] “IP blacklists (“blacklists” for short), which contain identities of prior known offenders, are usually used to aid more sophisticated defenses, such as spam filters or security information and event management (SIEM) systems, in identifying traffic that warrants further analysis” [page 6, section C] “The expansion phase starts with master blacklist candidates, which are all added to the BLAG master blacklist.” – As established by the limitations inherited from claim 12, the combined system positively detects an anomaly of the machine learning model and identifies the producer associated with the model. Ramanathan further teaches adding identified entities associated with undesirable activity to a blacklist. Applying Ramanathan’s blacklisting technique to the established anomaly-detection system provides adding the vendor associated with the anomalous machine learning model to a blacklist in accordance with the positive detection.).
Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having the combination of Nasr-Azadani, Chen, Whatley, Baldwin, and Ramanathan before them, to incorporate Ramanathan’s blacklisting technique into the machine-learning anomaly detection system of the combined references. One would have been motivated to do so in order to restrict or prevent further access by a vendor associated with a positively detected anomalous or malicious machine learning model, thereby reducing the opportunity for the vendor to conduct further attacks or tamper with the machine learning model or protected system.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Daravanh Phakousonh whose telephone number is (571)272-6324. The examiner can normally be reached Mon - Thurs 7 AM - 5 PM, Every other Friday 7 AM - 4PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at 571-272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Daravanh Phakousonh/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights