DETAILED ACTION
This action is in response to Applicant’s Amendment ("Response”) received on July 22, 2026 in response to the Office Action dated April 24, 2026. This action is made Final.
Claims 1-20 are pending in the case.
Claims 1, 10, and 19 are independent claims.
Claims 1-5, 7-14, and 16-20 are rejected.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Applicant’s Response
In Applicant’s Response, Applicant amended claims 1, 3-7, 10, 12, 14-16, and 19, and submitted arguments against the prior art in the Office Action dated April 24, 2026.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 9-12, and 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Djukic, et al., US Publication 2022/0330027 (“Djukic”), in view of Van Baalen et al., US Publication 2023/0058159 (“Baalen”), and further in view of Baharloui et al., US Publication 2023/0107463 (“Baharloui”)
Claim 1:
Djukic teaches or suggests a method comprising:
generating, using the neural network, an ensemble of neural networks having a plurality of ... members (see para. 0007 - machine learning models may include a Deep Neural Network (DNN) technique, an ensemble technique; para. 0051 -provides algorithmically combined outputs of multiple machine learning strategies (algorithms), some of which may include ensemble learning algorithms; para. 0074 - machine learning algorithms may include at least one of ensemble algorithms, DNNs; para. 0120 - classifier in the classifier layers 132. For example, this may include DNN, XGBoost, or the like. Also, the classifier layers 132 may be a DNN version that can be implemented in any number of ways; para. 0126 - attack classifiers 120, 140, 150 may be configured to perform a procedure that includes training K classifiers on the same dataset.); and
combining predictions of the plurality of ... members of the ensemble to harden the neural network … that protects the ensemble against one or more adversarial attacks (see para. 0006 - combining the outputs of the multiple analysis procedures; para. 0051 – provides algorithmically combined outputs of multiple machine learning strategies (algorithms), some of which may include ensemble learning algorithms; para. 0073 - decision point can use several machine algorithms and combine their outputs to detect the attack type; para. 0122 - classifiers 146 are configured to classify the data to determine a type of network intrusion or sub-attack and provide the results to an output combiner 148, which is configured to combine the outputs. In this way, the results of each of the classifiers 146 can be used in a single element to determine multiple types of sub-attacks; para. 0126 - attack classifiers 120, 140, 150 may be configured to perform a procedure that includes training K classifiers on the same dataset. Each of the classifiers may be configured to determine a score during testing for each of the sub-attack types. may use known test scores to decide on which algorithms perform preferable prediction results that can be used at inference time; para. 0127 - At inference time, the methods may take a given input and use all K classifiers to determine the output. Since the scores are available, indicating how well each of the classifiers can decide what the category of sub-attack is, the goal may be to create a prediction that is most likely to be correct or one that maximizes the overall score of the classifiers; para. 0131 - outputs of multiple machine learning procedures can be combined in many ways. In this way, the overall system can improve the performance of the groups of techniques, procedures, or algorithms over individual algorithms, such as the individual algorithms of the conventional systems.).
Djukic does not explicitly disclose applying stochastic quantization to one or more layers of a neural network; quantized; wherein at least one of weights or activations of each of quantized members have different bit precision; quantized … by establishing a bounded worst-case performance profile.
Baalen teaches or suggests applying stochastic quantization to one or more layers of a neural network; quantized; wherein at least one of weights or activations of each of quantized members have different bit precision; quantized (see Fig. 2, 3; para. 0016 - quantizing a first layer of the mixed-precision artificial neural network at a first precision determined using corresponding stochastic gates of quantization and pruning gates, and quantizing a second layer of the mixed-precision artificial neural network at a second precision determined using corresponding stochastic gates of the quantization and pruning gates, in which the first precision and the second precision are different; para. 0044 – Mixed precision quantization having different bit-widths for different tensors may yield improved accuracy to efficiency trade-offs compared to quantizing all tensors to the same bit-width; para. 0046 - size of the space for mixed-precision bit-width configurations in a neural network depends exponentially on the number of tensors. Lowering the bit-width of one tensor in a neural network affects the quantization sensitivity of other tensors. Thus, the bit-width for any given tensor cannot be chosen in isolation; para. 0056 - mixed-precision quantization and structured pruning may quantize weight values and/or activation values resulting in correspondingly quantized output channels 202a, 202b, 202c, 202d. For another example, mixed-precision quantization and structured pruning may prune weight values and/or activation values resulting in correspondingly pruned output channels.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Djukic, to include applying stochastic quantization to one or more layers of a neural network; quantized; wherein at least one of weights or activations of each of quantized members have different bit precision; quantized for the purpose of efficiently quantizing with different bitwidths to achieve quantized weight and/or activation values, improving model accuracy and efficiency, as taught by Baalen.
Baharloui teaches or suggests by establishing a bounded worst case performance profile (see Fig. 12; para. 0003 - obtaining a worst-case bound on a classification error and loss for perturbed versions of the input data; para. 0018 - training a neural network classification system with an abstain (rejection) option with provable robust (worst-case/adversarial) performance. defense mechanism is designed such that it will either (1) classify adversarial perturbed inputs as the abstain/rejection/detection class, or (2) it will correctly classify it as they class and thus prevents misclassification and fooling of the system; para. 0026 - minimize an upper bound of the worst-case loss of perturbed training samples (within a perturbation model) together with the traditional (robust) cross-entropy loss of the (clean) inputs. Thus, the increase in the crossentropy of perturbed samples (attacks or not) is bounded, and the effect of the attack is mitigated; para. 0057 – bounding the training objective (e.g., training objective function) by its worstcase upper bound via the interval bound propagation technique; para. 0121 - obtains a worst-case bound on a classification error and loss for perturbed versions of the input data; para. 0124 - detect at least one additional abstain class of the plurality of additional abstain classes in response to obtaining the worst-case bound; para. 0126 - wherein the classifier does not classify the input data as the original classes when the input data includes perturbations. In some embodiments, the method also includes bounding a training objective function by a worst-case upper bound utilizing an interval bound propagation (IBP) technique; para. 0128 – utilize interval bound propagation to compute a worst-case bound on a classification error and classification loss associated with perturbed versions of the input data; para. 0129 - additional abstain class of the plurality of additional abstain classes is determined in response to at least bounding input data including one or more perturbations.).
Accordingly, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Baharloui with those of Djukic. One would have been motivated to do so for the purpose of efficiently identifying and bounding worst case classification performance in order to identify adversarial data, securing model performance, as taught by Baharloui (0018, 0026, 0126, and 0128).
Claim(s) 19:
Claim(s) 19 correspond to Claim 1, and thus, Djukic, Baalen, and Baharloui teach or suggest the limitations of claim(s) 19 as well.
Claim 2:
Djukic further teaches or suggests extracting one or more features from one or more layers of the neural network (see para. 0010 - examining the extracted information using Machine Leaming (ML). Then, the process may further include determining if the extracted information includes one or more malicious threats; para. 0120 - The layers 126, 128, 130 may use fully-connected layers, convolutional layers, and/or other types of DNN layers. DNNs would understand that this layer can be implemented using a pretrained DNN model (e.g., Bidirectional Encoder Representations from Transformers (BERT), or the like) to perform a word embedding procedure to extract information from the messages; para. 0180 – examining the extracted information using Machine Learning (ML), as indicated in block 334. The process 330 further includes determining if the extracted information includes one or more malicious threats.).
Baalen further teaches or suggests applying stochastic quantization to an input layer of the neural network; applying stochastic quantization to the one or more layers of the neural network (see Fig. 2, 3; para. 0016 - quantizing a first layer of the mixed-precision artificial neural network at a first precision determined using corresponding stochastic gates of quantization and pruning gates, and quantizing a second layer of the mixed-precision artificial neural network at a second precision determined using corresponding stochastic gates of the quantization and pruning gates, in which the first precision and the second precision are different; para. 0044 – Mixed precision quantization having different bit-widths for different tensors may yield improved accuracy to efficiency trade-offs compared to quantizing all tensors to the same bit-width; para. 0046 - size of the space for mixed-precision bit-width configurations in a neural network depends exponentially on the number of tensors. Lowering the bit-width of one tensor in a neural network affects the quantization sensitivity of other tensors. Thus, the bit-width for any given tensor cannot be chosen in isolation; para. 0056 - mixed-precision quantization and structured pruning may quantize weight values and/or activation values resulting in correspondingly quantized output channels 202a, 202b, 202c, 202d. For another example, mixed-precision quantization and structured pruning may prune weight values and/or activation values resulting in correspondingly pruned output channels.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Djukic, to include applying stochastic quantization to an input layer of the neural network; applying stochastic quantization to the one or more layers of the neural network for the purpose of efficiently quantizing with different bitwidths to achieve quantized weight and/or activation values, improving model accuracy and efficiency, as taught by Baalen.
Claim(s) 20:
Claim(s) 20 correspond to Claim 2, and thus, Djukic, Baalen, and Baharloui teach or suggest the limitations of claim(s) 20 as well.
Claim 3:
Djukic further teaches or suggests wherein the plurality of … members have diversity with regard to sensitivity to changes in the input layer (see Fig. 9; para. 0006 - combining the outputs of the multiple analysis procedures; para. 0051 – provides algorithmically combined outputs of multiple machine learning strategies (algorithms), some of which may include ensemble learning algorithms; para. 0073 - decision point can use several machine algorithms and combine their outputs to detect the attack type; para. 0120 - The layers 126, 128, 130 may use fully-connected layers, convolutional layers, and/or other types of DNN layers; para. 0122 - classifiers 146 are configured to classify the data to determine a type of network intrusion or sub-attack and provide the results to an output combiner 148, which is configured to combine the outputs. In this way, the results of each of the classifiers 146 can be used in a single element to determine multiple types of sub-attacks; para. 0126 - attack classifiers 120, 140, 150 may be configured to perform a procedure that includes training K classifiers on the same dataset. Each of the classifiers may be configured to determine a score during testing for each of the sub-attack types. may use known test scores to decide on which algorithms perform preferable prediction results that can be used at inference time; para. 0127 - At inference time, the methods may take a given input and use all K classifiers to determine the output. Since the scores are available, indicating how well each of the classifiers can decide what the category of sub-attack is, the goal may be to create a prediction that is most likely to be correct or one that maximizes the overall score of the classifiers; para. 0131 - outputs of multiple machine learning procedures can be combined in many ways. In this way, the overall system can improve the performance of the groups of techniques, procedures, or algorithms over individual algorithms, such as the individual algorithms of the conventional systems.).
Baalen teaches or suggests quantized (see Fig. 2, 3; para. 0016 - quantizing a first layer of the mixed-precision artificial neural network at a first precision determined using corresponding stochastic gates of quantization and pruning gates, and quantizing a second layer of the mixed-precision artificial neural network at a second precision determined using corresponding stochastic gates of the quantization and pruning gates, in which the first precision and the second precision are different; para. 0044 – Mixed precision quantization having different bit-widths for different tensors may yield improved accuracy to efficiency trade-offs compared to quantizing all tensors to the same bit-width; para. 0046 - size of the space for mixed-precision bit-width configurations in a neural network depends exponentially on the number of tensors. Lowering the bit-width of one tensor in a neural network affects the quantization sensitivity of other tensors. Thus, the bit-width for any given tensor cannot be chosen in isolation; para. 0056 - mixed-precision quantization and structured pruning may quantize weight values and/or activation values resulting in correspondingly quantized output channels 202a, 202b, 202c, 202d. For another example, mixed-precision quantization and structured pruning may prune weight values and/or activation values resulting in correspondingly pruned output channels.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Djukic, to include quantized for the purpose of efficiently quantizing with different bitwidths to achieve quantized weight and/or activation values, improving model accuracy and efficiency, as taught by Baalen.
Claim 9:
Djukic further teaches or suggests wherein the ensemble of neural networks comprises a trained federated learning ensemble (see Fig. 1-11 and 18-22; para. 0006 - combining the outputs of the multiple analysis procedures; para. 0051 – provides algorithmically combined outputs of multiple machine learning strategies (algorithms), some of which may include ensemble learning algorithms; para. 0073 - decision point can use several machine algorithms and combine their outputs to detect the attack type; para. 0122 - classifiers 146 are configured to classify the data to determine a type of network intrusion or sub-attack and provide the results to an output combiner 148, which is configured to combine the outputs. In this way, the results of each of the classifiers 146 can be used in a single element to determine multiple types of sub-attacks; para. 0126 - attack classifiers 120, 140, 150 may be configured to perform a procedure that includes training K classifiers on the same dataset. Each of the classifiers may be configured to determine a score during testing for each of the sub-attack types. may use known test scores to decide on which algorithms perform preferable prediction results that can be used at inference time; para. 0127 - At inference time, the methods may take a given input and use all K classifiers to determine the output. Since the scores are available, indicating how well each of the classifiers can decide what the category of sub-attack is, the goal may be to create a prediction that is most likely to be correct or one that maximizes the overall score of the classifiers; para. 0131 - outputs of multiple machine learning procedures can be combined in many ways. In this way, the overall system can improve the performance of the groups of techniques, procedures, or algorithms over individual algorithms, such as the individual algorithms of the conventional systems.).
Claim 10:
Djukic teaches or suggests a computing system comprising: processing circuitry and memory for executing a machine learning system, wherein the machine learning system is configured to:
generate, using the neural network, an ensemble of neural networks having a plurality of ... members (see para. 0007 - machine learning models may include a Deep Neural Network (DNN) technique, an ensemble technique; para. 0051 -provides algorithmically combined outputs of multiple machine learning strategies (algorithms), some of which may include ensemble learning algorithms; para. 0074 - machine learning algorithms may include at least one of ensemble algorithms, DNNs; para. 0120 - classifier in the classifier layers 132. For example, this may include DNN, XGBoost, or the like. Also, the classifier layers 132 may be a DNN version that can be implemented in any number of ways; para. 0126 - attack classifiers 120, 140, 150 may be configured to perform a procedure that includes training K classifiers on the same dataset.);
combine predictions of the plurality of ... members of the ensemble to harden the neural network … that protects the ensemble against one or more adversarial attacks (see para. 0006 - combining the outputs of the multiple analysis procedures; para. 0051 – provides algorithmically combined outputs of multiple machine learning strategies (algorithms), some of which may include ensemble learning algorithms; para. 0073 - decision point can use several machine algorithms and combine their outputs to detect the attack type; para. 0122 - classifiers 146 are configured to classify the data to determine a type of network intrusion or sub-attack and provide the results to an output combiner 148, which is configured to combine the outputs. In this way, the results of each of the classifiers 146 can be used in a single element to determine multiple types of sub-attacks; para. 0126 - attack classifiers 120, 140, 150 may be configured to perform a procedure that includes training K classifiers on the same dataset. Each of the classifiers may be configured to determine a score during testing for each of the sub-attack types. may use known test scores to decide on which algorithms perform preferable prediction results that can be used at inference time; para. 0127 - At inference time, the methods may take a given input and use all K classifiers to determine the output. Since the scores are available, indicating how well each of the classifiers can decide what the category of sub-attack is, the goal may be to create a prediction that is most likely to be correct or one that maximizes the overall score of the classifiers; para. 0131 - outputs of multiple machine learning procedures can be combined in many ways. In this way, the overall system can improve the performance of the groups of techniques, procedures, or algorithms over individual algorithms, such as the individual algorithms of the conventional systems.); and
an output device configured to output the predictions of the plurality of … members (see Fig. 7-10; para. 0006 - combining the outputs of the multiple analysis procedures; para. 0051 – provides algorithmically combined outputs of multiple machine learning strategies (algorithms), some of which may include ensemble learning algorithms; para. 0073 - decision point can use several machine algorithms and combine their outputs to detect the attack type; para. 0122 - classifiers 146 are configured to classify the data to determine a type of network intrusion or sub-attack and provide the results to an output combiner 148, which is configured to combine the outputs. In this way, the results of each of the classifiers 146 can be used in a single element to determine multiple types of sub-attacks; para. 0126 - attack classifiers 120, 140, 150 may be configured to perform a procedure that includes training K classifiers on the same dataset. Each of the classifiers may be configured to determine a score during testing for each of the sub-attack types. may use known test scores to decide on which algorithms perform preferable prediction results that can be used at inference time; para. 0127 - At inference time, the methods may take a given input and use all K classifiers to determine the output. Since the scores are available, indicating how well each of the classifiers can decide what the category of sub-attack is, the goal may be to create a prediction that is most likely to be correct or one that maximizes the overall score of the classifiers; para. 0131 - outputs of multiple machine learning procedures can be combined in many ways. In this way, the overall system can improve the performance of the groups of techniques, procedures, or algorithms over individual algorithms, such as the individual algorithms of the conventional systems.).
Djukic does not explicitly disclose applying stochastic quantization to one or more layers of the neural network; quantized; wherein at least one of weights or activations of each of quantized members have different bit precision; quantized … by establishing a bounded worst-case performance profile; quantized.
Baalen teaches or suggests applying stochastic quantization to one or more layers of the neural network; quantized; wherein at least one of weights or activations of each of quantized members have different bit precision; quantized … quantized (see Fig. 2, 3; para. 0016 - quantizing a first layer of the mixed-precision artificial neural network at a first precision determined using corresponding stochastic gates of quantization and pruning gates, and quantizing a second layer of the mixed-precision artificial neural network at a second precision determined using corresponding stochastic gates of the quantization and pruning gates, in which the first precision and the second precision are different; para. 0044 – Mixed precision quantization having different bit-widths for different tensors may yield improved accuracy to efficiency trade-offs compared to quantizing all tensors to the same bit-width; para. 0046 - size of the space for mixed-precision bit-width configurations in a neural network depends exponentially on the number of tensors. Lowering the bit-width of one tensor in a neural network affects the quantization sensitivity of other tensors. Thus, the bit-width for any given tensor cannot be chosen in isolation; para. 0056 - mixed-precision quantization and structured pruning may quantize weight values and/or activation values resulting in correspondingly quantized output channels 202a, 202b, 202c, 202d. For another example, mixed-precision quantization and structured pruning may prune weight values and/or activation values resulting in correspondingly pruned output channels.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Djukic, to include applying stochastic quantization to one or more layers of the neural network; quantized; wherein at least one of weights or activations of each of quantized members have different bit precision; quantized … quantized for the purpose of efficiently quantizing with different bitwidths to achieve quantized weight and/or activation values, improving model accuracy and efficiency, as taught by Baalen.
Baharloui teaches or suggests by establishing a bounded worst-case performance profile (see Fig. 12; para. 0003 - obtaining a worst-case bound on a classification error and loss for perturbed versions of the input data; para. 0018 - training a neural network classification system with an abstain (rejection) option with provable robust (worst-case/adversarial) performance. defense mechanism is designed such that it will either (1) classify adversarial perturbed inputs as the abstain/rejection/detection class, or (2) it will correctly classify it as they class and thus prevents misclassification and fooling of the system; para. 0026 - minimize an upper bound of the worst-case loss of perturbed training samples (within a perturbation model) together with the traditional (robust) cross-entropy loss of the (clean) inputs. Thus, the increase in the crossentropy of perturbed samples (attacks or not) is bounded, and the effect of the attack is mitigated; para. 0057 – bounding the training objective (e.g., training objective function) by its worstcase upper bound via the interval bound propagation technique; para. 0121 - obtains a worst-case bound on a classification error and loss for perturbed versions of the input data; para. 0124 - detect at least one additional abstain class of the plurality of additional abstain classes in response to obtaining the worst-case bound; para. 0126 - wherein the classifier does not classify the input data as the original classes when the input data includes perturbations. In some embodiments, the method also includes bounding a training objective function by a worst-case upper bound utilizing an interval bound propagation (IBP) technique; para. 0128 – utilize interval bound propagation to compute a worst-case bound on a classification error and classification loss associated with perturbed versions of the input data; para. 0129 - additional abstain class of the plurality of additional abstain classes is determined in response to at least bounding input data including one or more perturbations.).
Accordingly, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Baharloui with those of Djukic. One would have been motivated to do so for the purpose of efficiently identifying and bounding worst case classification performance in order to identify adversarial data, securing model performance, as taught by Baharloui (0018, 0026, 0126, and 0128).
Claim 11:
Djukic further teaches or suggests extracting one or more features from one or more layers of the neural network (see para. 0010 - examining the extracted information using Machine Leaming (ML). Then, the process may further include determining if the extracted information includes one or more malicious threats; para. 0120 - The layers 126, 128, 130 may use fully-connected layers, convolutional layers, and/or other types of DNN layers. DNNs would understand that this layer can be implemented using a pretrained DNN model (e.g., Bidirectional Encoder Representations from Transformers (BERT), or the like) to perform a word embedding procedure to extract information from the messages; para. 0180 – examining the extracted information using Machine Learning (ML), as indicated in block 334. The process 330 further includes determining if the extracted information includes one or more malicious threats.).
Baalen further teaches or suggests applying stochastic quantization to an input layer of the neural network; applying stochastic quantization to the one or more layers of the neural network (see Fig. 2, 3; para. 0016 - quantizing a first layer of the mixed-precision artificial neural network at a first precision determined using corresponding stochastic gates of quantization and pruning gates, and quantizing a second layer of the mixed-precision artificial neural network at a second precision determined using corresponding stochastic gates of the quantization and pruning gates, in which the first precision and the second precision are different; para. 0044 – Mixed precision quantization having different bit-widths for different tensors may yield improved accuracy to efficiency trade-offs compared to quantizing all tensors to the same bit-width; para. 0046 - size of the space for mixed-precision bit-width configurations in a neural network depends exponentially on the number of tensors. Lowering the bit-width of one tensor in a neural network affects the quantization sensitivity of other tensors. Thus, the bit-width for any given tensor cannot be chosen in isolation; para. 0056 - mixed-precision quantization and structured pruning may quantize weight values and/or activation values resulting in correspondingly quantized output channels 202a, 202b, 202c, 202d. For another example, mixed-precision quantization and structured pruning may prune weight values and/or activation values resulting in correspondingly pruned output channels.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Djukic, to include applying stochastic quantization to an input layer of the neural network; applying stochastic quantization to the one or more layers of the neural network for the purpose of efficiently quantizing with different bitwidths to achieve quantized weight and/or activation values, improving model accuracy and efficiency, as taught by Baalen.
Claim 12:
Djukic further teaches or suggests wherein the plurality of … members have diversity with regard to sensitivity to changes in the input layer (see Fig. 9; para. 0006 - combining the outputs of the multiple analysis procedures; para. 0051 – provides algorithmically combined outputs of multiple machine learning strategies (algorithms), some of which may include ensemble learning algorithms; para. 0073 - decision point can use several machine algorithms and combine their outputs to detect the attack type; para. 0120 - The layers 126, 128, 130 may use fully-connected layers, convolutional layers, and/or other types of DNN layers; para. 0122 - classifiers 146 are configured to classify the data to determine a type of network intrusion or sub-attack and provide the results to an output combiner 148, which is configured to combine the outputs. In this way, the results of each of the classifiers 146 can be used in a single element to determine multiple types of sub-attacks; para. 0126 - attack classifiers 120, 140, 150 may be configured to perform a procedure that includes training K classifiers on the same dataset. Each of the classifiers may be configured to determine a score during testing for each of the sub-attack types. may use known test scores to decide on which algorithms perform preferable prediction results that can be used at inference time; para. 0127 - At inference time, the methods may take a given input and use all K classifiers to determine the output. Since the scores are available, indicating how well each of the classifiers can decide what the category of sub-attack is, the goal may be to create a prediction that is most likely to be correct or one that maximizes the overall score of the classifiers; para. 0131 - outputs of multiple machine learning procedures can be combined in many ways. In this way, the overall system can improve the performance of the groups of techniques, procedures, or algorithms over individual algorithms, such as the individual algorithms of the conventional systems.).
Baalen teaches or suggests quantized (see Fig. 2, 3; para. 0016 - quantizing a first layer of the mixed-precision artificial neural network at a first precision determined using corresponding stochastic gates of quantization and pruning gates, and quantizing a second layer of the mixed-precision artificial neural network at a second precision determined using corresponding stochastic gates of the quantization and pruning gates, in which the first precision and the second precision are different; para. 0044 – Mixed precision quantization having different bit-widths for different tensors may yield improved accuracy to efficiency trade-offs compared to quantizing all tensors to the same bit-width; para. 0046 - size of the space for mixed-precision bit-width configurations in a neural network depends exponentially on the number of tensors. Lowering the bit-width of one tensor in a neural network affects the quantization sensitivity of other tensors. Thus, the bit-width for any given tensor cannot be chosen in isolation; para. 0056 - mixed-precision quantization and structured pruning may quantize weight values and/or activation values resulting in correspondingly quantized output channels 202a, 202b, 202c, 202d. For another example, mixed-precision quantization and structured pruning may prune weight values and/or activation values resulting in correspondingly pruned output channels.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Djukic, to include quantized for the purpose of efficiently quantizing with different bitwidths to achieve quantized weight and/or activation values, improving model accuracy and efficiency, as taught by Baalen.
Claim 18:
Djukic further teaches or suggests wherein the ensemble of neural networks comprises a trained federated learning ensemble (see Fig. 1-11 and 18-22; para. 0006 - combining the outputs of the multiple analysis procedures; para. 0051 – provides algorithmically combined outputs of multiple machine learning strategies (algorithms), some of which may include ensemble learning algorithms; para. 0073 - decision point can use several machine algorithms and combine their outputs to detect the attack type; para. 0122 - classifiers 146 are configured to classify the data to determine a type of network intrusion or sub-attack and provide the results to an output combiner 148, which is configured to combine the outputs. In this way, the results of each of the classifiers 146 can be used in a single element to determine multiple types of sub-attacks; para. 0126 - attack classifiers 120, 140, 150 may be configured to perform a procedure that includes training K classifiers on the same dataset. Each of the classifiers may be configured to determine a score during testing for each of the sub-attack types. may use known test scores to decide on which algorithms perform preferable prediction results that can be used at inference time; para. 0127 - At inference time, the methods may take a given input and use all K classifiers to determine the output. Since the scores are available, indicating how well each of the classifiers can decide what the category of sub-attack is, the goal may be to create a prediction that is most likely to be correct or one that maximizes the overall score of the classifiers; para. 0131 - outputs of multiple machine learning procedures can be combined in many ways. In this way, the overall system can improve the performance of the groups of techniques, procedures, or algorithms over individual algorithms, such as the individual algorithms of the conventional systems.).
Claim(s) 4, 5, 7, 13, 14, and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Djukic, in view of Van Baalen, in view of Baharloui, and further in view of Akino et al., US Publication 2023/0162023 (“Akino”).
Claim 4:
Akino further teaches or suggests applying regularization to the wights of at least one of the plurality of quantized members (see para. 0040 - AI model can be realized by a reconfigurable deep neural network (DNN) model, whose architecture is specified by a set of hyperparameters; para. 0043 - used to regularize the affine-transform weights. Batch normalization and dropout techniques are also widely used as common regularization techniques to prevent overfitting; para. 0088 - MIGE provides an efficient method to estimate the gradient of mutual information directly. This suffices for regularization purposes, in which an objective function containing a mutual information term will be minimized by gradient descent; para. 0092 - domain adaptation techniques Including ... quantization; para. 0103 - augmentation policy includes but not limited to ... quantization; Claim 3 - pre-shot regularization methods ... mutual information censoring methods; Claim 18 - wherein the set of hyperparameters compromises a set of training schedules including an adaptive control of learning rates, regularization weights.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Djukic, to include applying regularization to the wights of at least one of the plurality of quantized members for the purpose of efficiently managing weights in a nn, reducing model complexity and preventing overfitting, improving model effectiveness on new input, as taught by Akino (0043, 0088).
Claim 5:
Akino further teaches or suggests wherein applying regularization further comprises applying mutual information (MI) using training data (see para. 0006 – adversarial training requires careful choice of hyperparameters because too strong censoring will hurt the main task performance as the main objective function is under-weighted. Moreover, adversarial censoring is not only sole regularization approach to promote independence from nuisance variables in feature space. For example, minimizing mutual information between nuisance and feature can be realized by mutual information gradient estimator (MIGE); para. 0040 - AI model can be realized by a reconfigurable deep neural network (DNN) model, whose architecture is specified by a set of hyperparameters; para. 0043 - used to regularize the affine-transform weights. Batch normalization and dropout techniques are also widely used as common regularization techniques to prevent overfitting; para. 0088 - MIGE provides an efficient method to estimate the gradient of mutual information directly. This suffices for regularization purposes, in which an objective function containing a mutual information term will be minimized by gradient descent; para. 0092 - domain adaptation techniques Including ... quantization; para. 0103 - augmentation policy includes but not limited to ... quantization; Claim 3 - pre-shot regularization methods ... mutual information censoring methods; Claim 18 - wherein the set of hyperparameters compromises a set of training schedules including an adaptive control of learning rates, regularization weights.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Djukic, to include wherein applying regularization further comprises applying mutual information (MI) using training data for the purpose of efficiently managing weights in a nn, reducing model complexity and preventing overfitting, improving model effectiveness on new input, as taught by Akino (0043, 0088).
Claim 7:
As indicated above, Djukic teaches or suggests using adversarial attacks generated from training data.
Akino teaches or suggests applying the MI regularization (see para. 0006 – adversarial training requires careful choice of hyperparameters because too strong censoring will hurt the main task performance as the main objective function is under-weighted. Moreover, adversarial censoring is not only sole regularization approach to promote independence from nuisance variables in feature space. For example, minimizing mutual information between nuisance and feature can be realized by mutual information gradient estimator (MIGE); para. 0040 - AI model can be realized by a reconfigurable deep neural network (DNN) model, whose architecture is specified by a set of hyperparameters; para. 0043 - used to regularize the affine-transform weights. Batch normalization and dropout techniques are also widely used as common regularization techniques to prevent overfitting; para. 0088 - MIGE provides an efficient method to estimate the gradient of mutual information directly. This suffices for regularization purposes, in which an objective function containing a mutual information term will be minimized by gradient descent; para. 0092 - domain adaptation techniques Including ... quantization; para. 0103 - augmentation policy includes but not limited to ... quantization; Claim 3 - pre-shot regularization methods ... mutual information censoring methods; Claim 18 - wherein the set of hyperparameters compromises a set of training schedules including an adaptive control of learning rates, regularization weights.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Djukic, to include applying the MI regularization for the purpose of efficiently managing weights in a nn, reducing model complexity and preventing overfitting, improving model effectiveness on new input, as taught by Akino (0043, 0088).
Claim 13:
Akino further teaches or suggests apply regularization to the wights of at least one of the plurality of quantized members (see para. 0040 - AI model can be realized by a reconfigurable deep neural network (DNN) model, whose architecture is specified by a set of hyperparameters; para. 0043 - used to regularize the affine-transform weights. Batch normalization and dropout techniques are also widely used as common regularization techniques to prevent overfitting; para. 0088 - MIGE provides an efficient method to estimate the gradient of mutual information directly. This suffices for regularization purposes, in which an objective function containing a mutual information term will be minimized by gradient descent; para. 0092 - domain adaptation techniques Including ... quantization; para. 0103 - augmentation policy includes but not limited to ... quantization; Claim 3 - pre-shot regularization methods ... mutual information censoring methods; Claim 18 - wherein the set of hyperparameters compromises a set of training schedules including an adaptive control of learning rates, regularization weights.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Djukic, to include apply regularization to the wights of at least one of the plurality of quantized members for the purpose of efficiently managing weights in a nn using regularization, reducing model complexity and preventing overfitting, improving model effectiveness on new input, as taught by Akino (0043, 0088).
Claim 14:
Akino further teaches or suggests wherein apply regularization further comprises apply mutual information (MI) using training data (see para. 0006 – adversarial training requires careful choice of hyperparameters because too strong censoring will hurt the main task performance as the main objective function is under-weighted. Moreover, adversarial censoring is not only sole regularization approach to promote independence from nuisance variables in feature space. For example, minimizing mutual information between nuisance and feature can be realized by mutual information gradient estimator (MIGE); para. 0040 - AI model can be realized by a reconfigurable deep neural network (DNN) model, whose architecture is specified by a set of hyperparameters; para. 0043 - used to regularize the affine-transform weights. Batch normalization and dropout techniques are also widely used as common regularization techniques to prevent overfitting; para. 0088 - MIGE provides an efficient method to estimate the gradient of mutual information directly. This suffices for regularization purposes, in which an objective function containing a mutual information term will be minimized by gradient descent; para. 0092 - domain adaptation techniques Including ... quantization; para. 0103 - augmentation policy includes but not limited to ... quantization; Claim 3 - pre-shot regularization methods ... mutual information censoring methods; Claim 18 - wherein the set of hyperparameters compromises a set of training schedules including an adaptive control of learning rates, regularization weights.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Djukic, to include wherein apply regularization further comprises apply mutual information (MI) using training data for the purpose of efficiently managing weights in a nn, reducing model complexity and preventing overfitting, improving model effectiveness on new input, as taught by Akino (0043, 0088).
Claim 16:
As indicated above, Djukic teaches or suggests using adversarial attacks generated from training data.
Akino teaches or suggests apply the MI regularization (see para. 0006 – adversarial training requires careful choice of hyperparameters because too strong censoring will hurt the main task performance as the main objective function is under-weighted. Moreover, adversarial censoring is not only sole regularization approach to promote independence from nuisance variables in feature space. For example, minimizing mutual information between nuisance and feature can be realized by mutual information gradient estimator (MIGE); para. 0040 - AI model can be realized by a reconfigurable deep neural network (DNN) model, whose architecture is specified by a set of hyperparameters; para. 0043 - used to regularize the affine-transform weights. Batch normalization and dropout techniques are also widely used as common regularization techniques to prevent overfitting; para. 0088 - MIGE provides an efficient method to estimate the gradient of mutual information directly. This suffices for regularization purposes, in which an objective function containing a mutual information term will be minimized by gradient descent; para. 0092 - domain adaptation techniques Including ... quantization; para. 0103 - augmentation policy includes but not limited to ... quantization; Claim 3 - pre-shot regularization methods ... mutual information censoring methods; Claim 18 - wherein the set of hyperparameters compromises a set of training schedules including an adaptive control of learning rates, regularization weights.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Djukic, to include apply the MI regularization for the purpose of efficiently managing weights in a nn, reducing model complexity and preventing overfitting, improving model effectiveness on new input, as taught by Akino (0043, 0088).
Claim(s) 8 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Djukic, in view of Van Baalen, in view of Baharloui, in view of Akino, and further in view of Li et al., US Publication 2022/0108615 (“Li”).
Claim 8:
Li further teaches or suggests wherein applying regularization further comprises applying Lipschitz regularization (see para. 0059 - better ensure the robustness of our model against input perturbations, the Lipschitz regularization may be employed to control the Lipschitz for the cerebellar embedding layer and the multilayer perceptron layer.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Djukic, to include wherein applying regularization further comprises applying Lipschitz regularization for the purpose of efficiently using a specific regularization scheme to ensure model robustness, improving model performance against input perturbations, as taught by Li (0059).
Claim 17:
Li further teaches or suggests wherein the machine learning system configured to apply regularization is further configured to apply Lipschitz regularization (see para. 0059 - better ensure the robustness of our model against input perturbations, the Lipschitz regularization may be employed to control the Lipschitz for the cerebellar embedding layer and the multilayer perceptron layer.).
Accordingly, it would have been obvious to one having ordinary skill before the effective filing date of the claimed invention to modify the system and method, taught in Djukic, to include wherein the machine learning system configured to apply regularization is further configured to apply Lipschitz regularization for the purpose of efficiently using a specific regularization scheme to ensure model robustness, improving model performance against input perturbations, as taught by Li (0059).
Allowable Subject Matter
Claims 6 and 15 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Response to Arguments
Applicant’s further arguments have been considered but are not persuasive because the arguments do not correspond to the rationales as used in the current rejection.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Andrew T McIntosh whose telephone number is (571)270-7790. The examiner can normally be reached M-Th 8:00am-5:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tamara Kyle can be reached at 571-272-4241. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW T MCINTOSH/Primary Examiner, Art Unit 2144
