DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Acknowledgement is made of Applicant’s claim amendments on 12/26/2025. The claim amendments are entered. Presently, claims 1-8 and 10-20 remain pending. Claims 1, 7, and 15 have been amended.
Response to Arguments
Applicant's arguments filed 12/09/2025 have been fully considered but they are not persuasive.
Applicant argues: Cited references do not teach "to perform a loss-error-aware weight quantization to quantize weight sets into ultra-low-bit versions after the loss-error-aware activation quantization to provide an ability to recover model accuracy, wherein the ULQ jointly regularizes weight and activation approximation error and accompanying loss perturbation, wherein at a training iteration, minimizing an explicit optimization objective that retains a basic loss function L with respect to the full-precision NN model having a loss representing a difference between a quantized NN model and the full precision NN model" of the independent claims (page 9-13 of remarks)
Examiner response: Examiner respectfully disagrees. The primary reference Zhou discloses a loss function quantizing weights and activations (Zhou pg. 4927; “With proximal Newton algorithm, a close-form solution can be derived, but it needs to estimate the second order Hessian matrix of the loss function w.r.t. the quantized weights and the input activations, bringing unacceptable computational complexity which prohibits its using in the training with a large-scale dataset such as ImageNet.”). Zhou further teaches wherein the ULQ jointly regularizes weight approximation error and accompanying loss perturbation (pg. 9427; “We present Explicit Loss-error-aware Quantization (ELQ), a new DNN quantization method that jointly regularizes the weight approximation error and the accompanying loss perturbation in an explicit manner. To train lossless quantized models, we further bridge our ELQ with an incremental quantization strategy.”), wherein at a training iteration, minimizing an explicit optimization objective that retains a basic loss function L with respect to the full-precision NN model having a loss representing a difference between a quantized NN model and the full precision NN model (Zhou pg. 9429; “
PNG
media_image1.png
88
474
media_image1.png
Greyscale
Here, L is the basic loss function w.r.t. the original full-precision model (It shall be noticed that this is different from the existing methods that only consider the loss function w.r.t. the quantized model at feed-forward stage during training. Retaining the loss function w.r.t. the full-precision model is critical for our ELQ, which will be clarified in the following paragraphs), Lp encodes the loss difference between the quantized and full-precision models, E represents the approximation error between the quantized weight sets and the full-precision counterparts, and a1 and a2 are two positive coefficients balancing the regularization.”)
Yang further teaches jointly regularizing a weight and activation approximation error using a loss function (pg. 2, section 3; “Afterwards, the regularization on both weights and activations is added in the elaborated JP loss for the following finetuning stage.” pg. 3, section 3.2; “The finetuning stage applies JP loss that integrates weight and activation regularization for a training batch {X, Y } as follows”).
While Zhou and Yang teach quantizing activations and weights using a loss function as disclosed above, Zhou does not explicitly teach that the activations are quantized before the weights. However, Choi teaches that weights can be quantized after quantizing the activations (Choi para [0019] “Quantization layers generate quantized activations during training and weights are optimized based on quantized activations.”)
Applicant argues modifying the DNN of Zhou with the mixed precision values of Zhu would mitigate the benefit of reduced computational complexity since Zhou’s DNN is represented with ternary or binary weights. Zhou discloses for a method for quantizing full-precision 32-bit floating-point DNNs into a quantized ternary or binary version (Zhou Abs. “In this paper, we propose Explicit Loss-error-aware Quantization (ELQ), a new method that can train DNN models with very low-bit parameter values such as ternary and binary ones to approximate 32-bit floating-point counterparts without noticeable loss of predication accuracy.”). Zhu discloses quantizing full-precision values for a wide range of precisions (i.e., mixed 16, 24, or 32 bit) including a 32-bit floating points (para [0044] “In particular, the precision of the representation is determined by the precision of the mantissa. Common floating-point representations use a mantissa of 10 (float 16), 24 (float 32), or 53 (float64) bits in width. The exponent modifies the magnitude of the mantissa.”). Modifying the DNN of Zhou with the mixed precision of Zhu would allow for quantizing a neural network with mixed precisions instead of only 32-bit floating points. Choi further discloses the problem with using quantized floating point precision (Choi para [0054] “The use of quantized-precision floating-point formats in this way can, however, have certain negative impacts on ANNs such as, but not limited to, a loss in accuracy.”). Choi instead teaches that the weights and activations may be quantized according to a desired bit width (Choi para [0059] “Specifically, hyperparameters 122 specify per layer bit widths for quantizing weights 302A, per step bit widths for quantizing weights 302B, and per gate bit widths for quantizing weights 302C. Hyperparameters 122 also define per layer bit widths for quantizing activation values 304A, per step bit widths for quantizing activation values 304B, and per gate bit widths for quantizing activation values 304C. In some configurations, hyperparameters 122 also specify a data type 306, which indicates whether and which type of float, integer, or other data type should be used.”). Arguments are not persuasive.
Applicant argues: Claim 6, cited references do not teach partitioning unquantized activations to be quantized into a first group and a second of activations that retain full-precision. (page 13-14 of remarks).
Examiner response: Examiner respectfully disagrees. Jung discloses dividing the training data into group A and group B (Jung pg. 4357, section 4.2; “We divide the dataset into two disjoint groups A (4 superclasses, 20 classes) and B (16 superclasses, 80 classes) where A is the original training set and B is used as a heterogeneous dataset for training low bit-width model (4/4-bit model in this experiment).”). First, Jung uses Group A is used to train a neural network with full precision (Jung pg. 4357, section 4.2; “First, we train the full-precision networks with A, then finetune the low bit-width networks with B by minimizing the mean-squared-errors between the outputs of the full precision model and the low bit-width model.”). Training a full-precision model results in unquantized activations (i.e., second group of activations) (Jung pg. 4352, section 3.1; “For the l-th layer of a full-precision convolutional neural network (CNN), the weight Wl is convolved with the input activation Xl where Wl and Xl are real-valued tensors”). Jung then uses the full-precision model to train a low bit-width model (Jung pg. 4357, section 4.1; “In this experiment, we train a low bit-width network from a pre-trained full-precision network without the original training dataset.”). Jung then uses group B to train a low bit-width network with training set B resulting in quantized activations (i.e., first group of activations) (Jung pg. 4352 “Reducing bit-widths inherently involves a quantization process, where we obtain the quantized weight ¯ w ∈ ¯ W and the quantized input activation ¯x ∈ ¯ X via quantizers”). Arguments are not persuasive.
Allowable Subject Matter
Claim 18 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 7-8, 10-12, 15, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al. (“Explicit Loss-Error-Aware Quantization for Low-Bit Deep Neural Networks”) in view of Zhu et al. (US-20200302283-A1), Choi et al. (US-20190138882-A1), and Yang et al. (“Joint Regularization on Activations and Weights for Efficient Neural Network Pruning”).
Regarding Claim 1,
Zhou teaches an apparatus for universal loss-error-aware quantization (ULQ) of a neural network (NN), the apparatus comprising:
a network processor coupled to the data storage, the network processor is configured to implement the ULQ (pg. 9427, section 1; “We present Explicit Loss-error-aware Quantization (ELQ), a new DNN quantization method that jointly regularizes the weight approximation error and the accompanying loss perturbation in an explicit manner.”) by constraining a low-precision NN model based on a full-precision NN model (pg. 9428, section 3.1; “Denote M={(Wl,Xl)|1 ≤ l ≤ L} as a full-precision (i.e., 32-bit floating-point) DNN model, where Wl is the weight set of the lth layer, Xl is the input set of the lth layer, and L is the number of layers in the DNN model M… In this paper, we aim to constrain DNN model M to only have very low-bit weight set Wlˆ whose entries are composed of Ql={αlck|1≤k≤K}. ” M denotes the full precision model.
PNG
media_image2.png
85
525
media_image2.png
Greyscale
in algorithm 1 denotes the low-precision NN.), …, to optimize the low-precision NN model with respect to a loss function that is based on the full-precision NN model (pg. 9429, section 3.2; “Here, L is the basic loss function w.r.t. the original full-precision model (It shall be noticed that this is different from the existing methods that only consider the loss function w.r.t. the quantized model at feed-forward stage during training. Retaining the loss function w.r.t. the full-precision model is critical for our ELQ, which will be clarified in the following paragraphs), Lp encodes the loss difference between the quantized and full-precision models, E represents the approximation error between the quantized weight sets and the full-precision counterparts, and a1 and a2 are two positive coefficients balancing the regularization.”), and to perform a loss-error-aware weight quantization to quantize weight sets into ultra-low-bit versions (pg. 9428, section 3.1; “In this paper, we aim to constrain DNN model M to only have very low-bit weight set Wlˆ whose entries are composed of Ql={αlck|1≤k≤K}.” Very low bit-weight (i.e. ultra low). "pg. 9429, section 3.2; “In our formulation defined as (5), besides the weight approximation error (i.e., the third term), we use L(Wl) instead of L(Wˆl) as the first term to emphasize their difference and use the loss difference between the quantized and full-precision models as the second term to encode the loss perturbation from the weight quantization, this is critical to obtain Equation (9) for weight update.”)
wherein the ULQ jointly regularizes weight approximation error and accompanying loss perturbation (pg. 9427; “We present Explicit Loss-error-aware Quantization (ELQ), a new DNN quantization method that jointly regularizes the weight approximation error and the accompanying loss perturbation in an explicit manner. To train lossless quantized models, we further bridge our ELQ with an incremental quantization strategy.”), wherein at a training iteration, minimizing an explicit optimization objective that retains a basic loss function L with respect to the full-precision NN model having a loss representing a difference between a quantized NN model and the full precision NN model (pg. 9429; “
PNG
media_image1.png
88
474
media_image1.png
Greyscale
Here, L is the basic loss function w.r.t. the original full-precision model (It shall be noticed that this is different from the existing methods that only consider the loss function w.r.t. the quantized model at feed-forward stage during training. Retaining the loss function w.r.t. the full-precision model is critical for our ELQ, which will be clarified in the following paragraphs), Lp encodes the loss difference between the quantized and full-precision models, E represents the approximation error between the quantized weight sets and the full-precision counterparts, and a1 and a2 are two positive coefficients balancing the regularization.”).
Zhou does not explicitly disclose
data storage to store data including activation sets and weight sets; and
to perform a loss-error-aware activation quantization to quantize activation sets into ultra- low-bit versions with given bit-width values,
to perform a loss-error-aware weight quantization to quantize weight sets into ultra-low-bit versions after the loss-error-aware activation quantization to provide an ability to recover model accuracy, wherein the ULQ jointly regularizes weight and activation approximation error and accompanying loss perturbation.
However, Zhu (US 20200302283 A1) teaches
data storage to store data including activation sets and weight sets (para [0009] Storage requirements are reduced by requiring fewer bits to store weights, activations, and other values used during ANN training.);
…to perform a loss-error-aware (para [0036] and para [0054] precisions are based on loss computed during training.) activation quantization to quantize activation sets into ultra-low-bit versions with given bit-width values (para [0059] Hyperparameters 122 also define per layer bit widths for quantizing activation values 304A, per step bit widths for quantizing activation values 304B, and per gate bit widths for quantizing activation values 304C. And para [0098] Example 1: A computer-implemented method, comprising: defining an artificial neural network (ANN) comprising a plurality of layers of nodes; setting a first bit width for activation values associated with a first layer of the plurality of layers of nodes; setting a second bit width for activation values associated with a second layer of the plurality of layers of nodes;),
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Zhou with the mixed-precision of Zhu.
The time needed to train and/or accuracy of ANNs can be improved by varying precision in different portions (e.g. different layers or other collection of neurons) of the ANN and/or during different training steps. Varying precision allows sensitive portions and/or steps to utilize higher precision weights, activations, etc., while less sensitive portions and/or steps may be satisfactorily processed using lower precision values (Zhu para [0007]).
Choi (US 20190138882 A1) teaches
to perform a loss-error-aware weight quantization to quantize weight sets into ultra-low-bit versions after the loss-error-aware activation quantization (para [0019] The MSQE for activations in each layer is minimized for optimization. Quantization layers generate quantized activations during training and weights are optimized based on quantized activations. In contrast, weights are not quantized during training iterations, but each weight gradually converges to a quantized value as training proceeds because of the MSQE regularization with an increasing regularization coefficient. Also see figure 3 wherein the activations are quantized in 303 before quantizing the weights in 311. See para [0061]-[0066].) to provide an ability to recover model accuracy (para [0054] The present system optimizes the cost function in network training and updates weights, quantization cell sizes, and the learnable regularization coefficient. Because of the penalty term on a small value of α, i.e.,—log α, where α causes the importance of the regularization term R.sub.n to increase continuously during training, which causes the regularization for weight quantization to increase, and causes the weights to be quantized by the completion of training. The regularization coefficient increases gradually so long as the network loss function does not significantly degrade.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Zhou with the quantization of Choi.
Doing so would allow for replacing full-precision floating-point arithmetic operations with fixed-point arithmetic operations to lower storage and memory requirements (Choi para [0085]).
Yang (“Joint Pruning on Activations and Weights for Efficient Neural Networks”) teaches
wherein the ULQ jointly regularizes weight and activation approximation error (pg. 2, section 3; “Afterwards, the regularization on both weights and activations is added in the elaborated JP loss for the following finetuning stage.” pg. 3, section 3.2; “The finetuning stage applies JP loss that integrates weight and activation regularization for a training batch {X, Y } as follows”)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Zhou with the weight and activation regularization of Yang.
Doing so would allow for substantially reducing the computational cost while maintaining a similar accuracy (Yang Abs. “With < 0.4% degradation on testing accuracy, a JPnet can save 71.1% ∼ 96.35% of computation cost, compared to the original dense models with up to 5.8× and 10× reductions in activation and weight numbers, respectively.”).
Regarding Claim 2,
Zhou, Zhu, Choi, and Yang teach the apparatus of claim 1. Zhou further teaches wherein the network processor is configured to perform a training update based on input from training data to generate the low-precision NN model with an increased number of quantized weights and activations in comparison to the full-precision NN model (pg. 9428, section 3.1; “BinaryConnect [3] and BinaryNet [4] apply stochastic binarization functions to transform trained full-precision weights into binary equivalents.” The final model in algorithm 1 has more quantized weights and activations than the full precision model.).
Regarding Claim 3,
Zhou, Zhu, Choi, and Yang teach the apparatus of claim 2. Zhou further teaches wherein the training update comprises a weight update scheme having a binary matrix to force quantized weights to be fixed while weights having full-precision are retraining to enhance accuracy of NN model and updated at backward propagation stage (pg. 9429-9430 section 3.3; “Its basic idea is to first split the weights of each layer of a DNN model into two disjoint groups, then the weights in one group are directly quantized and fixed, and the weights of the other group retaining 32-bit floating-point values are re-trained to compensate for model accuracy loss resulted from the quantization…Let Tl be a binary matrix having the same dimension to Wl,Wa be the weight group that needs to be quantized, Wb be the weight group that needs to be re-trained, And pg. 9429 section 3.2;“ On the one hand, the full-precision version of network weights is retained during training and updated at backward propagation stage.”).
Regarding Claim 4,
Zhou, Zhu, Choi, and Yang teach the apparatus of claim 3. Zhou further teaches wherein the network processor is configured to determine whether a maximum number of training iterations has been performed and to generate a final low-precision NN model when the maximum number of training iterations has been performed (pg. 9431;
PNG
media_image3.png
286
447
media_image3.png
Greyscale
the variable “L” denotes the maximum number of training iterations resulting in the final low precision model.).
Regarding Claim 5,
Zhou, Zhu, Choi, and Yang teach the apparatus of claim 1. Zhu further teaches wherein the ultra-low-bit versions of the activation sets comprises at least one of ternary or binary versions (para [0065] For example, epoch 502 may be associated with the precision “A5W6”, which allots ‘5’ bits to activation values and ‘6’ bits to weights. Epoch 504 may seek to increase accuracy in an attempt to refine the results generated by training epoch 502. For example, the epoch 504 may be associated with the precision “A6W6”, which increases the bit width of activation values to ‘6’.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Zhou with the mixed-precision of Zhu.
The time needed to train and/or accuracy of ANNs can be improved by varying precision in different portions (e.g. different layers or other collection of neurons) of the ANN and/or during different training steps. Varying precision allows sensitive portions and/or steps to utilize higher precision weights, activations, etc., while less sensitive portions and/or steps may be satisfactorily processed using lower precision values (Zhu para [0007]).
Regarding Claim 7,
Zhu (US 20200302283 A1) teaches a computer implemented method…, the method comprising:
providing a mixed-precision or all-same bit allocation for different layers of the low-precision DNN architecture (para [0055]-[0057] mixed precision layers); and
performing a loss-error-aware (para [0036] and para [0054] precisions are based on loss computed during training.) activation quantization to quantize activation sets into ultra-low-bit versions with a first bit-width for a first layer being allocated and a second bit-width for a second layer being allocated to support mixed-precision quantization (para [0059] Hyperparameters 122 also define per layer bit widths for quantizing activation values 304A, per step bit widths for quantizing activation values 304B, and per gate bit widths for quantizing activation values 304C. And para [0098] Example 1: A computer-implemented method, comprising: defining an artificial neural network (ANN) comprising a plurality of layers of nodes; setting a first bit width for activation values associated with a first layer of the plurality of layers of nodes; setting a second bit width for activation values associated with a second layer of the plurality of layers of nodes;).
Zhu does not explicitly disclose
a computer implemented method for universal loss-error-aware quantization (ULQ) of a deep neural network (DNN),
constraining a low-precision DNN model based on a full-precision DNN model having a plurality of layers including first and second layers;
and performing a loss-error-aware weight quantization to quantize weight sets into ultra-low-bit versions after the loss-error-aware activation quantization to provide an ability to recover model accuracy wherein the ULO jointly regularizes weight and activation approximation error and accompanying loss perturbation, wherein at a training iteration, minimizing an explicit optimization objective that retains a basic loss function L with respect to the full-precision NN model having a loss representing a difference between a quantized NN model and the full precision NN model.
However, Zhou teaches
a computer implemented method for universal loss-error-aware quantization (ULQ) of a deep neural network (DNN) (pg. 9427, section 1; “We present Explicit Loss-error-aware Quantization (ELQ), a new DNN quantization method that jointly regularizes the weight approximation error and the accompanying loss perturbation in an explicit manner.”),
constraining a low-precision DNN model based on a full-precision DNN model having a plurality of layers including first and second layers (pg. 9428, section 3.1; “Denote M={(Wl,Xl)|1 ≤ l ≤ L} as a full-precision (i.e., 32-bit floating-point) DNN model, where Wl is the weight set of the lth layer, Xl is the input set of the lth layer, and L is the number of layers in the DNN model M… In this paper, we aim to constrain DNN model M to only have very low-bit weight set Wlˆ whose entries are composed of Ql={αlck|1≤k≤K}. ” M denotes the full precision model.
PNG
media_image2.png
85
525
media_image2.png
Greyscale
in algorithm 1 denotes the low-precision NN.);
wherein the ULQ jointly regularizes weight approximation error and accompanying loss perturbation (pg. 9427; “We present Explicit Loss-error-aware Quantization (ELQ), a new DNN quantization method that jointly regularizes the weight approximation error and the accompanying loss perturbation in an explicit manner. To train lossless quantized models, we further bridge our ELQ with an incremental quantization strategy.”), wherein at a training iteration, minimizing an explicit optimization objective that retains a basic loss function L with respect to the full-precision NN model having a loss representing a difference between a quantized NN model and the full precision NN model (pg. 9429; “
PNG
media_image1.png
88
474
media_image1.png
Greyscale
Here, L is the basic loss function w.r.t. the original full-precision model (It shall be noticed that this is different from the existing methods that only consider the loss function w.r.t. the quantized model at feed-forward stage during training. Retaining the loss function w.r.t. the full-precision model is critical for our ELQ, which will be clarified in the following paragraphs), Lp encodes the loss difference between the quantized and full-precision models, E represents the approximation error between the quantized weight sets and the full-precision counterparts, and a1 and a2 are two positive coefficients balancing the regularization.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Zhu with the loss function of Zhou.
Doing so would allow for training DNN models with very low-bit parameter values without noticeable loss of prediction accuracy (Zhou Abs.).
Choi teaches
and performing a loss-error-aware weight quantization to quantize weight sets into ultra-low-bit versions after the loss-error-aware activation quantization (para [0019] The MSQE for activations in each layer is minimized for optimization. Quantization layers generate quantized activations during training and weights are optimized based on quantized activations. In contrast, weights are not quantized during training iterations, but each weight gradually converges to a quantized value as training proceeds because of the MSQE regularization with an increasing regularization coefficient. Also see figure 3 wherein the activations are quantized in 303 before quantizing the weights in 311. See para [0061]-[0066].) to provide an ability to recover model accuracy (para [0054] The present system optimizes the cost function in network training and updates weights, quantization cell sizes, and the learnable regularization coefficient. Because of the penalty term on a small value of α, i.e.,—log α, where α causes the importance of the regularization term R.sub.n to increase continuously during training, which causes the regularization for weight quantization to increase, and causes the weights to be quantized by the completion of training. The regularization coefficient increases gradually so long as the network loss function does not significantly degrade.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Zhou with the quantization of Choi.
Doing so would allow for replacing full-precision floating-point arithmetic operations with fixed-point arithmetic operations to lower storage and memory requirements (Choi para [0085]).
Yang (“Joint Pruning on Activations and Weights for Efficient Neural Networks”) teaches
wherein the ULQ jointly regularizes weight and activation approximation error (pg. 2, section 3; “Afterwards, the regularization on both weights and activations is added in the elaborated JP loss for the following finetuning stage.” pg. 3, section 3.2; “The finetuning stage applies JP loss that integrates weight and activation regularization for a training batch {X, Y } as follows”)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Zhou with the weight and activation regularization of Yang.
Doing so would allow for substantially reducing the computational cost while maintaining a similar accuracy (Yang Abs. “With < 0.4% degradation on testing accuracy, a JPnet can save 71.1% ∼ 96.35% of computation cost, compared to the original dense models with up to 5.8× and 10× reductions in activation and weight numbers, respectively.”).
Regarding Claim 8,
Zhou, Zhu, Choi, and Yang teach the computer-implemented method of claim 7. Zhou further teaches the method further comprising:
optimizing the DNN with respect to a loss function that is based on the full-precision DNN model (pg. 9428, section 3; “In this section, we give a detailed view of our Explicit Loss-error-aware Quantization (ELQ), show how to formulate the optimization, how to bridge our basic quantization algorithm with an incremental strategy, and how to train very low-bit DNNs from the full-precision reference models with our ELQ.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Zhu with the loss function of Zhou.
Doing so would allow for training DNN models with very low-bit parameter values without noticeable loss of prediction accuracy (Zhou Abs.).
Regarding Claim 10,
Zhou, Zhu, Choi, and Yang teach the computer-implemented method of claim 8. Zhou further teaches the method further comprising:
performing a training update based on input from training data to generate the low- precision DNN model with an increased number of quantized weights and activations in comparison to the full-precision DNN model (pg. 9428, section 3.1; “BinaryConnect [3] and BinaryNet [4] apply stochastic binarization functions to transform trained full-precision weights into binary equivalents.” The final model in algorithm 1 has more quantized weights and activations than the full precision model.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Zhu with the loss function of Zhou.
Doing so would allow for training DNN models with very low-bit parameter values without noticeable loss of prediction accuracy (Zhou Abs.).
Regarding Claim 11,
Zhou, Zhu, Choi, and Yang teach the computer-implemented method of claim 10. Zhou further teaches wherein the training update comprises a weight update scheme having a binary matrix to force quantized weights to be fixed while weights having full-precision are retraining to enhance accuracy of the DNN model, wherein the weight update scheme is at least partially performed during backward propagation stage (pg. 9429-9430 section 3.3; “Its basic idea is to first split the weights of each layer of a DNN model into two disjoint groups, then the weights in one group are directly quantized and fixed, and the weights of the other group retaining 32-bit floating-point values are re-trained to compensate for model accuracy loss resulted from the quantization…Let Tl be a binary matrix having the same dimension to Wl,Wa be the weight group that needs to be quantized, Wb be the weight group that needs to be re-trained,” And pg. 9429 section 3.2;“ On the one hand, the full-precision version of network weights is retained during training and updated at backward propagation stage.”).
Regarding Claim 12,
Zhou, Zhu, Choi, and Yang teach the computer-implemented method of claim 11. Zhou further teaches the method further comprising:
determining whether a maximum number of training iterations has been performed; and generating a final low-precision DNN model when the maximum number of training iterations has been performed (pg. 9431;
PNG
media_image3.png
286
447
media_image3.png
Greyscale
the variable “L” denotes the maximum number of training iterations resulting in the final low precision model.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Zhu with the loss function of Zhou.
Doing so would allow for training DNN models with very low-bit parameter values without noticeable loss of prediction accuracy (Zhou Abs.).
Regarding Claim 15,
Zhu (US 20200302283 A1) teaches a graphics processing unit (para [0039]), comprising:
a data storage device to store data including activation sets and weights (para [0009] Storage requirements are reduced by requiring fewer bits to store weights, activations, and other values used during ANN training.); and
to perform a loss-error-aware (para [0036] and para [0054] precisions are based on loss computed during training.) quantization of a neural network (NN) to quantize activation sets into ultra-low-bit versions with given bit-width values (para [0059] Hyperparameters 122 also define per layer bit widths for quantizing activation values 304A, per step bit widths for quantizing activation values 304B, and per gate bit widths for quantizing activation values 304C. And para [0098] Example 1: A computer-implemented method, comprising: defining an artificial neural network (ANN) comprising a plurality of layers of nodes; setting a first bit width for activation values associated with a first layer of the plurality of layers of nodes; setting a second bit width for activation values associated with a second layer of the plurality of layers of nodes;).
Zhu does not explicitly disclose
a processor coupled to the data storage device, the processor is configured to constrain a low-precision deep neural network (DNN) model based on a full-precision DNN model having a plurality of layers,
and to perform a loss-error-aware weight quantization to quantize weight sets into ultra-low-bit versions after the loss-error-aware quantization of the activation sets to provide an ability to recover model accuracy, wherein the ULO jointly regularizes weight and activation approximation error and accompanying loss perturbation, wherein at a training iteration, minimizing an explicit optimization objective that retains a basic loss function L with respect to the full-precision NN model having a loss representing a difference between a quantized NN model and the full precision NN model.
However, Zhou teaches
a processor coupled to the data storage device, the processor is configured to constrain a low-precision deep neural network (DNN) model based on a full-precision DNN model having a plurality of layers (pg. 9428, section 3.1; “Denote M={(Wl,Xl)|1 ≤ l ≤ L} as a full-precision (i.e., 32-bit floating-point) DNN model, where Wl is the weight set of the lth layer, Xl is the input set of the lth layer, and L is the number of layers in the DNN model M… In this paper, we aim to constrain DNN model M to only have very low-bit weight set Wlˆ whose entries are composed of Ql={αlck|1≤k≤K}.” M denotes the full precision model.
PNG
media_image2.png
85
525
media_image2.png
Greyscale
in algorithm 1 denotes the low-precision NN.);
wherein the ULQ jointly regularizes weight approximation error and accompanying loss perturbation, (pg. 9427; “We present Explicit Loss-error-aware Quantization (ELQ), a new DNN quantization method that jointly regularizes the weight approximation error and the accompanying loss perturbation in an explicit manner. To train lossless quantized models, we further bridge our ELQ with an incremental quantization strategy.”) wherein at a training iteration, minimizing an explicit optimization objective that retains a basic loss function L with respect to the full-precision NN model having a loss representing a difference between a quantized NN model and the full precision NN model (pg. 9429; “
PNG
media_image1.png
88
474
media_image1.png
Greyscale
Here, L is the basic loss function w.r.t. the original full-precision model (It shall be noticed that this is different from the existing methods that only consider the loss function w.r.t. the quantized model at feed-forward stage during training. Retaining the loss function w.r.t. the full-precision model is critical for our ELQ, which will be clarified in the following paragraphs), Lp encodes the loss difference between the quantized and full-precision models, E represents the approximation error between the quantized weight sets and the full-precision counterparts, and a1 and a2 are two positive coefficients balancing the regularization.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Zhu with the loss function of Zhou.
Doing so would allow for training DNN models with very low-bit parameter values without noticeable loss of prediction accuracy (Zhou Abs.).
However, Choi teaches
and to perform a loss-error-aware weight quantization to quantize weight sets into ultra-low-bit versions after the loss-error-aware quantization of the activation sets (para [0019] The MSQE for activations in each layer is minimized for optimization. Quantization layers generate quantized activations during training and weights are optimized based on quantized activations. In contrast, weights are not quantized during training iterations, but each weight gradually converges to a quantized value as training proceeds because of the MSQE regularization with an increasing regularization coefficient. Also see figure 3 wherein the activations are quantized in 303 before quantizing the weights in 311. See para [0061]-[0066].) to provide an ability to recover model accuracy (para [0054] The present system optimizes the cost function in network training and updates weights, quantization cell sizes, and the learnable regularization coefficient. Because of the penalty term on a small value of α, i.e.,—log α, where α causes the importance of the regularization term R.sub.n to increase continuously during training, which causes the regularization for weight quantization to increase, and causes the weights to be quantized by the completion of training. The regularization coefficient increases gradually so long as the network loss function does not significantly degrade.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Zhou with the quantization of Choi.
Doing so would allow for replacing full-precision floating-point arithmetic operations with fixed-point arithmetic operations to lower storage and memory requirements (Choi para [0085]).
Yang (“Joint Pruning on Activations and Weights for Efficient Neural Networks”) teaches
wherein the ULQ jointly regularizes weight and activation approximation error (pg. 2, section 3; “Afterwards, the regularization on both weights and activations is added in the elaborated JP loss for the following finetuning stage.” pg. 3, section 3.2; “The finetuning stage applies JP loss that integrates weight and activation regularization for a training batch {X, Y } as follows”)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Zhou with the weight and activation regularization of Yang.
Doing so would allow for substantially reducing the computational cost while maintaining a similar accuracy (Yang Abs. “With < 0.4% degradation on testing accuracy, a JPnet can save 71.1% ∼ 96.35% of computation cost, compared to the original dense models with up to 5.8× and 10× reductions in activation and weight numbers, respectively.”).
Regarding Claim 20,
Zhou, Zhu, Choi, and Yang teach the graphics processing unit of claim 15. Zhou further teaches wherein the processor is configured to optimize the low-precision deep neural network (DNN) model with respect to a loss function (pg. 9428, section 3; “In this section, we give a detailed view of our Explicit Loss-error-aware Quantization (ELQ), show how to formulate the optimization, how to bridge our basic quantization algorithm with an incremental strategy, and how to train very low-bit DNNs from the full-precision reference models with our ELQ.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Zhu with the loss function of Zhou.
Doing so would allow for training DNN models with very low-bit parameter values without noticeable loss of prediction accuracy (Zhou Abs.).
Claims 6 and 16-17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Zhou/Zhu/Choi/Yang, as applied above, and further in view of Jung et al. ("Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Loss.").
Regarding Claim 6,
Zhou, Zhu, Choi, and Yang teach the apparatus of claim 1 Zhou further teaches wherein the ultra-low-bit versions of the weight sets comprise at least one of ternary or binary versions (pg. 9426; “Specifically, we intend to address the problem of how to train DNN models whose weights are forced to be very low-bit values such as ternary and binary ones without noticeable loss of model accuracy when compared with full-precision (i.e., 32-bit floating-point) counterparts”),
Zhou, Zhu, and Choi do not explicitly disclose
wherein the loss-error-aware activation quantization comprises an incremental loss-error-aware activation quantization that includes partitioning unquantized activations into a first group of activations to be quantized into fixed ultra-low-bit versions and a second group of activations that retain full-precision to be retrained to compensate for model accuracy loss resulting from the quantization.
However, Jung teaches
wherein the loss-error-aware activation quantization comprises an incremental loss-error-aware activation quantization that includes partitioning unquantized activations into a first group of activations to be quantized into fixed ultra-low-bit versions and a second group of activations that retain full-precision to be retrained to compensate for model accuracy loss resulting from the quantization (pg. 5 Algorithm 1; “
PNG
media_image4.png
172
534
media_image4.png
Greyscale
” Low bit-width network quantizes activations. pg. 8, section 4.2; “First, we train the full-precision networks with A, then finetune the low bit-width networks with B by minimizing the mean-squared-errors between the outputs of the fullprecision model and the low bit-width model.” Low-bit width network contains quantized activations (i.e. first group). Full-precision networks have full-precision activations.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Zhou with the quantization of Jung.
Doing so would allow for quantizing the weights and activations without degrading the accuracy of the neural network (Jung Abs.).
Regarding Claim 16,
Zhou, Zhu, Choi, and Yang teach the graphics processing unit of claim 15.
Zhou, Zhu, Choi, and Yang do not explicitly disclose
wherein the processor is configured to perform an incremental loss-error-aware activation quantization by partitioning unquantized activations into a first group of activations to be quantized into fixed ultra-low-bit versions and a second group of activations that retain full-precision to be retrained to compensate for model accuracy loss resulting from the quantization.
However, Jung teaches
wherein the processor is configured to perform an incremental loss-error-aware activation quantization by partitioning unquantized activations into a first group of activations to be quantized into fixed ultra-low-bit versions and a second group of activations that retain full-precision to be retrained to compensate for model accuracy loss resulting from the quantization (pg. 5 Algorithm 1; “
PNG
media_image4.png
172
534
media_image4.png
Greyscale
” Low bit-width network quantizes activations. pg. 8, section 4.2; “First, we train the full-precision networks with A, then finetune the low bit-width networks with B by minimizing the mean-squared-errors between the outputs of the fullprecision model and the low bit-width model.” Low-bit width network contains quantized activations (i.e. first group). Full-precision networks have full-precision activations.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Zhou with the quantization of Jung.
Doing so would allow for quantizing the weights and activations without degrading the accuracy of the neural network (Jung Abs.).
Regarding Claim 17,
Zhu, Zhou, Choi, Yang, and Jung teach the graphics processing unit of claim 16. Zhou further teaches wherein the processor is configured to partition unquantized network weights into a first group of network weights to be quantized and a second group of network weights to be retrained, to process network weights to calculate a loss with respect to a loss function, and to quantize the first group of network weights to generate low-bit network weights corresponding to the first group of network weights, wherein the processor is further configured to partition unquantized network weights of the second group into a third group of network weights to be quantized and a fourth group of network weights to be retrained, to process network weights to calculate a loss with respect to a loss function, and to quantize the third group of network weights to generate low-bit network weights corresponding to the second group of network weights (pg. 9429 section 3.3; “Its basic idea is to first split the weights of each layer of a DNN model into two disjoint groups, then the weights in one group are directly quantized and fixed, and the weights of the other group retaining 32-bit floating-point values are re-trained to compensate for model accuracy loss resulted from the quantization. The operations of weight partition, group-wise quantization and re-training are repeated until all network weights are quantized.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Zhu with the loss function of Zhou.
Doing so would allow for training DNN models with very low-bit parameter values without noticeable loss of prediction accuracy (Zhou Abs.).
Regarding Claim 19,
Zhu, Zhou, Choi, Yang, and Jung teach the graphics processing unit of claim 16. Zhou further teaches wherein the processor is configured to calculate a ternary or binary scaling factor (pg. 9428, section 3.1; “Here for the lth layer, αl is a corresponding positive scaling factor that needs to be determined during training, ck is an integer value, and K is the number of the quantized weight centers. Specifically, for binary networks, ck∈{−1,1}, while for ternary networks, ck∈{−1,0,1}.”) and set interval bound factors at successive partition operations (pg. 9430, section 3.3; “Inspired by this, for each layer, we define an interval bound factor set {σn|1≤n≤N} where 0≤σn≤1, guiding successive weight partition, quantization and re-training steps. An illustration of our method for ternary DNN quantization is shown in Figure 1. Algorithm 1 summarizes the procedure of our ELQ to train a ternary or binary DNN for approximating the full-precision reference model.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Zhu with the loss function of Zhou.
Doing so would allow for training DNN models with very low-bit parameter values without noticeable loss of prediction accuracy (Zhou Abs.).
Claim(s) 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Zhou/Zhu/Choi/Yang, as applied above, and further in view of Chung et al. (US-20200193273-A1).
Regarding Claim 13,
Zhou, Zhu, Choi, and Yang teach the computer-implemented method of claim 7.
Zhou, Zhu, Choi, and Yang do not explicitly disclose
wherein the first bit-width for the first layer comprises at least one of 1 bit-width, 2 bit-width, or 3 bit-width and the second bit-width for the second layer comprises full precision.
However, Chung (US 20200193273 A1) teaches
wherein the first bit-width for the first layer comprises at least one of 1 bit-width, 2 bit-width, or 3 bit-width (para [0059] For example, the full emulation version can check for underflow or overflow conditions for a limited, quantized bit width (e.g., 3-, 4-, or 5-bit wide mantissas).) and the second bit-width for the second layer comprises full precision (para [0102] The bulk of the computational work within a layer can be performed in the quantized floating-point domain and less computationally expensive operations of the layer, such as adding a bias value or calculating an activation function, can be performed in the normal-precision floating-point domain. The values that interface between the layers can be passed from one layer to the other layer in the normal-precision floating-point domain. Values can be passed between quantized and normal (i.e. full precision layers).).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Zhu with quantized floating point format of Chung.
Doing so would allow for the quantized floating-point format can be selected to reduce the complexity of the computer arithmetic circuits to make the computer logic potentially faster and/or more energy efficient (Chung para [0102]).
Regarding Claim 14,
Zhou, Zhu, Choi, and Yang teach the computer-implemented method of claim 7,
Zhou, Zhu, Choi, and Yang do not explicitly disclose
wherein a third bit-width for a third layer being allocated and the second bit-width for a fourth layer being allocated to support mixed-precision quantization.
However, Chung (US 20200193273 A1) teaches
wherein a third bit-width for a third layer being allocated and the second bit-width for a fourth layer being allocated to support mixed-precision quantization (para [0059] For example, the full emulation version can check for underflow or overflow conditions for a limited, quantized bit width (e.g., 3-, 4-, or 5-bit wide mantissas).) and the second bit-width for the second layer comprises full precision (para [0102] The bulk of the computational work within a layer can be performed in the quantized floating-point domain and less computationally expensive operations of the layer, such as adding a bias value or calculating an activation function, can be performed in the normal-precision floating-point domain. The values that interface between the layers can be passed from one layer to the other layer in the normal-precision floating-point domain. Values can be passed between quantized and normal (i.e. full precision layers).).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Zhu with quantized floating-point format of Chung.
Doing so would allow for the quantized floating-point format can be selected to reduce the complexity of the computer arithmetic circuits to make the computer logic potentially faster and/or more energy efficient (Chung para [0102]).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HENRY K NGUYEN whose telephone number is (571)272-0217. The examiner can normally be reached Mon - Fri 7:00am-4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at 5712723768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HENRY NGUYEN/Examiner, Art Unit 2121