Notice of Pre-AIA or AIA Status
This Non-Final communication is in response to application no. 18/666,958 filed on 5/17/2024, and claims benefit to provisional application 63/503,207 filed on 5/19/2023. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 16, 31 and are rejected under 35 U.S.C. 103 as being unpatentable by Wiedemann (NPL ‘Entropy-Constrained Training of Deep Neural Networks’ (2018)) in view of Choi (NPL ‘Towards the limit of Network Quantization’ (2017)).
Regarding claim 1, Wiedemann teaches:
A method of training a deep neural network, the deep neural network comprising a plurality of layers, the plurality of layers including a plurality of intermediate layers arranged between an input layer and an output layer, wherein each intermediate layer has a plurality of weight parameters, and wherein each intermediate layer is operable to output one or more activations to an adjacent layer in the plurality of layers, the method comprising: (Section 3 describes training the deep neural network. In DNN architecture it is standard to have many layers including input and output layers with intermediate layers in between each having weights.)
inputting a plurality of training data samples into the input layer of the deep neural network, wherein the plurality of training data samples are contained within a training set used to train the deep neural network; and (Section 2. Page 2 left column “Assume a supervised learning setting where given a particular data set, DN = {(xi,yi)|xi ∈ X,yi ∈ Y,i ∈ {1,...,N = |X| = |Y|}}, we want to learn to predict the elements yi of the output (or label) set Y from the elements xi of the input set X.”)
generating a trained deep neural network using the plurality of training data samples by iteratively updating the plurality of weight parameters (Section 3 describes training a deep neural network using gradient based optimization techniques) to optimize an entropy constrained objective function, wherein the entropy constrained objective function is defined to jointly minimize a quantized loss function of the deep neural network and an entropy of a plurality of quantized weight values (Section 3 on page 3 (right column) teaches an objective function that both minimizes loss and entropy. Specifically equation 6. “Thus, we now aim to minimize the averaged prediction error of the network, as taken relative to our model Pθ, constrained by the respective relaxation of the entropy (5). However, we can still not apply gradient-based optimization techniques in order to minimize (6) as calculating the mean of the log-likelihood is infeasible for deep neural networks”) ,
Wiedemann does not teach:
the optimization is based on the quantized weights.
wherein the plurality of quantized weight values correspond to the plurality of weight parameters and each quantized weight value is a quantized representation of a corresponding weight parameter in the plurality of weight parameters.
However Choi does:
wherein the plurality of quantized weight values correspond to the plurality of weight parameters and each quantized weight value is a quantized representation of a corresponding weight parameter in the plurality of weight parameters. (Section 4.2 page 6 “In particular, optimal coding closely achieves the lower limit of the average source code length, i.e., entropy, and then we approximately have [Equation 10] where H is the entropy of the quantized network parameters after clustering (i.e., source), given that pi = |Ci|/N is the ratio of the number of network parameters in cluster Ci to the number of all network parameters (i.e., source distribution).”)
Wiedemann and Choi are considered analogous art to the claimed invention because
they are in the same field of endeavor being neural network quantization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the entropy and loss optimization of Wiedemann with the quantized weights of Choi. One would be motivated to do this for better compression of models.
Regarding claim 16, Wiedemann teaches:
A computer program product for training a deep neural network, the deep neural network comprising a plurality of layers, the plurality of layers including a plurality of intermediate layers arranged between an input layer and an output layer, wherein each intermediate layer has a plurality of weight parameters, and wherein each intermediate layer is operable to output one or more activations to an adjacent layer in the plurality of layers, the computer program product comprising a non-transitory computer readable medium having computer executable instructions stored thereon, the instructions for configuring one or more processors to perform a method of training the deep neural network, wherein the method comprises:: (Section 3 describes training the deep neural network. In DNN architecture it is standard to have many layers including input and output layers with intermediate layers in between each having weights. Section 5 describes experiment results and thus there must be a computer product and computer readable medium to run experiments. )
inputting a plurality of training data samples into the input layer of the deep neural network, wherein the plurality of training data samples are contained within a training set used to train the deep neural network; and (Section 2. Page 2 left column “Assume a supervised learning setting where given a particular data set, DN = {(xi,yi)|xi ∈ X,yi ∈ Y,i ∈ {1,...,N = |X| = |Y|}}, we want to learn to predict the elements yi of the output (or label) set Y from the elements xi of the input set X.”)
generating a trained deep neural network using the plurality of training data samples by iteratively updating the plurality of weight parameters (Section 3 describes training a deep neural network using gradient based optimization techniques) to optimize an entropy constrained objective function, wherein the entropy constrained objective function is defined to jointly minimize a quantized loss function of the deep neural network and an entropy of a plurality of quantized weight values (Section 3 on page 3 (right column) teaches an objective function that both minimizes loss and entropy. Specifically equation 6. “Thus, we now aim to minimize the averaged prediction error of the network, as taken relative to our model Pθ, constrained by the respective relaxation of the entropy (5). However, we can still not apply gradient-based optimization techniques in order to minimize (6) as calculating the mean of the log-likelihood is infeasible for deep neural networks”) ,
Wiedemann does not teach:
the optimization is based on the quantized weights.
wherein the plurality of quantized weight values correspond to the plurality of weight parameters and each quantized weight value is a quantized representation of a corresponding weight parameter in the plurality of weight parameters.
However Choi does:
wherein the plurality of quantized weight values correspond to the plurality of weight parameters and each quantized weight value is a quantized representation of a corresponding weight parameter in the plurality of weight parameters. (Section 4.2 page 6 “In particular, optimal coding closely achieves the lower limit of the average source code length, i.e., entropy, and then we approximately have [Equation 10] where H is the entropy of the quantized network parameters after clustering (i.e., source), given that pi = |Ci|/N is the ratio of the number of network parameters in cluster Ci to the number of all network parameters (i.e., source distribution).”)
Wiedemann and Choi are considered analogous art to the claimed invention because
they are in the same field of endeavor being neural network quantization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the entropy and loss optimization of Wiedemann with the quantized weights of Choi. One would be motivated to do this for better compression of models.
Regarding claim 31, Wiedemann teaches:
A system for training a deep neural network, the deep neural network comprising a plurality of layers, the plurality of layers including a plurality of intermediate layers arranged between an input layer and an output layer, wherein each intermediate layer has a plurality of weight parameters, and wherein each intermediate layer is operable to output one or more activations to an adjacent layer in the plurality of layers, the system comprising: one or more processors; and one or more non-transitory storage mediums; wherein the one or more processors are configured to: (Section 3 describes training the deep neural network. In DNN architecture it is standard to have many layers including input and output layers with intermediate layers in between each having weights. Section 5 describes experiment results and thus there must be a computer system with processors and computer readable medium to run experiments. )
input a plurality of training data samples into the input layer of the deep neural network, wherein the plurality of training data samples are contained within a training set used to train the deep neural network; and (Section 2. Page 2 left column “Assume a supervised learning setting where given a particular data set, DN = {(xi,yi)|xi ∈ X,yi ∈ Y,i ∈ {1,...,N = |X| = |Y|}}, we want to learn to predict the elements yi of the output (or label) set Y from the elements xi of the input set X.”)
generate a trained deep neural network using the plurality of training data samples by iteratively updating the plurality of weight parameters (Section 3 describes training a deep neural network using gradient based optimization techniques) to optimize an entropy constrained objective function, wherein the entropy constrained objective function is defined to jointly minimize a quantized loss function of the deep neural network and an entropy of a plurality of quantized weight values (Section 3 on page 3 (right column) teaches an objective function that both minimizes loss and entropy. Specifically equation 6. “Thus, we now aim to minimize the averaged prediction error of the network, as taken relative to our model Pθ, constrained by the respective relaxation of the entropy (5). However, we can still not apply gradient-based optimization techniques in order to minimize (6) as calculating the mean of the log-likelihood is infeasible for deep neural networks”) ,
Wiedemann does not teach:
the optimization is based on the quantized weights.
wherein the plurality of quantized weight values correspond to the plurality of weight parameters and each quantized weight value is a quantized representation of a corresponding weight parameter in the plurality of weight parameters.
storing the trained deep neural network by storing a plurality of trained weight parameters in the one or more non-transitory data storage mediums.
However Choi does:
wherein the plurality of quantized weight values correspond to the plurality of weight parameters and each quantized weight value is a quantized representation of a corresponding weight parameter in the plurality of weight parameters. (Section 4.2 page 6 “In particular, optimal coding closely achieves the lower limit of the average source code length, i.e., entropy, and then we approximately have [Equation 10] where H is the entropy of the quantized network parameters after clustering (i.e., source), given that pi = |Ci|/N is the ratio of the number of network parameters in cluster Ci to the number of all network parameters (i.e., source distribution).”)
storing the trained deep neural network by storing a plurality of trained weight parameters in the one or more non-transitory data storage mediums. (Choi discusses reducing size for storage implying they store their models. Introduction page 1 “The large sizes of deep neural networks make it difficult to deploy them on resource-limited devices, e.g., mobile or portable devices, and network compression is of great interest in recent years to reduce computational cost and memory requirements for deep neural networks. Our interest in this paper is mainly on curtailing the size of the storage (memory) for network parameters (weights and biases). In particular, we focus on the network size compression by reducing the number of distinct network parameters by quantization.”)
Wiedemann and Choi are considered analogous art to the claimed invention because
they are in the same field of endeavor being neural network quantization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the entropy and loss optimization of Wiedemann with the quantized weights of Choi. One would be motivated to do this for better compression of models.
Claims 2-3, 17-18 are rejected under 35 U.S.C. 103 as being unpatentable by Wiedemann in view of Choi and Esser (NPL ‘Learned Step Size Quantization’ (2020))
Regarding claim 2, Wiedemann in view of Choi teaches claim 1 as outlined above. Neither of them teaches the elements of claim 2. However, Esser does:
the entropy constrained objective function is defined using a plurality of quantization function trainable parameters, and generating the trained deep neural network comprises iteratively updating the quantization function trainable parameters along with the plurality of weight parameters to optimize the entropy constrained objective function. (Section 2.3 pages 4-5 “Model quantizers are trained with LSQ by making their step sizes learnable parameters with loss gradient computed using the quantizer gradient described above, while other model parameters can be trained using existing techniques. Here, we employ a common means of training quantized networks (Courbariaux et al., 2015), where full precision weights are stored and updated, quantized weights and activations are used for forward and backward passes, the gradient through the quantizer round function is computed using the straight through estimator (Bengio et al., 2013) such that [equation 5] and stochastic gradient descent is used to updated parameters”).
Wiedemann, Choi and Esser are considered analogous art to the claimed invention because they are in the same field of endeavor being neural network quantization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the entropy and loss optimization of Wiedemann with the quantized weights of Choi and the trainable parameters of Esser. One would be motivated to do this for better performance of the quantizer itself (Esser Abstract).
Regarding claim 3, Wiedemann in view of Choi and Esser teaches claim 2 as outlined above. Esser further teaches:
the plurality of quantization function trainable parameters comprises a plurality of quantization function trainable parameter sets, wherein each quantization function trainable parameter set corresponds to a particular quantization function trainable parameter type (Section 2.1 page 3 “LSQ provides a means to learn s based on the training loss by introducing the following gradient through the quantizer to the step size parameter”), and the quantization function trainable parameter set for each quantization function trainable parameter type includes a plurality of layer-specific quantization function trainable parameters, wherein each layer-specific quantization function trainable parameter is associated with a particular layer of the plurality of layers of the deep neural network. (Section 2.1 page 4 “For this work, each layer of weights and each layer of activations has a distinct step size, represented as an fp32 value, initialized to 2 |v|/√QP , computed on either the initial weights values or the first batch of activations, respectively.”).
Regarding claim 17, Wiedemann in view of Choi teaches claim 16 as outlined above. Neither of them teaches the elements of claim 17. However, Esser does:
the entropy constrained objective function is defined using a plurality of quantization function trainable parameters, and generating the trained deep neural network comprises iteratively updating the quantization function trainable parameters along with the plurality of weight parameters to optimize the entropy constrained objective function. (Section 2.3 pages 4-5 “Model quantizers are trained with LSQ by making their step sizes learnable parameters with loss gradient computed using the quantizer gradient described above, while other model parameters can be trained using existing techniques. Here, we employ a common means of training quantized networks (Courbariaux et al., 2015), where full precision weights are stored and updated, quantized weights and activations are used for forward and backward passes, the gradient through the quantizer round function is computed using the straight through estimator (Bengio et al., 2013) such that [equation 5] and stochastic gradient descent is used to updated parameters”).
Wiedemann, Choi and Esser are considered analogous art to the claimed invention because they are in the same field of endeavor being neural network quantization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the entropy and loss optimization of Wiedemann with the quantized weights of Choi and the trainable parameters of Esser. One would be motivated to do this for better performance of the quantizer itself (Esser Abstract).
Regarding claim 18, Wiedemann in view of Choi and Esser teaches claim 17 as outlined above. Esser further teaches:
the plurality of quantization function trainable parameters comprises a plurality of quantization function trainable parameter sets, wherein each quantization function trainable parameter set corresponds to a particular quantization function trainable parameter type (Section 2.1 page 3 “LSQ provides a means to learn s based on the training loss by introducing the following gradient through the quantizer to the step size parameter”), and the quantization function trainable parameter set for each quantization function trainable parameter type includes a plurality of layer-specific quantization function trainable parameters, wherein each layer-specific quantization function trainable parameter is associated with a particular layer of the plurality of layers of the deep neural network. (Section 2.1 page 4 “For this work, each layer of weights and each layer of activations has a distinct step size, represented as an fp32 value, initialized to 2 |v|/√QP , computed on either the initial weights values or the first batch of activations, respectively.”).
Claims 4-6, 12-13, 15, 19-21, 27-28, 30 are rejected under 35 U.S.C. 103 as being unpatentable by Wiedemann in view of Choi and Agustsson (NPL ‘Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations’ (2017))
Regarding claim 4, Wiedemann in view of Choi teaches claim 1 as outlined above. Neither of them teaches the elements of claim 4. However, Agustsson does:
each quantized weight value is determined from the corresponding weight parameter using a probabilistic weight quantization function that defines a random mapping between a given weight parameter and the corresponding quantized weight value. (Section 3.2 Page 5 soft assignment is creating a probability distribution).
Wiedemann, Choi and Agustsson are considered analogous art to the claimed invention because they are in the same field of endeavor being neural network quantization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the entropy and loss optimization of Wiedemann with the quantized weights of Choi and probabilistic quantization of Agustsson. One would be motivated to do this for making the scheme differentiable (Agustsson Introduction).
Regarding claim 5, Wiedemann in view of Choi and Agustsson teaches claim 4 as outlined above. Agustsson further teaches:
iteratively updating the plurality of weight parameters comprises, for each iteration, calculating a gradient of the entropy constrained objective function, wherein calculating the gradient comprises a combination of backpropagation over the layers of the deep neural network and using a deterministic weight quantization function as an approximation of the probabilistic weight quantization function to calculate partial derivatives of the entropy constrained objective function. (Section 3.2 page 6 “We have therefore obtained a differentiable “soft entropy” loss (w.r.t. q), which is an upper bound on the sample entropy H(p). Hence, we can indirectly minimize H(p) by minimizing ˜ H(φ), treating the histogram probabilities of p as constants for gradient computation. However, we note that while qj is additive over the training data and the symbol sequence, log(qj) is not. This prevents the use of mini-batch gradient descent on ˜ H(φ), which can be an issue for large scale learning problems. In this case, we can instead re-define the soft entropy ˜ H(φ) as H(q,p). As before, ˜ H(φ) → H(p) as σ →∞,but ˜H(φ) ceases to be an upper bound for H(p).”
Regarding claim 6, Wiedemann in view of Choi and Agustsson teaches claim 4 as outlined above. Agustsson further teaches:
each quantized weight value corresponds to a potential quantized weight value from a plurality of potential quantized weight values, and the random mapping is defined using a conditional probability mass function defined to calculate a plurality of mapping probabilities for each weight parameter, wherein each mapping probability for a given weight parameter indicates a probability of that given weight parameter being quantized to one of the potential quantized weight values in the plurality of potential quantized weight values. (Section 3.2 page 6 “This gives us a valid probability mass function q = (q1,··· ,qL), which is differentiable”).
Regarding claim 12, Wiedemann in view of Choi teaches claim 1 as outlined above. Neither of the teaches the elements of claim 12. However Agustsson does:
each quantized weight value is a partially quantized weight value determined from the corresponding weight parameter using a deterministic weight quantization function. (Section 3.2 teaches the soft quantization which is a partially quantized value)
Wiedemann, Choi and Agustsson are considered analogous art to the claimed invention because they are in the same field of endeavor being neural network quantization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the entropy and loss optimization of Wiedemann with the quantized weights of Choi and probabilistic quantization of Agustsson. One would be motivated to do this for making the scheme differentiable (Agustsson Introduction).
Regarding claim 13, Wiedemann in view of Choi and Agustsson teaches claim 12 as outlined above. Agustsson further teaches:
the deterministic weight quantization function is configured to define the partially quantized weight value as a weighted average value of a probabilistic quantized weight value determined from the corresponding weight parameter using a probabilistic weight quantization function that defines a random mapping between a given weight parameter and the corresponding quantized weight value. (Section 3.2 teaches the soft quantization/soft assignment. Specifically equation 7.)
Regarding claim 15, Wiedemann in view of Choi teaches claim 1 as outlined above. Choi teaches:
storing the trained deep neural network by storing the quantized plurality of trained weight parameters in one or more non-transitory data storage elements. (Choi discusses reducing size for storage implying they store their models. Introduction page 1 “The large sizes of deep neural networks make it difficult to deploy them on resource-limited devices, e.g., mobile or portable devices, and network compression is of great interest in recent years to reduce computational cost and memory requirements for deep neural networks. Our interest in this paper is mainly on curtailing the size of the storage (memory) for network parameters (weights and biases). In particular, we focus on the network size compression by reducing the number of distinct network parameters by quantization.”)
Neither of Wiedemann nor Choi teaches:
generating a quantized plurality of trained weight parameters by, after generating the trained deep learning model, quantizing the plurality of weight parameters using a probabilistic weight quantization function that defines a random mapping between a given weight parameter and the corresponding quantized weight value; and
However, Agustsson does:
generating a quantized plurality of trained weight parameters by, after generating the trained deep learning model, quantizing the plurality of weight parameters using a probabilistic weight quantization function that defines a random mapping between a given weight parameter and the corresponding quantized weight value; and (Section 3.2 Page 5 soft assignment is creating a probability distribution for probabilistic weight quantization).
Wiedemann, Choi and Agustsson are considered analogous art to the claimed invention because they are in the same field of endeavor being neural network quantization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the entropy and loss optimization of Wiedemann with the quantized weights of Choi and probabilistic quantization of Agustsson. One would be motivated to do this for making the scheme differentiable (Agustsson Introduction).
Regarding claim 19, Wiedemann in view of Choi teaches claim 16 as outlined above. Neither of them teaches the elements of claim 19. However, Agustsson does:
each quantized weight value is determined from the corresponding weight parameter using a probabilistic weight quantization function that defines a random mapping between a given weight parameter and the corresponding quantized weight value. (Section 3.2 Page 5 soft assignment is creating a probability distribution).
Wiedemann, Choi and Agustsson are considered analogous art to the claimed invention because they are in the same field of endeavor being neural network quantization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the entropy and loss optimization of Wiedemann with the quantized weights of Choi and probabilistic quantization of Agustsson. One would be motivated to do this for making the scheme differentiable (Agustsson Introduction).
Regarding claim 20, Wiedemann in view of Choi and Agustsson teaches claim 19 as outlined above. Agustsson further teaches:
iteratively updating the plurality of weight parameters comprises, for each iteration, calculating a gradient of the entropy constrained objective function, wherein calculating the gradient comprises a combination of backpropagation over the layers of the deep neural network and using a deterministic weight quantization function as an approximation of the probabilistic weight quantization function to calculate partial derivatives of the entropy constrained objective function. (Section 3.2 page 6 “We have therefore obtained a differentiable “soft entropy” loss (w.r.t. q), which is an upper bound on the sample entropy H(p). Hence, we can indirectly minimize H(p) by minimizing ˜ H(φ), treating the histogram probabilities of p as constants for gradient computation. However, we note that while qj is additive over the training data and the symbol sequence, log(qj) is not. This prevents the use of mini-batch gradient descent on ˜ H(φ), which can be an issue for large scale learning problems. In this case, we can instead re-define the soft entropy ˜ H(φ) as H(q,p). As before, ˜ H(φ) → H(p) as σ →∞,but ˜H(φ) ceases to be an upper bound for H(p).”
Regarding claim 21, Wiedemann in view of Choi and Agustsson teaches claim 19 as outlined above. Agustsson further teaches:
each quantized weight value corresponds to a potential quantized weight value from a plurality of potential quantized weight values, and the random mapping is defined using a conditional probability mass function defined to calculate a plurality of mapping probabilities for each weight parameter, wherein each mapping probability for a given weight parameter indicates a probability of that given weight parameter being quantized to one of the potential quantized weight values in the plurality of potential quantized weight values. (Section 3.2 page 6 “This gives us a valid probability mass function q = (q1,··· ,qL), which is differentiable”).
Regarding claim 27, Wiedemann in view of Choi teaches claim 16 as outlined above. Neither of the teaches the elements of claim 27. However Agustsson does:
each quantized weight value is a partially quantized weight value determined from the corresponding weight parameter using a deterministic weight quantization function. (Section 3.2 teaches the soft quantization which is a partially quantized value)
Wiedemann, Choi and Agustsson are considered analogous art to the claimed invention because they are in the same field of endeavor being neural network quantization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the entropy and loss optimization of Wiedemann with the quantized weights of Choi and probabilistic quantization of Agustsson. One would be motivated to do this for making the scheme differentiable (Agustsson Introduction).
Regarding claim 28, Wiedemann in view of Choi and Agustsson teaches claim 27 as outlined above. Agustsson further teaches:
the deterministic weight quantization function is configured to define the partially quantized weight value as a weighted average value of a probabilistic quantized weight value determined from the corresponding weight parameter using a probabilistic weight quantization function that defines a random mapping between a given weight parameter and the corresponding quantized weight value. (Section 3.2 teaches the soft quantization/soft assignment. Specifically equation 7.)
Regarding claim 30, Wiedemann in view of Choi teaches claim 16 as outlined above. Choi teaches:
storing the trained deep neural network by storing the quantized plurality of trained weight parameters in one or more non-transitory data storage elements. (Choi discusses reducing size for storage implying they store their models. Introduction page 1 “The large sizes of deep neural networks make it difficult to deploy them on resource-limited devices, e.g., mobile or portable devices, and network compression is of great interest in recent years to reduce computational cost and memory requirements for deep neural networks. Our interest in this paper is mainly on curtailing the size of the storage (memory) for network parameters (weights and biases). In particular, we focus on the network size compression by reducing the number of distinct network parameters by quantization.”)
Neither of Wiedemann nor Choi teaches:
generating a quantized plurality of trained weight parameters by, after generating the trained deep learning model, quantizing the plurality of weight parameters using a probabilistic weight quantization function that defines a random mapping between a given weight parameter and the corresponding quantized weight value; and
However, Agustsson does:
generating a quantized plurality of trained weight parameters by, after generating the trained deep learning model, quantizing the plurality of weight parameters using a probabilistic weight quantization function that defines a random mapping between a given weight parameter and the corresponding quantized weight value; and (Section 3.2 Page 5 soft assignment is creating a probability distribution for probabilistic weight quantization).
Wiedemann, Choi and Agustsson are considered analogous art to the claimed invention because they are in the same field of endeavor being neural network quantization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the entropy and loss optimization of Wiedemann with the quantized weights of Choi and probabilistic quantization of Agustsson. One would be motivated to do this for making the scheme differentiable (Agustsson Introduction).
Claims 14 and 29 are rejected under 35 U.S.C. 103 as being unpatentable by Wiedemann in view of Choi, Agustsson and Alistarh (NPL ‘QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding’ (2017)).
Regarding claim 14, Wiedemann in view of Choi and Agustsson teaches claim 12 as outlined above. None of them teach the elements of claim 14. However, Alistarh does:
the trained deep neural network is generated using a plurality of training computing devices, wherein each training computing device is associated with a device specific batch of training data samples in the plurality of training data samples, and generating the trained deep neural network comprises transmitting weight parameters between the plurality of training computing devices, wherein prior to transmitting the weight parameters, each weight parameter is quantized into a corresponding quantized weight value determined from the corresponding weight parameter using a probabilistic weight quantization function that defines a random mapping between a given weight and the corresponding quantized weight value. (Section 5. Pages 7-8 “Setup. We performed experiments on Amazon EC2 p2.16xlarge instances, with 16 NVIDIA K80 GPUs. Instances have GPUDirect peer-to-peer communication, but do not currently support NVIDIA NCCL extensions. We have implemented QSGD on GPUs using the Microsoft Cognitive Toolkit (CNTK)[3]. This package provides efficient (MPI-based) GPU-to-GPU communication, and implements an optimized version of 1bit-SGD[35].”)
Wiedemann, Choi, Agustsson and Alistarh are considered analogous art to the claimed invention because they are in the same field of endeavor being neural network quantization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the entropy and loss optimization of Wiedemann with the quantized weights of Choi and probabilistic quantization of Agustsson and the distributed training of Alistarh. One would be motivated to do this for faster training (Alistarh Abstract).
Regarding claim 29, Wiedemann in view of Choi and Agustsson teaches claim 27 as outlined above. None of them teach the elements of claim 29. However, Alistarh does:
the trained deep neural network is generated using a plurality of training computing devices, wherein each training computing device is associated with a device specific batch of training data samples in the plurality of training data samples, and generating the trained deep neural network comprises transmitting weight parameters between the plurality of training computing devices, wherein prior to transmitting the weight parameters, each weight parameter is quantized into a corresponding quantized weight value determined from the corresponding weight parameter using a probabilistic weight quantization function that defines a random mapping between a given weight and the corresponding quantized weight value. (Section 5. Pages 7-8 “Setup. We performed experiments on Amazon EC2 p2.16xlarge instances, with 16 NVIDIA K80 GPUs. Instances have GPUDirect peer-to-peer communication, but do not currently support NVIDIA NCCL extensions. We have implemented QSGD on GPUs using the Microsoft Cognitive Toolkit (CNTK)[3]. This package provides efficient (MPI-based) GPU-to-GPU communication, and implements an optimized version of 1bit-SGD[35].”)
Wiedemann, Choi, Agustsson and Alistarh are considered analogous art to the claimed invention because they are in the same field of endeavor being neural network quantization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the entropy and loss optimization of Wiedemann with the quantized weights of Choi and probabilistic quantization of Agustsson and the distributed training of Alistarh. One would be motivated to do this for faster training (Alistarh Abstract).
Claims 7 and 22 are rejected under 35 U.S.C. 103 as being unpatentable by Wiedemann in view of Choi and Park (NPL ‘Weighted-Entropy-based Quantization for Deep Neural Networks’ (2017)).
Regarding claim 7, Wiedemann in view of Choi teaches claim 1 as outlined above. Wiedemann teaches minimizing based of loss and entropy as outlined above. Wiedemann does not teach optimizing based on the activation values. However Park does:
the entropy constrained objective function is defined to jointly minimize the quantized loss function, the entropy of the plurality of quantized weight values, and an entropy of a plurality of quantized activation values, wherein the plurality of quantized activation values correspond to a plurality of activations, wherein the plurality of activations includes the one or more activations provided by each intermediate layer in the plurality of layers, and each quantized activation value is a quantized representation of a corresponding activation in the plurality of activations. (Introduction page 2: “In order to address these two limitations, we propose a new quantization scheme based on the concept of weighted entropy. Our approach addresses both of the aforementioned limitations while quantizing weights and activations. Our contributions can be summarized as follows: 1. We propose a new multi-bit quantization method for both weights and activations. Unlike binary quantization approaches, our scheme is able to produce quantization results for any number of bits per weight/ activation, thereby realizing much more flexibility for exploiting accuracy-performance trade-off. 2. Our scheme facilitates automated quantization of the entire neural network. It does not require any modifications to the network except for activation quantization, and thus, it can be easily integrated into conventional training algorithms for neural networks”)
Wiedemann, Choi and Park are considered analogous art to the claimed invention because they are in the same field of endeavor being neural network quantization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the entropy and loss optimization of Wiedemann with the quantized weights of Choi and activation quantization of Park. One would be motivated to reduce model size (Park Abstract).
Regarding claim 22, Wiedemann in view of Choi teaches claim 16 as outlined above. Wiedemann teaches minimizing based of loss and entropy as outlined above. Wiedemann does not teach optimizing based on the activation values. However Park does:
the entropy constrained objective function is defined to jointly minimize the quantized loss function, the entropy of the plurality of quantized weight values, and an entropy of a plurality of quantized activation values, wherein the plurality of quantized activation values correspond to a plurality of activations, wherein the plurality of activations includes the one or more activations provided by each intermediate layer in the plurality of layers, and each quantized activation value is a quantized representation of a corresponding activation in the plurality of activations. (Introduction page 2: “In order to address these two limitations, we propose a new quantization scheme based on the concept of weighted entropy. Our approach addresses both of the aforementioned limitations while quantizing weights and activations. Our contributions can be summarized as follows: 1. We propose a new multi-bit quantization method for both weights and activations. Unlike binary quantization approaches, our scheme is able to produce quantization results for any number of bits per weight/ activation, thereby realizing much more flexibility for exploiting accuracy-performance trade-off. 2. Our scheme facilitates automated quantization of the entire neural network. It does not require any modifications to the network except for activation quantization, and thus, it can be easily integrated into conventional training algorithms for neural networks”)
Wiedemann, Choi and Park are considered analogous art to the claimed invention because they are in the same field of endeavor being neural network quantization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the entropy and loss optimization of Wiedemann with the quantized weights of Choi and activation quantization of Park. One would be motivated to reduce model size (Park Abstract).
Claims 8-10 and 23-25 are rejected under 35 U.S.C. 103 as being unpatentable by Wiedemann in view of Choi, Park and Agustsson.
Regarding claim 8, Wiedemann in view of Choi and Park teaches claim 7 as outlined above. Wiedemann teaches the entropy and weight optimization, Choi teaches the quantized weights and Park teaches the activation optimization. None of them teaches the probabilistic quantization. However, Agustsson does:
each quantized activation value is determined from the corresponding activation using a probabilistic activation quantization function that defines a random mapping between a given activation and the corresponding quantized activation value, and iteratively updating the plurality of weight parameters comprises, for each iteration, calculating a gradient of the entropy constrained objective function, wherein calculating the gradient comprises a combination of backpropagation over the layers of the deep neural network and using a deterministic activation quantization function as an approximation of the probabilistic activation quantization function to calculate partial derivatives of the entropy constrained objective function. (Page 2 “Our method is simple and intuitively appealing, relying on soft assignments of a given scalar or vector to be quantized to quantization levels. A parameter controls the “hardness” of the assignments and allows to gradually transition from soft to hard assignments during training. In contrast to rounding-based or stochastic quantization schemes, our coding scheme is directly differentiable, thus trainable end-to-end.”)
Wiedemann, Choi, Park and Agustsson are considered analogous art to the claimed invention because they are in the same field of endeavor being neural network quantization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the entropy and loss optimization of Wiedemann with the quantized weights of Choi and activation quantization of Park and the probabilistic quantization of Agustsson. One would be motivated to do this for making the scheme differentiable (Agustsson Introduction).
Regarding claim 9, Wiedemann in view of Choi and Park teaches claim 7 as outlined above. None of them teaches the elements of claim 9. However Agustsson does:
each quantized activation value is a partially quantized activation value determined from the corresponding activation using a deterministic activation quantization function. (Section 3.2 teaches the soft quantization which is a partially quantized value)
Regarding claim 10, Wiedemann in view of Choi, Park and Agustsson teaches claim 9 as outlined above. Agustsson further teaches:
the deterministic activation quantization function is configured to define the partially quantized activation value as a weighted average value of a probabilistic quantized activation value determined from the corresponding activation using a probabilistic activation quantization function that defines a random mapping between a given activation and the corresponding quantized activation value. (Section 3.2 teaches the soft quantization/soft assignment. Specifically equation 7.)
Regarding claim 23, Wiedemann in view of Choi and Park teaches claim 22 as outlined above. Wiedemann teaches the entropy and weight optimization, Choi teaches the quantized weights and Park teaches the activation optimization. None of them teaches the probabilistic quantization. However, Agustsson does:
each quantized activation value is determined from the corresponding activation using a probabilistic activation quantization function that defines a random mapping between a given activation and the corresponding quantized activation value, and iteratively updating the plurality of weight parameters comprises, for each iteration, calculating a gradient of the entropy constrained objective function, wherein calculating the gradient comprises a combination of backpropagation over the layers of the deep neural network and using a deterministic activation quantization function as an approximation of the probabilistic activation quantization function to calculate partial derivatives of the entropy constrained objective function. (Page 2 “Our method is simple and intuitively appealing, relying on soft assignments of a given scalar or vector to be quantized to quantization levels. A parameter controls the “hardness” of the assignments and allows to gradually transition from soft to hard assignments during training. In contrast to rounding-based or stochastic quantization schemes, our coding scheme is directly differentiable, thus trainable end-to-end.”)
Wiedemann, Choi, Park and Agustsson are considered analogous art to the claimed invention because they are in the same field of endeavor being neural network quantization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the entropy and loss optimization of Wiedemann with the quantized weights of Choi and activation quantization of Park and the probabilistic quantization of Agustsson. One would be motivated to do this for making the scheme differentiable (Agustsson Introduction).
Regarding claim 24, Wiedemann in view of Choi and Park teaches claim 22 as outlined above. None of them teaches the elements of claim 24. However Agustsson does:
each quantized activation value is a partially quantized activation value determined from the corresponding activation using a deterministic activation quantization function. (Section 3.2 teaches the soft quantization which is a partially quantized value)
Regarding claim 25, Wiedemann in view of Choi, Park and Agustsson teaches claim 24 as outlined above. Agustsson further teaches:
the deterministic activation quantization function is configured to define the partially quantized activation value as a weighted average value of a probabilistic quantized activation value determined from the corresponding activation using a probabilistic activation quantization function that defines a random mapping between a given activation and the corresponding quantized activation value. (Section 3.2 teaches the soft quantization/soft assignment. Specifically equation 7.)
Claims 11 and 26 are rejected under 35 U.S.C. 103 as being unpatentable by Wiedemann in view of Choi, Park and Agustsson and Alistarh.
Regarding claim 11, Wiedemann in view of Choi, Park and Agustsson teaches claim 9 as outlined above. None of them teach the elements of claim 11. However, Alistarh does:
the trained deep neural network is generated using a plurality of training computing devices, wherein each training computing device is associated with one or more layers of the plurality of layers and generating the trained deep neural network comprises transmitting activations between the plurality of training computing devices, wherein prior to transmitting the activations, each activation is quantized into a corresponding quantized activation value determined from the corresponding activation using a probabilistic activation quantization function that defines a random mapping between a given activation and the corresponding quantized activation value. (Section 5. Pages 7-8 “Setup. We performed experiments on Amazon EC2 p2.16xlarge instances, with 16 NVIDIA K80 GPUs. Instances have GPUDirect peer-to-peer communication, but do not currently support NVIDIA NCCL extensions. We have implemented QSGD on GPUs using the Microsoft Cognitive Toolkit (CNTK)[3]. This package provides efficient (MPI-based) GPU-to-GPU communication, and implements an optimized version of 1bit-SGD[35].”)
Wiedemann, Choi, Park and Agustsson are considered analogous art to the claimed invention because they are in the same field of endeavor being neural network quantization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the entropy and loss optimization of Wiedemann with the quantized weights of Choi and activation quantization of Park and the probabilistic quantization of Agustsson and the distributed training of Alistarh. One would be motivated to do this for faster training (Alistarh Abstract).
Regarding claim 26, Wiedemann in view of Choi, Park and Agustsson teaches claim 24 as outlined above. None of them teach the elements of claim 26. However, Alistarh does:
the trained deep neural network is generated using a plurality of training computing devices, wherein each training computing device is associated with one or more layers of the plurality of layers and generating the trained deep neural network comprises transmitting activations between the plurality of training computing devices, wherein prior to transmitting the activations, each activation is quantized into a corresponding quantized activation value determined from the corresponding activation using a probabilistic activation quantization function that defines a random mapping between a given activation and the corresponding quantized activation value. (Section 5. Pages 7-8 “Setup. We performed experiments on Amazon EC2 p2.16xlarge instances, with 16 NVIDIA K80 GPUs. Instances have GPUDirect peer-to-peer communication, but do not currently support NVIDIA NCCL extensions. We have implemented QSGD on GPUs using the Microsoft Cognitive Toolkit (CNTK)[3]. This package provides efficient (MPI-based) GPU-to-GPU communication, and implements an optimized version of 1bit-SGD[35].”)
Wiedemann, Choi, Park and Agustsson are considered analogous art to the claimed invention because they are in the same field of endeavor being neural network quantization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the entropy and loss optimization of Wiedemann with the quantized weights of Choi and activation quantization of Park and the probabilistic quantization of Agustsson and the distributed training of Alistarh. One would be motivated to do this for faster training (Alistarh Abstract).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL P GRUSZKA whose telephone number is (571)272-5259. The examiner can normally be reached M-F 9:00 AM - 6:00 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li Zhen can be reached at (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL GRUSZKA/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights