DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Remarks page 10-12, Applicant contends:
The claims, as amended, are not taught by the prior art of record. The prior art of record does not teach that a deep learning model is trained using quantization with noise training that includes dynamically varying a training dataset by repeatedly and randomly selecting sub-blocks for during-training quantization.
Response:
Applicant’s arguments with respect to claim(s) 1 and other independent claims have been considered but are moot because the new ground of rejection contain elements that have not been previously examined or does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Prima facie case is given in regards to the amendments in this office action’s rejections.
Claim Rejections - 35 USC § 112
Regarding 112(a):
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-4, 6-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
In regard to Claim 1:
Claim 1 recites “training the deep learning model includes using randomly-selected training samples obtained from a training dataset such that the training dataset varies dynamically throughout the training of the deep learning model”, which is seen as failing to comply for not having any description within the specification or disclosure within the figures for the elements introduced in the recited limitation. There is no recitation in the specification indicating that training samples are randomly selected from a dataset. One of ordinary skill in the art would be unable to determine what “the training dataset varies dynamically throughout the training of the deep learning model” is supposed to mean or how the randomly selected training samples achieves the dynamic variation. The specification indicates each sample is applied to the model, not randomly selected samples
([Current specification 00090]: “The training of the model applies each training sample in a training dataset to each layer of the model (block 602). The quantization with noise training engine randomly selects the block in each weight matrix at each layer to quantize as each training sample is applied to the model (block 604). Thereafter, the training consists of a forward pass (block 606), a loss calculation (block 608), and a backward pass (block 610).).
The remarks indicate that the amendments “includes dynamically varying a training dataset by repeatedly and randomly selected sub-blocks of a first portion of weight matrices for during training quantization” (Remarks page 10). This argument does not appear to note random sampling of data from a training dataset to dynamically vary training. The specification does not appear to indicate elements related to dynamically varying.
In regards to claims 9 and 16:
Claims 9 and 16 recite the same 112(a) rejected limitation as claim 1, thus are rejected for the same reason as claim 1.
In regards to dependent claims:
Claims dependent upon claims rejected under 112(a) are rejected for being dependent upon a rejected claim.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-4, 6-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gholami et al (“A survey of quantization methods for efficient neural network inference”), referred to as Gholami in this document, and further in view of Fan et al (“Training With Quantization Noise for Extreme Model Compression”), referred to as Fan in this document, and further in view of Sriram et al (US 20220044114 A1), referred to as Sriram in this document, and further in view of Baker (US 20200143240 A1), referred to as Baker in this document.
Regarding Claim 1:
Gholami teaches:
one or more processors;
([Gholami V. Quantization and Hardware Processors page 17]: “GAP-8 [64], a RISC-V SoC (System on Chip) for edge inference with a dedicated CNN accelerator, is another example of an edge processor that only supports integer arithmetic. While programmable general-purpose processors [one or more processors;] are widely adopted due to their flexibility, Google Edge TPU, a purpose-built ASIC chip, is another emerging solution for running inference at the edge.”)
obtain the deep learning model, which includes a plurality of layers, each layer comprising a plurality of weight matrices;
train the deep learning model to determine a value for each weight of each of the plurality of weight matrices, said value minimizes a loss function through application of training samples to each layer of the plurality of layers,
([Gholami 1) Quantization-Aware Training page 8]: “Given a trained model [obtain the deep learning model,], quantization may introduce a perturbation to the trained model parameters, and this can push the model away from the point to which it had converged when it was trained with floating point precision. It is possible to address this by re-training the NN model with quantized parameters so that the model can converge to a point with better loss [train the deep learning model to determine a value…]. One popular approach is to use Quantization- Aware Training (QAT), in which the usual forward and backward pass are performed on the quantized model in floating point, but the model parameters are quantized after each gradient update (similar to projected gradient descent).”)
([Gholami A. Problem Setup and Notations page 4]: “Assume that the NN has L layers with learnable parameters [which includes a plurality of layers, each layer comprising a plurality of weight matrices], denoted as {W1, W2, …, WL}, [for each weight of each of the plurality of weight matrices] with
θ
denoting the combination of all such parameters. Without loss of generality, we focus on the supervised learning problem, where the nominal goal is to optimize the following empirical risk minimization function
PNG
media_image1.png
86
233
media_image1.png
Greyscale
[said value minimizes a loss function] where (x, y) is the input data [through application of training samples where the input data is shown to be training data according to figure 4 of Gholami] and the corresponding label, l(x, y,
θ
) is the loss function (e.g., Mean Squared Error or Cross Entropy loss), and N is the total number of data points. Let us also denote the input hidden activations of the ith layer [to each layer of the plurality of layers] as hi, and the corresponding output hidden activation as ai. We assume that we have the trained model parameters
θ
, stored in floating point precision. In quantization, the goal is to reduce the precision of both the parameters (
θ
), as well as the intermediate activation maps (i.e., hi, ai) to low-precision, with minimal impact on the generalization power/accuracy of the model.”)
[Gholami figure 4 page 8]
PNG
media_image2.png
223
349
media_image2.png
Greyscale
after completion of the training of the deep learning model, perform the post-training quantization operation by quantizing each weight matrix of the plurality of weight matrices with reduced bit-width weights
([Gholami 2) Post-Training Quantization page 10]: “In PTQ, all the weights and activations quantization parameters are determined [after completion of the training of the deep learning model (training is just for a model that is trained here. The specific parts of training in these limitations are taught under the respective limitations), perform the post-training quantization operation by quantizing each weight matrix of the plurality of weight matrices with reduced bit-width weights] without any re-training of the NN model. As such, PTQ is a very fast method for quantizing NN models. ”)
Reduced bit-width being a part of the quantization is shown in Gholami and supported by the current application in the quotes below, as quantization is noted to use fixed-point integers, and fixed-point integers are said to be reduced bit-width.
([Gholami A. Simulated and Integer-only Quantization page 12]: “There are two common approaches to deploy a quantized NN model, simulated quantization (aka fake quantization) and integer-only quantization (aka fixed-point quantization).”
[Current Application 000124]: “In an aspect, the program includes instructions to perform acts that: randomly select weights in the first portion of each weight matrix to quantized with reduced bit-widths. In an aspect, the reduced bit-width weights are fixed-point integers. In an aspect, the reduced bit-width weights are INT4 or INT8 data types. In an aspect, the deep learning model is a neural transformer model with attention.”)
generated by stochastic gradient descent
[Gholami F. Non-Uniform Quantization page 8]: “and the quantization steps/levels are generally trained with iterative optimization [258, 276] or gradient descent [generated by stochastic gradient descent]”
Gholami does not explicitly teach:
each weight matrix of the plurality of weight matrices includes a first portion and a second portion,
one or more randomly-selected sub-blocks of each first portion of each weight are quantized as a part of said during-training quantization operation with reduced bit-width weights of a determined data type,
the second portion of each weight matrix of the plurality of weight matrices comprises unquantized weights having full-precision floating point values,
each quantized weight of the first portion of each weight matrix of the plurality of weight matrices is updated based on a first gradient generated by an estimator,
each unquantized weight of the second portion of each weight matrix of the plurality of weight matrices is updated based on a second gradient generated by stochastic gradient descent
A system that (i) trains a deep learning model through quantization with noise training, such that during-training quantization operation is performed during said training and (ii) after said training is complete, performs a post-training quantization operation, said system comprising:
one or more hardware storage devices that store instructions that are executable by the one or more processors to cause the system to:
that are also of the same determined data type, such that the during-training quantization operation and the post-training quantization operation use the same determined data type when quantizing data
Fan teaches:
each weight matrix of the plurality of weight matrices includes a first portion and a second portion,
one or more randomly-selected sub-blocks of each first portion of each weight are selected for inclusion in the training dataset and are quantized as a part of said during-training quantization operation with reduced bit-width weights of a determined data type
the second portion of each weight matrix of the plurality of weight matrices comprises unquantized weights having full-precision floating point values,
each quantized weight of the first portion of each weight matrix of the plurality of weight matrices is updated based on a first gradient generated by an estimator,
each unquantized weight of the second portion of each weight matrix of the plurality of weight matrices is updated based on a second gradient
quantization with noise training
([Fan 4.1 Training Networks with Quantization Noise page 5]: “We consider the case of a real matrix W as in Section 3. During the training of a network, our proposed Quant-Noise [quantization with noise training] method works as follows: first, we compute blocks bkl related to a target quantization method. Then, during each forward pass, we randomly select a subset of these blocks and apply some distortion to them [each weight matrix of the plurality of weight matrices includes a first portion and a second portion][the second portion of each weight matrix of the plurality of weight matrices comprises unquantized weights having full-precision floating point values,].)
The distortion is noted to be noise, where the noise is noted to “noise function φ simulates the change in the weights produced by the target quantization method” below. The equation for the noise function is also listed to show that said function is representing quantization. The prior art uses the terms “distortion” and “noise” to refer to quantization in cases as a result of the effects quantization has on models. This is supported by [Current Application 0004]: “A deep learning model is trained through quantization with noise training to learn to perform a target software engineering task. During the quantized with noise training, a portion of the weights of a weight matrix are quantized into integer data types. By reducing the bit-width of a portion of the weights during training makes the model more resilient to quantization and reduces the noise or discrepancy between the quantized and full-precision model outputs.”
[Fan 4.1 Training Networks with Quantization Noise page 5]: “More formally, given a set of tuples of indices
PNG
media_image3.png
33
340
media_image3.png
Greyscale
and a distortion or noise function
φ
acting on a block, we define an operator
PNG
media_image4.png
27
61
media_image4.png
Greyscale
such that, for each block bkl, we apply the following transformation
PNG
media_image5.png
66
307
media_image5.png
Greyscale
The noise function φ simulates the change in the weights produced by the target quantization method (see Section 4.2 for details). We replace the matrix W by the resulting noisy matrix Wnoise during the forward pass to compute a noisy output ynoise, i.e.,
PNG
media_image6.png
40
407
media_image6.png
Greyscale
where x is an input vector. During the backward pass, we apply STE [each quantized weight of the first portion of each weight matrix of the plurality of weight matrices is updated based on a first gradient generated by an estimator,], which amounts to replacing the distorted weights Wnoise by their non-distorted counterparts. Note that our approach is equivalent to QAT when J contains all the tuples of indices. However, an advantage of Quant-Noise over QAT is that unbiased gradients continue to flow via blocks unaffected by the noise. As these blocks are randomly selected for each forward, we guarantee that each weight regularly sees gradients that are not affected by the nature of the function
φ
. [each unquantized weight of the second portion of each weight matrix of the plurality of weight matrices is updated based on a second gradient where the idea is supported by there being a portion of weights that is not quantized, and those weights are not using the estimator so those weights are able to use more conventional gradient methods such as gradient descent] As a side effect, our quantization noise regularizes the network in a similar way as DropConnect (Wan et al., 2013) or LayerDrop (Fan et al., 2019).”)
The equation for noise that is quantization [Fan 4.2 Adding Noise To Specific Quantization Methods page 6]:
PNG
media_image7.png
166
766
media_image7.png
Greyscale
[are quantized as a part of said during-training quantization operation with reduced bit-width weights of a determined data type as the Quant-Noise method is noted to be “QAT” and to be for “int 8”]
one or more randomly-selected sub-blocks of each first portion of each weight are selected for inclusion in the training dataset
([Fan 4.1 Training Networks with Quantization Noise page 5]: “We consider the case of a real matrix W as in Section 3. During the training of a network, our proposed Quant-Noise method works as follows: first, we compute blocks bkl related to a target quantization method. Then, during each forward pass, we randomly select a subset of these blocks and apply some distortion to them [one or more randomly-selected sub-blocks of each first portion of each weight are selected for inclusion in the training dataset as the distortion is noted to be noise, where the noise is noted to “noise function φ simulates the change in the weights produced by the target quantization method” as shown by the references in claim 1. The training dataset aspect is noted by this quote indicating this is an element of training].)
One of ordinary skill in the art prior to the effective filing date, would have been motivated to combine Gholami and Fan to incorporate having the matrices be split into two portions for quantization. Gholami and Fan are of the same field of endeavor of machine learning. One of ordinary skill in the art would have been motivated to combine Gholami and Fan to split the matrices into two for quantization or quantization noise, as the method involving the process is used to allow unbiased gradients to flow ([Fan 4.1 Training Networks with Quantization Noise]: “Note that our approach is equivalent to QAT when J contains all the tuples of indices. However, an advantage of Quant-Noise over QAT is that unbiased gradients continue to flow via blocks unaffected by the noise. As these blocks are randomly selected for each forward, we guarantee that each weight regularly sees gradients that are not affected by the nature of the function
φ
. As a side effect, our quantization noise regularizes the network in a similar way as DropConnect (Wan et al., 2013) or LayerDrop (Fan et al., 2019).”).
Sriram teaches:
A system that (i) trains a deep learning model through quantization with noise training, such that during-training quantization operation is performed during said training and (ii) after said training is complete, performs a post-training quantization operation, said system comprising:
that are also of the same determined data type, such that the during-training quantization operation and the post-training quantization operation use the same determined data type when quantizing data
[Sriram 0063]: “When QAT 108 is applied, GPU 116 trains one or more DNNs for lower precision INT8 deployment, without compromising on accuracy. This may be achieved by modeling quantization errors during training which helps in maintaining accuracy as compared to floating-point 16 (FP16) or floating point 32 (FP32). In at least one embodiment, GPU 116, using one or more processors, first applies QAT 108, during training 106 [A system that (i) trains a deep learning model through quantization with noise training, such that during-training quantization operation is performed during said training], to output a first trained model 110. The first trained model 110 may also be referred to herein as an intermediate trained model, a QAT trained model, QAT quantized model, or a QAT model. After the training is complete with a satisfactory model, accuracy of the first trained model 110 is then calibrated using an INT8 entropy calibrator (calibration is described in more detail below with respect to FIG. 2). The INT8 entropy calibrator calibrates the first trained model 110 when building an INT8 engine to output a satisfactory trained model where PTQ can then be applied. In at least one embodiment, GPU 116, using one or more processors, applies PTQ 112 [and (ii) after said training is complete, performs a post-training quantization operation, said system comprising] on the first trained model 110 to output a second trained model 114 that has both QAT and PTQ applied. The second trained model 114 may comprise parameters (e.g., weights and activations) that are represented by 8-bit integers [that are also of the same determined data type, such that the during-training quantization operation and the post-training quantization operation use the same determined data type when quantizing data as the QAT is noted as being for 8 bit integers at the beginning of the quote (“When QAT 108 is applied, GPU 116 trains one or more DNNs for lower precision INT8 deployment”) and the current part of the ref notes PTQ and a resulting model of 8 bit integers, thus both quantizations are 8 bit integers].”
one or more hardware storage devices that store instructions that are executable by the one or more processors to cause the system to:
[Sriram 0111]: “In at least one embodiment, code is stored on a computer-readable storage medium in form of a computer program comprising a plurality of computer-readable instructions [one or more hardware storage devices that store instructions that are executable by the one or more processors to cause the system to:] executable by one or more processors. In at least one embodiment, a computer-readable storage medium is a non-transitory computer-readable medium.”
One of ordinary skill in the art, prior to the effective filing date, would have been motivated to combine Gholami and Sriram. Gholami and Sriram are in the same field of endeavor of machine learning and quantization. One of ordinary skill in the art would have been motivated to combine Gholami and Sriram in order to utilize both QAT (quantization aware training) and PTQ (post training quantization) to maintain accuracy of the model while reducing memory usage and computation on things such as edge devices ([Sriram 0057]: “Techniques described herein provide a way to transform (e.g., quantize) a model to have weights represented by lower-precision values (e.g., low-bit integers) instead of using higher-precision values (e.g., values with full floating point-precision) to conserve memory usage and reduces computation when the trained model is deployed. In one example, an object detection or classification network is trained using full floating-point precision (either single-precision 32-bit floating point or double-precision 64-bit floating point) to represent the weights and activations of a deep neural network (DNN). However, when these trained models are deployed to edge devices that have less computational resources, it can tie up resources and be quite inefficient. In some instances, quantizing models such as rounding the weights and activations to lower-bit integers (e.g., 8-bit integers) is performed during training (e.g., quantization-aware training (QAT)), and in some other instances, it can be performed post-training (e.g., post-training quantization ((PTQ)). However, performing quantization either during training or post-training by itself can result in some drawbacks (e.g., the trained model is less accurate). The techniques described herein generate a trained quantized model to use 8-bit integer numbers by using a combination of both QAT and PTQ in a way that maintains accuracy of the model.”).
The use of a storage medium is motivated from the storage of instructions to enable a processor to perform the steps of the invention ([Sriram 0111]: “In at least one embodiment, code is stored on a computer-readable storage medium in form of a computer program comprising a plurality of computer-readable instructions executable by one or more processors.)
Baker teaches:
training the deep learning model includes using randomly-selected training samples obtained from a training dataset, such that the training dataset varies dynamically throughout the training of the deep learning model
[Baker 0099]: "If the data items for each minibatch are random samples independently selected [training the deep learning model includes using randomly-selected training samples obtained from a training dataset, such that the training dataset varies dynamically throughout the training of the deep learning model] from the same distribution of training data examples, the standard deviation of the estimate of each component of the gradient will vary in inverse proportion to the size of the minibatch. Thus, a larger minibatch will tend to be more accurate in the sense that the estimate will have a lower standard deviation."
One of ordinary skill in the art, prior to the effective filing date, would have been motivated to combine Gholami and Baker. Gholami and Baker are in the same field of endeavor of machine learning. One of ordinary skill in the art would have been motivated to combine Gholami and Baker in order to utilize random samples selected from training data, as this results in the gradient to vary inverse proportion to the size, allowing larger minibatches to have more accurate results ([Baker 0099]: "If the data items for each minibatch are random samples independently selected from the same distribution of training data examples, the standard deviation of the estimate of each component of the gradient will vary in inverse proportion to the size of the minibatch. Thus, a larger minibatch will tend to be more accurate in the sense that the estimate will have a lower standard deviation.).
Regarding Claim 2:
The system of claim 1 is taught by Gholami, Fan, Sriram, Baker.
Gholami teaches:
wherein the instructions are further executable to cause the system to: generate a codebook for each of the plurality of weight matrices, wherein the codebook includes a plurality of uniformly-distributed range of values
([Gholami IV Advanced Concepts: Quantization Below 8 bits F. Vector Quantization page 16]: “Having said that, there are a lot of interesting ideas in the classical quantization methods in DSP that have been applied to NN quantization, and in particular vector quantization [9]. In particular, the work of [1, 30, 74, 84, 117, 170, 180, 189, 256] clusters the weights into different groups and use the centroid of each group as quantized values during inference. As shown in Eq. 13, i is the index of weights in a tensor, c1, …, ck are the k centroids found by the clustering, and cj is the corresponding centroid to wi. After clustering, weight wi will have a cluster index j related to cj in the codebook [wherein the instructions are further executable to cause the system to: generate a codebook for each of the plurality of weight matrices] (look-up table)… It has been found that using a k-means clustering is sufficient to reduce the model size up to 8x without significant accuracy degradation [74].”)
(Gholami Figure 2 on page 5 [wherein the codebook includes a plurality of uniformly-distributed range of values])
PNG
media_image8.png
343
1021
media_image8.png
Greyscale
Quote from current application for the specification’s interpretation [Current Application 00020]: “In the case of scalar quantization 108, each quantized weight matrix is transformed into a quantized weight matrix of indices 110 to a codebook 112. The full-precision floating point values in the matrix are rescaled into a uniformly-distributed range of values where the number of ranges k is based on the n-bit integer data type (k = 2n-1). For example, when the weights are quantized to INT8 data types, there are 255 ranges.”
Regarding Claim 3:
The system of claim 1 is taught by Gholami, Fan, Sriram, Baker.
Gholami teaches:
wherein the instructions are further executable to cause the system to: generate a codebook for each of the plurality of weight matrices, wherein the codebook includes a plurality of centroids, wherein each centroid of the plurality of centroids is generated from K-means clustering of weights of a respective weight matrix
([Gholami IV Advanced Concepts: Quantization Below 8 bits F. Vector Quantization page 16]: “Having said that, there are a lot of interesting ideas in the classical quantization methods in DSP that have been applied to NN quantization, and in particular vector quantization [9]. In particular, the work of [1, 30, 74, 84, 117, 170, 180, 189, 256] clusters the weights into different groups and use the centroid of each group as quantized values during inference. As shown in Eq. 13, i is the index of weights in a tensor, c1, …, ck are the k centroids [wherein the codebook includes a plurality of centroids] found by the clustering, and cj is the corresponding centroid to wi. After clustering, weight wi will have a cluster index j related to cj in the codebook [wherein the instructions are further executable to cause the system to: generate a codebook for each of the plurality of weight matrices] (look-up table)… It has been found that using a k-means clustering [wherein each centroid of the plurality of centroids is generated from K-means clustering of weights of a respective weight matrix] is sufficient to reduce the model size up to 8x without significant accuracy degradation [74].”)
Regarding Claim 4:
The system of claim 3 is taught by Gholami, Fan, Sriram, Baker.
Gholami teaches:
wherein the instructions are further executable to cause the system to: generate an index matrix that maps a weight of a respective weight matrix into a select one of the centroids of the codebook
([Gholami IV Advanced Concepts: Quantization Below 8 bits F. Vector Quantization page 16]: “Having said that, there are a lot of interesting ideas in the classical quantization methods in DSP that have been applied to NN quantization, and in particular vector quantization [9]. In particular, the work of [1, 30, 74, 84, 117, 170, 180, 189, 256] clusters the weights into different groups and use the centroid of each group as quantized values during inference. As shown in Eq. 13, i is the index of weights in a tensor, c1, …, ck are the k centroids found by the clustering, and cj is the corresponding centroid to wi [wherein the instructions are further executable to cause the system to: generate an index matrix that maps a weight of a respective weight matrix into a select one of the centroids of the codebook]. After clustering, weight wi will have a cluster index j related to cj in the codebook (look-up table)… It has been found that using a k-means clustering is sufficient to reduce the model size up to 8x without significant accuracy degradation [74].”)
Regarding Claim 6:
The system of claim 1 is taught by Gholami, Fan, Sriram, Baker.
Gholami teaches:
wherein the determined type for the reduced bit-width weights include fixed-point integers
([Gholami A. Simulated and Integer-only Quantization page 12]: “There are two common approaches to deploy a quantized NN model, simulated quantization (aka fake quantization) and integer-only quantization (aka fixed-point quantization [wherein the determined type for the reduced bit-width weights include fixed-point integers]).”)
Regarding Claim 7:
The system of claim 1 is taught by Gholami, Fan, Sriram, Baker.
Gholami teaches:
wherein the determined type for the reduced bit-width weights include INT4 or INT8 data types
(Gholami Figure 2 on page 5 description [wherein the determined type for the reduced bit-width weights include INT4 or INT8 data types])
PNG
media_image8.png
343
1021
media_image8.png
Greyscale
Regarding Claim 8:
The system of claim 1 is taught by Gholami, Fan, Sriram, Baker.
Gholami teaches:
wherein the deep learning model is a neural transformer model with attention
([Gholami E. Quantization Granularity page 7]: “This could be helpful for cases where the distribution of the parameters across a single convolution/activation varies a lot. For instance, this approach was found useful in Q-BERT [219] for quantizing Transformer [243] models that consist of fully-connected attention layers [wherein the deep learning model is a neural transformer model with attention].”)
Regarding Claim 9:
Claim 9 is analogous to claim 1 aside from the limitations noted below.
Gholami teaches:
computing an error loss from the computations;
([Gholami A. Problem Setup and Notations page 4]: “Assume that the NN has L layers with learnable parameters, denoted as {W1, W2, …, WL}, with
θ
denoting the combination of all such parameters. Without loss of generality, we focus on the supervised learning problem, where the nominal goal is to optimize the following empirical risk minimization function
PNG
media_image1.png
86
233
media_image1.png
Greyscale
where (x, y) is the input data and the corresponding label, l(x, y,
θ
) is the loss function [computing an error loss from the computations] (e.g., Mean Squared Error or Cross Entropy loss), and N is the total number of data points. Let us also denote the input hidden activations of the ith layer as hi, and the corresponding output hidden activation as ai. We assume that we have the trained model parameters
θ
, stored in floating point precision. In quantization, the goal is to reduce the precision of both the parameters (
θ
), as well as the intermediate activation maps (i.e., hi, ai) to low-precision, with minimal impact on the generalization power/accuracy of the model.”)
[Gholami figure 4 page 8]
PNG
media_image2.png
223
349
media_image2.png
Greyscale
Gholami does not explicitly teach:
performing computations at each layer of the plurality of layers with the first type of fixed-point integer representation
Fan teaches:
performing computations at each layer of the plurality of layers with the first type of fixed-point integer representation
([Fan 4.1 Training Networks with Quantization Noise page 5]: “We consider the case of a real matrix W as in Section 3. During the training of a network, our proposed Quant-Noise method works as follows: first, we compute blocks bkl related to a target quantization method. Then, during each forward pass, we randomly select a subset of these blocks and apply some distortion to them.)
The distortion is noted to be noise, where the noise is noted to “noise function φ simulates the change in the weights produced by the target quantization method” below. The equation for the noise function is also listed to show that said function is representing quantization. The prior art uses the terms “distortion” and “noise” to refer to quantization in cases as a result of the effects quantization has on models. This is supported by [Current Application 0004]: “A deep learning model is trained through quantization with noise training to learn to perform a target software engineering task. During the quantized with noise training, a portion of the weights of a weight matrix are quantized into integer data types. By reducing the bit-width of a portion of the weights during training makes the model more resilient to quantization and reduces the noise or discrepancy between the quantized and full-precision model outputs.”
[Fan 4.1 Training Networks with Quantization Noise page 5]: “More formally, given a set of tuples of indices
PNG
media_image3.png
33
340
media_image3.png
Greyscale
and a distortion or noise function
φ
acting on a block, we define an operator
PNG
media_image4.png
27
61
media_image4.png
Greyscale
such that, for each block bkl, we apply the following transformation
PNG
media_image5.png
66
307
media_image5.png
Greyscale
The noise function φ simulates the change in the weights produced by the target quantization method (see Section 4.2 for details). We replace the matrix W by the resulting noisy matrix Wnoise during the forward pass [performing computations at each layer of the plurality of layers with the first type of fixed-point integer representation where the first type of fixed point integer representation is shown below by this section referring to QAT, as the first type is used for in training quantization, and int N and int 8 being noted] to compute a noisy output ynoise, i.e.,
PNG
media_image6.png
40
407
media_image6.png
Greyscale
where x is an input vector. During the backward pass, we apply STE, which amounts to replacing the distorted weights Wnoise by their non-distorted counterparts. Note that our approach is equivalent to QAT when J contains all the tuples of indices. However, an advantage of Quant-Noise over QAT is that unbiased gradients continue to flow via blocks unaffected by the noise. As these blocks are randomly selected for each forward, we guarantee that each weight regularly sees gradients that are not affected by the nature of the function
φ
. As a side effect, our quantization noise regularizes the network in a similar way as DropConnect (Wan et al., 2013) or LayerDrop (Fan et al., 2019).”)
The equation for noise that is quantization [Fan 4.2 Adding Noise To Specific Quantization Methods page 6]:
PNG
media_image7.png
166
766
media_image7.png
Greyscale
The motivations are the same as claim 1 to combine with Fan and Sriram.
Regarding Claim 10:
The method of claim 9 is taught by Gholami, Fan, Sriram, Baker.
Fan teaches:
decomposing each weight matrix into sub-blocks;
([Fan 4.1 Training Networks with Quantization Noise page 5]: “We consider the case of a real matrix W as in Section 3. During the training of a network, our proposed Quant-Noise method works as follows: first, we compute blocks bkl related to a target quantization method. Then, during each forward pass, we randomly select a subset of these blocks and apply some distortion to them [decomposing each weight matrix into sub-blocks].)
The motivation is the same as claim 9 to combine with Fan.
Regarding Claim 11:
The method of claim 9 is taught by Gholami, Fan, Sriram, Baker.
Gholami teaches:
generating a codebook for a first weight matrix, wherein the codebook includes a plurality of uniformly-distributed range of values based on an n-bit representation of the first type of fixed-point integer representation; and mapping a weight of the first weight matrix into a value of the codebook
([Gholami F. Vector Quantization page 16]: “Having said that, there are a lot of interesting ideas in the classical quantization methods in DSP that have been applied to NN quantization, and in particular vector quantization [9]. In particular, the work of [1, 30, 74, 84, 117, 170, 180, 189, 256] clusters the weights into different groups and use the centroid of each group as quantized values during inference. As shown in Eq. 13, i is the index of weights in a tensor, c1, …, ck are the k centroids found by the clustering, and cj is the corresponding centroid to wi. After clustering, weight wi will have a cluster index j related to cj in the codebook [generating a codebook for a first weight matrix… and mapping a weight of the first weight matrix into a value of the codebook] (look-up table)… It has been found that using a k-means clustering is sufficient to reduce the model size up to 8x without significant accuracy degradation [74].”)
(Gholami Figure 2 on page 5 [wherein the codebook includes a plurality of uniformly-distributed range of values based on an n-bit representation of the first type of fixed-point integer representation])
PNG
media_image8.png
343
1021
media_image8.png
Greyscale
Quote from current application for the specification’s interpretation [Current Application 00020]: “In the case of scalar quantization 108, each quantized weight matrix is transformed into a quantized weight matrix of indices 110 to a codebook 112. The full-precision floating point values in the matrix are rescaled into a uniformly-distributed range of values where the number of ranges k is based on the n-bit integer data type (k = 2n-1). For example, when the weights are quantized to INT8 data types, there are 255 ranges.”
Regarding Claim 12:
The method of claim 9 is taught by Gholami, Fan, Sriram, Baker.
Gholami teaches:
generating a codebook for a second weight matrix, wherein the codebook includes a plurality of centroids, wherein each centroid of the plurality of centroids is generated from K- means clustering of weights of the second weight matrix.
([Gholami F. Vector Quantization page 16]: “Having said that, there are a lot of interesting ideas in the classical quantization methods in DSP that have been applied to NN quantization, and in particular vector quantization [9]. In particular, the work of [1, 30, 74, 84, 117, 170, 180, 189, 256] clusters the weights into different groups and use the centroid of each group as quantized values during inference. As shown in Eq. 13, i is the index of weights in a tensor, c1, …, ck are the k centroids found by the clustering, and cj is the corresponding centroid to wi. After clustering, weight wi will have a cluster index j related to cj in the codebook [generating a codebook for a second weight matrix, wherein the codebook includes a plurality of centroids] (look-up table)… It has been found that using a k-means clustering [wherein each centroid of the plurality of centroids is generated from K- means clustering of weights of the second weight matrix] is sufficient to reduce the model size up to 8x without significant accuracy degradation [74].”)
Regarding Claim 13:
The method of claim 9 is taught by Gholami, Fan, Sriram, Baker.
Gholami teaches:
generating an index matrix to map a weight of the second weight matrix into the select centroid of the codebook
([Gholami F. Vector Quantization page 16]: “Having said that, there are a lot of interesting ideas in the classical quantization methods in DSP that have been applied to NN quantization, and in particular vector quantization [9]. In particular, the work of [1, 30, 74, 84, 117, 170, 180, 189, 256] clusters the weights into different groups and use the centroid of each group as quantized values during inference. As shown in Eq. 13, i is the index of weights in a tensor, c1, …, ck are the k centroids found by the clustering, and cj is the corresponding centroid to wi [generating an index matrix to map a weight of the second weight matrix into the select centroid of the codebook]. After clustering, weight wi will have a cluster index j related to cj in the codebook (look-up table)… It has been found that using a k-means clustering is sufficient to reduce the model size up to 8x without significant accuracy degradation [74].”)
Regarding Claim 14:
The method of claim 9 is taught by Gholami, Fan, Sriram, Baker.
This claim is analogous to claim 7.
Regarding Claim 15:
The method of claim 9 is taught by Gholami, Fan, Sriram, Baker.
This claim is analogous to claim 8.
Claims 16-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gholami et al (“A survey of quantization methods for efficient neural network inference”), referred to as Gholami in this document, and further in view of Fan et al (“Training With Quantization Noise for Extreme Model Compression”), referred to as Fan in this document, and further in view of Sriram et al (US 20220044114 A1), referred to as Sriram in this document, and further in view of Baker (US 20200143240 A1), referred to as Baker in this document, and even further in view of Kaitha et al (US 11416224 B1), referred to as Kaitha in this document.
Regarding Claim 16:
Claim 16 is analogous to claim 1 aside from the limitations mapped below.
Gholami teaches:
and memory, wherein the memory includes instructions that are executable by the processor to cause the device to
([Gholami C. Hardware Aware Quantization page 14]: “One of the goals of quantization is to improve the inference latency. However, not all hardware provide the same speed up after a certain layer/operation is quantized. In fact, the benefits from quantization is hardware-dependant, with many factors such as on-chip memory [and memory, wherein the memory includes instructions that are executable by the processor to cause the device to], bandwidth, and cache hierarchy affecting the quantization speed up.”)
by computing values for each of the plurality of weights of the at least one weight matrix of each of the plurality of layers that minimizes an error function
([Gholami A. Problem Setup and Notations page 4]: “Assume that the NN has L layers with learnable parameters, denoted as {W1, W2, …, WL}, with
θ
denoting the combination of all such parameters. Without loss of generality, we focus on the supervised learning problem, where the nominal goal is to optimize the following empirical risk minimization function
PNG
media_image1.png
86
233
media_image1.png
Greyscale
where (x, y) is the input data and the corresponding label, l(x, y,
θ
) is the loss function [by computing values for each of the plurality of weights of the at least one weight matrix of each of the plurality of layers that minimizes an error function] (e.g., Mean Squared Error or Cross Entropy loss), and N is the total number of data points. Let us also denote the input hidden activations of the ith layer as hi, and the corresponding output hidden activation as ai. We assume that we have the trained model parameters
θ
, stored in floating point precision. In quantization, the goal is to reduce the precision of both the parameters (
θ
), as well as the intermediate activation maps (i.e., hi, ai) to low-precision, with minimal impact on the generalization power/accuracy of the model.”)
[Gholami figure 4 page 8]
PNG
media_image2.png
223
349
media_image2.png
Greyscale
Gholami does not explicitly teach:
train the deep learning model to learn to generate source code
determine values for weights of the at least one weight matrix through multiple iterations of a forward pass, backward pass, and weight update, wherein the forward pass computes a loss
Fan teaches:
determine values for weights of the at least one weight matrix through multiple iterations of a forward pass, backward pass, and weight update,
([Fan 4.1 Training Networks with Quantization Noise page 5]: “We consider the case of a real matrix W as in Section 3. During the training of a network [and weight update], our proposed Quant-Noise method works as follows: first, we compute blocks bkl related to a target quantization method. Then, during each forward pass [determine values for weights of the at least one weight matrix through multiple iterations of a forward pass], we randomly select a subset of these blocks and apply some distortion to them.)
The distortion is noted to be noise, where the noise is noted to “noise function φ simulates the change in the weights produced by the target quantization method” below. The equation for the noise function is also listed to show that said function is representing quantization. The prior art uses the terms “distortion” and “noise” to refer to quantization in cases as a result of the effects quantization has on models. This is supported by [Current Application 0004]: “A deep learning model is trained through quantization with noise training to learn to perform a target software engineering task. During the quantized with noise training, a portion of the weights of a weight matrix are quantized into integer data types. By reducing the bit-width of a portion of the weights during training makes the model more resilient to quantization and reduces the noise or discrepancy between the quantized and full-precision model outputs.”
[Fan 4.1 Training Networks with Quantization Noise page 5]: “More formally, given a set of tuples of indices
PNG
media_image3.png
33
340
media_image3.png
Greyscale
and a distortion or noise function
φ
acting on a block, we define an operator
PNG
media_image4.png
27
61
media_image4.png
Greyscale
such that, for each block bkl, we apply the following transformation
PNG
media_image5.png
66
307
media_image5.png
Greyscale
The noise function φ simulates the change in the weights produced by the target quantization method (see Section 4.2 for details). We replace the matrix W by the resulting noisy matrix Wnoise during the forward pass to compute a noisy output ynoise, i.e.,
PNG
media_image6.png
40
407
media_image6.png
Greyscale
where x is an input vector. During the backward pass [backward pass], we apply STE, which amounts to replacing the distorted weights Wnoise by their non-distorted counterparts. Note that our approach is equivalent to QAT when J contains all the tuples of indices. However, an advantage of Quant-Noise over QAT is that unbiased gradients continue to flow via blocks unaffected by the noise. As these blocks are randomly selected for each forward, we guarantee that each weight regularly sees gradients that are not affected by the nature of the function
φ
. As a side effect, our quantization noise regularizes the network in a similar way as DropConnect (Wan et al., 2013) or LayerDrop (Fan et al., 2019).”)
The equation for noise that is quantization [Fan 4.2 Adding Noise To Specific Quantization Methods page 6]:
PNG
media_image7.png
166
766
media_image7.png
Greyscale
wherein the forward pass computes a loss
[Fan 3.2 Product Quantization page 4]: “L is the loss function [wherein the forward pass computes a loss where the mapping here is to show loss functions are taught, as the forward pass is taught earlier] and n > 0 is a learning rate. This adapts the upper layers to the drift appearing in their inputs, reducing the impact of the quantization approximation on the overall performance.”
The motivations are the same as claim 1 to combine with Fan and Sriram.
Kaitha teaches:
train the deep learning model to learn to generate source code
([Kaitha Column 2, line 31]: “Using a machine learning model, source code is automatically generated based on the one or more application specific inputs and the one or more external inputs. The source code can embody the one or more application specific inputs and the one or more external inputs. One or more revisions to the automatically generated source code are received, and a difference is determined between the generated source code and the revised source code. The machine learning model may further be trained to generate a future source code [train the deep learning model to learn to generate source code] based on the differences it determines.”)
One of ordinary skill in the art, prior to the effective filing date, would have been motivated to combine Gholami and Kaitha to generate source code. Gholami and Kaitha are in the same field of endeavor of machine learning. One of ordinary skill would have been motivated to combine Gholami and Kaitha to generate source code, as source code generating can help automate the process of code generation as well as improve efficiency ([Kaitha Column 1, Line 34]: “Embodiments disclosed herein provide artificial intelligence systems that perform methods for the generation of source code. The systems and methods improve conventional systems by enabling the automation of source code generation, and as a result increase efficiency when developing software, reduce duplication of efforts, and increase uniformity of source code.”).
Regarding Claim 17:
The device of claim 16 is taught by Gholami, Fan, Sriram, Baker and Kaitha.
This claim is analogous to claim 3.
Regarding Claim 18:
The device of claim 17 is taught by Gholami, Fan, Sriram, Baker, and Kaitha.
This claim is analogous to claim 4.
Regarding Claim 19:
The device of claim 16 is taught by Gholami, Fan, Sriram, Baker, and Kaitha.
This claim is analogous to claim 7.
Regarding Claim 20:
The device of claim 16 is taught by Gholami, Fan, Sriram, Baker, and Kaitha.
This claim is analogous to claim 9.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Bhandare et al (“Efficient 8-bit Quantization of Transformer Neural Machine Language Translation Model”) is relevant art as the reference covers similar aspects as the current application. Bhandare et al covers quantizing neural networks values such as float 32 utilizing values such as INT8, the use of attention models, only quantizing parts of a model, and creating a codebook.
Gao et al (“Beyond Product Quantization: Deep Progressive Quantization for Image Retrieval”) is relevant art as Gao et al covers aspects covered by the current application such as quantization of neural networks, codebooks, k means clustering, and centroids.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHRISTOPHER D DEVORE whose telephone number is (703)756-1234. The examiner can normally be reached Monday-Friday 7:30 am - 5 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael J Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.D.D./Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129
Prosecution Insights