DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements submitted on March 15, 2023 and March 19, 2024 have been considered by the Examiner.
Claim Objections
Claim 5 is objected to because of the following informalities: the phrase “one or more power or two values,” which occurs twice in claim 5, understandably comprises a typographical error. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 4, 10, 11, 13, 15, 16, 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over the article entitled “RMSMP: A Novel Deep Neural Network Quantization Framework with Row-wise Mixed Schemes and Multiple Precisions” by Chang et al. (“Chang”), and also over the article entitled, “Mix and Match: A Novel FPGA-Centric Deep Neural Network Quantization Framework” by Chang et al. (“Chang 2”).
Regarding claims 1, 11 and 16, Chang introduces “a novel Deep Neural Network (DNN) quantization framework, namely RMSMP, with a Row-wise Mixed-Scheme and Multi-Precision approach,” which “assign[s] mixed quantization schemes and multiple precisions within layers – among rows of the DNN weight matrix.” (Abstract). Chang particularly teaches:
selecting a first group of one or more weights from a weight tensor of a layer of the DNN, the weight tensor comprising the first group of one or more weights and a second group of one or more weights (Chang discloses that each filter in a weight tensor or each row in a weight matrix is assigned a quantization scheme and precision:
This work proposes a novel DNN quantization frame work, namely RMSMP, with a Row-wise Mixed-Scheme and Multi-Precision approach. Figure 1 illustrates the proposed quantization framework on the DNN (a) weight tensor and (b) weight matrix. Specifically, each filter in the weight tensor or each row in the weight matrix is assigned a combination of quantization scheme and precision. The candidates of schemes and precisions are derived practically for facilitating hardware implementation, and effectively for speeding up inference with the given resources on the hardware platforms/devices, while with the capability to preserve the accuracy as the unquantized (32-bit floating point) models. This highly hardware-informative quantization strategy significantly reduces the search space of the DNN quantization problem, making our framework distinctive from existing multi-precision quantization works.
(Section 1. Introduction. Emphasis Added).
The quantization schemes comprise Power-of-Two (PoT) and Fixed-point (Fixed), and the precisions comprise 4-bit and 8-bit:
This is the first effort to apply mixed quantization schemes and multiple precisions within layers, targeting for simplified operations in hardware inference, while preserving the accuracy. Specifically, two quantization schemes i.e., Power-of-Two (PoT) and Fixed-point (Fixed), and two precisions i.e., 4-bit and 8-bit are adopted and explored for quantization on weights and activations, to reduce inference computation and preserve accuracy.
(Section 1. Introduction).
Chang further discloses that the rows/filters within the weight tensor are assigned a quantization scheme and precision based on the results of a Hessian-based method and the variances of the weights in each row/filter:
For the assignment of quantization schemes and precisions to the filters of each layer, we use the Hessian-based method to determine which filters should use Fixed-W8A4 (higher precision). And for the rest filters, we determine PoT-W4A4 vs Fixed-W4A4 based on the variances of the weights in each filter.
(Section 3.3. RMSMP Quantization Algorithm).
Chang thus teaches selecting a first number of rows/filters from a weight tensor of a DNN to quantize with a PoT quantization scheme, and a second number of rows/filters from the weight tensor to quantize with a Fixed quantization scheme. The first number of rows/filters is considered a “first group of one or more weights” like claimed, and the second number of rows/filters is considered a “second group of one or more weights” like claimed.);
quantizing a weight in the first group to a power of two value (as noted above, Chang teaches that a first number of rows/filters, i.e. a first group, within a weight tensor is assigned a PoT quantization scheme. The weights in the first group are thus understandably quantized to a power of two value.);
quantizing a weight in the second group to fixed-point number (as noted above, Chang teaches that a second number of rows/filters, i.e. a second group, within a weight tensor is assigned a Fixed quantization scheme. The weights in the second group are thus understandably quantized to a fixed-point number.);
shifting an activation of the layer by an exponent of the power of two value (Chang teaches that multiplying activations with the weights of a layer during inference, when the weights are quantized to a power of two value, comprises shifting the activations according to the power of two value:
Different from binary, ternary, and Fixed, PoT is a non-linear quantization scheme, where with quantization levels as power-of-two numbers, multiplications can be replaced with bit shifting operations, thereby reducing computation to speedup inference.
(Section 1. Introduction. Emphasis Added.).
Non-linear quantization schemes use non-uniform quantization levels, such as the Power-of-Two (PoT) scheme, where quantization levels are power-of-two numbers. The multiplication of a weight (in PoT) and an input (in Fixed) can be replaced with the bit shifting operation. And therefore PoT can have higher speedup than the Fixed scheme for the DNN inference.
(Section 2.1.2 Non-Linear Quantization Schemes. Emphasis added.).
The Fixed-point (Fixed) quantization scheme has superior accuracy performance, and the Power-of-Two (PoT) is the most computationally efficient quantization scheme (with still acceptable accuracy performance) to speedup inference since multiplications can be replaced by bit shifting operations. Therefore, this work proposes a novel row wise mixed-scheme quantization approach with Fixed for preserving accuracy and PoT for reducing computation of inference. Specifically, for each row in the weight matrix or each filter in the weight tensor, the weights are either quantized into the Fixed scheme or the PoT scheme. The row wise scheme assignment instead of a layer-wise scheme assignment is used, because the hardware inference execution is conducted layer by layer on the same pieces of computing resource – GEMMFixed i.e., the GEMM (general matrix multiply) core for processing Fixed weights, and GEMMPoT i.e., the GEMM core for processing PoT weights. (Note that in PoT, activations are also quantized into Fixed to support the bit shifting operation in replacement of multiplication.)
(Section 3.1. Mixed Schemes for Simplified Operations).
Chang thus teaches shifting an activation of the layer by an exponent of the power of two value.); and
multiplying the fixed-point number with another activation of the layer (Chang suggests that, when the weights are quantized to fixed-point numbers, inference comprises multiplying the activations of the layer with the fixed-point numbers:
Fixed-point (Fixed) quantization schemes use more bits than binary and ternary to preserve accuracy, and have been implemented with different methods/algorithms. DoReFa Net [38] first explored it by introducing hyperbolic tangent transformation to weights and activations, with scaling factors to minimize quantization error. PACT [4] improved this method by adding a parameterized clipping threshold to activations. DSQ [12] developed differentiable soft quantization, which evolves training method to gradually approximate the uniform quantizer. QIL [16] parameterized the quantization interval and trained it with task loss, avoiding access to the original training data. µL2Q [3] introduced data distribution loss during training to minimize quantization error. LSQ [11] proposed a differentiable method to learn the quantizer for each layer jointly with parameters. Generally, 4-bit fixed-point quantized models have negligible accuracy loss comparing with 32-bit floating-point models, although cannot get rid of the expensive multiplications.
(Section 2.2.2 Linear Quantization Schemes. Emphasis Added).
The Fixed-point (Fixed) quantization scheme has superior accuracy performance, and the Power-of-Two (PoT) is the most computationally efficient quantization scheme (with still acceptable accuracy performance) to speedup inference since multiplications can be replaced by bit shifting operations. Therefore, this work proposes a novel row wise mixed-scheme quantization approach with Fixed for preserving accuracy and PoT for reducing computation of inference. Specifically, for each row in the weight matrix or each filter in the weight tensor, the weights are either quantized into the Fixed scheme or the PoT scheme. The row wise scheme assignment instead of a layer-wise scheme assignment is used, because the hardware inference execution is conducted layer by layer on the same pieces of computing resource – GEMMFixed i.e., the GEMM (general matrix multiply) core for processing Fixed weights, and GEMMPoT i.e., the GEMM core for processing PoT weights. (Note that in PoT, activations are also quantized into Fixed to support the bit shifting operation in replacement of multiplication.)
(Section 3.1. Mixed Schemes for Simplified Operations).
Chang thus teaches multiplying the fixed-point number with another activation of the layer.).
Accordingly, Chang teaches a method similar to that of claim 1. Chang suggests that such teachings can be implemented via computer program instructions that are executable by a processor of an apparatus, which also comprises a non-transitory computer-readable memory for storing the instructions (see e.g. section 4.1. Experiment Setup). Such an apparatus for implementing the above-described teachings of Chang is considered an apparatus similar to that of claim 11, and the non-transitory computer-readable memory storing the program instructions for implementing the above-described teachings is considered one or more non-transitory computer-readable media similar to that of claim 16. Chang, however, does not explicitly disclose that the above-noted fixed-point numbers are integers like required by claims 1, 11 and 16 (e.g. claim 1 recites, “quantizing a weight in the second group to an integer” and “multiplying the integer with another activation of the layer.”).
Similar to Chang, Chang 2 teaches applying different quantization schemes to different portions (i.e. rows) of a weight tensor (see e.g. the Abstract, which recites, “[u]nlike existing methods that use the same quantization scheme for all weights, we propose the first solution that applies different quantization schemes for different rows of the weight matrix.”). Chang 2 discloses that one of these quantization schemes is fixed-point quantization (see e.g. section I. Introduction, which recites, “[t]o fully explore the FPGA resources, we propose an FPGA centric mixed scheme quantization (MSQ) with an ensemble of the proposed SP2 [sum-of-power-of-2] and the fixed-point schemes.”). Particularly regarding claimed invention, Chang 2 teaches that the fixed-point quantization scheme entails representing each weight with an integer value, which is multiplied with an activation during forward inference:
Next, we analyze the effect of SP2 quantization scheme on the computation of weight-activation multiplication. In Table I, we compare fixed-point and SP2 quantization schemes of the weights, while throughout this paper we use fixed-point quantization for the activation. In the first scheme with m-bit fixed-point quantization for the weight and n-bit fixed-point quantization on the activation, the weight operand is actually represented as the (m−1)-bit unsigned integer, since 1 bit is for the sign. Although a quantization level is within [−1,+1], the actual weight operand is the (m−1)-bit unsigned integer. And the activation operand is directly represented as the n-bit unsigned integer, because activations are non-negative. The operations for implementing weight-activation multiplication are therefore n-bit additions for (m − 2) times.
(Section III.A SP2 Quantization Scheme. Emphasis added.).
Chang 2 thus teaches quantizing a weight to an integer, and multiplying the integer with an activation of the layer.
It would have been obvious to one of ordinary skill in the art, having the teachings of Chang and Chang 2 before the effective filing date of the claimed invention, to modify the method, apparatus and non-transitory computer-readable media taught by Chang such that the fixed-point representation of each weight is represented as integer value like taught by Chang 2. Each weight in the second group would thus be quantized to a respective integer, which during forward inference, is multiplied by an activation of the layer. It would have been advantageous to one of ordinary skill to utilize such an integer because it would facilitate the multiplication of the weights with the activations, as is evident from Chang 2 (see e.g. Section III.A SP2 Quantization Scheme.). Accordingly, Chang and Chang 2 are considered to teach, to one of ordinary skill in the art, a method like that of claim 1, an apparatus like that of claim 11, and one or more non-transitory computer-readable media like in claim 16.
As per claims 4, 13 and 18, Chang further teaches selecting the first group of one or more weights (i.e. the weights to be quantized to a power-of-two value) from the weight tensor based on a predetermined partition parameter, the partition parameter indicating a ratio of a number of weight or weights in the first group to a total number of weights in the weight tensor:
Figure 2 demonstrates the proposed multi-precision approach aligned with the row-wise mixed schemes. The majority of the rows use the 4-bit precision for weights/activations i.e., PoT-W4A4 and Fixed-W4A4, because 2-bit has large accuracy loss and 3-bit is not suitable for hardware implementation, which prefers operands in 2 bit, 4-bit, 8-bit, etc. To boost accuracy, a higher precision with 8-bit weights and 4-bit activations is used on the Fixed scheme, i.e., Fixed-W8A4. The PoT scheme is not applied the higher precision because of its rigid resolution issue. The ratio of PoT-W4A4 : Fixed-W4A4 : Fixed-W8A4 can be determined offline, and is the same across different layers, in order to keep the layer-wise uniformality in the hardware implementation, such that little overhead is incurred during the layer-by-layer inference execution and inference speedup can be guaranteed.
(Section 3.2. Multiple Precisions for Boosting Accuracy. Emphasis added.).
The RMSMP quantization algorithm can train a DNN model from scratch or quantize a pre-trained model into a quantized one, such that for each layer, the numbers of filters quantized into PoT-W4A4, Fixed-W4A4, and Fixed-W8A4 follow the predefined ratio of
S
P
o
T
-
4
:
S
F
i
x
e
d
-
4
:
S
F
i
x
e
d
-
8
=
A
:
B
:
C
, where
A
+
B
+
C
=
100
.
(Section 3.3. RMSMO Quantization Algorithm).
Accordingly, the above-described combination of Chang and Chang 2 is further considered to teach a method like that of claim 4, an apparatus like that of claim 13, and one or more non-transitory computer-readable media like in claim 18.
As per claims 10, 15 and 20, Chang suggests that the exponent of the power of two value is stored in memory in lieu of the weight in the first group, and that the fixed-point value is stored in memory in lieu of the weight in the second group (see e.g. section 1. Introduction: Chang discloses quantization is a form of model compression that reduces memory and storage requirements. It is therefore apparent that the smaller quantized values, i.e. the power-of-two exponents and the fixed-point values, are stored in memory in lieu of the non-quantized weights.). As noted above, it would have been obvious to modify the method, apparatus and non-transitory computer-readable media taught by Chang such that the fixed-point representation of each weight is represented as integer value like taught by Chang 2. Accordingly, the above-described combination of Chang and Chang 2 is further considered to teach a method like that of claim 10, an apparatus like that of claim 15, and one or more non-transitory computer-readable media like in claim 20.
Claims 2, 3, 6, 12 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over the above-described combination of Chang and Chang 2, and also over U.S. Patent Application Publication No. 2023/0075643 to Wang et al. (“Wang”).
Regarding claims 2, 12 and 17, Chang and Chang 2 teach a method like that of claim 1, an apparatus like that of claim 11, and one or more non-transitory computer-readable media like that of claim 16, which like described above, entail selecting a first group of one or more weights from a weight tensor, and quantizing a weight in the first group to a power-of-two value. Chang further demonstrates that the weight tensor can comprise a plurality of weights arranged in one or more rows and one or columns (see e.g. the weight matrix in Figure 1). Chang and Chang 2, however, do not explicitly disclose that a number of weights in a row of the weight tensor is equal to a number of channels in an input feature map of the layer, and that a number of weights in a column of the weight tensor is equal to a number of channels in an output feature map of the layer, as is required by claims 2, 12 and 17.
Wang generally describes a method for compressing a deep neural network (DNN) by applying independent row and column pruning to each block of a weight matrix of a DNN layer (see e.g. paragraph 0015). Like claimed, Wang particularly demonstrates that the weight matrix (i.e. a weight tensor) can comprise a plurality of weights arranged in one or more rows and one or more columns, wherein a number of weights in a row of the weight tensor is equal to a number of channels in an input feature map of the layer, and a number of weights in a column of the weight tensor is equal to a number of channels in an output feature map of the layer (see e.g. paragraphs 0038-0039 and FIGS. 1 and 2).
It would have been obvious to one of ordinary skill in the art, having the teachings of Chang, Chang 2 and Wang before the effective filing date of the claimed invention, to modify the method, apparatus and non-transitory computer-readable media taught by Chang and Chang 2 such that the weight tensor comprises a plurality of weights arranged in one or more rows and one or more columns, wherein a number of weights in a row of the weight tensor is equal to a number of channels in an input feature map of the layer, and a number of weights in a column of the weight tensor is equal to a number of channels in an output feature map of the layer, as is taught by Wang. It would have been advantageous to one of ordinary skill to utilize such a combination, because the resulting weight tensor format can facilitate computation with the weights therein, as is suggested by Wang (see e.g. paragraphs 0038-0039). Accordingly, Chang, Chang 2 and Wang are considered to teach, to one of ordinary skill in the art, a method like that of claim 2, an apparatus like that of claim 12, and one or more non-transitory computer-readable media like in claim 17.
Regarding claim 3, Chang and Chang 2 do not teach that selecting the first group of one or more weights from the weight tensor comprises selecting a same number of weight or weights from each respective row of the weight tensor, as is required by claim 3.
Like noted above, Wang demonstrates a weight matrix (i.e. a weight tensor) that comprises a plurality of weights arranged in one or more rows and one or more columns, wherein a number of weights in a row of the weight tensor is equal to a number of channels in an input feature map, and a number of weights in a column of the weight tensor is equal to a number of channels in an output feature map (see e.g. paragraphs 0038-0039 and FIGS. 1 and 2). Wang further discloses that weights within the weight matrix can be pruned (see e.g. paragraphs 0038-0039). Similar to quantization, weight pruning is a model compression technique, but where particular weights are removed instead of being reduced in bit-width (see e.g. paragraphs 0008 and 0037). With respect to claim 3, Wang particularly teaches that selecting a group of weights for pruning can entail selecting a same number of weights (i.e. weights corresponding to a particular one or more input channels) from each respective row of the weight tensor (see e.g. paragraph 0039 and FIG. 2).
It would have been obvious to one of ordinary skill in the art, having the teachings of Chang, Chang 2 and Wang before the effective filing date of the claimed invention, to further modify the method taught by Chang, Chang 2 and Wang such that the first group can be selected (i.e. for quantization with a power-of-two value) by selecting a same number of weight or weights from each respective row of the weight tensor, as is done with the weight pruning taught by Wang. It would have been advantageous to one of ordinary skill to utilize such a combination, because such a technique can address the importance (or lack thereof) of particular input channels, as is demonstrated by Wang (see e.g. paragraph 0039). Accordingly, Chang, Chang 2 and Wang are further considered to teach, to one of ordinary skill in the art, a method like that of claim 3.
Regarding claim 6, Chang and Chang 2 teach a method like that of claim 1, which like described above, comprises selecting a first group of one or more weights from a weight tensor, wherein the weight tensor comprises the first group of one or more weights and a second group of one or more weights. Chang and Chang 2, however, do not explicitly teach: (i) dividing a whole weight tensor of the layer into the weight tensor and additional weight tensor; (ii) selecting a third group of one or more weights from the additional weight tensor; and (iii) quantizing each respective weight in the third group to a power of two value, wherein a ratio of a number of weight or weights in the first group to a number of weights in the weight tensor is different from a ratio of a number of weight or weights in the third group to a number of weights in the additional weight tensor, as is required by claim 6.
Like noted above, Wang generally describes a method for compressing a deep neural network (DNN) by applying independent row and column pruning to each block of a weight matrix of a DNN layer (see e.g. paragraph 0015). Similar to quantization, weight pruning is a model compression technique, but where particular weights are removed instead of being reduced in bit-width (see e.g. paragraphs 0008 and 0037). Wang particularly teaches: (i) selecting a first group of one or more weights from a weight tensor of a layer of the DNN (i.e. selecting a first group of one or more weights to prune), wherein the weight tensor comprises the first group of one or more weights and a second group of one or more weights (i.e. a second group of weights that are not to be pruned); (ii) dividing the whole weight tensor of the layer into the weight tensor (i.e. a first “block” of the weight tensor) and an additional weight tensor (i.e. a second block of the weight tensor); (iii) selecting a third group of one or more weights from the additional weight tensor (i.e. a plurality of weights within the second block to prune); and (iv) pruning each respective weight in the third group to a 0 value, (v) wherein a ratio of a number of weight or weights in the first group to a number of weights in the weight tensor is different from a ratio of a number of weight or weights in the third group to a number of weights in the additional weight tensor (i.e. the number of pruned weights in each block can be different) (see e.g. paragraph 0044).
It would have been obvious to one of ordinary skill in the art, having the teachings of Chang, Chang 2 and Wang before the effective filing date of the claimed invention, to modify the method taught by Chang and Chang 2 so as to quantize the weights within different blocks of the weight tensor like done with the pruning taught by Wang. That is, it would have been obvious to: (i) divide the whole weight tensor of the layer into the weight tensor (i.e. a first “block” of the weight tensor) and an additional weight tensor (i.e. a second block of the weight tensor), wherein the first block comprises the first group (i.e. weights to be quantized to a power-of-two value); (ii) select a third group of one or more weights from the additional weight tensor (i.e. a plurality of weights within the second block to quantize to the power-of-two value); and (iii) compress each weight in the third group (i.e. quantize each respective weight in the third group to a power-of-two value), wherein a ratio of a number of weight or weights in the first group to a number of weights in the weight tensor (i.e. the first block) is different from a ratio of a number of weight or weights in the third group to a number of weights in the additional weight tensor (i.e. the second block), as is done with the pruning taught by Wang. It would have been advantageous to one of ordinary skill to utilize such a combination, because it would provide more flexibility as is suggested by Wang (see e.g. paragraphs 0043-0044). Accordingly, Chang, Chang 2 and Wang are considered to teach, to one of ordinary skill in the art, a method like that of claim 6.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over the above-described combination of Chang and Chang 2, and also over the article entitled, “Retraining-Based Iterative Weight Quantization for Deep Neural Networks” by Lee et al. (“Lee”).
Regarding claim 5, Chang and Chang 2 teach a method like that of claim 1, which like described above, comprises selecting a first group of one or more weights from a weight tensor and quantizing a weight in the first group to a power of two value, and quantizing a weight within a second group of one or more weights of the weight tensor to an integer value. Chang further suggests selecting the first group of one or more weights from the weight tensor by minimizing a difference between the weight and a tensor comprising one or more integers and one or more power of two values (see e.g. section 3.1. Mixed Schemes for Simplified Operations, and section 3.3. RMSMP Quantization Algorithm: Chang teaches that the weights identified to be quantized to a power of two value are those having a smaller variance. Chang suggests that this minimizes the accuracy loss from quantization, which would understandably minimize the difference between the original non-quantized tensor and the quantized tensor comprising the fixed-point integers and power-of-two values). Chang and Chang 2 are thus further considered to teach a method similar to that of claim 5, but do not explicitly disclose that the one or more integers are generated by quantizing the one or more weights in the first group, and that the one or more power of two values are generated by quantizing the one or more weights in the second group, as is required by claim 5.
Lee nevertheless generally teaches identifying quantized values for neural network weights by iteratively quantizing the weights and then retraining the neural network (see e.g. the Abstract, which recites: “In this work, we introduce an iterative technique to apply quantization….In the proposed technique, weight quantization is followed by retraining the model with full precision weights. We show that iterative retraining generates new sets of weights which can be quantized with decreasing quantization loss at each iteration.”).
It would have been obvious to one of ordinary skill in the art, having the teachings of Chang, Chang 2 and Lee before the effective filing date of the claimed invention, to modify the method taught by Chang and Chang 2 so as to generate the quantized values (i.e. the fixed-point integer values and the power-of-two values) of the weights by quantizing the weights (i.e. the one or more weights in the first group and the one or more weights in the second group) and retaining the neural network, as is taught by Lee. It would have been advantageous to one of ordinary skill to utilize such a combination because it can reduce the accuracy degradation from quantization, as is suggested by Lee (see e.g. the Abstract). Accordingly, Chang, Chang 2 and Lee are considered to teach, to one of ordinary skill in the art, a method like that of claim 5.
Claims 7-9, 14 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over the above-described combination of Chang and Chang 2, and also over U.S. Patent Application Publication No. 2022/0108157 to Hunter et al. (“Hunter”).
Regarding claims 7, 14 and 19, Chang and Chang 2 teach a method like that of claim 1, an apparatus like that of claim 11, and one or more non-transitory computer-readable media like that of claim 16, which like described above, entail quantizing a weight in a first group of weights of a weight tensor to a power of two value, and quantizing a weight in a second group of weights of the weight tensor to an integer. Chang and Chang 2, however, do not teach generating a bitmap for the weight tensor, wherein the bitmap comprises a plurality of bits, wherein each bit corresponds to a weight in the weight tensor and indicates whether the weight is quantized to an integer or a power of two value, as is required by claims 7, 14 and 19.
Hunter nevertheless teaches generating a bitmap (i.e. a bit vector) for a weight tensor, wherein the bitmap comprises a plurality of bits and each bit corresponds to a weight in the weight tensor (see e.g. paragraph 0116). In particular, each bit indicates a corresponding characteristic associated with its corresponding weight within the weight tensor (e.g. if it is active or not) (see e.g. paragraph 0116).
It would have been obvious to one of ordinary skill in the art, having the teachings of Chang, Chang 2 and Hunter before the effective filing date of the claimed invention, to modify the method taught by Chang and Chang 2 so as to generate a bitmap for the weight tensor, wherein the bitmap comprises a plurality of bits, and each bit corresponds to a weight in the weight tensor and indicates a characteristic of the corresponding weight in the weight tensor, as is taught by Hunter. As described above, Chang and Chang 2 teach that weights within the weight tensor are characterized by a quantization scheme, i.e. fixed-point integer or power-of-two, and so it particularly would have been obvious to have each bit within the bitmap indicate whether the corresponding weight is quantized to an integer or a power of two value. It would have been advantageous to one of ordinary skill to utilize such a bitmap, because it can be used to direct the processing with the weight tensor, as is demonstrated by Hunter (see e.g. paragraphs 0117-0120). Accordingly, Chang, Chang 2 and Hunter are considered to teach, to one of ordinary skill in the art, a method like that of claim 7, an apparatus like that of claim 14, and one or more non-transitory computer-readable media like in claim 19.
As per claim 8, it would have been obvious, as is described above, to modify the method taught by Chang and Chang 2 so as to generate a bitmap for the weight tensor, wherein the bitmap comprises a plurality of bits, and each bit corresponds to a weight in the weight tensor and indicates a characteristic of the corresponding weight in the weight tensor, as is taught by Hunter. As further noted above, it particularly would have been obvious to have each bit within the bitmap indicate whether the corresponding weight is quantized to an integer or a power of two value. Hunter generally teaches that a bit within the bitmap having a value of zero indicates a first characteristic of the associated weight (e.g. that the weight is active), and that a value of one indicates a different characteristic (e.g. that the weight is inactive). It thus follows that it would have been apparent to one of ordinary skill to configure the bitmap such that a value of zero indicates a first characteristic of the corresponding weight, i.e. that the weight is quantized to a power of two value. Under a similar rationale, it would have been apparent to one of ordinary skill to configure the bitmap such that a value of one indicates another characteristic of the corresponding weight, i.e. that the weight is quantized to an integer. The above-described combination of Chang, Chang 2 and Hunter is thus further considered to teach a method like that of claim 8.
As per claim 9, it would have been obvious, as is described above, to modify the method taught by Chang and Chang 2 so as to generate a bitmap for the weight tensor, wherein the bitmap comprises a plurality of bits, and each bit corresponds to a weight in the weight tensor and indicates a characteristic of the corresponding weight in the weight tensor, as is taught by Hunter. Hunter generally teaches that such a bitmap can be used to identify the characteristics of the corresponding weights, and identify the corresponding weights for multiply-and-accumulate operations (see e.g. paragraphs 0117-0120). Moreover, Chang suggests that the multiply-and-accumulate operations entail transmitting the first group of one or more weights (i.e. the weights having a power of two value) from a memory to one or more shifters, whereas the second group of weights (i.e. the weights having fixed-point values/integers) are transmitted to one or more multipliers (see e.g. Section 3.1. Mixed Schemes for Simplified Operations). Chang 2 provides a similar teaching (see e.g. section V.B. Architecture with Heterogeneous GEMM Engines). It thus follows that it would have been apparent to one of ordinary skill in the art utilize the bitmap to identify the characteristics of the associated weights (i.e. whether they have power-of-two values or fixed-point/integers) and appropriately direct the weights (i.e. transmit them to one or more shifters or multipliers) based on these characteristics. Consequently, the above-described combination of Chang, Chang 2 and Hunter is further considered to teach: (i) transmitting, based on the bitmap, the first group of one or more weights from a memory to one or more shifters; and (ii) transmitting, based on the bitmap, the second group of one or more weights from the memory to one or more multipliers, as is required by claim 9.
Conclusion
The prior art made of record on form PTO-892 and not relied upon is considered pertinent to applicant’s disclosure. The applicant is required under 37 C.F.R. §1.111(C) to consider these references fully when responding to this action. In particular, the article by Przewlocka-Rus cited therein (“Power-of-Two Quantization for Low Bitwidth and Hardware Compliant Neural Networks”) describes a Quantization Aware Training (QAT) algorithm that allows training of low bit width Power-of-Two (PoT) networks. The article by Sun et al. cited therein (“FILM-QNN: Efficient FPGA Acceleration of Deep Neural Networks with Intra-Layer, Mixed-Precision Quantization”) describes an intra-layer, mixed precision quantization algorithm that assigns different precisions onto the filters of each of a plurality of deep neural network layers.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BLAINE T BASOM whose telephone number is (571)272-4044. The examiner can normally be reached Monday-Friday, 9:00 am - 5:30 pm, EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Ell can be reached at (571)270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BTB/
5/30/2026
/MATTHEW ELL/Supervisory Patent Examiner, Art Unit 2141
Prosecution Insights