DETAILED ACTION
This communication is in response to the Application filed on 12/06/2024. Claims 1-20 are pending and have been examined. Claims 1, 10 and 19 are independent. This Application was published as U.S. Pub. 2025/0285613.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority based on application CN202410257364.1 filed in China National Intellectual Property Administration (CNIPA) on 03/06/2024, and receipt of a certified copy thereof.
.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract idea without significantly more.
Regarding Claims 1, 10 and 19,
Claim 1 recites a quantization method, which falls under the statutory category of process. Claims 10 and 19 recite a device and a computer program product, which fall under the statutory categories of machine and manufacture, respectively (Step 1: Yes).
Claims recite limitations, “(a) determining a weight matrix…”, “(b) adjusting an order of the plurality of blocks in the weight matrix…”, “(c) quantizing the plurality of blocks…”, and “(d) quantizing the plurality of blocks…”.
Except for the recitation of the speech recognition model, under its broadest reasonable interpretation when read in light of the specification, limitations (a)-(d) encompass the mathematical concept or calculation since limitations involve mathematical operations such as matrix calculation, sorting, clustering, and approximation. According, the claim recites an abstract idea (Step 2A, Prong one).
The judicial exception is not integrated into a practical application. In particular, limitation(s) (X) recites an additional element of “the speech recognition model,” but it is recited at a high level of generality (i.e., speech recognition models or modules, combination of hardware and software are a generic computing device and generic computer components performing a generic computer functions such as processing and storing data from given input) such that it amounts to no more than mere instructions to apply to the exception using a generic computer component.
Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea and the claim is therefore directed to the judicial exception. (Step 2A: YES).
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because they do not include subject matter that could not be performed by a human, as discussed above with respect to integration of the abstract idea into a practical application, the additional element of using the generic computing elements to perform the claimed elements amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept.
As noted previously, the claim as a whole merely describes how to generally linking the use of the aforementioned concept to a particular technological environment or field of use. Thus, even when viewed as a whole, nothing in the claim adds significantly more (i.e., an inventive concept) to the abstract idea. The claim is not patent eligible. (Step 2B: NO).
Regarding Dependent Claims 2-9, 11-18 and 20,
Claims 2-9, 11-18 and 20 are dependent on supra claims and includes all the limitations of the claim(s) and further limits the elements of Claims 1, 10 and 19.
Therefore, the dependent claims recite the same abstract idea. The claim recites the additional limitations, which are no more than mere instructions to apply the exception using a generic computer component, generally linking the use of the judicial exception to a particular technological environment or field of use, insignificant extra-solution activity, or that are well understood, routine and conventional activities previously known to the industry.
No additional elements beyond the use of generic computing elements are claimed, therefore the judicial exception is not integrated into a practical application nor are the claim elements sufficient to amount to significantly more than the judicial exception. Therefore, claims are not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Frantar et al. ("GPTQ: Accurate post-training quantization for generative pre-trained transformers." arXiv preprint arXiv:2210.17323 (2022)) in view of Dong et al. ("HAWQ: Hessian aware quantization of neural networks with mixed-precision." Proceedings of the IEEE/CVF international conference on computer vision. 2019) further in view of Yuan et al. ("RPTQ: Reorder-based post-training quantization for large language models." arXiv preprint arXiv:2304.01089 (2023)).
Regarding Claim 1,
Frantar discloses a quantization method for a speech recognition model (Title, Abstract, "…GPTQ, a new one-shot weight quantization method...GPTQ can quantize GPT models"), the method comprising:
determining a weight matrix for a network layer of the speech recognition model (Title, 3 Background, Layer-Wise Quantization, "…by performing quantization layer-by-layer, the objective is to find a matrix of quantized weights
W
^
which minimizes the squared error..."), wherein the weight matrix comprises a plurality of blocks divided into a plurality of groups (4 THE GPTQ ALGORITHM, Fig.2: GPTQ quantization procedure, "…Blocks of consecutive columns (bolded) are quantized at a given step...", "…we apply the algorithm to B = 128 columns at a time, keeping updates contained to those columns and the corresponding B x B block of H-1 (see also Figure 2)");
Frantar does not explicitly discloses "adjusting an order of the plurality of blocks in the weight matrix…" and "…quantizing the plurality of blocks in the adjusted weight matrix", and "restoring the order of the plurality of blocks…".
Dong, in the analogous field of the computational resources required for Neural Network (NN) training and inference, discloses adjusting an order of the plurality of blocks in the weight matrix according to the plurality of groups and based on a plurality of block parameters of the plurality of blocks (Abstract, 3.1. Second-Order Information, "…We compute the eigenvalues of the Hessian (i.e., the second-order operator) of each block in the network..."; 3.2. Algorithm, "…after Si (i.e., Si = λi/ni, where λi is the top eigenvalue of Hi) computed, we sort Si in descending order and use it as a metric to determine the quantization precision..."), wherein the block parameter indicates how much a corresponding block affects the speech recognition model ( Fig.1, 3. Methodology, "…We can get a better metrics for sensitivity (to quantization) by using second-order information, based on the Hessian matrix…"); and
quantizing the plurality of blocks in the adjusted weight matrix according to the adjusted order (3.2. Algorithm, Algorithm 2: Hessian Aware Quantization);
Therefore, it would have been obvious to one of ordinary skill in the art, before effective filing date of the claimed invention, to have modified a post quantization for generative pre-trained transformers (GPT) of Frantar with a Hessian Aware Quantization (HAWQ), a novel second-order quantization allowing for the mixed -precision quantization at different neural network layers of Dong with a reasonable expectation of success to reduce the number of bits of all weights/activations of a general convolutional network to ultra-low-precision without significant accuracy loss (Dong, Abstract, 1. Introduction).
Neither Frantar nor Dong explicitly teaches the limitation, "restoring the order of the plurality of blocks in the quantized weight matrix," but Yuan, in the analogous field of endeavor, discloses
restoring the order of the plurality of blocks in the quantized weight matrix (4.2 Avoid Explicit Reordering and Misalignment, Fig.3, "…To minimize the overhead of the reorder operation, Firstly, we fuse the reorder operation into the layer norm operation...Secondly, we adjust the weight parameters of the network to allow linear layers to directly accept reordered activations and output reordered activations...The new weight matrix W is obtained by rearranging the rows and columns of the original weight matrix W...The channel ordering of output tensor
Y
~
adheres to the same order as the channel ordering of dimension C2 of the weight. Note that the weight reordering can be completed before deployment, resulting in zero overhead related to reordering during inference…"; i.e., the combination of weight reordering and inverse reordering is construed to restore the original weight layout for inference).
Therefore, it would have been obvious to one of ordinary skill in the art, before effective filing date of the claimed invention, to have modified a Hessian-based post quantization for generative pre-trained transformers (GPT) as being taught by Frantar in view of Dong with a clustering and reordering of channel and weight of Yuan with a reasonable expectation of success to overcome the challenge of quantizing the activations attributed to the significant variations across different channel and minimize the overhead of the reorder operation (Yuan, Abstract, 1. Introduction).
Regarding Claim 2,
Frantar in view of Dong further in view of Yuan discloses the method according to claim 1, wherein adjusting an order of the plurality of blocks in the weight matrix according to the plurality of groups and based on a plurality of block parameters of the plurality of blocks comprises:
determining a first sorting result of the plurality of block parameters of the plurality of blocks in each group (Dong, 3.1. Second Order Information, "…We compute the eigenvalues of the Hessian (i.e., the second-order operator) of each block in the network..."); and
adjusting the order of the plurality of blocks in the weight matrix according to the plurality of groups and based on a plurality of first sorting results for the plurality of groups (Dong, Abstract, 3.1. Second-Order Information, "…We compute the eigenvalues of the Hessian (i.e., the second-order operator) of each block in the network..."; 3.2. Algorithm, "…after Si (i.e.,Si = λi/ni, where λi is the top eigenvalue of Hi) computed, we sort Si in descending order and use it as a metric to determine the quantization precision...").
Regarding Claim 3,
Frantar in view of Dong further in view of Yuan discloses the method according to claim 2, wherein adjusting the order of the plurality of blocks in the weight matrix according to the plurality of groups and based on a plurality of first sorting results for the plurality of groups comprises:
determining a group parameter of each group based on the plurality of block parameters of the plurality of blocks in each group (Yuan, Fig.2(d), 4.1 Clustering and Reordering of Channels, "…main idea of this approach is to cluster the channels in the activations and reorganize them for quantization as shown in Figure 2(d)...");
determining a second sorting result of a plurality of group parameters of the plurality of groups (Yuan, "…we employ the K-Means algorithm to categorize the distinct channels into g clusters, based on the points formed by each channel’s maximum and minimum values..."); and
adjusting the order of the plurality of blocks in the weight matrix according to the plurality of groups and based on the plurality of first sorting results and the second sorting result for the plurality of groups (Dong, "…we proceed with channel reordering by positioning channels from the same cluster together...").
Regarding Claim 4,
Frantar in view of Dong further in view of Yuan discloses the method according to claim 1, wherein the block comprises a column block (Frantar, Fig.2, Weight Matrix/Blocks), and the plurality of block parameters are determined by:
obtaining a plurality of input matrices of the network layer (Frantar,3 BACKGROUND, Layer-Wise Quantization, "…let Wl be the weights corresponding to a linear layer l and let Xl denote the layer input corresponding to a small set of m data points running through the network..." );
determining a second derivative matrix for the weight matrix based on products of the plurality of input matrices and a plurality of corresponding transpose matrices (Frantar,3 BACKGROUND, "…Hessian is
H
F
=
2
X
F
X
F
T
, where F denotes the set of remaining full-precision weights..."); and
determining the plurality of block parameters of a plurality of column blocks based on diagonal elements of the second derivative matrix (Dong, 3.2. Algorithm, "…We approximate the Hessian as a block diagonal matrix, scaled by its top eigenvalue λ as
{
H
i
≈
λ
i
I
}
i
=
1
m
, where m is the number of blocks in the network…").
Regarding Claim 5,
Frantar in view of Dong further in view of Yuan discloses the method according to claim 4, wherein quantizing the plurality of blocks in the adjusted weight matrix according to the adjusted order comprises:
quantizing a first column block in the plurality of column blocks in the adjusted weight matrix (Frantar, 4 THE GPTQ ALGORITHM, Fig.2, "…Blocks of consecutive columns (bolded) are quantized at a given step...");
adjusting unquantized column blocks in the plurality of column blocks based on the second derivative matrix (Frantar, 4 THE GPTQ ALGORITHM, Fig.2, "…using the inverse Hessian information stored in the Cholesky decomposition, and the remaining weights (blue) are updated at the end of the step."); and
updating the first column block to a first column block of the unquantized column blocks (Frantar, 4 THE GPTQ ALGORITHM, Fig.2, "…The quantization procedure is applied recursively inside each block: the white middle column is currently being quantized...").
Regarding Claim 6,
Frantar in view of Dong further in view of Yuan discloses the method according to claim 1, wherein quantizing the plurality of blocks in the adjusted weight matrix according to the adjusted order comprises:
determining a first quantization parameter for each group of the plurality of groups, the first quantization parameter comprising at least a first scale and a first offset (Yuan, 3.1 Post-training Quantization, "…
we use uniform quantization function Qk to transform a float value x to k bits integer xq, where s represents the scaling factor, z denotes the zero point...");
determining a group corresponding to the block (Yuan, 4.1 Clustering and Reordering of Channels, "…we employ the K-Means algorithm to categorize the distinct channels into g clusters, based on the points formed by each channel’s maximum and minimum values..."); and
quantizing the block based on the first quantization parameter for the group corresponding to the block (Yuan, "…Following the reordering process, we quantize the activations within each cluster. Specifically, we calculate the quantization parameters (scale s and zero point z) individually for each cluster...").
Regarding Claim 7,
Frantar in view of Dong further in view of Yuan discloses the method according to claim 6, wherein the first scale is determined by:
determining a difference between a maximum weight and a minimum weight in each group of the plurality of groups (Yuan, 3.1 Post-training Quantization, "…One of the commonly used methods is Min-Max method. This method involves computing the maximum value Xmax = max(X) and minimum value Xmin = min(X) of tensor X that share the quantization parameters...");
determining a quantization range for each group in a predetermined quantization manner ( "…the clamp function constrains the value within the range of a k-bit integer, specifically [-2k1, 2k1-1]. For 4 bit integer, the range is [-8,7]..."); and
determining the first scale for each group based on the difference between the maximum weight and the minimum weight and the quantization range ("…scaling factor
s
=
X
m
a
x
-
X
m
i
n
2
k
").
Regarding Claim 8,
Frantar in view of Dong further in view of Yuan discloses the method according to claim 7, wherein the offset is determined by:
determining a plurality of ratios of a plurality of weights in each group to the first scale (Yuan, 3.1 Post-training Quantization, "…we use uniform quantization function Qk to transform a float value x to k bits integer xq ...");
determining a minimum rounding result from a plurality of rounding results corresponding to the plurality of ratios in each group ("…
x
q
=
Q
k
x
,
s
,
z
=
c
l
a
m
p
(
r
o
u
n
d
x
s
+
z
,
-
2
k
-
1
,
2
k
-
1
-
1
)
where s represents the scaling factor, z denotes the zero point, and the clamp function constrains the value within the range of a k-bit integer..."); and
determining the first offset for each group based on a ratio of the minimum rounding result to the first scale ("…zero point z
=
-
r
o
u
n
d
(
X
m
a
x
+
X
m
i
n
2
)
").
Regarding Claim 9,
Frantar in view of Dong further in view of Yuan discloses the method according to claim 6, wherein the method further comprises:
generating a fake quantized block for a quantized block in each group based on the first quantization parameter for each group (Yuan, 3.1 Post-training Quantization, "…The integer xq can be de-quantized to
x
^
=
s
(
x
q
-
z
)
≈
x
. The de-quantized value
x
^
is a float..."; A.4 Computation under cluster-based quantization, "…We describe two methods for computing the output tensor Y using the quantized activations and weights. The first method dequantizes the weight and activation values back to floating-point numbers...
The dequantized values are then concatenated to form the full activations
X
^
and the full weights
W
^
. Matrix multiplication is performed using the dequantized values: Y =
X
^
W
^
...");and
after restoring the order of the plurality of blocks in the quantized weight matrix, the method further comprises: determining a second quantization parameter for the restored weight matrix, the second quantization parameter comprising at least a second scale and a second offset (par [0], e.g., transforming the weight matrix into type int4 using uniform quantification function
x
q
=
Q
k
x
,
s
,
z
=
c
l
a
m
p
(
r
o
u
n
d
x
s
+
z
,
-
8,7
)
"); and
quantizing the restored weight matrix based on the second quantization parameter (Yuan, "…Following the reordering process, we quantize the activations within each cluster. Specifically, we calculate the quantization parameters (scale s and zero point z) individually for each cluster...").
Claim 10 is a device claim with limitations similar to the limitations of Claim 1 and is rejected under similar rationale. Additionally,
Frantar discloses an electronic device, comprising: a processor; and a memory coupled to the processor, wherein the memory has instructions stored therein, which, when executed by the processor, cause the electronic device to (Frantar, 5 Experimental Validation, "...We quantized all models (including the 175 billion parameter variants) using a single NVIDIA A100 GPU with 80GB of memory...")
…
Rationale for combination is similar to that provided for Claim 1.
Claim 11 is a device claim with limitations similar to the limitations of Claim 2 and is rejected under similar rationale.
Claim 12 is a device claim with limitations similar to the limitations of Claim 3 and is rejected under similar rationale.
Claim 13 is a device claim with limitations similar to the limitations of Claim 4 and is rejected under similar rationale.
Claim 14 is a device claim with limitations similar to the limitations of Claim 5 and is rejected under similar rationale.
Claim 15 is a device claim with limitations similar to the limitations of Claim 6 and is rejected under similar rationale.
Claim 16 is a device claim with limitations similar to the limitations of Claim 7 and is rejected under similar rationale.
Claim 17 is a device claim with limitations similar to the limitations of Claim 8 and is rejected under similar rationale.
Claim 18 is a device claim with limitations similar to the limitations of Claim 9 and is rejected under similar rationale.
Claim 19 is a computer program product claim with limitations similar to the limitations of Claim 1 and is rejected under similar rationale.
Rationale for combination is similar to that provided for Claim 1.
As for additional elements a computer program product, a person of ordinary skill in the field of neural network quantization and deep learning system would understand as common general knowledge that any published pre-trained quantization algorithm is necessarily implemented as computer-executable instructions stored on a non-transitory computer-readable storage medium and executed on a computer system comprising at minimum a processor, memory, and a GPU.
Claim 20 is a computer program product claim with limitations similar to the limitations of Claim 2 and is rejected under similar rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Brothers et al. (US Pub 2018/0082181) discloses a neural network is trained to generate feature maps and associated weights. Reordering is performed to generate a functionally equivalent network. The reordering may be performed to improve at least one of compression of the weights, load balancing, and execution (Brothers, Abstract).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JANGWOEN LEE whose telephone number is (703)756-5597. The examiner can normally be reached Monday-Friday 8:00 am - 5:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, BHAVESH MEHTA can be reached at (571)272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JANGWOEN LEE/ Examiner, Art Unit 2656
/BHAVESH M MEHTA/ Supervisory Patent Examiner, Art Unit 2656
Prosecution Insights