APPARATUS AND METHOD FOR POST TRAINING QUANTIZATION OF HYBRID VISION TRANSFORMER

§101 §103

DETAILED ACTION This action is responsive to the Application filed on 05/30/2024. Claims 1-19 are pending in the case. Claims 1, 6 and 11 are independent claims. Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claim Rejections - 35 USC § 101 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claims 1-19 are rejected under 35 U.S.C. 101 because the claim(s) is directed to an abstract idea Regarding Claim 1/6/11: Under step 1, Claim 1 is directed to an apparatus which is directed to a machine, one of the statutory categories. Under step 1, Claim 6 is directed to a method which is directed to a process, one of the statutory categories. Under step 1, Claim 11 is directed to a method which is directed to an process, one of the statutory categories. Under Step 2A Prong 1, the claim recites the following limitations which are considered mental evaluations: calculates a quantization parameter for quantizing a pretrained neural network model, and calculates the quantization parameter optimized based on a reconstruction error between a first output value of the neural network model before quantization and a second output value of the neural network model after quantization by inputting a predetermined number of pieces of data, [from claim 11] and calculating a quantization parameter optimized based on a reconstruction error between the first output value and a second output value of a quantized neural network model acquiring a first output value of a pretrained neural network model by inputting a predetermined number of pieces of data Calculations of abstract parameters amount to mental processes that are able to be performed in the human mind. Further, the limitation which describes acquiring does not describe how the values are acquired, determining values by inputting abstract data into some unnamed process is again an evaluation of abstract data. Therefore the claim recites an abstract idea. Under Step 2A Prong 2, The claim recites the following additional elements: [from claim 1] memory in which at least one program is recorded; and a processor for executing the program, wherein: the program (amounts to mere instructions to apply a computer technology to an abstract idea, see MPEP 2106.05(f)) the neural network model is configured with a convolutional neural network and a transformer, and the convolutional neural network and the transformer are connected by a bridge block. (describes an application at a high degree of generality which makes use of the recited exception, see MPEP 2106.05(f) and/or is generally linking the use of the judicial exception to a particular technological environment or field of use, see MPEP 2106.05(h). Examiner notes the functions of the labeled components are not described in any particular detail. At most the claim recites that the calculated parameters are “of” such a neural network model. Therefore, the claim is directed to a judicial exception Under Step 2B, the recited additional elements when considered alone or in combination neither integrates the abstract idea into a practical application nor provides significantly more than the abstract idea itself. Regarding Claim 2/7 The claim depends on a rejected base claim. Each of the limitations described in the claim, under Step 2A Prong 1, only serve to describe the abstract ideas addressed in the independent claim, in particular the limitations describe mental evaluations. Under Step 2B, the recited additional elements when considered alone or in combination neither integrates the abstract idea into a practical application nor provides significantly more than the abstract idea itself. Regarding Claim 3/8/12 The claim depends on a rejected base claim. Each of the limitations described in the claim, under Step 2A Prong 1, only serve to describe the abstract ideas addressed in the independent claim, in particular the limitations describe mental evaluations. Under Step 2B, the recited additional elements when considered alone or in combination neither integrates the abstract idea into a practical application nor provides significantly more than the abstract idea itself. Regarding Claim 4/9/13 The claim depends on a rejected base claim. Each of the limitations described in the claim, under Step 2A Prong 1, only serve to describe the abstract ideas addressed in the independent claim, in particular the limitations describe mental evaluations. Under Step 2B, the recited additional elements when considered alone or in combination neither integrates the abstract idea into a practical application nor provides significantly more than the abstract idea itself. Regarding Claim 5/10/14 The claim depends on a rejected base claim. Each of the limitations described in the claim, under Step 2A Prong 1, only serve to describe the abstract ideas addressed in the independent claim, in particular the limitations describe mental evaluations. Under Step 2B, the recited additional elements when considered alone or in combination neither integrates the abstract idea into a practical application nor provides significantly more than the abstract idea itself. Regarding Claim 15 The claim depends on a rejected base claim. Each of the limitations described in the claim, under Step 2A Prong 1, only serve to describe the abstract ideas addressed in the independent claim, in particular the limitations describe mental evaluations. Under Step 2B, the recited additional elements when considered alone or in combination neither integrates the abstract idea into a practical application nor provides significantly more than the abstract idea itself. Regarding Claim 16 The claim depends on a rejected base claim. Under Step 2A Prong 1, the claim recites the following limitations which are considered mental evaluations: wherein calculating the quantization parameter further includes acquiring a gradient value for each layer Calculations of abstract parameters amount to mental processes that are able to be performed in the human mind. Further, acquiring new data is the consequence of performing a mental calculation on abstract data which further describes the identified mental process. Under Step 2A Prong 2, The claim recites the following additional elements: …through error backpropagation of a second neural network model after a predetermined number of pieces of data is input to the pretrained second neural network model and propagated in a forward direction (is generally linking the use of the judicial exception to a particular technological environment or field of use, see MPEP 2106.05(h). The limitation does not describe how backpropagation is performed nor how data is propagated in a forward direction. Instead, the limitations generally links the recited abstract idea to a general neural network process.) Each of the limitations described in the claim, under Step 2A Prong 1, only serve to describe the abstract ideas addressed in the independent claim, in particular the limitations describe mental evaluations. Under Step 2B, the recited additional elements when considered alone or in combination neither integrates the abstract idea into a practical application nor provides significantly more than the abstract idea itself. Regarding Claim 17 The claim depends on a rejected base claim. Each of the limitations described in the claim, under Step 2A Prong 1, only serve to describe the abstract ideas addressed in the independent claim, in particular the limitations describe mental evaluations. Under Step 2B, the recited additional elements when considered alone or in combination neither integrates the abstract idea into a practical application nor provides significantly more than the abstract idea itself. Regarding Claim 18 The claim depends on a rejected base claim. Each of the limitations described in the claim, under Step 2A Prong 1, only serve to describe the abstract ideas addressed in the independent claim, in particular the limitations describe mental evaluations. Under Step 2B, the recited additional elements when considered alone or in combination neither integrates the abstract idea into a practical application nor provides significantly more than the abstract idea itself. Regarding Claim 19 The claim depends on a rejected base claim. Each of the limitations described in the claim, under Step 2A Prong 1, only serve to describe the abstract ideas addressed in the independent claim, in particular the limitations describe mental evaluations. Under Step 2B, the recited additional elements when considered alone or in combination neither integrates the abstract idea into a practical application nor provides significantly more than the abstract idea itself. Claim Rejections - 35 USC § 103 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. Claim(s) 1-19 are rejected under 35 U.S.C. § 103 as being unpatentable over Wu “EasyQuant: Post-training Quantization via Scale Optimization” further in view of Michielli et al. GB2626819A “Method for personalisation of ASR models” Claim 1/6/11 Wu teaches, An apparatus for post-training quantization of a hybrid vision transformer, comprising: memory in which at least one program is recorded; and a processor for executing the program [from claim 6] A method for post-training quantization of a hybrid vision transformer, comprising [from claim 11] A method for post-training quantization of a hybrid vision transformer, comprising ( abstract “we present an efficient and simple post-training method…Then, we lower down bit width to INT7 both for weights and activations, and adopt INT16 intermediate storage and integer Winograd convolution implementation to accelerate inference. Experimental results on various computer vision tasks show that EQ outperforms the TensorRT method and can achieve near INT8 accuracy in 7 bits width post-training.” The method works on various hybrid vision tasks based on post training quantizationpg 13 Table 6 caption “The latency (ms) performance on RK3399, which inside is a 1.8 GHz 64-bit Dual-core ARM Cortex-A72. #k means k threads.” The system is tested and implemented on a processor, thus the method/system is performed on a computer having memory) the program calculates a quantization parameter for quantizing a pretrained neural network model…and calculates the quantization parameter optimized based on a reconstruction error between a first output value of the neural network model before quantization and a second output value of the neural network model after quantization by inputting a predetermined number of pieces of data [claim 6] calculating a quantization parameter for quantizing a pretrained neural network model; [claim 11] acquiring a first output value of a pretrained neural network model by inputting a predetermined number of pieces of data; and calculating a quantization parameter optimized based on a reconstruction error between the first output value and a second output value of a quantized neural network model (“pg 6 “To solve this problem, we present a simple but efficient scale optimization method, which jointly optimize both scales of activation and weights targeting at the loss on the similarity between original and quantized convolutional outputs.” pg 6 “According to Equation (2), we optimize scale factors {Sl}L l=1 for a pre-trained convolutional neural network {Al,Wl}L l=1, where activations {Al}L l=1 are generated from a given calibration dataset D with N samples. It is much less than the common training dataset. In the l-th layer of the neural network, our approach can be expressed to maximize output feature map cosine similarity”) the neural network model is configured with a convolutional neural network (Figure 1 caption pg 6 “The quantization process of a convolutional layer. The whole process contains three parts: quantization of input activations and weights, convolutional operation” a network with convolutional operations is a convolutional network) Wu does not explicitly teach, and a transformer, and the convolutional neural network and the transformer are connected by a bridge block. Michielli however when addressing quantization for hybrid neural networks having convolutional and transformer layer teaches, and a transformer, and the convolutional neural network and the transformer are connected by a bridge block. (paragraph 66 “Figure 2a shows more detail of the personalisation method of Figure 1. The general-purpose model 210 and personalised model 240 are both neural networks and both comprise a plurality of layers (1, N). The models may be any type of neural network, including for example convolution neural networks (CNNs) or transformers… The layers may thus be any type of layer, including for example convolutional, linear or transformer layers” the network quantized has any type of layer convolutional or transformer, the bridge block are simply the layers connected or bridging the convolutional and transformer layers. Further convolutional layers themselves may reasonably be considered transformers themselves as they transform input into new output. However, in the interest of compact prosecution the rejection is made based on the combination of Wu and Michielli.) Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the neural network quantization method with the neural network quantization method described by Michielli which quantizes both convolutional and transformer layers. One would have been motivated to make such a combination because both Wu and Michielli address methods for post training quantization of neural network layers of various architecture. Further, Michielli notes “Model quantization is effective to produce compressed general-purpose models…DF-PTQ has been tackled in the computer vision (CV) domain via layerwise knowledge distillation (KD) between original and quantized layers for compressing CNNs and transformer network…” (Michielli paragraph 3-4) Claim 2/7 Wu/Michielli teach claim 1 Wu teaches, the program optimizes the quantization parameter individually for each of layers included in the convolutional neural network (pg 5 “Quantization process of a neural network model could be divided into each layer, where weights and activations are quantized respectively and prepared for convolutional operation. The quantization of a convolutional layer is shown in Figure 1.” Quantization for each convolutional layer) Michielli teaches, and the transformer, and optimizes the quantization parameter for all of layers included in the bridge block. (paragraph 66 “As schematically illustrated in Figure 2a, each layer in the general-purpose model is quantized in the quantization module 230 and the resulting quantized layer is used in the personalised model 240. Thus, there is layer-wise personalised quantization. The number of layers in the personalised model is the same as the general-purpose model” as previously noted each layer of the network includes convolutional, transformer and bridge block. Each and all layers are quantized on a per layer basis as claimed.) Claim 3/8/12 Wu/Michielli teach claim 2/7/11 Wu teaches, the quantization parameter includes a scaling factor for a weight and a scaling factor for an input activation, (pg 5 “Let us define a quantized L-layer neural network as {Al,Wl,Sl}L l=1. Al, Wl, and Sl are the l-th layer input activation, weight, and quantization scale factors in FP32 range, respectively. Specifically, quantization scale factors (Sl) contain two parts. The scale number for the input activations in the l-th layer is denoted as Sa l . The scale number for the weights in the l-th layer is denoted as Sw l . Sa l is a non-negative real number applied for each elements in feature maps. Sw l is a non-negative real number for input weights.”) and each of the scaling factors is set within a predetermined search range. (pg 6 “where Sl scales the activations and weights to the fixed bits width range. Generally, the larger the scale is, the more high value elements will be saturated to the maximum of the quantized domain. While the smaller the scale is, the more low value elements will be rounded to zeros.”) Claim 4/9/13 Wu/Michielli teach claim 2/13/12 Wu teaches, the quantization parameter further includes granularity for determining a number of scaling factors for each layer. (pg 5 “For the convenience of discussion, we illustrate the proposed method in per-layer quantization scheme. For per channel quantization scheme, each filter in Wl should have independent scale number.” Depending on the quantization scheme a number of scale factors is determined indicating granularity, multiple independent parameters of per layer parameters in the per-layer scheme indicative of the number of scaling factors for each layer.) Claim 5/10/14 Wu/Michielli teach claim 3/8/12 Michielli teaches, wherein the quantization parameter further includes a quantization scheme for determining whether a quantization section is symmetric or asymmetric. ( paragraph 14 Generating a set of quantized weights may comprise using a quantization function to restrict the general weights to finite integer values by scaling, using the scaling factor and rounding. An example quantization function for each layer! is defined as PNG media_image1.png 83 314 media_image1.png Greyscale … Z1 corrects the zero point of the quantized output” as explained in paragraph 0069 of the disclosure the zero point in a quantization function is what determines whether quantization is symmetric or asymmetric.) Claim 15 Wu/Michielli teach claim 11 Michielli teaches, wherein calculating the quantization parameter includes acquiring the second output value of the quantized neural network model individually for each of layers included in the convolutional neural network and the transformer and thereby calculating an error between the second output value and the first output value, and acquiring the second output value of the quantized neural network model for all of layers included in the bridge block and thereby calculating an error between the second output value and the first output value. (paragraph 66 “As schematically illustrated in Figure 2a, each layer in the general-purpose model is quantized in the quantization module 230 and the resulting quantized layer is used in the personalised model 240. Thus, there is layer-wise personalised quantization. The number of layers in the personalised model is the same as the general-purpose model” para 18-19 The calibration data set may be the set of local data samples and the two phrases may be used interchangeably…. PNG media_image2.png 78 205 media_image2.png Greyscale as previously noted each layer of the network includes convolutional, transformer and bridge block. Each and all layers, as claimed, are quantized on a per layer basis as claimed. According to the per layer minimization function to determine corresponding scale factors.) Claim 16 Wu/Michielli teach claim 15 Michielli teaches, wherein calculating the quantization parameter further includes acquiring a gradient value for each layer or the bridge block through error backpropagation of a second neural network model after a predetermined number of pieces of data is input to the pretrained second neural network model and propagated in a forward direction. (paragraph 87 “The Hessian-based calibration optimization for layer 1 is defined by PNG media_image3.png 93 321 media_image3.png Greyscale where PNG media_image4.png 35 114 media_image4.png Greyscale is the distance between the output of the full precision layer and the PNG media_image5.png 42 35 media_image5.png Greyscale represents the loss of the full precision network over the calibration data” here the determination of the parameters is based on the hessian equation, which is in fact the same equation (8) in the instant disclosure. The equation describes a forward propagation on data for both quantized on non-quantized models to arrive at outputs for computing distance, and additionally a gradient value for each layer 1 to L in the equation is through error backpropagation of each layer output.) Claim 17 Wu/Michielli teach claim 12 Wu teaches, calculating the quantization parameter includes generating a predetermined number of candidates for the quantization parameter for each layer of the neural network model within a predetermined range. (pg 7 “For search space of Sw l and Sa l ,we linearly divide interval of [αSl,βSl] into n candidate options and conduct a simple search strategy on them.” ) Claim 18 Wu/Michielli teach claim 17 Wu teaches, wherein calculating the quantization parameter further includes searching for a quantization parameter that minimizes an error by alternately selecting quantization parameters included in the candidates for the quantization parameter for each layer. ( pg 6-7 “We adopt alternating optimization method here to solve this problem in two folds. First, Sa l is fixed, and solve Sw l for weight scales adjustment. Second, Sw l is fixed, and solve Sa l to finetune activations scales. Sw l and Sa l are alternately optimized until cos(Oi l, ˆOi l) converges” as described maximizing the similarity between the quantized and non-quantized layer outputs amounts to minimizing the error between the outputs as claimed) Claim 19 Wu/Michielli teach claim 18 Michielli teaches, wherein calculating the quantization parameter further includes acquiring a gradient value for each layer or the bridge block through error backpropagation of a second neural network model after a predetermined number of pieces of data is input to the pretrained second neural network model and propagated in a forward direction. (paragraph 87 “The Hessian-based calibration optimization for layer 1 is defined by PNG media_image3.png 93 321 media_image3.png Greyscale where PNG media_image4.png 35 114 media_image4.png Greyscale is the distance between the output of the full precision layer and the PNG media_image5.png 42 35 media_image5.png Greyscale represents the loss of the full precision network over the calibration data” here the determination of the parameters is based on the hessian equation, which is in fact the same equation (8) in the instant disclosure. Which is by using a diagonal matrix indicated by diag[] of the squared gradient values for each layer L.) Conclusion Prior art not relied upon: Nagel et al “A White Paper on Neural Network Quantization” describes fundamental methods for neural network post training quantization. Li et al. “BRECQ: pushing the limit of post-training quantization by block reconstruction” describes algorithms for post training quantization. Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHNATHAN R GERMICK whose telephone number is (571)272-8363. The examiner can normally be reached M-F 9:30-4:30. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached on 571-272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /J.R.G./ Examiner, Art Unit 2122 /KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122