Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-10 and 12-20 are rejected under 35 U.S.C. 103 as being unpatentable over “MIXED PRECISION QUANTIZATION OF TRANSFORMER LANGUAGE MODELS FOR SPEECH RECOGNITION”, Xu et al (2021) in view of “Up or Down? Adaptive Rounding for Post-Training Quantization”, Nagel et al, 2020.
Referring to claims 1, 8 and 18, Xu discloses a method for mixed precision quantization of an artificial intelligence (AI) model by an electronic device, the method comprising:
performing, by the electronic device, perturbation in weights of each layer of a plurality of layers of the AI model for a pre-defined number of times; (page 3 of Xu, “when using a given quantization precision, is expressed in the form of Hessian trace weighted squared quantization error. In simple terms, for each cluster of weight parameters, given the same amount of weight perturbation resulted from quantization, the smaller the associated Hessian matrix trace, the lower the performance sensitivity to quantization.” Here, a pre-defined number of times can be one time of perturbation in weights)
determining, by the electronic device, a sensitivity metric for each layer of the plurality of layers of the AI model as a measure of the change in the output of each layer; (page 3 and Fig. 1 of Xu, “An example of auto-configured mixed precision quantization of a transformer LM using a minimum performance sensitivity measure.”)
assigning, by the electronic device, a bit-precision to each layer of the plurality of layers of the AI model based on the determined sensitivity metric; (page 4 and Table 1 and 2 of Xu, bit precious performance sensitivity based quantization) and
performing, by the electronic device, the mixed precision quantization of the AI model using the bit-precision assigned to each layer of the plurality of layers of the AI model. (page 4 and Table 1 and 2 of Xu, bit precious performance sensitivity based quantization)
Xu does not specifically disclose “determining, by the electronic device, a change in an output of each layer of the plurality of layers of the AI model based on the perturbation in the weights of each layer of the plurality of layers.”
However, Nagel discloses determining, by the electronic device, a change in an output of each layer of the plurality of layers of the AI model based on the perturbation in the weights of each layer of the plurality of layers (pages 2-5 of Nagel, where the precision is based on perturbed weight of the layers).
Xu and Nagel are analogous art because both references concern mixed precision quantization. Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Xu’s sensitivity measurement of the bit-precision with running output through a weighted layer as taught by Nagel. The motivation for doing so would have been improving the measurement of bit-precision sensitivity based on manipulating the weight of each layer.
Referring to claims 2 and 9, Xu in view of Nagel disclose the method of claim 1, wherein the determining, by the electronic device, of the change in the output of each layer of the plurality of layers of the AI model comprises:
determining, by the electronic device, gradients of loss of each layer of the plurality of layers based on the perturbing weights of each layer of the plurality of layers; and determining, by the electronic device, the change in the output of each layer of the plurality of layers of the AI model based on the gradients of loss of each layer of the plurality of layers, wherein the change in the output indicates loss with respect to each layer of the plurality of layers. (page 2 of Xu, “Assuming the parameters of a neural network is twice differentiable and converged to a local optimum, it was proved in [16] that the expected performance loss, when using a given quantization precision, is expressed in the form of Hessian trace weighted squared quantization error. In simple terms, for each cluster of weight parameters, given the same amount of weight perturbation resulted from quantization, the smaller the associated Hessian matrix trace, the lower the performance sensitivity to quantization.”)
Referring to claims 3, 10 and 19, Xu in view of Nagel disclose the method of claim 2, wherein the sensitivity metric for each layer of the plurality of layers of the AI model is determined based on the gradients of loss. (page 2 of Xu, “Assuming the parameters of a neural network is twice differentiable and converged to a local optimum, it was proved in [16] that the expected performance loss, when using a given quantization precision, is expressed in the form of Hessian trace weighted squared quantization error. In simple terms, for each cluster of weight parameters, given the same amount of weight perturbation resulted from quantization, the smaller the associated Hessian matrix trace, the lower the performance sensitivity to quantization.”)
Referring to claims 5 and 12, Xu in view of Nagel disclose the method of claim 1, wherein the performing, by the electronic device, of the mixed precision quantization of the AI model using the bit-precision assigned to each layer of the plurality of layers of the AI model comprises: enabling, by the electronic device, each layer of the plurality of layers to be on the assigned bit-precision to obtain an optimal mixed-precision quantized AI model by performing a post training quantization of the AI model using the assigned bit-precision to each of the plurality of layers. (page 1 of Xu, The optimal local precision settings are automatically learned using two techniques. The first is based on a quantization sensitivity metric in the form of Hessian trace weighted quantization perturbation. It can be efficiently computed using Hessian-free approaches. The second is based on mixed precision Transformer architecture search.”)
Referring to claims 6 and 13, Xu in view of Nagel disclose the method of claim 5, wherein the optimal AI model obtains an optimal performance of each layer of the plurality of layers of the AI model in terms of at least one of a power level, an amount of memory usage, a level of computational efficiency, and On-Device learning on the electronic device. (page 1 of Xu, improvement on memory foodprint and computation cost)
Referring to claims 7 and 14, Xu in view of Nagel disclose the method of claim 1, wherein the bit-precision is assigned to each layer of the plurality of layers of the AI model by selecting at least one bit from a bit-precision set based on the sensitivity metric. (page 3 of Xu, “this requires transformer LMs using uniform precision, for example 1-bit, 2-bit, 4-bit and 8-bit be separately trained off-line first via ADMM”)
Referring to claim 15, Xu in view of Nagel disclose the electronic device of claim 8, wherein, to assign the bit-precision to each layer of the plurality of layers of the AI model based on the sensitivity metric, the one or more computer programs further comprise computer-executable instructions to: decide a final bit configuration for each layer considering total-loss from a loss-estimator and model six constraints. (pages 7 and 8 of Nagel)
Referring to claim 16, Xu in view of Nagel disclose the electronic device of claim 15, wherein the loss-estimator is configured to assign 2 bit, 4 bit, or 8 bit to the AI model. (page 2 of Xu, “when the local quantization table in equation (8) is shared across all the layers, leads to the traditional uniform precision quantization approach. The only remaining factor affecting the system performance is the bit length #bit which is also globally set to be 1,2,4,8 etc.”)
Referring to claim 17, Xu in view of Nagel disclose the electronic device of claim 8, wherein the weights of each layer of a plurality of layers is modified by a random noise in a weight tensor. (pages 1 and 7 of Nagel, weight tensor with introduced noise)
Referring to claim 20, Xu in view of Nagel disclose the one or more non-transitory computer-readable storage media of claim 18, wherein the assigning, by the electronic device, of the bit-precision to each layer of the plurality of layers of the AI model based on the sensitivity metric further comprising: constructing, by the electronic device, a constrained optimization problem model using the sensitivity metric for each layer of the plurality of layers and a net compression ratio, and assigning, by the electronic device, the bit precision to each layer of the plurality of layers based on the constrained optimization problem model, wherein the performing, by the electronic device, of the mixed precision quantization of the AI model using the bit-precision assigned to each layer of the plurality of layers of the AI model further comprising: enabling, by the electronic device, each layer of the plurality of layers to be on the assigned bit-precision to obtain an optimal mixed-precision quantized AI model by performing a post training quantization of the AI model using the assigned bit-precision to each of the plurality of layers, wherein the optimal AI model obtains an optimal performance of each layer of the plurality of layers of the AI model in terms of at least one of a power level, an amount of memory usage, a level of computational efficiency, and On-Device learning on the electronic device, and wherein the bit-precision is assigned to each layer of the plurality of layers of the AI model by selecting at least one bit from a bit-precision set based on the sensitivity metric. (see citations for claims 1, 6 and 7)
Allowable Subject Matter
Claims 4, and 11 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The prior art made of record and not relied upon is considered pertinent to Applicant's disclosure:
“BMPQ: Bit-Gradient Sensitivity-Driven Mixed-Precision Quantization of DNNs from Scratch,” Kundu et al (2021): Large DNNs with mixed-precision quantization can achieve ultra-high compression while retaining high classification performance. However, because of the challenges in finding an accurate metric that can guide the optimization process, these methods either sacrifice significant performance compared to the 32-bit floating-point (FP-32) baseline or rely on a compute-expensive, iterative training policy that requires the availability of a pre-trained baseline. To address this issue, this paper presents BMPQ, a training method that uses bit gradients to analyze layer sensitivities and yield mixed-precision quantized models. BMPQ requires a single training iteration but does not need a pre-trained baseline. It uses an integer linear program (ILP) to dynamically adjust the precision of layers during training, subject to a fixed hardware budget. To evaluate the efficacy of BMPQ, we conduct extensive experiments with VGG16 and ResNet18 on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets. Compared to the baseline FP-32 models, BMPQ can yield models that have 15.4x fewer parameter bits with a negligible drop in accuracy. Compared to the SOTA "during training", mixed-precision training scheme, our models are 2.1x, 2.2x, and 2.9x smaller, on CIFAR-10, CIFAR-100, and Tiny-ImageNet, respectively, with an improved accuracy of up to 14.54%.
Yao et al (US 20210019630 A1): loss-error-aware quantization of a low-bit neural network are disclosed. An example apparatus includes a network weight partitioner to partition unquantized network weights of a first network model into a first group to be quantized and a second group to be retrained. The example apparatus includes a loss calculator to process network weights to calculate a first loss. The example apparatus includes a weight quantizer to quantize the first group of network weights to generate low-bit second network weights. In the example apparatus, the loss calculator is to determine a difference between the first loss and a second loss. The example apparatus includes a weight updater to update the second group of network weights based on the difference. The example apparatus includes a network model deployer to deploy a low-bit network model including the low-bit second network weights.
Applicant is required under 37 C.F.R. § 1.111(c) to consider these references fully when responding to this action.
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
In the interests of compact prosecution, Applicant is invited to contact the examiner via electronic media pursuant to USPTO policy outlined MPEP § 502.03. All electronic communication must be authorized in writing. Applicant may wish to file an Internet Communications Authorization Form PTO/SB/439. Applicant may wish to request an interview using the Interview Practice website: http://;www.uspto.gov/patent/laws-and-regulations/interview-practice.
Applicant is reminded Internet e-mail may not be used for communication for matters under 35 U.S.C. § 132 or which otherwise require a signature. A reply to an Office action may NOT be communicated by Applicant to the USPTO via Internet e- mail. If such a reply is submitted by Applicant via Internet e-mail, a paper copy will be placed in the appropriate patent application file with an indication that the reply is NOT ENTERED. See MPEP § 502.03(II).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HAIMEI JIANG whose telephone number is (571)270-1590. The examiner can normally be reached M-F 9-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela D Reyes can be reached at 571-270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HAIMEI JIANG/Primary Examiner, Art Unit 2142
Prosecution Insights