DETAILED ACTION
Status of Claims
This Office action is responsive to communications filed on 2026-01-21. Claims 2-4, 6, 9-10, 12-14, 16, 20-21, and 23 were cancelled. Claim(s) 1, 5, 7-8, 11, 15, 17-19, 22, and 24-25 is/are pending and are examined herein.
Claim(s) 1, 5, 7-8, 11, 15, 17-19, 22, and 24-25 is/are rejected under 35 USC 112(b).
Claim(s) 1, 5, 7-8, 11, 15, 17-19, 22, and 24-25 is/are rejected under 35 USC 112(a).
Claim(s) 1, 5, 7-8, 11, 15, 17-19, 22, and 24-25 is/are rejected under 35 USC 103.
Response to Arguments
The applicant asserts that that they have “not acquiesced to any characterizations of the invention, nor any rejections or objections of the claims, made by the Examiner” [remarks, page 8]. No rationale is provided in support of this blanket assertion of non-acquiescence, and a bare assertion unsupported by any rationale fails to comply with 37 CFR 1.111(b) as it does not distinctly and specifically point out the reasons therefor.
Regarding the rejections under 35 USC 112, the applicant states that “[a]ll the pending claims have been amended to correct the indefiniteness and comply with the written description requirement” [remarks, page 9]. The examiner respectfully disagrees. The pending claims continue to include issues of indefiniteness and lack of written description as described below. In particular, the applicant has not addressed the substance of the rejections regarding how to define a single L2 regularization term using an infinite set of values that results from evaluating the argmin. Moreover, the applicant has amended the argmin formula appearing in the claim to deviate from the specification. The applicant asserts [remarks, page 8] that the amendments are supported by [specification, 0056, 0064], but neither of these paragraphs in the specification includes the formula that is now recited in the independent claims.
Regarding the rejections under 35 USC 103, the applicant’s remarks have been fully considered but they are not persuasive. The substance of the applicant’s remarks appears to be the observation that the clipping function PACT(x) [Choi, section 4 equation (1)] from Choi has only one parameter α, whereas the clipping function f(x) appearing in the claim has two parameters α and β [remarks, page 11]. However, the clipping function f(x) of the claim has not been mapped using PACT(x) from Choi; rather it has been mapped using the function clamp(x; a, c) [Nagel, section 2.2 equation (5)] from Nagel, which does in fact have two parameters that correspond precisely to the parameters in the applicant’s clipping function (namely, a and c). As explained in the previous Office action, Choi was brought into the rejection to disclose the indefinite use of L2 regularization which appears to be described in the claim (not for the clipping function of the claim). The examiner maintains that all of the claim elements would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention having before them the references made of record, and the complete prior art mapping, updated in view of the applicant’s amendments, is given below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after 2013-03-16, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC 112(b)
The following is a quotation of 35 USC 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 USC 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 1, 5, 7-8, 11, 15, 17-19, 22, and 24-25 is/are rejected under 35 USC 112(b) or 35 USC 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 USC 112, the applicant), regards as the invention.
Claims 1, 11, and 19 recite
a loss function including an L2 regularization term of the respective range from the respective minimum value to the respective maximum value for each channel of the plurality of channels of the neural network:
PNG
media_image1.png
39
169
media_image1.png
Greyscale
wherein α_l is the respective minimum value for channel l, β_l is the respective maximum value for the channel l, and x is the respective value used for the channel l of the neural network; [emphasis added]
This is indefinite for at least the following reasons. First, the underlined phrases lack antecedent basis. Secondly, and more substantially, the fact that α_l, β_l, and x appear below the argmin in the formula means that they are to be treated as varying quantities when evaluating sum_l ‖ α_l - β_l ‖, and this expression evidently achieves its minimum value of 0 if and only if α_l = β_l. In other words, the argmin in the claim necessarily evaluates to an infinite set of triples (namely, the infinite set of triples (α_l, β_l, x) where α_l = β_l), but it is not clear how to use an infinite set of triples as a single L2 regularization term. In other words, the claim does not clearly define how formula appearing in the claim relates to the L2 regularization term recited in the same limitation. Thirdly, the variable x is defined twice in the claim (first as “the respective value used within the neural network” and then as “the respective value used for the channel l of the neural network”), rendering unclear if these are two descriptions of the same quantity or of two distinct quantities. Dependent claims 5, 7-8, 15, 17-18, 22, and 24-25 inherit the rejection. The examiner notes that neither the formula nor its verbal description appear in the specification, nor does the specification indicate how an infinite set of triples is to be used as a single L2 regularization term (cf. 112(a) rejections). For the purpose of compact prosecution, the claim is interpreted broadly as encompassing at least a situation where some quantity depending on a difference between minimum and maximum values is used as L2 regularization term in the loss function.
Claim Rejections - 35 USC 112(a)
The following is a quotation of the first paragraph of 35 USC 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1, 5, 7-8, 11, 15, 17-19, 22, and 24-25 are rejected under 35 USC 112(a) or 35 USC 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 1, 11, and 19 were amended to recite
a loss function including an L2 regularization term of the respective range from the respective minimum value to the respective maximum value for each channel of the plurality of channels of the neural network:
PNG
media_image1.png
39
169
media_image1.png
Greyscale
wherein α_l is the respective minimum value for channel l, β_l is the respective maximum value for the channel l, and x is the respective value used for the channel l of the neural network;
This claim element lacks adequate written description in the originally filed specification. As best understood by the examiner (even in view of the applicant’s remarks [remarks, 8]), this limitation appears to be an attempt to incorporate content from [specification, 0056] into the claims. However, the formula in [specification, 0056] is not the same as the formula in claim. The formula in the claim includes α_l and β_l below the argmin, whereas the formula in the specification does not. Secondly, the specification does not describe α_l, β_l, and x as representing the quantities they are indicated as representing in the claim. Thirdly, the specification does not clarify how to compute an L2 regularization term from an argmin that evaluates to an infinite set (cf. 112(b) rejections). MPEP 2161.01 indicates that a computer-implemented functional claim limitation may lack adequate written description when the claims define the invention in functional language specifying a desired result but the specification does not sufficiently describe how the function is performed or the result is achieved. In other words, the algorithm or steps/procedure taken to perform the function must be described with sufficient detail so that one of ordinary skill in the art would understand how the inventor intended the function to be performed. The L2 regularization as claimed is not adequately described in the originally filed specification and the claims are therefore rejected for lack of written description. Dependent claims 5, 7-8, 15, 17-18, 22, and 24-25 inherit the rejection.
Claim Rejections - 35 USC 103
The following is a quotation of 35 USC 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 5, 7-8, 11, 15, 17-19, 22, and 24-25 is/are rejected under 35 USC 103 as being unpatentable over Markus NAGEL et al. (A White Paper on Neural Network Quantization, published 2021-06-15; hereafter “Nagel”) in view of Sangil JUNG et al. (Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Loss, published 2018-11-23; hereafter “Jung”) and Jungwook CHOI et al. (PACT: Parametrized Clipping Activation for Quantized Neural Networks; published 2018-07-17; hereafter “Choi”).
Claim 1
Nagel discloses:
A method of quantizing a neural network performed by an electronic apparatus, comprising: ([Nagel, abstract]: Nagel discloses methods of “[n]eural network quantization” [Nagel, abstract], indicating that the methods are performed on “hardware” [Nagel, abstract; see also, section 2.1]. This hardware is the “electronic apparatus” of the claim.)
clipping a respective value used within the neural network to be within a respective range from a respective minimum value to a respective maximum value ([Nagel, section 2.2 equation (5)]: Nagel discloses the use of the function clamp(x; a, c) for quantization. This clamping is the same as the “clipping” recited by the claim, with the variable a mapping to the “respective minimum value” of the claim and c to the “respective maximum value” of the claim.) for each channel of a plurality of channels of the neural network ([Nagel, section 2.2.3]: Nagel discloses “a separate quantizer for individual segments of a tensor (e.g., output channels of a weight tensor)”, calling this strategy “per-channel quantization” [Nagel, section 2.2.3]. See also: [Nagel, sections 2.4.2 and 4.2].) by a clipping function defined as:
PNG
media_image2.png
55
160
media_image2.png
Greyscale
wherein α is the respective minimum value, β is the respective maximum value, and x is the respective value used within the neural network, ([Nagel, section 2.2 equation (5)]: The function clamp(x; a, c) used in Nagel is equal to the function f(x) from the claim (i.e., it takes on exactly the same values for every possible input), with the variable a from Nagel mapping to α in the claim and c in Nagel mapping to β in the claim.)
wherein the respective value used within the neural network is one of a weight, a layer activation, and an intermediate feature in the neural network and has a corresponding pair of a minimum value and a maximum value; ([Nagel, section 2.1]: Nagel discloses “quantizing weights and activations” [Nagel, section 2.1 page 3 paragraph beginning “To move”] in neural networks [Nagel, section 2.1 first paragraph].)
quantizing the clipped respective value f(x) from a 32-bit Floating-Point (FP32) format to an 8-bit signed Integers (INT8) format; ([Nagel, sections 2.1-2.2]: Nagel discloses that “[n]eural networks are commonly trained using FP32 weights and activations” [Nagel, section 2.1 page 2 paragraph beginning “The above operation”]. It also discloses quantizing values to “a lower bit fixed-point or quantized representation… such as INT8” [Nagel, section 2.1 paragraph beginning “The above operation”]. The quantized value x_{int} is obtained from the clamped/clipped value [Nagel, section 2.2 equation (4)].)
training the neural network having the clipped respective value f(x) in the quantized INT8 format with input training data ([Nagel, abstract and sections 2.1-2 and 4]: As noted above, Nagel discloses a neural network having clipped and quantized values in INT8 format [Nagel, section 2.1]. It also discloses quantization-aware training (QAT) [Nagel, section 4], in which quantization parameters (i.e., parameters which are used for quantizing values) are updated alongside model parameters (e.g., weights) at each backward pass of training [Nagel, section 4.1 and figure 11]. Nagel also indicates that QAT uses “labeled training data” [Nagel, abstract]. In other words, QAT maps to the “training” of the claim, and the labeled training data used during QAT maps to the “input training data” of the claim.)
[and updating the neural network by repeating] (i) clipping of the respective value used within the neural network [according to the updated minimum value and the updated maximum value] and (ii) quantizing of the clipped respective value f(x) used within the neural network from the FP32 format to the INT8 format [according to the updated minimum value and the updated maximum value.] ([Nagel, section 4.1, and figures 10-11]: Nagel discloses performing quantization during the forward and backward passes of the QAT procedure [Nagel, section 4.1 and figure 10]. The quantization includes applying the clamping/clipping function clamp(x; a, c) to x and quantizing the clamped/clipped value to obtain the quantized value x_{int}, and it is indicated in particular that the original values are FP32 and the quantized values are INT8 [Nagel, sections 2.1-2]. Moreover, it discloses an example where training occurs over 20 epochs [Nagel, figure 11]. An epoch includes at least one forward pass and at least one backward pass. In other words, the repeated forward and backward passes performed across the plurality of epochs of QAT map to the “updating the neural network by repeating” of the claim, the clamping maps to the “clipping” of the claim, and the quantizing to the “quantizing” of the claim.)
As noted above, Nagel discusses QAT, i.e., training where quantization parameters are updated alongside model parameters. However, it does not distinctly disclose QAT where the minimum and maximum values of the clamping/clipping function are among the quantization parameters being updated. In other words, Nagel does not distinctly disclose:
[training the neural network…] to minimize a range between the minimum value and the maximum value corresponding to the clipped respective value by updating the minimum value and the maximum value using a loss function including an L2 regularization term of the respective range from the respective minimum value to the respective maximum value for each channel of the plurality of channels of the neural network:
PNG
media_image1.png
39
169
media_image1.png
Greyscale
wherein α_l is the respective minimum value for channel l, β_l is the respective maximum value for the channel l, and x is the respective value used for the channel l of the neural network;
[and updating the neural network by repeating (i) clipping of the respective value used within the neural network] according to the updated minimum value and the updated maximum value [and (ii) quantizing of the clipped respective value f(x) used within the neural network from the FP32 format to the INT8 format] according to the updated minimum value and the updated maximum value.
Jung is in the field of neural network quantization. It discloses a method called “quantization-interval-learning (QIL)” which “parameterize[s] the quantization intervals and obtain[s] their optimal values by directly minimizing the task loss of the network” during training [Jung, abstract]. This is a form of QAT. In other words, Nagel in view of Jung discloses:
[training the neural network…] to minimize a range between the minimum value and the maximum value corresponding to the clipped respective value by updating the minimum value and the maximum value using a loss function ([Jung, abstract, sections 1 and 3]: As noted above, Jung discloses a method of “quantization-interval-learning (QIL)” which “parameterize[s] the quantization intervals and obtain[s] their optimal values by directly minimizing the task loss of the network” during training [Jung, abstract]. The quantization intervals are parameterized by parameters c_Δ and d_Δ which “indicate the center of the interval and the distance from the center, respectively” but it is also indicated that “this is simply a design choice and other types of parameterization, such as parameterization with lower and upper bound, are also possible” [Jung, section 3.2 paragraph beginning “We define”; emphasis added] (because there is a bijective correspondence between c_Δ and d_Δ and the maximum and minimum values of interval; the minimum value is c_ Δ - d_ Δ and the maximum value is c_ Δ + d_ Δ) [Jung, section 3.2 equations (3) and (5)]). The parameters c_Δ and d_Δ are updated during training [Jung, algorithm 1 line 11], which means that so too are the minimum and maximum values of the interval. Jung discloses that that their “trainable quantizer adaptively finds the optimal intervals for quantization that minimize the task loss” while still ensuring that the interval is “as compact as possible” in order to “maintain or increase the quantization resolution” [Jung, section 1 paragraph beginning “Since reducing”]. Ensuring that the interval is “as compact as possible” is a form of “minimizing the range” as recited by the claim. The training algorithm also makes use of a loss ℓ [Jung, algorithm 1 lines 6-7, 9, and 12], which maps to the “loss function” of the claim.)
[and updating the neural network by repeating (i) clipping of the respective value used within the neural network] according to the updated minimum value and the updated maximum value [and (ii) quantizing of the clipped respective value f(x) used within the neural network from the FP32 format to the INT8 format] according to the updated minimum value and the updated maximum value. ([Jung, abstract and algorithm 1; Nagel, sections 2.1-2 and 4 and figures 10-11]: As noted above, Nagel already discloses QAT across a plurality of epochs having forward and backward passes that include clamping/clipping quantization steps [Nagel, sections 2.1-2 and 4 and figures 10-11]. Jung, like Nagel, discloses that the original network is “full precision (32-bit)” [Jung, abstract]. Moreover, each training iteration in Jung’s algorithm [Jung, algorithm 1 lines 3-12] updates the quantization parameters [Jung, algorithm 1 line 11] and performs clipping and quantizing with respect to the quantization parameters [Jung, algorithm 1 lines 4-5]. In other words, the combination discloses repeatedly clipping “according to the updated minimum value and the updated maximum value” and quantizing “the clipped respective value… according to the updated minimum value and the updated maximum value” as recited by the claim.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine neural network quantization as disclosed by Nagel with QIL as disclosed by Jung because learning quantization intervals results in a quantizer which produces quantized networks that “maintain the accuracy of the full precision (32-bit) networks” and which “outperforms existing methods to achieve… state-of-the-art accuracy” [Jung, abstract], so the combination would be an effective method of neural network quantization.
Nagel in view of Jung does not distinctly disclose:
[a loss function] including an L2 regularization term of the respective range from the respective minimum value to the respective maximum value for each channel of the plurality of channels of the neural network:
PNG
media_image1.png
39
169
media_image1.png
Greyscale
wherein α_l is the respective minimum value for channel l, β_l is the respective maximum value for the channel l, and x is the respective value used for the channel l of the neural network;
Choi is in the field of neural network quantization and discloses a technique called “PArameterized Clipping activation (PACT)” which includes a “clipping parameter α that is optimized during training to find the right quantization scale” [Choi, abstract], where the clipping parameter α that is optimized is the width of the quantization interval [Choi, section 4 first paragraph]. In other words, PACT is a form of QAT which is substantially similar to QIL in that the quantization parameters being updated during training include parameters related to the quantization interval. Moreover, Nagel in view of Jung and Choi discloses:
[a loss function] including an L2 regularization term of the respective range from the respective minimum value to the respective maximum value for each channel of the plurality of channels of the neural network:
PNG
media_image1.png
39
169
media_image1.png
Greyscale
wherein α_l is the respective minimum value for channel l, β_l is the respective maximum value for the channel l, and x is the respective value used for the channel l of the neural network; ([Choi, section 4; Nagel, section 2.2.3]: Choi discloses “includ[ing] a L2-regularizer for α in the loss function” in order to “avoid large quantization errors due to a wide dynamic range” [Choi, section 4 first paragraph]. As noted above, the clipping parameter α represents a width of the “range between the minimum value and the maximum value” as recited in the claim. As noted above, Nagel already discloses per-channel quantization [Nagel, section 2.2.3], so the combination of the L2-regularizer for α as in Choi with per-channel quantization as in Nagel discloses the “L2 regularization of the respective range… for each channel of the plurality of channels of the neural network” recited by the claim as best understood by the examiner in view of the 112(b) rejections.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine neural network quantization techniques as disclosed by Nagel in view of Jung with PACT as disclosed by Choi because it “enables neural networks to work well with ultra low precision weights and activations without any significant accuracy degradation”, “achiev[es] much better accuracy relative to published state-of-the-art quantization schemes”, and “enable[s] a super-linear improvement in inferencing performance due to a significant reduction in the area of accelerator compute engines coupled with the ability to retain the quantized model and activation data in on-chip memories” [Choi, abstract], so the combination would be an effective method of neural network quantization.
Claim 5
Nagel in view of Jung and Choi already discloses the elements of the parent claim(s). It also discloses:
[The method according to claim 1, wherein] clipping the respective value used within the neural network to be within the respective range from the respective minimum value to the respective maximum value includes at least one of symmetrical clipping, asymmetric clipping, positive clipping, and negative clipping. ([Nagel, section 2.2]: Nagel discloses “asymmetric quantization” [Nagel, section 2.2 page 4 first paragraph] as well as “[s]ymmetric quantization” [Nagel, section 2.2.1 first paragraph]. See also: [Nagel, figure 3].)
The same motivation to combine applies.
Claim 7
Nagel in view of Jung and Choi already discloses the elements of the parent claim(s). It also discloses:
[The method according to claim 1, wherein] clipping the respective value used within the neural network to be within the range from the respective minimum value to the respective maximum value is performed in a forward propagation. ([Nagel, section 2.2 and figure 4]: Nagel discloses a “quantized forward pass” [Nagel, figure 4 caption] in a neural network using a quantizer [Nagel, figure 4(b)]. It was already noted above that the quantization disclosed by Nagel uses a clamping/clipping operation [Nagel, section 2.2].)
The same motivation to combine applies.
Claim 8
Nagel in view of Jung and Choi already discloses the elements of the parent claim(s). It also discloses:
[The method according to claim 1, wherein] updating the neural network is performed in a backward propagation. ([Nagel, section 4.1 and figure 10]: As noted above, Nagel discloses QAT in which quantization parameters are updating during “backward pass[es]” which are performed during training [Nagel, section 4.1 and figure 10].)
The same motivation to combine applies.
Claim 11
Nagel discloses:
An electronic apparatus comprising one or more processing units, memory coupled to the one or more processing units, and a plurality of programs stored in the memory that, when executed by the one or more processing units, cause the electronic apparatus to perform a plurality of operations of quantizing a neural network ([Nagel, abstract and section 2]: Nagel discloses methods of “[n]eural network quantization” [Nagel, abstract], indicating that the methods are performed on “hardware” [Nagel, abstract; see also, section 2.1]. The hardware includes “processing units” [Nagel, section 2.1 second paragraph] and/or a “compute core” [Nagel, section 2.3.2 first paragraph] as well as a “memory” [Nagel, section 1 second paragraph, section 2.1 second paragraph, section 2.1 last paragraph, figure 2, section 2.3.2 first paragraph]. This hardware is the “electronic apparatus” of the claim, the processing units and/or compute core map to the “one or more processing units” of the claim, and the memory to the “memory” of the claim. The examiner notes that these hardware elements are also disclosed in the combination (e.g., “multiple cores, interfaced with an external memory” [Choi, section 6 first paragraph]).)
clipping a respective value used within the neural network to be within a respective range from a respective minimum value to a respective maximum value ([Nagel, section 2.2 equation (5)]: Nagel discloses the use of the function clamp(x; a, c) for quantization. This clamping is the same as the “clipping” recited by the claim, with the variable a mapping to the “respective minimum value” of the claim and c to the “respective maximum value” of the claim.) for each channel of a plurality of channels of the neural network ([Nagel, section 2.2.3]: Nagel discloses “a separate quantizer for individual segments of a tensor (e.g., output channels of a weight tensor)”, calling this strategy “per-channel quantization” [Nagel, section 2.2.3]. See also: [Nagel, sections 2.4.2 and 4.2].) by a clipping function defined as:
PNG
media_image2.png
55
160
media_image2.png
Greyscale
wherein α is the respective minimum value, β is the respective maximum value, and x is the respective value used within the neural network, ([Nagel, section 2.2 equation (5)]: The function clamp(x; a, c) used in Nagel is equal to the function f(x) from the claim (i.e., it takes on exactly the same values for every possible input), with the variable a from Nagel mapping to α in the claim and c in Nagel mapping to β in the claim.)
wherein the respective value used within the neural network is one of a weight, a layer activation, and an intermediate feature in the neural network and has a corresponding pair of a minimum value and a maximum value; ([Nagel, section 2.1]: Nagel discloses “quantizing weights and activations” [Nagel, section 2.1 page 3 paragraph beginning “To move”] in neural networks [Nagel, section 2.1 first paragraph].)
quantizing the clipped respective value f(x) from a 32-bit Floating-Point (FP32) format to an 8-bit signed Integers (INT8) format; ([Nagel, sections 2.1-2.2]: Nagel discloses that “[n]eural networks are commonly trained using FP32 weights and activations” [Nagel, section 2.1 page 2 paragraph beginning “The above operation”]. It also discloses quantizing values to “a lower bit fixed-point or quantized representation… such as INT8” [Nagel, section 2.1 paragraph beginning “The above operation”]. The quantized value x_{int} is obtained from the clamped/clipped value [Nagel, section 2.2 equation (4)].)
training the neural network having the clipped respective value f(x) in the quantized INT8 format with input training data ([Nagel, abstract and sections 2.1-2 and 4]: As noted above, Nagel discloses a neural network having clipped and quantized values in INT8 format [Nagel, section 2.1]. It also discloses quantization-aware training (QAT) [Nagel, section 4], in which quantization parameters (i.e., parameters which are used for quantizing values) are updated alongside model parameters (e.g., weights) at each backward pass of training [Nagel, section 4.1 and figure 11]. Nagel also indicates that QAT uses “labeled training data” [Nagel, abstract]. In other words, QAT maps to the “training” of the claim, and the labeled training data used during QAT maps to the “input training data” of the claim.)
[and updating the neural network by repeating] (i) clipping of the respective value used within the neural network [according to the updated minimum value and the updated maximum value] and (ii) quantizing of the clipped respective value f(x) used within the neural network from the FP32 format to the INT8 format [according to the updated minimum value and the updated maximum value.] ([Nagel, section 4.1, and figures 10-11]: Nagel discloses performing quantization during the forward and backward passes of the QAT procedure [Nagel, section 4.1 and figure 10]. The quantization includes applying the clamping/clipping function clamp(x; a, c) to x and quantizing the clamped/clipped value to obtain the quantized value x_{int}, and it is indicated in particular that the original values are FP32 and the quantized values are INT8 [Nagel, sections 2.1-2]. Moreover, it discloses an example where training occurs over 20 epochs [Nagel, figure 11]. An epoch includes at least one forward pass and at least one backward pass. In other words, the repeated forward and backward passes performed across the plurality of epochs of QAT map to the “updating the neural network by repeating” of the claim, the clamping maps to the “clipping” of the claim, and the quantizing to the “quantizing” of the claim.)
As noted above, Nagel discusses QAT, i.e., training where quantization parameters are updated alongside model parameters. However, it does not distinctly disclose QAT where the minimum and maximum values of the clamping/clipping function are among the quantization parameters being updated. In other words, Nagel does not distinctly disclose:
[training the neural network…] to minimize a range between the minimum value and the maximum value corresponding to the clipped respective value by updating the minimum value and the maximum value using a loss function including an L2 regularization term of the respective range from the respective minimum value to the respective maximum value for each channel of the plurality of channels of the neural network:
PNG
media_image1.png
39
169
media_image1.png
Greyscale
wherein α_l is the respective minimum value for channel l, β_l is the respective maximum value for the channel l, and x is the respective value used for the channel l of the neural network;
[and updating the neural network by repeating (i) clipping of the respective value used within the neural network] according to the updated minimum value and the updated maximum value [and (ii) quantizing of the clipped respective value f(x) used within the neural network from the FP32 format to the INT8 format] according to the updated minimum value and the updated maximum value.
Jung is in the field of neural network quantization. It discloses a method called “quantization-interval-learning (QIL)” which “parameterize[s] the quantization intervals and obtain[s] their optimal values by directly minimizing the task loss of the network” during training [Jung, abstract]. This is a form of QAT. In other words, Nagel in view of Jung discloses:
[training the neural network…] to minimize a range between the minimum value and the maximum value corresponding to the clipped respective value by updating the minimum value and the maximum value using a loss function ([Jung, abstract, sections 1 and 3]: As noted above, Jung discloses a method of “quantization-interval-learning (QIL)” which “parameterize[s] the quantization intervals and obtain[s] their optimal values by directly minimizing the task loss of the network” during training [Jung, abstract]. The quantization intervals are parameterized by parameters c_Δ and d_Δ which “indicate the center of the interval and the distance from the center, respectively” but it is also indicated that “this is simply a design choice and other types of parameterization, such as parameterization with lower and upper bound, are also possible” [Jung, section 3.2 paragraph beginning “We define”; emphasis added] (because there is a bijective correspondence between c_Δ and d_Δ and the maximum and minimum values of interval; the minimum value is c_ Δ - d_ Δ and the maximum value is c_ Δ + d_ Δ) [Jung, section 3.2 equations (3) and (5)]). The parameters c_Δ and d_Δ are updated during training [Jung, algorithm 1 line 11], which means that so too are the minimum and maximum values of the interval. Jung discloses that that their “trainable quantizer adaptively finds the optimal intervals for quantization that minimize the task loss” while still ensuring that the interval is “as compact as possible” in order to “maintain or increase the quantization resolution” [Jung, section 1 paragraph beginning “Since reducing”]. Ensuring that the interval is “as compact as possible” is a form of “minimizing the range” as recited by the claim. The training algorithm also makes use of a loss ℓ [Jung, algorithm 1 lines 6-7, 9, and 12], which maps to the “loss function” of the claim.)
[and updating the neural network by repeating (i) clipping of the respective value used within the neural network] according to the updated minimum value and the updated maximum value [and (ii) quantizing of the clipped respective value f(x) used within the neural network from the FP32 format to the INT8 format] according to the updated minimum value and the updated maximum value. ([Jung, abstract and algorithm 1; Nagel, sections 2.1-2 and 4 and figures 10-11]: As noted above, Nagel already discloses QAT across a plurality of epochs having forward and backward passes that include clamping/clipping quantization steps [Nagel, sections 2.1-2 and 4 and figures 10-11]. Jung, like Nagel, discloses that the original network is “full precision (32-bit)” [Jung, abstract]. Moreover, each training iteration in Jung’s algorithm [Jung, algorithm 1 lines 3-12] updates the quantization parameters [Jung, algorithm 1 line 11] and performs clipping and quantizing with respect to the quantization parameters [Jung, algorithm 1 lines 4-5]. In other words, the combination discloses repeatedly clipping “according to the updated minimum value and the updated maximum value” and quantizing “the clipped respective value… according to the updated minimum value and the updated maximum value” as recited by the claim.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine neural network quantization as disclosed by Nagel with QIL as disclosed by Jung because learning quantization intervals results in a quantizer which produces quantized networks that “maintain the accuracy of the full precision (32-bit) networks” and which “outperforms existing methods to achieve… state-of-the-art accuracy” [Jung, abstract], so the combination would be an effective method of neural network quantization.
Nagel in view of Jung does not distinctly disclose:
[a loss function] including an L2 regularization term of the respective range from the respective minimum value to the respective maximum value for each channel of the plurality of channels of the neural network:
PNG
media_image1.png
39
169
media_image1.png
Greyscale
wherein α_l is the respective minimum value for channel l, β_l is the respective maximum value for the channel l, and x is the respective value used for the channel l of the neural network;
Choi is in the field of neural network quantization and discloses a technique called “PArameterized Clipping activation (PACT)” which includes a “clipping parameter α that is optimized during training to find the right quantization scale” [Choi, abstract], where the clipping parameter α that is optimized is the width of the quantization interval [Choi, section 4 first paragraph]. In other words, PACT is a form of QAT which is substantially similar to QIL in that the quantization parameters being updated during training include parameters related to the quantization interval. Moreover, Nagel in view of Jung and Choi discloses:
[a loss function] including an L2 regularization term of the respective range from the respective minimum value to the respective maximum value for each channel of the plurality of channels of the neural network:
PNG
media_image1.png
39
169
media_image1.png
Greyscale
wherein α_l is the respective minimum value for channel l, β_l is the respective maximum value for the channel l, and x is the respective value used for the channel l of the neural network; ([Choi, section 4; Nagel, section 2.2.3]: Choi discloses “includ[ing] a L2-regularizer for α in the loss function” in order to “avoid large quantization errors due to a wide dynamic range” [Choi, section 4 first paragraph]. As noted above, the clipping parameter α represents a width of the “range between the minimum value and the maximum value” as recited in the claim. As noted above, Nagel already discloses per-channel quantization [Nagel, section 2.2.3], so the combination of the L2-regularizer for α as in Choi with per-channel quantization as in Nagel discloses the “L2 regularization of the respective range… for each channel of the plurality of channels of the neural network” recited by the claim as best understood by the examiner in view of the 112(b) rejections.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine neural network quantization techniques as disclosed by Nagel in view of Jung with PACT as disclosed by Choi because it “enables neural networks to work well with ultra low precision weights and activations without any significant accuracy degradation”, “achiev[es] much better accuracy relative to published state-of-the-art quantization schemes”, and “enable[s] a super-linear improvement in inferencing performance due to a significant reduction in the area of accelerator compute engines coupled with the ability to retain the quantized model and activation data in on-chip memories” [Choi, abstract], so the combination would be an effective method of neural network quantization.
Claims 15 and 17-18 inherit limitations from claim 11 and recite further limitations which are substantially similar to those recited by claims 5 and 7-8, respectively, so they are rejected by the same rationale.
Claim 19
Nagel discloses:
A non-transitory computer readable storage medium storing a plurality of programs for execution by an electronic apparatus having one or more processing units, wherein the plurality of programs, when executed by the one or more processing units, cause the electronic apparatus to perform a plurality of operations of quantizing a neural network, comprising: ([Nagel, abstract and section 2]: Nagel discloses methods of “[n]eural network quantization” [Nagel, abstract], indicating that the methods are performed on “hardware” [Nagel, abstract; see also, section 2.1]. The hardware includes “processing units” [Nagel, section 2.1 second paragraph] and/or a “compute core” [Nagel, section 2.3.2 first paragraph] as well as a “memory” [Nagel, section 1 second paragraph, section 2.1 second paragraph, section 2.1 last paragraph, figure 2, section 2.3.2 first paragraph]. This hardware is the “electronic apparatus” of the claim, the processing units and/or compute core map to the “one or more processing units” of the claim, and the memory to the “storage medium” of the claim. The examiner notes that these hardware elements are also disclosed in the combination (e.g., “multiple cores, interfaced with an external memory” [Choi, section 6 first paragraph]).)
clipping a respective value used within the neural network to be within a respective range from a respective minimum value to a respective maximum value ([Nagel, section 2.2 equation (5)]: Nagel discloses the use of the function clamp(x; a, c) for quantization. This clamping is the same as the “clipping” recited by the claim, with the variable a mapping to the “respective minimum value” of the claim and c to the “respective maximum value” of the claim.) for each channel of a plurality of channels of the neural network ([Nagel, section 2.2.3]: Nagel discloses “a separate quantizer for individual segments of a tensor (e.g., output channels of a weight tensor)”, calling this strategy “per-channel quantization” [Nagel, section 2.2.3]. See also: [Nagel, sections 2.4.2 and 4.2].) by a clipping function defined as:
PNG
media_image2.png
55
160
media_image2.png
Greyscale
wherein α is the respective minimum value, β is the respective maximum value, and x is the respective value used within the neural network, ([Nagel, section 2.2 equation (5)]: The function clamp(x; a, c) used in Nagel is equal to the function f(x) from the claim (i.e., it takes on exactly the same values for every possible input), with the variable a from Nagel mapping to α in the claim and c in Nagel mapping to β in the claim.)
wherein the respective value used within the neural network is one of a weight, a layer activation, and an intermediate feature in the neural network and has a corresponding pair of a minimum value and a maximum value; ([Nagel, section 2.1]: Nagel discloses “quantizing weights and activations” [Nagel, section 2.1 page 3 paragraph beginning “To move”] in neural networks [Nagel, section 2.1 first paragraph].)
quantizing the clipped respective value f(x) from a 32-bit Floating-Point (FP32) format to an 8-bit signed Integers (INT8) format; ([Nagel, sections 2.1-2.2]: Nagel discloses that “[n]eural networks are commonly trained using FP32 weights and activations” [Nagel, section 2.1 page 2 paragraph beginning “The above operation”]. It also discloses quantizing values to “a lower bit fixed-point or quantized representation… such as INT8” [Nagel, section 2.1 paragraph beginning “The above operation”]. The quantized value x_{int} is obtained from the clamped/clipped value [Nagel, section 2.2 equation (4)].)
training the neural network having the clipped respective value f(x) in the quantized INT8 format with input training data ([Nagel, abstract and sections 2.1-2 and 4]: As noted above, Nagel discloses a neural network having clipped and quantized values in INT8 format [Nagel, section 2.1]. It also discloses quantization-aware training (QAT) [Nagel, section 4], in which quantization parameters (i.e., parameters which are used for quantizing values) are updated alongside model parameters (e.g., weights) at each backward pass of training [Nagel, section 4.1 and figure 11]. Nagel also indicates that QAT uses “labeled training data” [Nagel, abstract]. In other words, QAT maps to the “training” of the claim, and the labeled training data used during QAT maps to the “input training data” of the claim.)
[and updating the neural network by repeating] (i) clipping of the respective value used within the neural network [according to the updated minimum value and the updated maximum value] and (ii) quantizing of the clipped respective value f(x) used within the neural network from the FP32 format to the INT8 format [according to the updated minimum value and the updated maximum value.] ([Nagel, section 4.1, and figures 10-11]: Nagel discloses performing quantization during the forward and backward passes of the QAT procedure [Nagel, section 4.1 and figure 10]. The quantization includes applying the clamping/clipping function clamp(x; a, c) to x and quantizing the clamped/clipped value to obtain the quantized value x_{int}, and it is indicated in particular that the original values are FP32 and the quantized values are INT8 [Nagel, sections 2.1-2]. Moreover, it discloses an example where training occurs over 20 epochs [Nagel, figure 11]. An epoch includes at least one forward pass and at least one backward pass. In other words, the repeated forward and backward passes performed across the plurality of epochs of QAT map to the “updating the neural network by repeating” of the claim, the clamping maps to the “clipping” of the claim, and the quantizing to the “quantizing” of the claim.)
As noted above, Nagel discusses QAT, i.e., training where quantization parameters are updated alongside model parameters. However, it does not distinctly disclose QAT where the minimum and maximum values of the clamping/clipping function are among the quantization parameters being updated. In other words, Nagel does not distinctly disclose:
[training the neural network…] to minimize a range between the minimum value and the maximum value corresponding to the clipped respective value by updating the minimum value and the maximum value using a loss function including an L2 regularization term of the respective range from the respective minimum value to the respective maximum value for each channel of the plurality of channels of the neural network:
PNG
media_image1.png
39
169
media_image1.png
Greyscale
wherein α_l is the respective minimum value for channel l, β_l is the respective maximum value for the channel l, and x is the respective value used for the channel l of the neural network;
[and updating the neural network by repeating (i) clipping of the respective value used within the neural network] according to the updated minimum value and the updated maximum value [and (ii) quantizing of the clipped respective value f(x) used within the neural network from the FP32 format to the INT8 format] according to the updated minimum value and the updated maximum value.
Jung is in the field of neural network quantization. It discloses a method called “quantization-interval-learning (QIL)” which “parameterize[s] the quantization intervals and obtain[s] their optimal values by directly minimizing the task loss of the network” during training [Jung, abstract]. This is a form of QAT. In other words, Nagel in view of Jung discloses:
[training the neural network…] to minimize a range between the minimum value and the maximum value corresponding to the clipped respective value by updating the minimum value and the maximum value using a loss function ([Jung, abstract, sections 1 and 3]: As noted above, Jung discloses a method of “quantization-interval-learning (QIL)” which “parameterize[s] the quantization intervals and obtain[s] their optimal values by directly minimizing the task loss of the network” during training [Jung, abstract]. The quantization intervals are parameterized by parameters c_Δ and d_Δ which “indicate the center of the interval and the distance from the center, respectively” but it is also indicated that “this is simply a design choice and other types of parameterization, such as parameterization with lower and upper bound, are also possible” [Jung, section 3.2 paragraph beginning “We define”; emphasis added] (because there is a bijective correspondence between c_Δ and d_Δ and the maximum and minimum values of interval; the minimum value is c_ Δ - d_ Δ and the maximum value is c_ Δ + d_ Δ) [Jung, section 3.2 equations (3) and (5)]). The parameters c_Δ and d_Δ are updated during training [Jung, algorithm 1 line 11], which means that so too are the minimum and maximum values of the interval. Jung discloses that that their “trainable quantizer adaptively finds the optimal intervals for quantization that minimize the task loss” while still ensuring that the interval is “as compact as possible” in order to “maintain or increase the quantization resolution” [Jung, section 1 paragraph beginning “Since reducing”]. Ensuring that the interval is “as compact as possible” is a form of “minimizing the range” as recited by the claim. The training algorithm also makes use of a loss ℓ [Jung, algorithm 1 lines 6-7, 9, and 12], which maps to the “loss function” of the claim.)
[and updating the neural network by repeating (i) clipping of the respective value used within the neural network] according to the updated minimum value and the updated maximum value [and (ii) quantizing of the clipped respective value f(x) used within the neural network from the FP32 format to the INT8 format] according to the updated minimum value and the updated maximum value. ([Jung, abstract and algorithm 1; Nagel, sections 2.1-2 and 4 and figures 10-11]: As noted above, Nagel already discloses QAT across a plurality of epochs having forward and backward passes that include clamping/clipping quantization steps [Nagel, sections 2.1-2 and 4 and figures 10-11]. Jung, like Nagel, discloses that the original network is “full precision (32-bit)” [Jung, abstract]. Moreover, each training iteration in Jung’s algorithm [Jung, algorithm 1 lines 3-12] updates the quantization parameters [Jung, algorithm 1 line 11] and performs clipping and quantizing with respect to the quantization parameters [Jung, algorithm 1 lines 4-5]. In other words, the combination discloses repeatedly clipping “according to the updated minimum value and the updated maximum value” and quantizing “the clipped respective value… according to the updated minimum value and the updated maximum value” as recited by the claim.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine neural network quantization as disclosed by Nagel with QIL as disclosed by Jung because learning quantization intervals results in a quantizer which produces quantized networks that “maintain the accuracy of the full precision (32-bit) networks” and which “outperforms existing methods to achieve… state-of-the-art accuracy” [Jung, abstract], so the combination would be an effective method of neural network quantization.
Nagel in view of Jung does not distinctly disclose:
[a loss function] including an L2 regularization term of the respective range from the respective minimum value to the respective maximum value for each channel of the plurality of channels of the neural network:
PNG
media_image1.png
39
169
media_image1.png
Greyscale
wherein α_l is the respective minimum value for channel l, β_l is the respective maximum value for the channel l, and x is the respective value used for the channel l of the neural network;
Choi is in the field of neural network quantization and discloses a technique called “PArameterized Clipping activation (PACT)” which includes a “clipping parameter α that is optimized during training to find the right quantization scale” [Choi, abstract], where the clipping parameter α that is optimized is the width of the quantization interval [Choi, section 4 first paragraph]. In other words, PACT is a form of QAT which is substantially similar to QIL in that the quantization parameters being updated during training include parameters related to the quantization interval. Moreover, Nagel in view of Jung and Choi discloses:
[a loss function] including an L2 regularization term of the respective range from the respective minimum value to the respective maximum value for each channel of the plurality of channels of the neural network:
PNG
media_image1.png
39
169
media_image1.png
Greyscale
wherein α_l is the respective minimum value for channel l, β_l is the respective maximum value for the channel l, and x is the respective value used for the channel l of the neural network; ([Choi, section 4; Nagel, section 2.2.3]: Choi discloses “includ[ing] a L2-regularizer for α in the loss function” in order to “avoid large quantization errors due to a wide dynamic range” [Choi, section 4 first paragraph]. As noted above, the clipping parameter α represents a width of the “range between the minimum value and the maximum value” as recited in the claim. As noted above, Nagel already discloses per-channel quantization [Nagel, section 2.2.3], so the combination of the L2-regularizer for α as in Choi with per-channel quantization as in Nagel discloses the “L2 regularization of the respective range… for each channel of the plurality of channels of the neural network” recited by the claim as best understood by the examiner in view of the 112(b) rejections.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine neural network quantization techniques as disclosed by Nagel in view of Jung with PACT as disclosed by Choi because it “enables neural networks to work well with ultra low precision weights and activations without any significant accuracy degradation”, “achiev[es] much better accuracy relative to published state-of-the-art quantization schemes”, and “enable[s] a super-linear improvement in inferencing performance due to a significant reduction in the area of accelerator compute engines coupled with the ability to retain the quantized model and activation data in on-chip memories” [Choi, abstract], so the combination would be an effective method of neural network quantization.
Claims 22 and 24-25 inherit limitations from claim 19 and recite further limitations which are substantially similar to those recited by claims 5 and 7-8, respectively, so they are rejected by the same rationale.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Shishir AGRAWAL whose telephone number is +1 703-756-1183. The examiner can normally be reached Monday through Thursday, 08:30-14:30 Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey SHMATOV can be reached on +1 571-270-3428. The fax phone number for the organization where this application or proceeding is assigned is +1 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at +1 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call +1 800-786-9199 (IN USA OR CANADA) or +1 571-272-1000.
/S.A./Examiner, Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights