DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
(Submitted 1/22/2026)
In regard to 101 rejections
- The applicant traverses the 101 rejections on Page 7. The applicant argues that the claim has demonstrated an improvement of machine learning models operation as a result of splitting of the neural network onto two devices and using quantization of the data of transmission between devices.
Examiners’ Response
The examiner submits that the claims focuses on calculation steps to determine the quantization
levels. These steps are abstract ideas (mental processes). The claims do not require performing
processing for the neural neatwork. Fig 3 describes the series of steps to determine quantization
level that includes series of calculation steps. Therefore, the examiner reaffirms that the claims are
directed to an abstract idea.
In CONCLUSION, the examiner MAINTAINS the 101 rejections on claims 1-20.
In regard to 103 rejections
- On Page 7, the applicant traverses the rejection on claim 1 but states that nevertheless the applicant has amended the claim 1 that the “ the data is drawn from layer of a neural network” . On Page 8, the applicant argues that reference Diril in [0024] does not teach the amendment argues that [0024] appears to be for quantization weight.
Examiners’ Response
The examiner respectfully “disagree” with the above arguments. The examiner submits that
quantization parameters as taught by Diril in [0024] do represent “quantization levels”. They may be
used to determine the precision of the model weight and then perform any precision changes. The
data value is the “quantization level” and not what is the post processing. The examiner does not
understand how the applicant interpreted that [0024] is interpreting the data value as a “weight”.
With respect to claims 10 and 15, the applicant’s argument is MOOT in the context of new reference
“Chen” teaching the amendment related to “decoder data output”.
In CONCLUSION, the examiner rejects the claims 1-20 under 103 and MOVES the application to FINAL
REJECTION under 103.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Step 1:
According to the first part of the analysis, in the instant case, claims 1-9 and 10-14 are directed to a method claim. Thus, each of the claims falls within one of the four statutory categories (i.e. process, machine, manufacture, or composition of matter.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
In regard to claim 1: (Currently Amended)
Step 2A Prong 1:
“ the data values being outputs from a layer of the neural network” is a mental step data calculation that can be performed as process of steps skin the way human can process based on the received data.
“ determine modified quantization levels to include: the at least one non-modifiable quantization level; and at least one level obtained by adjusting the at least one modifiable quantization level based on a cost function indicative of a distortion between the obtained data values and the preliminary quantization levels,” is a mental step data calculation and manipulation.
“ wherein the at least one non-modifiable quantization level includes a minimum clipping value (cmin) to which all lower data values are clipped or a maximum clipping value (cmax) to which all higher data values are clipped” is a mental step data calculation and manipulation.
Additional Elements
Step 2A Prong 2:
“ A method for modifying quantization levels of a quantizer for a neural network, the method
comprising the steps of: “recited in the preamble does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
“ obtaining data values and preliminary quantization levels attributed to the data values, the preliminary quantization levels including at least one non-modifiable quantization level and at least one modifiable quantization level “ does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
Step 2B:
“ A method for modifying quantization levels of a quantizer for a neural network, the method
comprising the steps of: “recited in the preamble does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
“ obtaining data values and preliminary quantization levels attributed to the data values, the preliminary quantization levels including at least one non-modifiable quantization level and at least one modifiable quantization level “ does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
In regard to claim 2 : (Original)
Step 2A Prong 1:
“ including the minimum clipping value (cmin) and the maximum clipping value (cmax” is a mental step data calculation using intersection points.
Step 2A Prong 2:
“wherein there are at least two non-modifiable quantization levels,” does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
Step 2B:
“wherein there are at least two non-modifiable quantization levels,” does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
In regard to claim 3 : (Original)
Step 2A Prong 2:
“ wherein there are at least two non-modifiable quantization levels, wherein among the at least two non-modifiable quantization levels, one has the value of zero” does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
Step 2B:
“ wherein there are at least two non-modifiable quantization levels, wherein among the at least two non-modifiable quantization levels, one has the value of zero” does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
In regard to claim 4 : (Original)
Step 2A Prong 2:
“ computing a level as a predefined function of the data values assigned to the level.” does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
Step 2B:
“ computing a level as a predefined function of the data values assigned to the level” does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
In regard to claim 5 : (Original)
Step 2A Prong 2:
“the cost function includes a linear combination of the distortion and a rate, the rate is represented by codeword length of codewords representing the respective preliminary quantization levels” does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
“ and the codewords representing at least two respective preliminary quantization levels having mutually different lengths” does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
Step 2B:
“the cost function includes a linear combination of the distortion and a rate, the rate is represented by codeword length of codewords representing the respective preliminary quantization levels” does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
“ and the codewords representing at least two respective preliminary quantization levels having mutually different lengths” does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
In regard to claim 6 : (Original)
Step 2A Prong 2:
“ wherein the steps of obtaining the preliminary quantization levels and determining the modified quantization levels are iterated K times, K being larger than 1, a j-th iteration out of the K iterations comprises:” does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
“ obtaining the preliminary quantization levels corresponding to the modified quantization levels determined by the (j-1)-th iteration” does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
“ determining the modified quantization levels to include the at least one non- modifiable quantization level and by modifying the at least one modifiable quantization level obtained by the (j-1)-the iteration based on the cost function” does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
Step 2B:
“ wherein the steps of obtaining the preliminary quantization levels and determining the modified quantization levels are iterated K times, K being larger than 1, a j-th iteration out of the K iterations comprises:” does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
“ obtaining the preliminary quantization levels corresponding to the modified quantization levels determined by the (j-1)-th iteration” does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
“ determining the modified quantization levels to include the at least one non- modifiable quantization level and by modifying the at least one modifiable quantization level obtained by the (j-1)-the iteration based on the cost function” does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
In regard to claim 7 : (Original)
Step 2A Prong 2:
“ stopping the iterations after the number of iterations K, in response to: - the value of the cost function in the K-th iteration being lower than a first threshold “does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
“ or the difference between the value of the cost function in the (K-1)-th iteration and the value of the cost function in the K-th iteration being lower than a second threshold” does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
Step 2B:
“ stopping the iterations after the number of iterations K, in response to: - the value of the cost function in the K-th iteration being lower than a first threshold “does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
“ or the difference between the value of the cost function in the (K-1)-th iteration and the value of the cost function in the K-th iteration being lower than a second threshold” does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
In regard to claim 8: (Currently Amended)
Step 2A Prong 2:
“ wherein the data values are: values of a feature map output by a layer of the neural network” does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
Step 2B:
“ wherein the data values are: values of a feature map output by a layer of the neural network” does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
In regard to claim 9 : (Original)
Step 2A Prong 2:
“ further comprising the steps of: computing an adjustment distance between the modified quantization levels and the quantization levels before the modification, adjusting the modified quantization levels by, for each modified quantization level, adopting either said modified quantization level or the quantization level before the modification, depending on the adjustment distance” does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
Step 2B:
“ further comprising the steps of: computing an adjustment distance between the modified quantization levels and the quantization levels before the modification, adjusting the modified quantization levels by, for each modified quantization level, adopting either said modified quantization level or the quantization level before the modification, depending on the adjustment distance” does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
In regard to claim 10: (Currently Amended)
Step 2A Prong 1:
“ a method of decoding data output from a layer of a neural network” is a mental step data calculation that can be performed as process of steps skin the way human can process based on the received data.
“ obtaining one or more supplementary quantization levels corresponding to a respective one or more of the original quantization levels, computing an adjustment distance between the original quantization levels and the corresponding supplementary quantization levels, determining modified quantization levels by, for each modified quantization level, adopting either an original quantization level or the corresponding supplementary quantization level, depending on the adjustment distance” a mental step data calculation and manipulation
Step 2A Prong 2:
“ the method including: obtaining original quantization levels with which the data were quantized” does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
“ obtaining, from codewords of the bitstream, the data for the neural network based on the obtained modified quantization levels” does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
“ and process the obtained data for the neural network by at least one layer of the neural network” does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
Step 2B:
“ the method including: obtaining original quantization levels with which the data were quantized” does not integrate the judicial exception into a practical application. does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
“ obtaining one or more supplementary quantization levels corresponding to a respective one or more of the original quantization levels, computing an adjustment distance between the original quantization levels and the corresponding supplementary quantization levels, determining modified quantization levels by, for each modified quantization level, adopting either an original quantization level or the corresponding supplementary quantization level, depending on the adjustment distance,” does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
“ obtaining, from codewords of the bitstream, the data for the neural network based on the obtained modified quantization levels” does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
“ and process the obtained data for the neural network by at least one layer of the neural network” does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
In regard to claim 11 : (Original)
Step 2A Prong 1:
“ minimum clipping value to which all lower data values are clipped, a maximum clipping value to which all higher data values are clipped, or a zero value.” is a mental step data calculation using intersection points.
Step 2A Prong 2:
“wherein the one or more supplementary quantization levels include at least one of a “does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
Step 2B:
“wherein the one or more supplementary quantization levels include at least one of a “does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
In regard to claim 12 : (Original)
Step 2A Prong 2:
“ wherein the adjustment distance includes a difference between the original and the corresponding supplementary quantization levels and an offset by a predetermined constant” does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
Step 2B:
“ wherein the adjustment distance includes a difference between the original and the corresponding supplementary quantization levels and an offset by a predetermined constant”
does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
In regard to claim 13 : (Original)
Step 2A Prong 2:
“ wherein the determining of the modified quantization levels includes the following steps performed for each of the original quantization levels: setting a modified quantization level corresponding to said original quantization level to the supplementary quantization level, when the adjustment distance is smaller than an adjustment threshold, or otherwise setting a modified quantization level to said original quantization level” does not integrate the judicial exception into a practical application. This element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
Step 2B:
“ wherein the determining of the modified quantization levels includes the following steps performed for each of the original quantization levels: setting a modified quantization level corresponding to said original quantization level to the supplementary quantization level, when the adjustment distance is smaller than an adjustment threshold, or otherwise setting a modified quantization level to said original quantization level” does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
In regard to claim 14: (Original)
Regarding claim 14:
Step 2A Prong 1:
“ decoding from the bitstream at least one of: the predetermined constant, the adjustment threshold, or the supplementary levels” ” is a mental step of data translation.
Step 2A Prong 2: no additional elements
Step 2B: no additional elements
In regard to claim 15: (Currently Amended)
Step 2A Prong 1:
“a decoder for decoding data output from a layer of a neural network from a network” is a mental step data calculation that can be performed as process of steps skin the way human can process based on the received data.
“ obtain original quantization levels with which the data were quantized; obtain one or more supplementary quantization levels corresponding to a respective one or more of the original quantization levels” is a mental step of data translation
“ and determine modified quantization levels by, for each modified quantization level, adopting either an original quantization level or the corresponding supplementary quantization level, depending on the adjustment distance” is a mental step of data translation
Step 2A Prong 2:
“ the decoder comprising” does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
“ compute an adjustment distance between the original quantization levels and the corresponding supplementary quantization levels” does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
“ inverse quantizer configured to obtain, from codewords of the bitstream, the data for the neural network based on the obtained modified quantization levels, and neural network circuitry configured to process the obtained data for the neural network by at least one layer of the neural network” does not integrate the judicial exception into a practical application. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
Step 2B:
“ the decoder comprising” does not amount to significantly more than the judicial exception in the claim. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
“ compute an adjustment distance between the original quantization levels and the corresponding supplementary quantization levels” does not amount to significantly more than the judicial exception in the claim. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
“ inverse quantizer configured to obtain, from codewords of the bitstream, the data for the neural network based on the obtained modified quantization levels, and neural network circuitry configured to process the obtained data for the neural network by at least one layer of the neural network” does not amount to significantly more than the judicial exception in the claim. This additional element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
In regard to claim 16 : (Original)
Step 2A Prong 2:
“ wherein the decoder is a cloud-based system” does not integrate the judicial exception into a practical application. This element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
Step 2B:
“ wherein the decoder is a cloud-based system” does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
In regard to claim 17: (Original)
Step 2A Prong 1:
“ include at least one of a minimum clipping value to which all lower data values are clipped, a maximum clipping value to which all higher data values are clipped, or a zero value” is a mental step data calculation using intersection points.
Step 2A Prong 2:
“ wherein the one or more supplementary quantization levels” does not integrate the judicial exception into a practical application. This element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
Step 2B:
“ wherein the one or more supplementary quantization levels” does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
In regard to claim 18: (Original)
Step 2A Prong 1:
“ wherein the adjustment distance includes a difference between the original and the corresponding supplementary quantization levels and an offset by a predetermined constant. “ is a mental step of data compare.
Step 2A Prong 2: no additional elements
Step 2B: no additional elements
In regard to claim 19 : (Original)
Step 2A Prong 2:
“ wherein the quantization adjustment circuitry is configured to perform the following steps for each of the original quantization levels:” does not integrate the judicial exception into a practical application. This element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
“ setting a modified quantization level corresponding to said original quantization level to the supplementary quantization level, when the adjustment distance is smaller than an adjustment threshold, or otherwise setting a modified quantization level to said original quantization level” does not integrate the judicial exception into a practical application. This element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
Step 2B:
“ wherein the quantization adjustment circuitry is configured to perform the following steps for each of the original quantization levels:” does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
“ setting a modified quantization level corresponding to said original quantization level to the supplementary quantization level, when the adjustment distance is smaller than an adjustment threshold, or otherwise setting a modified quantization level to said original quantization level” does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
In regard to claim 20: (Original)
Step 2A Prong 1:
“decode from the bitstream at least one of: the predetermined constant, the adjustment threshold, or the supplementary levels” is a mental step of data translation.
Step 2A Prong 2:
“ wherein the decoder is further configured to “does not integrate the judicial exception into a practical application. This element is merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(h).
Step 2B:
“ wherein the decoder is further configured to “does not amount to significantly more than the judicial exception in the claim. The additional element of “a computer system” is merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(h)).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4 and 6-8 are rejected under 35 U.S.C. 103 as being unpatentable over
Soroosh Khoram et.al (hereinafter Khoram) ADAPTIVE QUANTIZATION OF NEURAL NETWORKS, ICLR 2018.
In view of Abdulkadir Utku Diril et.al (hereinafter Diril) US 2019/0171927 A1.
In regard to claim 1: (Currently Amended)
Khoram discloses:
- A method for modifying quantization levels of a quantizer for a neural network, the method comprising the steps of:
In [Introduction, Page 2]:
The state-of-the-art neural network compression techniques include weight quantization,
In [Introduction, Page 2]:
For instance, weight quantization has previously shown good accuracy with fixed-point 16-bit and 8-bit precisions,
In [Introduction, Page 2]:
Recent works attempt to push that even further towards reduced precision and have trained models with 4-bit, 2-bit, and 1-bit parameters using quantized training methods
In [Abstract, Page 1]:
adaptive quantization, which simplifies a trained DNN model by finding a unique, optimal precision for each network parameter such that the increase in loss is minimized. The optimization problem at the core of this method iteratively uses the loss function gradient to determine an error margin for each parameter and assigns it a precision accordingly.
- determine modified quantization levels to include: the at least one non-modifiable quantization level;
In [2.1 , Page 4]:
When computing the output of a quantized layer, then, we multiply the result by r to account for the scale.
In [2.1, Page 2]:
We minimize the aggregate bit-widths of all network parameters while monitoring the training loss function. Due to the quantization noise, this function deviates from its optimum which was achieved through training. This noise effect can be controlled by introducing an upper bound on the loss function in order to maintain it reasonably close to the optimum. Consequently, the solution of this minimization problem represents critical parameters with high precision to sustain high accuracy and
In [2.1, Page 3]:
assigns low precisions to ineffectual ones or prunes them. The problem described here can be defined in formal terms as below.
PNG
media_image1.png
122
815
media_image1.png
Greyscale
Here, n is the number of model parameters, W =
[
w
1
,
…
w
n
]
T
is a vector of all model parameters,
w
i
is the i-th model parameter, and the function
N
q
(
w
i
) is the minimum number of bits required to represent
w
i
in its fixed-point format. As a result,
N
q
(
W
)
is the total number of bits required to represent the model. In addition, L(W) is the loss function value of the model W over the training set (or a minibatch of the training set). Finally,
l
-
≥ L(
W
0
)) is a constant upper bound on the loss of the neural network, which is used to bound the accuracy loss, and
W
0
is the vector of initial model parameters before quantization
The optimization problem presented in Equations 1 and 2 is difficult to solve because of its
non-smooth objective function. However, a smooth upper limit can be found for it. In
lemma below we derive such a bound.
In [2.1, Page 3]:
PNG
media_image2.png
316
928
media_image2.png
Greyscale
In [2.1, Page 3]:
The problem of Equation 3 can be easily solved using Algorithm 1, which simply checks all possible solutions one-by-one. Here, we allocate at most 32 bits to each parameter. This should be more than enough as previous works have demonstrated that typical DNNs can be easily quantized to even 8 bits without a significant loss of accuracy
PNG
media_image3.png
262
841
media_image3.png
Greyscale
In [2.1, Page 5]:
Algorithm 2 summarizes the proposed method of solving the original optimization problem (Equation 1). This algorithm first initializes the initial trust region radius
∆
0
and the loss function bound
l
-
. It also quantizes the input floating-point parameters W0 with 32 bits to generate the initial quantized parameters
W
0
. Here the function Quantize(.,.) refers to a pass of Algorithm 1 over all parameters. Thus, using 0 tolerance results in quantization with 32 bits.
(BRI: In the context of quantization, "zero tolerance" typically refers to the zero-point, a parameter that guarantees that the floating-point value 0.0 is represented exactly by an integer in the quantized range)
- and at least one level obtained by adjusting the at least one modifiable quantization level based on a cost function indicative of a distortion between the obtained data values and the preliminary quantization levels,
In [Abstract, Page 1]:
The optimization problem at the core of this method iteratively uses the loss function gradient to determine an error margin for each parameter and assigns it a precision accordingly
In [2.1, Page 1]:
We minimize the aggregate bit-widths of all network parameters while monitoring the training loss function. Due to the quantization noise, this function deviates from its optimum which was achieved through training. This noise effect can be controlled by introducing an upper bound on the loss function in order to maintain it reasonably close to the optimum.
(BRI: the distortion is the accuracy loss)
Khoram does not explicitly disclose:
- obtaining data values and preliminary quantization levels attributed to the data values, the preliminary quantization levels including at least one non-modifiable quantization level and at least one modifiable quantization level;
- wherein the at least one non-modifiable quantization level includes a minimum clipping value (cmin) to which all lower data values are clipped or a maximum clipping value (cmax) to which all higher data values are clipped.
However, Diril discloses:
- obtaining data values and preliminary quantization levels attributed to the data values, the preliminary quantization levels including at least one non-modifiable quantization level and at least one modifiable quantization level;
In [0051]:
The systems and methods described herein may quantize any number represented by a particular bit width to a number represented by a narrower bit width.
In [0051]:
processor 814 may quantize a single-precision floating point number (e.g., a 32-bit wide number with one sign bit, eight exponent bits, and 23 fraction bits, as represented by single-precision floating point number 510 in FIG. 5A) to a half-precision floating point number
in [0051]:
to an eight-bit unsigned integer (e.g., integer 610 in FIG. 6A), to an eight-bit signed integer (e.g., a number represented by a single sign bit and seven significand bits, such as integer 620 in FIG. 6B),
(BRI: the bit width initial is the preliminary quantization level such as float 32 which is a non-modifiable quantization level)
In [0024]:
The present disclosure is generally directed to implementing layer-level quantization within neural networks by dynamically adjusting (e.g., for a particular dataset or group of datasets) quantization parameters for network layers. Embodiments of the instant disclosure may, while performing inference (and/or training) on a dataset, identify minimum and maximum values for activation layers (i.e., hidden or intermediate layers) of a neural network and then update scaling factors for the layers based on the identified values. For example, a layer-level quantization system (e.g., a system with quantization that is guided by data profiling)
(BRI: updating scaling factors during inference is a form of adaptive quantization (dynamic) that, determines the appropriate scaling factors on-the-fly based on the range of the incoming data)
- the data values being outputs from a layer of the neural network
In [0024]:
The present disclosure is generally directed to implementing layer-level quantization within neural networks by dynamically adjusting (e.g., for a particular dataset or group of datasets) quantization parameters for network layers
(BRI: the output data value is the quantization level within a “Quantization parameter”).
In [0024]:
For example, a layer-level quantization system (e.g., a system with quantization that is guided by data profiling) may include a hardware accelerator that tracks the minimum and maximum output values for activation layers and analyzes the output values to determine how (or whether) to adjust a quantization scaling factor.
- wherein the at least one non-modifiable quantization level includes a minimum clipping value (cmin) to which all lower data values are clipped or a maximum clipping value (cmax) to which all higher data values are clipped.
In [0024]:
For example, a layer-level quantization system (e.g., a system with quantization that is guided by data profiling) may include a hardware accelerator that tracks the minimum and maximum output values for activation layers and analyzes the output values to determine how (or whether) to adjust a quantization scaling factor.
In [0052]:
processor 814 may be configured to dynamically update the scaling factor by, for example, performing the steps of method 400 for every inference (or at any interval of inferences, as noted above) of an activation layer. In some examples, processing unit 765 may compare the current minimum and maximum values and replace one or both with new respective values if a difference between the respective old and new values is greater than a threshold.
In [0046]:
The minimum and maximum values may span all or most of the dynamic values of the activation layer, and the scaling factor may be used to scale numbers between the minimum and maximum values linearly (e.g., in fixed quantization intervals) or non-linearly (e.g., in variable quantization intervals, such as logarithmic intervals) down to a smaller range, thereby quantizing a range of data to a range that can be represented by within a bit width,
In [0046]:
The quantization scheme for determining the scaling factor may be designed to preserve as much accuracy as possible while reducing the bit width to a predetermined size or an optimal size for a dataset.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Khoram and Diril.
Khoram teaches quantization and cost function (error function)
Diril teaches clipping.
One of ordinary skill would have motivation to combine Khoram and Diril that can provide reduced bit-width and achieve an optimized quantization [(Diril [0074]).
In regard to claim 2: (Original)
Khoram does not explicitly disclose:
- wherein there are at least two non-modifiable quantization levels, including the minimum clipping value (cmin) and the maximum clipping value (cmax).
However, Diril discloses:
- wherein the at least two non-modifiable quantization level, including the minimum clipping value (cmin) and the maximum clipping value (cmax).
In [0051]:
quantize a single-precision floating point number (e.g., a 32-bit wide number with one sign bit, eight exponent bits, and 23 fraction bits, as represented by single-precision floating point number 510 in FIG. 5A) to a half-precision floating point number (e.g., a number represented by a sign bit, five exponent bits, and 10 fraction bits, such as half-precision floating-point number 520 in FIG. 5B),
in [0051]:
to an eight-bit unsigned integer (e.g., integer 610 in FIG. 6A.
in [0052]:
layer-level may observe that the minimum and maximum values have not changed outside a predetermined range over a predetermined amount of time and may therefore determine that the scaling factor no longer needs to be adjusted. This determination may, for example, be made on a per-layer basis or simultaneously for all layers within a neural network.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Khoram and Diril.
Khoram teaches quantization and cost function (error function)
Diril teaches clipping.
One of ordinary skill would have motivation to combine Khoram and Diril that can provide reduced bit-width and achieve an optimized quantization [(Diril [0074]).
In regard to claim 3: (Original)
Khoram discloses:
- wherein there are at least two non-modifiable quantization levels, wherein among the at least two non-modifiable quantization levels, one has the value of zero.
In [2.1, Page 3]:
assigns low precisions to ineffectual ones or prunes them. The problem described here can be defined in formal terms as below.
PNG
media_image1.png
122
815
media_image1.png
Greyscale
Here, n is the number of model parameters, W =
[
w
1
,
…
w
n
]
T
is a vector of all model parameters,
w
i
is the i-th model parameter, and the function
N
q
(
w
i
) is the minimum number of bits required to represent
w
i
in its fixed-point format.
(BRI: the adaptation provides at least to on modifiable quantization levels (
w
i
)
In [2.1, Page 3]:
The problem of Equation 3 can be easily solved using Algorithm 1, which simply checks all possible solutions one-by-one. Here, we allocate at most 32 bits to each parameter. This should be more than enough as previous works have demonstrated that typical DNNs can be easily quantized to even 8 bits without a significant loss of accuracy
PNG
media_image3.png
262
841
media_image3.png
Greyscale
In [2.1, Page 5]:
Algorithm 2 summarizes the proposed method of solving the original optimization problem (Equation 1). This algorithm first initializes the initial trust region radius
∆
0
and the loss function bound
l
-
. It also quantizes the input floating-point parameters W0 with 32 bits to generate the initial quantized parameters
W
0
. Here the function Quantize(.,.) refers to a pass of Algorithm 1 over all parameters. Thus, using 0 tolerance results in quantization with 32 bits.
(BRI: In the context of quantization, "zero tolerance" typically refers to the zero-point, a parameter that guarantees that the floating-point value 0.0 is represented exactly by an integer in the quantized range)
In regard to claim 4: (Original)
Khoram does not explicitly disclose:
- further comprising computing a level as a predefined function of the data values assigned to the level.
However, Diril discloses:
- further comprising computing a level as a predefined function of the data values assigned to the level.
In [0045]:
At step 430, one or more of the systems described herein may store a second limit value of the activation layer in the data storage system. For instance, accelerator 700 may store the second limit value in register 790B or in any other part of a data storage subsystem. This second limit value may correspond to a maximum value for the activation layer, such as an absolute maximum weight or filter value (e.g., the highest value of an activation layer, which may be identified by passing output values through a min-max unit) or an estimated maximum weight or filter value (e.g., an approximate maximum that discards outliers, a maximum within a predetermined standard deviation of values for a particular layer, etc.). One of functional units 770 may be a processing element for determining the maximum value of the activation layer. In certain implementations, a single functional unit 770 may determine the minimum value and the maximum value.
In [0046]:
the minimum and maximum values may span all or most of the dynamic values of the activation layer, and the scaling factor may be used to scale numbers between the minimum and maximum values linearly (e.g., in fixed quantization intervals) or non-linearly (e.g., in variable quantization intervals, such as logarithmic intervals) down to a smaller range, thereby quantizing a range of data to a range that can be represented by within a bit width of the arithmetic operators of a system or subsequent layer.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Khoram and Diril.
Khoram teaches quantization and cost function (error function)
Diril teaches clipping.
One of ordinary skill would have motivation to combine Khoram and Diril that can provide reduced bit-width and achieve an optimized quantization [(Diril [0074]).
In regard to claim 6: (Original)
Khoram discloses:
- wherein the steps of obtaining the preliminary quantization levels and determining the modified quantization levels are iterated K times, K being larger than 1, a j-th iteration out of the K iterations comprises:
In [2.2, Page 5]:
Algorithm 2 summarizes the proposed method of solving the original optimization problem (Equation 1). This algorithm first initializes the initial trust region radius
∆
0
and the loss function bound
l
-
. It also quantizes the input floating-point parameters
W
0
with 32 bits to generate the initial quantized parameters
W
0
.
In [2.1, Page 3]:
The problem of Equation 3 can be easily solved using Algorithm 1, which simply checks all possible solutions one-by-one. Here, we allocate at most 32 bits to each parameter. This should be more than enough as previous works have demonstrated that typical DNNs can be easily quantized to even 8 bits without a significant loss of accuracy
PNG
media_image3.png
262
841
media_image3.png
Greyscale
(BRI: the algorithm 1 shows the iteration to adaptive quantization , in which the number of iterations K is greater than 1 as a result of initializing the parameters to “0”)
- obtaining the preliminary quantization levels corresponding to the modified quantization levels determined by the (j-1)-th iteration,
In [2.1, Page 3]:
PNG
media_image3.png
262
841
media_image3.png
Greyscale
(BRI: the modified quantization is line
w
i
q
←
round(.. ))
- determining the modified quantization levels to include the at least one non- modifiable quantization level and by modifying the at least one modifiable quantization level obtained by the (j-1)-the iteration based on the cost function.
In [2.2, Page 5]:
We use two measures to examine convergence of the solution: the loss function and the trust region radius. We declare convergence, if the loss function of the quantized model converges to the loss bound
l
-
or the trust region radius becomes very small, indicated by values
ε
and η, respectively. Note that the protocol to update the trust region radius as well as the trust region convergence condition used in this algorithm are commonly used in trust region methods.
In [2.2, Page 5]:
Subsequently, Algorithm 2 iteratively solves the subproblem of Equation 16 and calculates the quantized parameters
PNG
media_image4.png
538
946
media_image4.png
Greyscale
(BRI: loss bound
l
-
is the cost function and the iteration terminates on the meeting loss after quantization less than the bound)
In regard to claim 7: (Original)
Khoram discloses:
- including stopping the iterations after the number of iterations K, in response to: the value of the cost function in the K-th iteration being lower than a first threshold ,
In [2.1, Page 3]:
PNG
media_image2.png
316
928
media_image2.png
Greyscale
In [2.2, Page 4]:
We sidestep the first challenge by using the upper bound of the objective function, Φ(T). Particularly, we approximate the minimum of
N
q
W
by first finding the optimal T for Φ(T) and then calculating the quantized parameters using Algorithm 1. This optimization problem can be defined as: minimize Φ(T) such that if each
w
i
is perturbed by, at most,
τ
i
, the loss constraint of Equation 2 is not violated:
PNG
media_image5.png
103
796
media_image5.png
Greyscale
in [2.1, Page 3]:
PNG
media_image3.png
262
841
media_image3.png
Greyscale
(BRI: See line |quantized-original|
≤
τ
i
) as a criterion for stopping the iteration)
In regard to claim 8: (Currently Amended)
Khoram does not explicitly disclose:
- wherein the data values are: values of a feature map output by a layer of the neural network
However, Diril discloses:
- wherein the data values are: values of a feature map output by a layer of the neural network
In [0038]:
Scaling unit 250 may multiply an input value by a scaling factor (e.g., sf.sub.0) to quantize the input value to correspond to a bit width of operators within activation layer 214.
In [0036]:
In the example shown in FIG. 2A, data flows from input layer 202 thorough activation layers 212-220 to output layer 204 (i.e., from left to right). As shown, each value from the nodes of input layer 202 may be duplicated and sent to the nodes of activation layer 212. At activation layer 212, a set of weights (i.e., a filter) may be applied to the layer inputs, and each node may output a weighted sum that may be scaled
(BRI: loss bound
l
-
is the cost function (can be interpreted as threshold) and the iteration terminates (stops) on the meeting loss after quantization less than the bound)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Khoram and Diril.
Khoram teaches quantization and cost function (error function)
Diril teaches clipping.
One of ordinary skill would have motivation to combine Khoram and Diril that can provide reduced bit-width and achieve an optimized quantization [(Diril [0074]).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over
In view of Soroosh Khoram et.al (hereinafter Khoram) ADAPTIVE QUANTIZATION OF NEURAL NETWORKS, ICLR 2018,
In view of Abdulkadir Utku Diril et.al (hereinafter Diril) US 2019/0171927 A1,
further in view of Paul HAASE1 et.al(hereinafter HAASE1) US 2022/0393986 A1.
In regard to claim 5: (Original)
Khoram and Diril not explicitly disclose:
- wherein the cost function includes a linear combination of the distortion and a rate, the rate is represented by codeword length of codewords representing the respective preliminary quantization levels
- and the codewords representing at least two respective preliminary quantization levels having mutually different lengths.
However, HAASE 1 discloses:
- wherein the cost function includes a linear combination of the distortion and a rate, the rate is represented by codeword length of codewords representing the respective preliminary quantization levels
In [0119]:
Better results in rate-distortion sense can be obtained if the quantization process minimizes a Lagrangian function D+λ.R, where D represent the distortion (e.g., MSE distortion or MAE distortion) of the set of neural network parameters, R specifies the number of bits that are required for transmitting the quantization indexes 56, and λ is a Lagrange multiplier.
(BRI: A linear combination of distortion and rate is a cost function used in rate-distortion optimization (RDO). The rate is the number of bits used for encoding)
In [0189] :
in dependent quantization, the reconstruction order of neural network parameters 13 has to be uniquely defined. The performance of the overall neural network codec can typically be improved if the knowledge about the set of reconstruction levels associated with a quantization index q.sub.k 56 is also exploited in the entropy coding. That means, it is typically advantageous to switch contexts (probability models) or codeword tables based on the set of reconstruction levels that applies to a neural network parameter.
In [0134]:
a variable rem=X−|q| could be encoded using k bits. Or alternatively, a variable rem′ could be defined as rem′=(1«k)−rem−1 which is encoded using k bits. Any other mapping of the variable rem to a fixed length code of k bits may alternatively be used.
(BRI: a codeword table based on the set of reconstruction levels for a neural network parameter does represent a number of bits)
- and the codewords representing at least two respective preliminary quantization levels having mutually different lengths.
In [0131]:
the bin indicates if the current weight level is positive (e.g., bin=0) or negative (e.g., bin=1). In other words, the at least one bin of the binarization 82 of the quantization index 56 shown in FIG. 4 may comprise a sign bin 86 indicative of the quantization index 56 of the current neural network parameter being greater than zero or lower than zero.
(BRI: the binarization based on the parameter greater than zero or less than zero represents two preliminary quantization levels)
In[ 0132]:
Next, a unary sequence of bins is encoded
In [0134]:
a variable rem=X−|q| could be encoded using k bits. Or alternatively, a variable rem′ could be defined as rem′=(1«k)−rem−1 which is encoded using k bits. Any other mapping of the variable rem to a fixed length code of k bits may alternatively be used.
In [0139]:
Decoding of the quantized weight levels 56 (integer representation) works analogously to the encoding. The decoder first decodes the sig_flag. If it is equal to one, a sign_flag and a unary sequence of abs_level_greater_X follows, where the updates of k, (and thus increments of X) has to follow the same rule as in the encoder. Finally, the fixed length code of k bits is decoded and interpreted as integer number (e.g. as rem or rem′, depending on which of both was encoded).
(BRI: a "unary sequence of bins followed by a fixed-length sequence" is a form of variable-length coding used in data compression)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Khoram, Diril and HAASE1.
Khoram teaches quantization and cost function (error function)
Diril teaches clipping.
HAASE1 teaches codeword.
One of ordinary skill would have motivation to combine Khoram, Diril and HAASE1 that can improve the performance of the overall neural network codec by using set of reconstruction levels exploited in the entropy coding (HASSE1[0189]).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over
In view of Soroosh Khoram et.al (hereinafter Khoram) ADAPTIVE QUANTIZATION OF NEURAL NETWORKS, ICLR 2018,
In view of Abdulkadir Utku Diril et.al (hereinafter Diril) US 2019/0171927 A1,
further in view of in view of Raghuraman Krishnamoorthi (hereinafter Krish) Quantizing deep convolutional networks for efficient inference: A whitepaper, arXiv:1806.08342v1 [cs.LG] 21 Jun 2018.
In regard to claim 9: (Original)
Khoram discloses:
- adjusting the modified quantization levels by, for each modified quantization level, adopting either said modified quantization level or the quantization level before the modification, depending on the adjustment distance
In [2.1, Page 4]:
PNG
media_image2.png
316
928
media_image2.png
Greyscale
(BRI: |
w
i
q
-
w
i
| is the adjustment distance)
In [2.1, Page 4]:
Algorithm 1 estimates with a rounding error of up to
PNG
media_image6.png
52
62
media_image6.png
Greyscale
Thus, for the worst case to be consistent with the constraint of Equation 3, we need:
PNG
media_image7.png
55
676
media_image7.png
Greyscale
In this case the minimization guarantees that the smallest
N
q
(
w
i
)
is chosen. This entails:
PNG
media_image8.png
32
620
media_image8.png
Greyscale
In general, we can expect
N
q
(
w
i
) to be smaller than this worst case. Consequently
PNG
media_image9.png
70
671
media_image9.png
Greyscale
Khoram and Diril do not explicitly disclose:
- further comprising the steps of: computing an adjustment distance between the modified quantization levels and the quantization levels before the modification,
However, Krish discloses:
- further comprising the steps of: computing an adjustment distance between the modified quantization levels(NPL) and the quantization levels before the modification,
In [2.1, Page 4]:
Consider a floating point variable with range (
x
m
i
n
,
x
m
a
x
) that needs to be quantized to the range (0,
N
l
e
v
e
l
s
− 1) where
N
l
e
v
e
l
s
= 256 for 8-bits of precision. We derive two parameters: Scale (∆) and Zero-point(z) which map the floating point values to integers
In [2.1, Page 4]:
The scale specifies the step size of the quantizer and floating point zero maps to zero-point. Zero-point is an integer, ensuring that zero is quantized with no error. This is important to ensure that common operations like zero padding do not cause quantization error. For one sided distributions, therefore, the range
(
x
m
i
n
,
x
m
a
x
) is relaxed to include zero. For example, a floating point variable with the range (2.1,3.5) will be relaxed to 4 the range (0, 3.5) and then quantized. Note that this can cause a loss of precision in the case of extreme one-sided distribution
Once the scale and zero-point are defined, quantization proceeds as follows:
PNG
media_image10.png
90
645
media_image10.png
Greyscale
where
PNG
media_image11.png
95
531
media_image11.png
Greyscale
The de-quantization operation is :
PNG
media_image12.png
32
732
media_image12.png
Greyscale
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Khoram, Diril and Krish.
Khoram teaches quantization and cost function (error function)
Diril teaches clipping.
Krish teaches computing an adjustment distance.
One of ordinary skill would have motivation to combine Khoram, Diril, and Krish
that can provide an improved accuracy over the post training at 8 bits of precision [(Krish [3.2, Page 13]).
Claims 10-20 are rejected under 35 U.S.C. 103 as being unpatentable over
Soroosh Khoram et.al (hereinafter Khoram) ADAPTIVE QUANTIZATION OF NEURAL NETWORKS, ICLR 2018.
in view of Jie CHEN et.al (hereinafter CHEN) US 2021/0235085 A1.
[Foreign Priority: CN 201910087815.0 Filed: 2019-01-29]
in view of Paul HAASE et.al(hereinafter HAASE1) US 2022/0393986 A1.
In regard to claim 10: (Currently Amended)
Khoram discloses:
- obtaining one or more supplementary quantization levels corresponding to a respective one or more of the original quantization levels
In [2.1, Page 2]:
We minimize the aggregate bit-widths of all network parameters while monitoring the training loss function. Due to the quantization noise, this function deviates from its optimum which was achieved through training. This noise effect can be controlled by introducing an upper bound on the loss function in order to maintain it reasonably close to the optimum. Consequently, the solution of this minimization problem represents critical parameters with high precision to sustain high accuracy
In [2.1, Page 3]:
and assigns low precisions to ineffectual ones or prunes them. The problem described here can be defined in formal terms as below.
PNG
media_image13.png
131
850
media_image13.png
Greyscale
Here, n is the number of model parameters, W =
w
1
…
w
n
T
is a vector of all model parameters,
w
i
is the i-th model parameter, and the function
N
q
(
w
i
)
is the minimum number of bits required to represent
w
i
in its fixed-point format. As a result,
N
Q
(W) is the total number of bits required to represent the model. In addition, L(W) is the loss function value of the model W over the training set (or a minibatch of the training set). Finally,
PNG
media_image14.png
32
132
media_image14.png
Greyscale
is a constant upper bound on the loss of the neural network, which is used to bound the accuracy loss, and
W
0
is the vector of initial model parameters before quantization.
In [2.2, Page 5]:
We note that we did not make any assumptions regarding the point
W
0
and the result can be extended to any point in
R
n
Consequently, if we define such a trust region, we can simply use the following problem as a subproblem that we repeatedly solve in successive iterations.
PNG
media_image15.png
138
728
media_image15.png
Greyscale
Algorithm 2 summarizes the proposed method of solving the original optimization problem (Equation 1).
PNG
media_image16.png
681
1101
media_image16.png
Greyscale
In [2.1, Page 4]:
Consider a floating point variable with range (
x
m
i
n
,
x
m
a
x
) that needs to be quantized to the range (0,
N
l
e
v
e
l
s
− 1) where
N
l
e
v
e
l
s
= 256 for 8-bits of precision. We derive two parameters: Scale (∆) and Zero-point(z) which map the floating point values to integers
In [2.1, Page 4]:
The scale specifies the step size of the quantizer and floating point zero maps to zero-point. Zero-point is an integer, ensuring that zero is quantized with no error. This is important to ensure that common operations like zero padding do not cause quantization error. For one sided distributions, therefore, the range
(
x
m
i
n
,
x
m
a
x
) is relaxed to include zero. For example, a floating point variable with the range (2.1,3.5) will be relaxed to 4 the range (0, 3.5) and then quantized. Note that this can cause a loss of precision in the case of extreme one-sided distribution
Once the scale and zero-point are defined, quantization proceeds as follows:
PNG
media_image10.png
90
645
media_image10.png
Greyscale
where
PNG
media_image11.png
95
531
media_image11.png
Greyscale
The de-quantization operation is :
PNG
media_image12.png
32
732
media_image12.png
Greyscale
(BRI: the zero-point is the constant and combining clamping with quantization using an integer offset can provide a form of "adjustment distance" that allows shifting the entire quantization range. The supplementary quantization level directly relates to quantization steps. Smaller step size results in more quantization levels and lowers quantization error and larger size results in lower supplementary quantization levels. Using an adaptive quantization, this can be selected (if and then in Algorithm 2))
- determining modified quantization levels by, for each modified quantization level, adopting either an original quantization level or the corresponding supplementary quantization level, depending on the adjustment distance
In [2.1, Page 3]:
PNG
media_image2.png
316
928
media_image2.png
Greyscale
(BRI: |
w
i
q
-
w
i
| is the adjustment distance)
In [2.1, Page 4]:
Algorithm 1 estimates with a rounding error of up to
PNG
media_image6.png
52
62
media_image6.png
Greyscale
Thus, for the worst case to be consistent with the constraint of Equation 3, we need:
PNG
media_image7.png
55
676
media_image7.png
Greyscale
In this case the minimization guarantees that the smallest
N
q
(
w
i
)
is chosen. This entails:
PNG
media_image8.png
32
620
media_image8.png
Greyscale
In general, we can expect
N
q
(
w
i
) to be smaller than this worst case. Consequently
PNG
media_image9.png
70
671
media_image9.png
Greyscale
- computing an adjustment distance between the original quantization levels and the corresponding supplementary quantization levels,
In [2.1, Page 2]:
We minimize the aggregate bit-widths of all network parameters while monitoring the training loss function. Due to the quantization noise, this function deviates from its optimum which was achieved through training. This noise effect can be controlled by introducing an upper bound on the loss function in order to maintain it reasonably close to the optimum. Consequently, the solution of this minimization problem represents critical parameters with high precision to sustain high accuracy
In [2.1, Page 3]:
and assigns low precisions to ineffectual ones or prunes them. The problem described here can be defined in formal terms as below.
PNG
media_image13.png
131
850
media_image13.png
Greyscale
Here, n is the number of model parameters, W =
w
1
…
w
n
T
is a vector of all model parameters,
w
i
is the i-th model parameter, and the function
N
q
(
w
i
)
is the minimum number of bits required to represent
w
i
in its fixed-point format. As a result,
N
Q
(W) is the total number of bits required to represent the model. In addition, L(W) is the loss function value of the model W over the training set (or a minibatch of the training set). Finally,
PNG
media_image14.png
32
132
media_image14.png
Greyscale
is a constant upper bound on the loss of the neural network, which is used to bound the accuracy loss, and
W
0
is the vector of initial model parameters before quantization.
In [2.1, Page 3]:
PNG
media_image2.png
316
928
media_image2.png
Greyscale
(BRI: |
w
i
q
-
w
i
| is the adjustment distance)
In [2.1, Page 4]:
Consider a floating point variable with range (
x
m
i
n
,
x
m
a
x
) that needs to be quantized to the range (0,
N
l
e
v
e
l
s
− 1) where
N
l
e
v
e
l
s
= 256 for 8-bits of precision. We derive two parameters: Scale (∆) and Zero-point(z) which map the floating point values to integers
In [2.1, Page 4]:
The scale specifies the step size of the quantizer and floating point zero maps to zero-point. Zero-point is an integer, ensuring that zero is quantized with no error. This is important to ensure that common operations like zero padding do not cause quantization error. For one sided distributions, therefore, the range
(
x
m
i
n
,
x
m
a
x
) is relaxed to include zero. For example, a floating point variable with the range (2.1,3.5) will be relaxed to 4 the range (0, 3.5) and then quantized. Note that this can cause a loss of precision in the case of extreme one-sided distribution
Once the scale and zero-point are defined, quantization proceeds as follows:
PNG
media_image10.png
90
645
media_image10.png
Greyscale
where
PNG
media_image11.png
95
531
media_image11.png
Greyscale
The de-quantization operation is :
PNG
media_image12.png
32
732
media_image12.png
Greyscale
- determining modified quantization levels by, for each modified quantization level, adopting either an original quantization level or the corresponding supplementary quantization level, depending on the adjustment distance
In [2.1, Page 3]:
PNG
media_image2.png
316
928
media_image2.png
Greyscale
(BRI: |
w
i
q
-
w
i
| is the adjustment distance)
In [2.1, Page 4]:
Algorithm 1 estimates with a rounding error of up to
PNG
media_image6.png
52
62
media_image6.png
Greyscale
Thus, for the worst case to be consistent with the constraint of Equation 3, we need:
PNG
media_image7.png
55
676
media_image7.png
Greyscale
In this case the minimization guarantees that the smallest
N
q
(
w
i
)
is chosen. This entails:
PNG
media_image8.png
32
620
media_image8.png
Greyscale
In general, we can expect
N
q
(
w
i
) to be smaller than this worst case. Consequently
PNG
media_image9.png
70
671
media_image9.png
Greyscale
Khoram does not explicitly disclose:
- A method for decoding data output from a layer of a neural network from a bitstream, the method including: obtaining original quantization levels with which the data were quantized
- and process the obtained data for the neural network by at least one layer of the neural network.
However, Chen discloses:
- A method for decoding data output from a layer of a neural network from a bitstream, the method including: obtaining original quantization levels with which the data were quantized
In [0020]:
FIG. 4a is a schematic diagram of an existing image decoder and encoder,
In [0063]:
encode the raw image to be processed to generate a compressed bit-stream for transmission. After receiving the compressed bit-stream, the cloud server decodes it and performs corresponding processing on the decoded and reconstructed raw image.
In [0074]:
the encoding parameters include the quantization parameters or other parameters. In the practical application, the first model and the second model are deep learning network models, for example, multilayer convolutional neural network. Particularly, the first model includes a first deep learning model and the second model includes a second deep learning model,
in [0074]:
Wherein, the encoding parameters include the quantization parameters or other parameters. In the practical application, the first model and the second model are deep learning network models, for example, multilayer convolutional neural network. Particularly, the first model includes a first deep learning model and the second model includes a second deep learning model,
- and process the obtained data for the neural network by at least one layer of the neural network.
In [0079]:
3) a training data layer in which the training data is a set of feature values of each coding unit in the processed raw image, wherein when a model is trained in real time, the processed raw image is a processed raw image previous to the raw image to be processed,
In [0082]:training process:
data or information obtained in the above hierarchies is the input for training to obtain the first deep learning model or the second deep learning model; and
[0083]: application process:
data or information obtained in the above hierarchies is the input to the trained deep learning model and the related coding information is the output.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Khoram, and Chen.
Khoram teaches adaptive quantization that includes quantization levels, modified quantization levels and adjustment distance.
Chen teaches decoding data output from a layer.
One of ordinary skill would have motivation to combine Khoram ,and Chen that can be further optimized by the semantic module to reduce the perceptual distortion caused by the image processing [(CHEN [0065]).
Khoram and Chen do not explicitly disclose:
- obtaining, from codewords of the bitstream, the data for the neural network based on the obtained modified quantization levels
However, HASSE1 discloses:
- obtaining, from codewords of the bitstream,
In [0117]:
Standards for compression of neural networks only specify the bitstream syntax and the reconstruction process. If we consider parameter coding for a given set of original neural network parameters 13 and given quantization step sizes (QP), the encoder has a lot a freedom. Given the quantization indexes q.sub.k 56 for a layer 10a, 10b, the entropy coding has to follow a uniquely defined algorithm for writing the data to the bitstream 14 (i.e., constructing the arithmetic codeword).
- the data for the neural network based on the obtained modified quantization levels
In [0184]:
the concept of dependent scalar quantization is combined with a modified entropy coding, in which the probability model selection (or, alternatively, the codeword table selection) for a neural network parameter depends on the set of admissible reconstruction level,
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Khoram, Chen and HASSE1.
Khoram teaches adaptive quantization that includes quantization levels, modified quantization levels and adjustment distance.
Chen teaches decoding data output from a layer.
HASSE1 teaches codeword from bitstream.
One of ordinary skill would have motivation to combine Khoram , Chen and HASSE1 that can
bitrate is improved compared to using a constant quantizer (HASSE1[0056]).
In regard to claim 11: (Original)
Khoram does not explicitly disclose:
- wherein the one or more supplementary quantization levels include at least one of a minimum clipping value to which all lower data values are clipped, a maximum clipping value to which all higher data values are clipped, or a zero value.
However, Chen discloses:
- wherein the one or more supplementary quantization levels include at least one of a minimum clipping value to which all lower data values are clipped, a maximum clipping value.
In [0234]:
clip the calculated value of the quantization parameter to the effective range, and the effective range in this embodiment is greater than or equal to 0, and less than or equal to 51. The current coding unit to be processed is encoded according to the quantization parameter.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Khoram, and Chen.
Khoram teaches adaptive quantization that includes quantization levels, modified quantization levels and adjustment distance.
Chen teaches decoding data output from a layer.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Khoram, and Chen.
Khoram teaches adaptive quantization that includes quantization levels, modified quantization levels and adjustment distance.
Chen teaches decoding data output from a layer.
One of ordinary skill would have motivation to combine Khoram ,and Chen that can be further optimized by the semantic module to reduce the perceptual distortion caused by the image processing [(CHEN [0065]).
In regard to claim 12: (Original)
Khoram discloses:
- wherein the adjustment distance includes a difference between the original and the corresponding supplementary quantization levels and an offset by a predetermined constant.
In [2.1, Page 4]:
For models with parameters in a range larger than [−1,1], say [−r, r] with r > 1, we perform the quantization similarly. The only difference is that we first scale the parameters and their tolerances by 1/r to bring them to [−1, 1] range and then quantize the parameters. When computing the output of a quantized layer, then, we multiply the result by r to account for the scale. Following this, the same equations as above will hold.
(BRI: parameters outside ([-1, 1]) are first scaled (along with their tolerances) into ([-1, 1]), quantized there, and then scaled back. The offset in such schemes represents the adjustment needed to recover the original parameter from its quantized form. The scaled factor r is the offset)
In regard to claim 13: (Original)
Khoram discloses:
- wherein the determining of the modified quantization levels includes the following steps performed for each of the original quantization levels:
setting a modified quantization level corresponding to said original quantization level to the supplementary quantization level, when the adjustment distance is smaller than an adjustment threshold,
in [2.1, Page 3]:
PNG
media_image3.png
262
841
media_image3.png
Greyscale
(BRI: |
w
i
q
-
w
i
| is the adjustment distance)
(BRI: Based on the interpretation as perhaps known to the art that a threshold for adjustment distance determines the maximum deviation an input value can have from an ideal quantized value, the algorithm 1 the tolerance is the “threshold” and compared to quantized to the difference of quantized and from ideal quantized value, the iteration stops when it is lower than the threshold and sets the modified level to the original (see
w
i
q
←
r
o
u
n
d
.
.
,
otherwise continues for adjusting)
In regard to claim 14: (Original)
Khoram, and Chen do not explicitly disclose:
- further comprising decoding from the bitstream at least one of: the predetermined constant, the adjustment threshold, or the supplementary levels.
However, HAASE1 discloses:
- further comprising decoding from the bitstream at least one of: the predetermined constant, the adjustment threshold, or the supplementary levels.
In [0027]:
neural networks, e.g. defined by neural network parameters, may be compressed and/or transmitted efficiently, e.g. with a low amount of data in a bitstream, using reconstruction-layers, for example sublayers, such as base-layers and enhancement-layers. The reconstruction layers may be defined, such that the neural network parameters are reconstructible by, for each neural network parameter, combining the first-reconstruction-layer neural network parameter value and the second-reconstruction-layer neural network parameter value. This distribution enables an efficient coding, e.g. encoding and/or decoding, and/or transmission of the neural network parameters.
in [0139]:
Decoding of the quantized weight levels 56 (integer representation) works analogously to the encoding. The decoder first decodes the sig_flag. If it is equal to one, a sign_flag and a unary sequence of abs_level_greater_X follows, where the updates of k, (and thus increments of X) has to follow the same rule as in the encoder. Finally, the fixed length code of k bits is decoded and interpreted as integer number (e.g. as rem or rem′, depending on which of both was encoded).
In [0060]:
the apparatus may be configured to parametrize 60 (parametrization) the plurality 50 of reconstruction level sets 52 (e.g., set 0, set 1) by way of a predetermined quantization step size (QP), for example denoted by Δ or Δk, and derive information on the predetermined quantization step size from the data stream 14. Therefore, a decoder according to embodiments may adapt to a variable step size (QP).
In [0139]:
Finally, the quantized weight w is reconstructed by multiplying the quantized weight level q with the step size Δ (QP).
(BRI: decoding fixed codeword length represents “predetermined constant” that the decoder uses to determine how many bits to read for each symbol. This constant is part of the decoding algorithm’s specification and is not derived from the bit stream itself)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Khoram, Chen and HASSE1.
Khoram teaches adaptive quantization that includes quantization levels, modified quantization levels and adjustment distance.
Chen teaches decoding data output from a layer.
HASSE1 teaches codeword from bitstream.
One of ordinary skill would have motivation to combine Khoram , Chen and HASSE1 that can
bitrate is improved compared to using a constant quantizer (HASSE1[0056]).
In regard to claim 15: (Currently Amended)
Khoram discloses:
- quantization adjustment circuitry configured to:
In [Abstract]:
In [3.2, Page 8]:
The key contributors to the computational load of the proposed technique are back propagation (to calculate gradients) and quantization (Algorithm 2).
In [3.2, Page 8]:
We implement the proposed quantization on Intel Core i7 CPU (3.5 GHz) with Titan X GPU performing training and quantization.
- obtain original quantization levels with which the data were quantized;
In [2.1, Page 2]:
We minimize the aggregate bit-widths of all network parameters while monitoring the training loss function. Due to the quantization noise, this function deviates from its optimum which was achieved through training. This noise effect can be controlled by introducing an upper bound on the loss function in order to maintain it reasonably close to the optimum. Consequently, the solution of this minimization problem represents critical parameters with high precision to sustain high accuracy
In [2.1, Page 3]:
and assigns low precisions to ineffectual ones or prunes them. The problem described here can be defined in formal terms as below.
PNG
media_image13.png
131
850
media_image13.png
Greyscale
Here, n is the number of model parameters, W =
w
1
…
w
n
T
is a vector of all model parameters,
w
i
is the i-th model parameter, and the function
N
q
(
w
i
)
is the minimum number of bits required to represent
w
i
in its fixed-point format. As a result,
N
Q
(W) is the total number of bits required to represent the model. In addition, L(W) is the loss function value of the model W over the training set (or a minibatch of the training set). Finally,
PNG
media_image14.png
32
132
media_image14.png
Greyscale
is a constant upper bound on the loss of the neural network, which is used to bound the accuracy loss, and
W
0
is the vector of initial model parameters before quantization.
In [2.2, Page 5]:
We note that we did not make any assumptions regarding the point
W
0
and the result can be extended to any point in
R
n
Consequently, if we define such a trust region, we can simply use the following problem as a subproblem that we repeatedly solve in successive iterations.
PNG
media_image15.png
138
728
media_image15.png
Greyscale
Algorithm 2 summarizes the proposed method of solving the original optimization problem (Equation 1).
PNG
media_image16.png
681
1101
media_image16.png
Greyscale
In [2.1, Page 4]:
Consider a floating point variable with range (
x
m
i
n
,
x
m
a
x
) that needs to be quantized to the range (0,
N
l
e
v
e
l
s
− 1) where
N
l
e
v
e
l
s
= 256 for 8-bits of precision. We derive two parameters: Scale (∆) and Zero-point(z) which map the floating point values to integers
In [2.1, Page 4]:
The scale specifies the step size of the quantizer and floating point zero maps to zero-point. Zero-point is an integer, ensuring that zero is quantized with no error. This is important to ensure that common operations like zero padding do not cause quantization error. For one sided distributions, therefore, the range
(
x
m
i
n
,
x
m
a
x
) is relaxed to include zero. For example, a floating point variable with the range (2.1,3.5) will be relaxed to 4 the range (0, 3.5) and then quantized. Note that this can cause a loss of precision in the case of extreme one-sided distribution
Once the scale and zero-point are defined, quantization proceeds as follows:
PNG
media_image10.png
90
645
media_image10.png
Greyscale
where
PNG
media_image11.png
95
531
media_image11.png
Greyscale
The de-quantization operation is :
PNG
media_image12.png
32
732
media_image12.png
Greyscale
(BRI: the zero-point is the constant and combining clamping with quantization using an integer offset can provide a form of "adjustment distance" that allows shifting the entire quantization range. The supplementary quantization level directly relates to quantization steps. Smaller step size results in more quantization levels and lowers quantization error and larger size results in lower supplementary quantization levels. Using an adaptive quantization, this can be selected (if and then in Algorithm 2))
- obtain one or more supplementary quantization levels corresponding to a respective one or more of the original quantization levels;
In [2.1, Page 2]:
We minimize the aggregate bit-widths of all network parameters while monitoring the training loss function. Due to the quantization noise, this function deviates from its optimum which was achieved through training. This noise effect can be controlled by introducing an upper bound on the loss function in order to maintain it reasonably close to the optimum. Consequently, the solution of this minimization problem represents critical parameters with high precision to sustain high accuracy
In [2.1, Page 3]:
and assigns low precisions to ineffectual ones or prunes them. The problem described here can be defined in formal terms as below.
PNG
media_image13.png
131
850
media_image13.png
Greyscale
Here, n is the number of model parameters, W =
w
1
…
w
n
T
is a vector of all model parameters,
w
i
is the i-th model parameter, and the function
N
q
(
w
i
)
is the minimum number of bits required to represent
w
i
in its fixed-point format. As a result,
N
Q
(W) is the total number of bits required to represent the model. In addition, L(W) is the loss function value of the model W over the training set (or a minibatch of the training set). Finally,
PNG
media_image14.png
32
132
media_image14.png
Greyscale
is a constant upper bound on the loss of the neural network, which is used to bound the accuracy loss, and
W
0
is the vector of initial model parameters before quantization.
In [2.2, Page 5]:
We note that we did not make any assumptions regarding the point
W
0
and the result can be extended to any point in
R
n
Consequently, if we define such a trust region, we can simply use the following problem as a subproblem that we repeatedly solve in successive iterations.
PNG
media_image15.png
138
728
media_image15.png
Greyscale
Algorithm 2 summarizes the proposed method of solving the original optimization problem (Equation 1).
PNG
media_image16.png
681
1101
media_image16.png
Greyscale
In [2.1, Page 4]:
Consider a floating point variable with range (
x
m
i
n
,
x
m
a
x
) that needs to be quantized to the range (0,
N
l
e
v
e
l
s
− 1) where
N
l
e
v
e
l
s
= 256 for 8-bits of precision. We derive two parameters: Scale (∆) and Zero-point(z) which map the floating point values to integers
In [2.1, Page 4]:
The scale specifies the step size of the quantizer and floating point zero maps to zero-point. Zero-point is an integer, ensuring that zero is quantized with no error. This is important to ensure that common operations like zero padding do not cause quantization error. For one sided distributions, therefore, the range
(
x
m
i
n
,
x
m
a
x
) is relaxed to include zero. For example, a floating point variable with the range (2.1,3.5) will be relaxed to 4 the range (0, 3.5) and then quantized. Note that this can cause a loss of precision in the case of extreme one-sided distribution
Once the scale and zero-point are defined, quantization proceeds as follows:
PNG
media_image10.png
90
645
media_image10.png
Greyscale
where
PNG
media_image11.png
95
531
media_image11.png
Greyscale
The de-quantization operation is :
PNG
media_image12.png
32
732
media_image12.png
Greyscale
(BRI: the zero-point is the constant and combining clamping with quantization using an integer offset can provide a form of "adjustment distance" that allows shifting the entire quantization range. The supplementary quantization level directly relates to quantization steps. Smaller step size results in more quantization levesl and lowers quantization error and larger size results in lower supplementary quantization levels. Using an adaptive quantization, this can be selected (if and then in Algorithm 2))
- compute an adjustment distance between the original quantization levels and the corresponding supplementary quantization levels;
In [2.1, Page 2]:
We minimize the aggregate bit-widths of all network parameters while monitoring the training loss function. Due to the quantization noise, this function deviates from its optimum which was achieved through training. This noise effect can be controlled by introducing an upper bound on the loss function in order to maintain it reasonably close to the optimum. Consequently, the solution of this minimization problem represents critical parameters with high precision to sustain high accuracy
In [2.1, Page 3]:
and assigns low precisions to ineffectual ones or prunes them. The problem described here can be defined in formal terms as below.
PNG
media_image13.png
131
850
media_image13.png
Greyscale
Here, n is the number of model parameters, W =
w
1
…
w
n
T
is a vector of all model parameters,
w
i
is the i-th model parameter, and the function
N
q
(
w
i
)
is the minimum number of bits required to represent
w
i
in its fixed-point format. As a result,
N
Q
(W) is the total number of bits required to represent the model. In addition, L(W) is the loss function value of the model W over the training set (or a minibatch of the training set). Finally,
PNG
media_image14.png
32
132
media_image14.png
Greyscale
is a constant upper bound on the loss of the neural network, which is used to bound the accuracy loss, and
W
0
is the vector of initial model parameters before quantization.
In [2.1, Page 3]:
PNG
media_image2.png
316
928
media_image2.png
Greyscale
(BRI: |
w
i
q
-
w
i
| is the adjustment distance)
In [2.1, Page 4]:
Consider a floating point variable with range (
x
m
i
n
,
x
m
a
x
) that needs to be quantized to the range (0,
N
l
e
v
e
l
s
− 1) where
N
l
e
v
e
l
s
= 256 for 8-bits of precision. We derive two parameters: Scale (∆) and Zero-point(z) which map the floating point values to integers
In [2.1, Page 4]:
The scale specifies the step size of the quantizer and floating point zero maps to zero-point. Zero-point is an integer, ensuring that zero is quantized with no error. This is important to ensure that common operations like zero padding do not cause quantization error. For one sided distributions, therefore, the range
(
x
m
i
n
,
x
m
a
x
) is relaxed to include zero. For example, a floating point variable with the range (2.1,3.5) will be relaxed to 4 the range (0, 3.5) and then quantized. Note that this can cause a loss of precision in the case of extreme one-sided distribution
Once the scale and zero-point are defined, quantization proceeds as follows:
PNG
media_image10.png
90
645
media_image10.png
Greyscale
where
PNG
media_image11.png
95
531
media_image11.png
Greyscale
The de-quantization operation is :
PNG
media_image12.png
32
732
media_image12.png
Greyscale
- and determine modified quantization levels by, for each modified quantization level, adopting either an original quantization level or the corresponding supplementary quantization level, depending on the adjustment distance,
In [2.1, Page 3]:
PNG
media_image2.png
316
928
media_image2.png
Greyscale
(BRI: |
w
i
q
-
w
i
| is the adjustment distance)
In [2.1, Page 4]:
Algorithm 1 estimates with a rounding error of up to
PNG
media_image6.png
52
62
media_image6.png
Greyscale
Thus, for the worst case to be consistent with the constraint of Equation 3, we need:
PNG
media_image7.png
55
676
media_image7.png
Greyscale
In this case the minimization guarantees that the smallest
N
q
(
w
i
)
is chosen. This entails:
PNG
media_image8.png
32
620
media_image8.png
Greyscale
In general, we can expect
N
q
(
w
i
) to be smaller than this worst case. Consequently
PNG
media_image9.png
70
671
media_image9.png
Greyscale
Khoram does not explicitly disclose:
- A decoder for decoding data output from a laver of a neural network from a bitstream, the decoder comprising:
- and neural network circuitry configured to process the obtained data for the neural network by at least one layer of the neural network.
However, Chen discloses:
- A decoder for decoding data output from a laver of a neural network from a bitstream, the decoder comprising:
In [0020]:
FIG. 4a is a schematic diagram of an existing image decoder and encoder,
In [0063]:
encode the raw image to be processed to generate a compressed bit-stream for transmission. After receiving the compressed bit-stream, the cloud server decodes it and performs corresponding processing on the decoded and reconstructed raw image.
In [0074]:
the encoding parameters include the quantization parameters or other parameters. In the practical application, the first model and the second model are deep learning network models, for example, multilayer convolutional neural network. Particularly, the first model includes a first deep learning model and the second model includes a second deep learning model,
in [0074]:
Wherein, the encoding parameters include the quantization parameters or other parameters. In the practical application, the first model and the second model are deep learning network models, for example, multilayer convolutional neural network. Particularly, the first model includes a first deep learning model and the second model includes a second deep learning model,
- and neural network circuitry configured to process the obtained data for the neural network by at least one layer of the neural network.
In [0170]:
4) a multilayer convolutional neural network layer configured to input the raw image corresponding to the historical processed image, the residual image of each coding unit, the set of feature values of each coding unit, and the perceptual distortion value of each coding unit to the multilayer convolutional neural network for training to obtain the first deep learning model.
In [0079]:
3) a training data layer in which the training data is a set of feature values of each coding unit in the processed raw image, wherein when a model is trained in real time, the processed raw image is a processed raw image previous to the raw image to be processed,
In [0082]:training process:
data or information obtained in the above hierarchies is the input for training to obtain the first deep learning model or the second deep learning model; and
[0083]: application process:
data or information obtained in the above hierarchies is the input to the trained deep learning model and the related coding information is the output.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Khoram, and Chen.
Khoram teaches adaptive quantization that includes quantization levels, modified quantization levels and adjustment distance.
Chen teaches decoding data output from a layer.
One of ordinary skill would have motivation to combine Khoram ,and Chen that can be further optimized by the semantic module to reduce the perceptual distortion caused by the image processing [(CHEN [0065]).
Khoram and Chen do not explicitly disclose:
- inverse quantizer configured to obtain, from codewords of the bitstream
However, HASSE1 discloses:
- inverse quantizer configured to obtain, from codewords of the bitstream
In [0117]:
Standards for compression of neural networks only specify the bitstream syntax and the reconstruction process. If we consider parameter coding for a given set of original neural network parameters 13 and given quantization step sizes (QP), the encoder has a lot a freedom. Given the quantization indexes q.sub.k 56 for a layer 10a, 10b, the entropy coding has to follow a uniquely defined algorithm for writing the data to the bitstream 14 (i.e., constructing the arithmetic codeword).
In [0008]:
the quantization index indicates one reconstruction level out of the selected set of reconstruction levels for the current neural network parameter, dequantizing the current neural network parameter onto the one reconstruction level of the selected set of reconstruction levels that is indicated by the quantization index for the current neural network parameter.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Khoram, Chen and HASSE1.
Khoram teaches adaptive quantization that includes quantization levels, modified quantization levels and adjustment distance.
Chen teaches decoding data output from a layer.
HASSE1 teaches codeword from bitstream.
One of ordinary skill would have motivation to combine Khoram , Chen and HASSE1 that can
bitrate is improved compared to using a constant quantizer (HASSE1[0056]).
In regard to claim 16: (Original)
Khoram does not explicitly disclose:
- wherein the decoder is a cloud-based system.
However, CHEN discloses:
- wherein the decoder is a cloud-based system.
In [0065] :
in the scenario of cloud camera system, after processed by the encoder and decoder, the raw image will be optimized by the cloud semantic analyzing module and the cloud ISP module in server
In [0065] :
as shown in FIG. 4b, the image encoder in the “cloud camera” system encodes a raw image to generate a compressed bit-stream which is transmitted to the cloud via network. The image decoder in the “cloud camera” system decodes the compressed bit-stream to obtain the reconstructed raw image,
in [0074]:
the encoding parameters include the quantization parameters or other parameters.
in [0154]:
In the quantization process, first, the specific processing in the step S510 is executed: an encoding parameter (for example, a quantization parameter) of each coding unit in the raw image to be processed is determined according to a set of feature values of each coding unit in the raw image to be processed and based on a trained second deep learning model.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Khoram, and Chen.
Khoram teaches adaptive quantization that includes quantization levels, modified quantization levels and adjustment distance.
Chen teaches cloud-based decoder.
One of ordinary skill would have motivation to combine Khoram ,and Chen that can be further optimized by the semantic module to reduce the perceptual distortion caused by the image processing [(CHEN [0065]).
In regard to claim 17: (Original)
Khoram does not explicitly disclose:
- wherein the one or more supplementary quantization levels include at least one of a minimum clipping value to which all lower data values are clipped, a maximum clipping value to which all higher data values are clipped, or a zero value.
However, Chen discloses:
- wherein the one or more supplementary quantization levels include at least one of a minimum clipping value to which all lower data values are clipped, a maximum clipping value.
In [0234]:
clip the calculated value of the quantization parameter to the effective range, and the effective range in this embodiment is greater than or equal to 0, and less than or equal to 51. The current coding unit to be processed is encoded according to the quantization parameter.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Khoram, and Chen.
Khoram teaches adaptive quantization that includes quantization levels, modified quantization levels and adjustment distance.
Chen teaches cloud-based decoder.
One of ordinary skill would have motivation to combine Khoram ,and Chen that can be further optimized by the semantic module to reduce the perceptual distortion caused by the image processing [(CHEN [0065]).
In regard to claim 18: (Original)
Khoram discloses:
- wherein the adjustment distance includes a difference between the original and the corresponding supplementary quantization levels and an offset by a predetermined constant.
In [2.1, Page 4]:
For models with parameters in a range larger than [−1,1], say [−r, r] with r > 1, we perform the quantization similarly. The only difference is that we first scale the parameters and their tolerances by 1/r to bring them to [−1, 1] range and then quantize the parameters. When computing the output of a quantized layer, then, we multiply the result by r to account for the scale. Following this, the same equations as above will hold.
(BRI: parameters outside ([-1, 1]) are first scaled (along with their tolerances) into ([-1, 1]), quantized there, and then scaled back. The offset in such schemes represents the adjustment needed to recover the original parameter from its quantized form. The scaled factor r is the offset)
In regard to claim 19: (Original)
Khoram discloses:
- wherein the determining of the modified quantization levels includes the following steps performed for each of the original quantization levels:
setting a modified quantization level corresponding to said original quantization level to the supplementary quantization level, when the adjustment distance is smaller than an adjustment threshold,
in [2.1, Page 3]:
PNG
media_image3.png
262
841
media_image3.png
Greyscale
In regard to claim 20: (Original)
Khoram, and Chen do not explicitly disclose:
- further comprising decoding from the bitstream at least one of: the predetermined constant, the adjustment threshold, or the supplementary levels.
However, HAASE1 discloses:
- further comprising decoding from the bitstream at least one of: the predetermined constant, the adjustment threshold, or the supplementary levels.
In [0027]:
neural networks, e.g. defined by neural network parameters, may be compressed and/or transmitted efficiently, e.g. with a low amount of data in a bitstream, using reconstruction-layers, for example sublayers, such as base-layers and enhancement-layers. The reconstruction layers may be defined, such that the neural network parameters are reconstructible by, for each neural network parameter, combining the first-reconstruction-layer neural network parameter value and the second-reconstruction-layer neural network parameter value. This distribution enables an efficient coding, e.g. encoding and/or decoding, and/or transmission of the neural network parameters.
in [0139]:
Decoding of the quantized weight levels 56 (integer representation) works analogously to the encoding. The decoder first decodes the sig_flag. If it is equal to one, a sign_flag and a unary sequence of abs_level_greater_X follows, where the updates of k, (and thus increments of X) has to follow the same rule as in the encoder. Finally, the fixed length code of k bits is decoded and interpreted as integer number (e.g. as rem or rem′, depending on which of both was encoded).
In [0060]:
the apparatus may be configured to parametrize 60 (parametrization) the plurality 50 of reconstruction level sets 52 (e.g., set 0, set 1) by way of a predetermined quantization step size (QP), for example denoted by Δ or Δk, and derive information on the predetermined quantization step size from the data stream 14. Therefore, a decoder according to embodiments may adapt to a variable step size (QP).
In [0139]:
Finally, the quantized weight w is reconstructed by multiplying the quantized weight level q with the step size Δ (QP).
(BRI: decoding fixed codeword length represents “predetermined constant” that the decoder uses to determine how many bits to read for each symbol. This constant is part of the decoding algorithm’s specification and is not derived from the bit stream itself)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Khoram, Chen and HASSE1.
Khoram teaches adaptive quantization that includes quantization levels, modified quantization levels and adjustment distance.
Chen teaches decoding data output from a layer.
HASSE1 teaches codeword from bitstream.
One of ordinary skill would have motivation to combine Khoram , Chen and HASSE1 that can
bitrate is improved compared to using a constant quantizer (HASSE1[0056]).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the
examiner should be directed to TIRUMALE KRISHNASWAMY RAMESH whose telephone number is (571)272-4605. The examiner can normally be reached by phone.
Examiner interviews are available via telephone, in-person, and video conferencing
using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at
http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached on phone (571-272-3768). The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be
obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit:
https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for
information about filing in DOCX format.
For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TIRUMALE K RAMESH/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121