Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 5/13/2026 have been fully considered but they are not persuasive.
Applicant argues, “the claims are not directed to an abstract idea, but instead to a specific technological improvement in neural network training. The claims address a concrete technical problem in binary neural networks, namely that binarization functions are non-differentiable and therefore prevent conventional gradient-based backpropagation, resulting in ineffective training under prior approaches such as the Straight Through Estimator.” Remarks 9. The rule is that “A claim reciting a judicial exception is not directed to the judicial exception if it also recites additional elements demonstrating that the claim as a whole integrates the exception into a practical application. One way to demonstrate such integration is when the claimed invention improves the functioning of a computer or improves another technology or technical field.” MPEP 2106.04(d). The claims don’t train, and therefore cannot solve the alleged problem in training. The claims only calculate error, the error has to be backpropagated. The alleged improvement has to be claimed in order to integrate the abstract idea into a practical application. The claimed invention does not train, therefore the alleged improvement to training is not in the claims, therefore the abstract idea is not integrated into a practical application.
Applicant argues, “This is not merely mathematical manipulation, but a specific implementation integrated into the neural network training process that improves its functioning.” Remarks 9. The claims don’t train, and therefore cannot solve the alleged problem in training. The claims only calculate error, the error has to be backpropagated.
Applicant argues, “Gong fails to disclose that the fitting function comprises a plurality of subfunctions and an error function that compensates for an error between the second gradient of the fitting function and the third gradient of the binarization function to improve accuracy of training of the neural network.” Remarks 11. The newly amended elements are taught by new art. This argument is moot.
Applicant argues Liu fails to cure the deficiencies of Gong, Examiner disagrees. Remarks 13.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of a mathematical concept without significantly more. The claims recite calculating the weights of a neural network using forward propagation, determining the fit using expansion and binarization, calculating the gradient of the error for the weights, calculating subfunction of the series expansion, fitting the error to a neural network, and using Fourier/wavelet/discrete Fourier as the expansion. This judicial exception is not integrated into a practical application because the claims to a specific data type merely link the abstract idea to the field of computers. The additional element of improving accuracy of a neural network does not actually improve accuracy, because the claims only calculate the error – the claims don’t do anything with the calculated error that would improve a neural network. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the components such as memory and processor are generic computer parts.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 1-20 are rejected under 35 USC 112(b). The term “improve” in claims 1, 8 and 15 is a relative term which renders the claim indefinite. The term “improve” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. It is unclear how the accuracy is improved, especially since the error never gets propagated to the weights. Further, it is unclear how the accuracy is improved – e.g. few training epochs, smaller error etc. Lastly, it is unclear what the claimed invention is being compared to – i.e. what is the claimed invention an improvement over?
The term “reduce” in claims 6, 13 and 20 is a relative term which renders the claim indefinite. The term “reduce” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 7-12 and 14-19 are rejected under 35 U.S.C. 103 as being unpatentable over Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks by Gong et al and Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm by Liu et al.
Claims 6, 13 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks by Gong et al, Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm by Liu et al and https://www.physics.smu.edu/scalise/P4321sp20/fs.pdf (Olver).
Gong teaches claims 1, 8 and 15. A neural network training method, comprising: (Gong abs. “DSQ can automatically evolve during training to gradually approximate the standard quantization.”)
performing, in a forward propagation process and using a binarization function, (Gong sec. 3.1 “For 1-bit binary quantization, the binary neural network (BNN) limits its activations and weights to either-1 or +1 usually using the binary function… sgn(x)…” The binarizing happens in the forward pass in algorithm 1 in section 3.5, see below.)
PNG
media_image1.png
146
342
media_image1.png
Greyscale
binarization processing on a target weight to obtain a weight of a first neural network layer in a neural network, (See above in Gong algorithm 1 where the wq- is the weight and it is a function of binarization of the weight wsq. These weights are the layer weights.) or on an activation value of a second neural network layer in neural network to obtain an input of the first neural network layer; (The inputs are quantized/binarized too because Gong sec. 4.1 says “When building up a quantized model, we simply insert DSQ function to all places that will be quantized ,e.g., the inputs and weights of a convolution layer.”)
determining a fitting function based on series expansion of the binarization function; and (Gong sec. 1 “DSQ employs a series of hyperbolic tangent functions to gradually approach the staircase function for low-bit quantization (e.g., sign for 1-bit case), and meanwhile keeps the smoothness for easy gradient calculation.” The binarization is aproximated by a series of tangent functions.)
calculating, in a backward propagation process, a first gradient of a loss function with respect to the target weight using a second gradient of the fitting function as a third gradient of the binarization function, (Gong sec. 1 “DSQ employs a series of hyperbolic tangent functions to gradually approach the staircase function for low-bit quantization (e.g., sign for 1-bit case), and meanwhile keeps the smoothness for easy gradient calculation.” Gong abs “Owing to its differentiable property, DSQ can help pursue the accurate gradients in backward propagation…” This shows that Gong uses a series of smooth differentiable tangent function to approximate the gradient used in backpropagation. This teaches taking the gradient of the fitting function instead of the gradient of the binarization function, because binaries are not differentiable.)
Gong doesn’t teach compensating for error between the binarization gradient and the fitting gradient.
However, Liu teaches the error function compensates for an error between the second gradient of the fitting function and the third gradient of the binarization function to improve accuracy of training of the neural network. (Liu Fig. 5 “(a)Sign function and its derivative, (b)Clip function and its derivative for approximating the derivative of the sign function, proposed in[7],(c) Proposed differentiable piecewise polynomial function and its triangle-shaped derivative for approximating the derivative of the sign function in gradients computation.” Sign derivative is the third gradient, clip gradient is the second derivative and the approximation function gradient is the compensation for the error between those gradients.)
Liu, Gong and the claims are all trying to train with a binarization function. It would have been obvious to a person having ordinary skill in the art, at the time of filing, because “All proposed techniques for initialization, weight update and activation backward improve the accuracy at various degrees.” Liu p. 12.
Gong teaches claims 2, 9 and 16. The neural network training method of claim 1, further comprising determining the plurality of subfunctions based on the series expansion, wherein the series expansion is truncated to a finite number of terms. (Gong sec. 1 “DSQ employs a series of hyperbolic tangent functions to gradually approach the staircase function for low-bit quantization (e.g., sign for 1-bit case), and meanwhile keeps the smoothness for easy gradient calculation.” The subfunctions are the tangent functions, the error function is included because the gradient is calculated and the gradient is a gradient of error. Gong eq 5, below, has a finite number of terms, Based on the asymptotic function ϕ, we can have our differentiable soft quantization (DSQ) function, approximating the uniform quantizer…”)
PNG
media_image2.png
78
312
media_image2.png
Greyscale
Gong teaches claims 3, 10 and 17. The neural network training method of claim 1, further comprising fitting the error function by using a (Gong sec. 4.1 “All convolution and fully-connected layers except the first and the last one are quantized with DSQ.”)
Gong doesn’t teach a two layer with a residual.
However, Liu teaches a two-layer fully connected neural network with a residual. (Liu p. 14 “keep the weights and activations in the first convolution and the last fully-connected layers to be real-valued.” Liu p. 2 “we propose to keep these real activations via adding a simple yet effective shortcut, dubbed Bi-Real net. As shown in Fig. 1(b), the shortcut connects the real activations to an addition operator with the real-valued activations of the next block.” The shortcut connection is the residual connection.)
Gong teaches claims 4, 11 and 18. The neural network training method of claim 1, further comprising fitting the error function using at least one neural network layer, wherein calculating the first gradient comprises:
calculating, in the backward propagation process, fourth gradients of the plurality of subfunctions with respect to the target weight;
calculating a fifth gradient of the at least one neural network layer with respect to the target weight; and
calculating the first gradient based on the fourth gradients and the fifth gradient. (Gong algorithm 1 and equation 6, see below. The gradient of alpha is the gradient of the subfunctions. It’s all with respect to weight because Gong sec. 4.1 says “When building up a quantized model, we simply insert DSQ function to all places that will be quantized ,e.g., the inputs and weights of a convolution layer.”)
PNG
media_image3.png
116
308
media_image3.png
Greyscale
Algorithm 1
PNG
media_image4.png
40
280
media_image4.png
Greyscale
Equation 6
Gong teaches claims 5, 12 and 19. The neural network training method of claim 1, further comprising determining a plurality of subfunctions based on the series expansion, wherein the fitting function comprises a finite superposition of sine functions. (Fig. 5 “(a)Sign function and its derivative, (b)Clip function and its derivative for approximating the derivative of the sign function, proposed in[7],(c) Proposed differentiable piecewise polynomial function and its triangle-shaped derivative for approximating the derivative of the sign function in gradients computation.”)
Gong teaches claims 6, 13 and 20. The neural network training method of claim 1, wherein the series expansion is a (Gong sec. 1 “DSQ employs a series of hyperbolic tangent functions to gradually approach the staircase function for low-bit quantization (e.g., sign for 1-bit case), and meanwhile keeps the smoothness for easy gradient calculation.” Binarization reduces storage by turning the weights into binaries. Reduced weight size reduces operation amount of the neural network.)
Gong doesn’t teach a Fourier series expansion of the binarization function.
However, Olver teaches a Fourier series expansion of the binarization function. (Olver p. 645 “Thus, the Fourier series converges, as expected, to f(x) at all points of continuity; at discontinuities, the Fourier series can’t decide whether to converge to the right or left hand limit, and so ends up ‘splitting the difference’ by converging to their average; see Figure 12.4.” Fig. 12.4 shows a fourier series of a binary/step function, below. And equation 12.41, shows a fourier series expansion of a step function.)
PNG
media_image5.png
250
314
media_image5.png
Greyscale
PNG
media_image6.png
86
652
media_image6.png
Greyscale
Olver, Gong and the claims all turn a non-continuous step function into a smooth function. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to use a Fourier series to do the conversion because “If f(x) is any piecewise continuous function, then its Fourier coefficients are well defined — the integrals (12.28) exist and are finite.” Olver p. 644
Gong teaches claims 7 and 14. The neural network training method of claim 1, wherein a data type of the target weight is a 32-bit floating point type. (Gong sec. 3.1 “given the bit width b and the floating-point activation/weight x following in the range (l,u),” b is a whole number that includes 32.)
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Austin Hicks whose telephone number is (571)270-3377. The examiner can normally be reached Monday - Thursday 8-4 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AUSTIN HICKS/Primary Examiner, Art Unit 2142
Prosecution Insights