Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Status of Claims
This action is in response to the application filed on October 27, 2023.
This application claims priority to Korean patent application KR10-2022-0147117, filed November 7, 2022.
Claims 1-19 are currently pending.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on October 27, 2023, and February 27, 2026 have been considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 4 and 14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 4 and 14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being incomplete for omitting essential steps, such omission amounting to a gap between the steps. See MPEP § 2172.01. The omitted steps are: calculating a cross-entropy loss. The claims do not detail computing a cross-entropy loss or which aspect is being measured for the loss.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
When considering subject matter eligibility under 35 U.S.C. § 101, it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter (Step 1). If the claim does fall within one of the statutory categories, the second step in the analysis is to determine whether the claim is directed to a judicial exception (Step 2A). The step 2A analysis is broken into two prongs. In the first prong (Step 2A, Prong 1), it is determined whether or not the claims recite a judicial exception (e.g. mathematical concepts, mental processes, certain methods of organizing human activity). If it is determined in step 2A, Prong 1 that the claims recite a judicial exception, the analysis proceeds to the second prong (Step 2A, Prong 2), where it is determined whether or not the claims integrate the judicial exception into a practical application. If it is determined that step 2A, Prong that the claims do not integrate the judicial exception into a practical application, the analysis proceeds to determining whether the claim is a patent-eligible application of the exception (Step 2B). If an abstract idea is present in the claim, any element or combination of elements in the claim must be sufficient to ensure that the claim integrates the judicial exception into a practical application, or else amounts to significantly more than the abstract idea itself.
According to Step 1 of the analysis, in the instant case claims 1-9 are directed to a method, claim 10 is directed to a non-transitory computer-readable storage medium, and claims 10-19 are directed to an apparatus. Thus, each of the claims falls within one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).
Independent claim 1: considering Step 2A, Prong One, the limitations in claim 1 including: “calculating a quantization error for each channel of a neural network using activation data output from a first layer of the neural network and a quantization scale of a second layer connected to the first layer; calculating a final loss using a regularization loss determined based on the quantization error for each channel; and updating a batch norm parameter of the first layer in a direction to decrease the final loss,” covers mental processes but for the recitation of generic computer components.
MPEP 2106.04.(a)(2)(III) details “the "mental processes" abstract idea grouping is defined as concepts performed in the human mind, and examples of mental processes include observations, evaluations, judgments, and opinions.” The claimed “calculating” and “updating” are observations, evaluations, and judgments.
Claim 1 does not contain any additional elements. Therefore, the claim is not integrated into a practical application under Step 2A, Prong Two, or amount to significantly more under Step 2B.
Therefore, claim 1 is ineligible.
Claims 2-8 recite details relating to the “calculating” and “updating” aspects of claim 1. These steps are observations, evaluations, judgments, and opinions and, therefore, also mental processes. Claims 2-8 do not include any additional elements.
Claims 2-8 are not eligible.
Claim 9 recites an equation that can be performed by a human mentally, or with pen and paper; see MPEP 2106.04.(a)(2)(III). The equation in claim 9 is also a mathematical concept; see MPEP 2106.04(a)(2).
Claim 10 includes a non-transitory computer-readable storage medium which is a generic computer component that does not integrate the judicial exception into a practical application; see MPEP 2106.05(b). Additionally, considering Step 2B, the storage medium is a generic computer component that does not amount to significantly more; see MPEP 2106.05(b).
Independent claim 11: considering Step 2A, Prong One, the limitations in claim 11 including: “calculate a quantization error for each channel of a neural network using activation data output from a first layer of the neural network and a quantization scale of a second layer connected to the first layer; calculate a final loss using a regularization loss determined based on the quantization error for each channel; and update a batch norm parameter of the first layer in a direction to decrease the final loss,” covers mental processes but for the recitation of generic computer components.
MPEP 2106.04.(a)(2)(III) details “the "mental processes" abstract idea grouping is defined as concepts performed in the human mind, and examples of mental processes include observations, evaluations, judgments, and opinions.” The claimed “calculating” and “updating” are observations, evaluations, and judgments.
Claim 11 contains additional elements “[a]n apparatus,” “one or more processors,” and “a memory.” These elements are generic computer component that does not integrate the judicial exception into a practical application; see MPEP 2106.05(b). Additionally, considering Step 2B, the apparatus, processors, and memory are generic computer components that do not amount to significantly more; see MPEP 2106.05(b).
Claims 12-19 are similar to claims 2-9 and rejected for the same reasons as disclosed above.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 6-8, 10, 11, and 16-18 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Nagel et al., U.S. Patent Application Publication 2020/0302299 (“Nagel”).
With respect to independent claim 1, Nagel teaches:
A processor-implemented method, the method comprising:
calculating a quantization error for each channel of a neural network using activation data output from a first layer of the neural network and a quantization scale of a second layer connected to the first layer (Nagel teaches equalizing ranges of output channel weights within a first layer of a neural network by scaling each of the output channel weights of the first layer by a scaling factor and scaling each of a second adjacent layer’s corresponding input channel weights by applying an inverse; see abstract. Further, Nagel teaches the scaling factor may be determined using a quantization error metric; see abstract.);
calculating a final loss using a regularization loss determined based on the quantization error for each channel (Nagel teaches a global loss that is determined by using a metric for the quantization error and black box optimizer that minimizes the error metric with respect to the scaling parameters; see [0033].); and
updating a batch norm parameter of the first layer in a direction to decrease the final loss (Nagel teaches training a neural network model that includes performing batch normalization operations on layers of the trained neural network; see figure 5 and [0072].).
With respect to independent claim 11, Nagel teaches:
An apparatus (Nagel teaches implementation on a computing device; see [0070].), the apparatus comprising:
one or more processors configured to execute instructions (Nagel teaches processors in [0074]-[0075].); and
a memory storing the instructions, wherein execution of the instructions by the one or
more processors configure the one or more processors to (Nagel teaches computer memory in [0076].):
calculate a quantization error for each channel of a neural network using activation data output from a first layer of the neural network and a quantization scale of a second layer connected to the first layer (Nagel teaches equalizing ranges of output channel weights within a first layer of a neural network by scaling each of the output channel weights of the first layer by a scaling factor and scaling each of a second adjacent layer’s corresponding input channel weights by applying an inverse; see abstract. Further, Nagel teaches the scaling factor may be determined using a quantization error metric; see abstract.);
calculate a final loss using a regularization loss term based on the quantization error for each channel (Nagel teaches a global loss that is determined by using a metric for the quantization error and black box optimizer that minimizes the error metric with respect to the scaling parameters; see [0033].); and
update a batch norm parameter of the first layer in a direction to decrease the final loss (Nagel teaches training a neural network model that includes performing batch normalization operations on layers of the trained neural network; see figure 5 and [0072].).
With respect to claims 6 and 16, the rejections of claims 1 and 11 are incorporated. Further, Nagel teaches:
wherein the updating of the batch norm parameter comprises fixing values of all parameters of the neural network other than the batch norm parameter when updating the batch norm parameter (Nagel teaches batch normalization including folding batch normalization parameters into a preceding layer because channel scale and shift operations do not have to be performed (i.e. are fixed); see [0050].).
With respect to claims 7 and 17, the rejections of claims 1 and 11 are incorporated. Further, Nagel teaches:
wherein the updating of the batch norm parameter comprises training a quantization scale of the first layer to reduce the final loss (Nagel teaches cross layer rescaling in [0043]-[0048].).
With respect to claims 8 and 18, the rejections of claims 1 and 11 are incorporated. Further, Nagel teaches:
wherein the updating of the batch norm parameter comprises measuring performance of the neural network having the updated batch norm parameter (Nagel teaches that batch normalization is used to reduce errors from quantizing the neural network and may also improve inference times; see [0050].).
With respect to claim 10, the rejections of claim 1 is incorporated. Further, Nagel teaches:
A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 (Nagel teaches various computer implementation options beginning in [0074].).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under pre-AIA 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 2 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nagel et al., U.S. Patent Application Publication 2020/0302299 (“Nagel”); in view of Ha et al., U.S. Patent Application Publication 2021/0201117 (“Ha”).
With respect to claims 2 and 12, the rejections of claims 1 and 11 are incorporated. Further, Nagel does not explicitly teach:
wherein the calculating of the quantization error for each channel further comprises quantifying, for each of the channels, the quantization error as an inverse of a signal-to-quantization-noise ratio (SQNR) for a corresponding scale.
However, Ha teaches this limitation:
wherein the calculating of the quantization error for each channel further comprises quantifying, for each of the channels, the quantization error as an inverse of a signal-to-quantization-noise ratio (SQNR) for a corresponding scale (Ha teaches determining a quantization error and a quantization noise for each channel based on a signal-to-quantization noise ratio (SQNR) in [0020].).
Nagel and Ha are analogous art directed towards neural network quantization. Nagel teaches quantization using scaling factors and weight adjustments and Ha teaches quantizing parameters into a range using, in part, signal-to-quantization noise ratios.
It would have been obvious for one of ordinary skill in neural network quantization to incorporate Ha’s methods into Nagel’s disclosed system at the time of filing. It would have been obvious because one of ordinary skill would be motivated to implement an improved quantization apparatus that takes into account the SQNR, as disclosed in [0094] of Ha.
Claim 3, 4, 13, and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nagel et al., U.S. Patent Application Publication 2020/0302299 (“Nagel”); in view of Liang et al., U.S. Patent Application Publication 2023/0401834 (“Liang”).
With respect to claims 3 and 13, the rejections of claims 1 and 11 are incorporated. Further, Nagel does not teach:
wherein the calculating of the final loss comprises calculating the regularization loss by calculating an average of the quantization error for each channel.
However, Liang teaches this feature:
wherein the calculating of the final loss comprises calculating the regularization loss by calculating an average of the quantization error for each channel (Liang (2023/0401834) teaches determining quantization errors including determining an average quantization error of all channels in [0091]).
Nagel and Liang are analogous art directed towards neural network quantization. Nagel teaches quantization using scaling factors and weight adjustments and Liang teaches quantizing a deep neural network model using various quantization policies.
It would have been obvious for one of ordinary skill in neural network quantization to incorporate Liang’s methods into Nagel’s disclosed system at the time of filing. It would have been obvious because one of ordinary skill would be motivated to implement a quantization system that shortens quantization time and ensures performance of the model is not affected by the quantization; see [0053] of Liang.
With respect to claims 4 and 14, the rejections of claims 1 and 11 are incorporated. Further, Liang teaches:
wherein the calculating of the final loss comprises calculating the final loss by summing the regularization loss and a cross-entropy loss (Liang (2023/0401834) teaches summing layer quantization errors in [0090]).
See the rejection of claims 3 and 13 for the motivation to combine references.
Claims 5 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nagel et al., U.S. Patent Application Publication 2020/0302299 (“Nagel”); in view of Bjorck et al., “Understanding Batch Normalization” (“Bjorck”).
With respect to claims 5 and 15, the rejections of claims 1 and 11 are incorporated. Further, Nagel does not explicitly disclose:
wherein the updating of the batch norm parameter comprises deriving the batch norm parameter in a direction reduces a value of the regularization loss by performing stochastic gradient descent using the final loss.
However, Bjorck teaches this limitation:
wherein the updating of the batch norm parameter comprises deriving the batch norm parameter in a direction reduces a value of the regularization loss by performing stochastic gradient descent using the final loss (Bjorck teaches batch normalization that implements gradient descent; see sections 1, 3, and 4.1. In particular, in section 3, Bjorck teaches looking at the loss landscape along the gradient direction during mini-batches.).
Nagel and Bjorck are analogous art directed towards neural network batch normalization. Nagel teaches batch normalization methods and Bjorck teaches batch normalization methods that implement gradient descent.
It would have been obvious for one of ordinary skill in neural network quantization to incorporate Bjorck’s methods into Nagel’s disclosed system at the time of filing. It would have been obvious because one of ordinary skill would be motivated to reduce diverging loss and activations growing uncontrollably with network depth; see Bjorck abstract.
Allowable Subject Matter
Claims 9 and 19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. In addition, the rejection of claims under 35 U.S.C. 101 must be overcome.
Prior Art of Record
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Hirose et al., “Quantization Error-Based Regularization in Neural Networks” – teaches reducing quantization errors in neural networks.
Conclusion
Claims 1-19 are rejected.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL T PELLETT whose telephone number is (571)270-7156. The examiner can normally be reached on Monday - Friday 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li Zhen can be reached on 571-272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL T PELLETT/Primary Examiner, Art Unit 2121
Prosecution Insights