DETAILED ACTION
This action is in response to the amendment filed 12/25/2025. Claims 1-22 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 12/25/2025 has been entered.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: (a) a layer control unit (LCU), comprising a state machine operative to: receive state information from said processing elements… (b) wherein said LCU is operative to generate a ‘ready’ signal to a previous layer… (c) wherein said LCU evaluates said early termination condition… , and (d) wherein said LCU evaluates said early termination condition … in claims 17, 20, 21 and 22 respectively. Regarding the layer control unit (LCU), the specification in paragraph [00254], states:
The LCU is operative to generate control signaling to implement the early termination mechanism. In particular, the LCU generates control signal 901 to the processing elements and control signal 907 to the APU. The LCU is adapted to receive information regarding the state or status 903 of the MACs in the PEs. Based on the feedback received from the MACs in the PEs, the LCU determines whether and when to terminate the layer calculation processing early. If a determination is made by the LCU to terminate the layer calculations, an early termination/inhibit signal 905 is generated and sent to the PEs and also an early termination/inhibit signal 909 is generated and sent to the APUs.
In addition to the above text, the specification also states, in paragraph [00220]:
A high-level block diagram illustrating a second example layer controller in more detail is shown in Figure 21. The example LC, generally referenced 550, comprises instruction memory 552 including a plurality of instructions 554, LCU 556, instruction decoders 566, trigger window cross connect 558, and trigger handler 560. The LCU 556 comprises a state machine 562, and instruction register 564.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-22 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1:
Subject Matter Eligibility Analysis Step 1:
Claim 1 recites a method and is thus a process, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 1 recites
for each layer of said ANN, sorting before inference a plurality of weight tensors based on a plurality of ANN output and/or input features and calculating early termination statistics therefrom for each layer; (This limitation is a mental process as it encompasses a human mentally sorting weight tensors and calculating statistics.)
setting, during inference, a termination threshold for each layer in the ANN provided by a user or determined dynamically (This limitation is a mental process as it encompasses a human mentally setting a threshold.)
calculating activations for each layer; (This limitation is a mental process as it encompasses a human mentally calculating activations.)
evaluating an early termination condition during inference in accordance with results of said calculations and the termination threshold; (This limitation is a mental process as it encompasses a human mentally evaluating a condition with calculations and a threshold.)
continuing said processing and calculating otherwise (This limitation is a mental process as it encompasses a human mentally continuing processing and calculating.)
determines based on feedback from the plurality of PEs whether to terminate early a particular layer, thereby reducing computational resource usage power consumption and latency of the neural network processor (This limitation is a mental process as it encompasses a human mentally determining whether to terminate a layer early.)
Therefore, claim 1 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 1 further recites additional elements of
early termination of layer processing in an artificial neural network (ANN) implemented on a neural network processor (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
on a compiler (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
loading the ANN model into the neural network processor (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g)).)
performing ANN processing on input data and said sorted plurality of weight tensors during inference on a plurality of processing elements (PEs) each PE comprising a plurality of multiply-accumulate (MAC) circuits (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
said evaluating comprising receiving feedback from said MAC circuits indicating saturation in MAC outputs; (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g)).)
during inference, terminating processing for a particular layer (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
sending trigger indications to a next layer before processing would normally complete if said termination condition exceeds the termination threshold (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g)).)
a hardware-based layer control unit (LCU) comprising a state machine (This element does not integrate the abstract idea into a practical application because it a generic computing component on which to perform the abstract idea (see MPEP 2106.05(f)).)
Therefore, claim 1 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 1 do not provide significantly more than the abstract idea itself, taken alone and in combination because
early termination of layer processing in an artificial neural network (ANN) implemented on a neural network processor uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
on a compiler uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
loading the ANN model into the neural network processor is the well understood, routine, and conventional activity of “transmitting or receiving data over a network” (see MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
performing ANN processing on input data and said sorted plurality of weight tensors during inference on a plurality of processing elements (PEs) each PE comprising a plurality of multiply-accumulate (MAC) circuits uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
said evaluating comprising receiving feedback from said MAC circuits indicating saturation in MAC outputs; is the well understood, routine, and conventional activity of “transmitting or receiving data over a network” (see MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
during inference, terminating processing for a particular layer uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
sending trigger indications to a next layer before processing would normally complete if said termination condition exceeds the termination threshold is the well understood, routine, and conventional activity of “transmitting or receiving data over a network” (see MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
a hardware-based layer control unit (LCU) comprising a state machine uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 1 is subject-matter ineligible.
Regarding Claim 2:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 2 recites
wherein said early termination statistics comprises a mathematical norm. (This limitation encompasses a mathematical concept. )
Therefore claim 2 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 2 has no additional elements that would integrate the abstract idea into a practical application. Therefore, Claim 2 is directed to the abstract idea.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements to provide significantly more than the abstract idea itself, taken alone and in combination, claim 2 is subject-matter ineligible.
Regarding Claim 3:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 3 recites
wherein said sorting comprising sorting either ascending or descending. (This limitation is a mental process as it encompasses a human mentally sorting either ascending or descending.)
Therefore claim 3 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 3 has no additional elements that would integrate the abstract idea into a practical application. Therefore, Claim 3 is directed to the abstract idea.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements to provide significantly more than the abstract idea itself, taken alone and in combination, claim 3 is subject-matter ineligible.
Regarding Claim 4:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 4 recites
wherein said early termination statistics are selected from a group consisting of norm(weights), lambdamax/lambamin, and variance. (This limitation is a mental process as it encompasses a human mentally calculating a statistic from the group.)
Therefore claim 4 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 4 has no additional elements that would integrate the abstract idea into a practical application. Therefore, Claim 4 is directed to the abstract idea.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements to provide significantly more than the abstract idea itself, taken alone and in combination, claim 4 is subject-matter ineligible.
Regarding Claim 5:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 5 recites
wherein said termination threshold can be configured by a user. (This limitation is a mental process as it encompasses a human mentally setting a threshold.)
Therefore claim 5 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 5 has no additional elements that would integrate the abstract idea into a practical application. Therefore, Claim 5 is directed to the abstract idea.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements to provide significantly more than the abstract idea itself, taken alone and in combination, claim 5 is subject-matter ineligible.
Regarding Claim 6:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 6 recites
wherein said termination threshold can be configured for each layer. (This limitation is a mental process as it encompasses a human mentally setting a threshold. )
Therefore claim 6 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 6 has no additional elements that would integrate the abstract idea into a practical application. Therefore, Claim 6 is directed to the abstract idea.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements to provide significantly more than the abstract idea itself, taken alone and in combination, claim 6 is subject-matter ineligible.
Regarding Claim 7:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 7 recites the same abstract ideas as claim 1. Therefore claim 7 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 7 further recites
wherein said terminating processing for a particular layer comprises generating a ‘ready’ signal to a previous layer and a ‘done’ signal to a next layer. (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g)).)
Therefore, Claim 7 is directed to the abstract idea.
Subject Matter Eligibility Analysis Step 2B:
The additional element of Claim 7 does not provide significantly more than the abstract idea itself, taken alone and in combination, because
wherein said terminating processing for a particular layer comprises generating a ‘ready’ signal to a previous layer and a ‘done’ signal to a next layer is a well understood, routine, and conventional activity of “transmitting or receiving data over a network” (see MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
Therefore, claim 7 is subject-matter ineligible.
Regarding Claim 8:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 8 recites
wherein said evaluating said early termination condition is performed in accordance with a selected strategy of detecting no change to the output of a neuron over at least N cycles. (This limitation is a mental process as it encompasses a human mentally selecting and performing a selected strategy of detecting none or minimal change to the output of a neuron.)
Therefore claim 8 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 8 has no additional elements that would integrate the abstract idea into a practical application. Therefore, Claim 8 is directed to the abstract idea.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements to provide significantly more than the abstract idea itself, taken alone and in combination, claim 8 is subject-matter ineligible.
Regarding Claim 9:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 9 recites
wherein said evaluating said early termination condition is performed in accordance with a selected strategy of the output of a neuron being either at or near zero or saturated over at least N cycles. (This limitation is a mental process as it encompasses a human mentally selecting and performing a selected strategy of the output of a neuron being at or near zero or saturated.)
Therefore claim 9 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 9 has no additional elements that would integrate the abstract idea into a practical application. Therefore, Claim 9 is directed to the abstract idea.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements to provide significantly more than the abstract idea itself, taken alone and in combination, claim 9 is subject-matter ineligible.
Regarding Claim 10:
Subject Matter Eligibility Analysis Step 1:
Claim 10 recites a method and is thus a process, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 10 recites
for each layer of said ANN, calculating before inference a first metric including early termination statistics from a first plurality of weight tensors across a plurality of output features (This limitation is a mental process as it encompasses a human mentally calculating statistics.);
sorting said first plurality of weight tensors before inference in accordance with said first metric (This limitation is a mental process as it encompasses a human mentally sorting weight tensors.);
for each layer of said ANN, calculating before inference a second metric from a second plurality of weight tensors across a plurality of input features (This limitation is a mental process as it encompasses a human mentally calculating statistics.);
sorting said second plurality of weight tensors before inference in accordance with said second metric (This limitation is a mental process as it encompasses a human mentally sorting weight tensors.);
setting, during inference, a termination threshold for each layer in the ANN provided by a user or determined dynamically (This limitation is a mental process as it encompasses a human mentally setting a threshold.)
performing calculations utilizing said sorted first plurality of weight tensors and said sorted second plurality of weight tensors (This limitation is a mental process as it encompasses a human mentally performing calculations.);
calculating activations for each layer; (This limitation is a mental process as it encompasses a human mentally calculating activations.)
evaluating an early termination condition during inference in accordance with results of said calculations and the termination threshold; (This limitation is a mental process as it encompasses a human mentally evaluating a condition with calculations and a threshold.)
continuing said processing and calculating otherwise (This limitation is a mental process as it encompasses a human mentally continuing processing and calculating.)
determines based on feedback from the plurality of PEs whether to terminate early a particular layer, thereby reducing computational resource usage power consumption and latency of the neural network processor (This limitation is a mental process as it encompasses a human mentally determining whether to terminate a layer early.)
Therefore, claim 10 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 10 further recites additional elements of
early termination of layer processing in an artificial neural network (ANN) implemented on a neural network processor (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
on a compiler (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
loading the ANN model into the neural network processor (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g)).)
performing ANN processing on input data and said sorted plurality of weight tensors during inference on a plurality of processing elements (PEs) each PE comprising a plurality of multiply-accumulate (MAC) circuits (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
said evaluating comprising receiving feedback from said MAC circuits indicating saturation in MAC outputs; (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g)).)
during inference, terminating processing for a particular layer (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
sending trigger indications to a next layer before processing would normally complete if said termination condition exceeds the termination threshold (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g)).)
a hardware-based layer control unit (LCU) comprising a state machine (This element does not integrate the abstract idea into a practical application because it a generic computing component on which to perform the abstract idea (see MPEP 2106.05(f)).)
Therefore, claim 10 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 10 do not provide significantly more than the abstract idea itself, taken alone and in combination because
early termination of layer processing in an artificial neural network (ANN) implemented on a neural network processor uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
on a compiler uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
loading the ANN model into the neural network processor is the well understood, routine, and conventional activity of “transmitting or receiving data over a network” (see MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
performing ANN processing on input data and said sorted plurality of weight tensors during inference on a plurality of processing elements (PEs) each PE comprising a plurality of multiply-accumulate (MAC) circuits uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
said evaluating comprising receiving feedback from said MAC circuits indicating saturation in MAC outputs; is the well understood, routine, and conventional activity of “transmitting or receiving data over a network” (see MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
during inference, terminating processing for a particular layer uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
sending trigger indications to a next layer before processing would normally complete if said termination condition exceeds the termination threshold is the well understood, routine, and conventional activity of “transmitting or receiving data over a network” (see MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
a hardware-based layer control unit (LCU) comprising a state machine uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 10 is subject-matter ineligible.
Regarding Claim 11:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 11 recites
wherein said first metric and said second metric comprise a mathematical norm. This limitation encompasses a mathematical concept.
Therefore claim 11 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 11 has no additional elements that would integrate the abstract idea into a practical application. Therefore, Claim 11 is directed to the abstract idea.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements to provide significantly more than the abstract idea itself, taken alone and in combination, claim 11 is subject-matter ineligible.
Regarding claim 12, claim 12 recites substantially similar limitations to claim 4, and is therefore rejected under the same analysis.
Regarding claim 13, claim 13 recites substantially similar limitations to claim 5, and is therefore rejected under the same analysis.
Regarding claim 14, claim 14 recites substantially similar limitations to claim 7, and is therefore rejected under the same analysis.
Regarding claim 15, claim 15 recites substantially similar limitations to claim 8, and is therefore rejected under the same analysis.
Regarding claim 16, claim 16 recites substantially similar limitations to claim 9, and is therefore rejected under the same analysis.
Regarding Claim 17:
Subject Matter Eligibility Analysis Step 1:
Claim 17 recites a method and is thus a process, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 17 recites
calculate an output and activations for each layer in accordance with input data weights ordered before inference based on output features and/or input features, and termination statistics calculated therefrom for each layer, (This limitation is a mental process as it encompasses a human mentally calculating outputs and activations.)
evaluate an early termination condition during inference in accordance with said state information and a termination threshold provided by a user or determined dynamically; (This limitation is a mental process as it encompasses a human mentally evaluating a condition with state information and a threshold.)
continuing to calculate an output and activations otherwise; (This limitation is a mental process as it encompasses a human mentally continuing calculations.)
Therefore, claim 17 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 17 further recites additional elements of
An apparatus for early termination of layer processing in an artificial neural network (ANN), the model of which is loaded and implemented on a neural network processor, (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
a plurality of processing elements, each having a multiply and accumulate (MAC) circuit operative to (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
a layer control unit (LCU) comprising a state machine operative to: (This element does not integrate the abstract idea into a practical application because it recites a generic computing component on which to perform the abstract idea (see MPEP 2106.05(f)).)
receive state information feedback from said processing elements during inference indicating a state of saturation of said MAC circuit; (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g)).)
generating an inhibit signal and sending trigger indications to a next layer if said termination condition exceeds said selected termination threshold; (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g)).)
applying said inhibit signal to one or more processing elements of a layer thereby terminating processing for a particular layer before it would normally complete, thereby reducing computational resource usage, power consumption and latency of the neural network processor. (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
Therefore, claim 17 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 17 do not provide significantly more than the abstract idea itself, taken alone and in combination because
An apparatus for early termination of layer processing in an artificial neural network (ANN), the model of which is loaded and implemented on a neural network processor, uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
a plurality of processing elements, each having a multiply and accumulate (MAC) circuit operative to uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
a layer control unit (LCU) comprising a state machine operative to: uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
receive state information feedback from said processing elements during inference indicating a state of saturation of said MAC circuit; is the well understood, routine, and conventional activity of “transmitting or receiving data over a network” (see MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
generating an inhibit signal and sending trigger indications to a next layer if said termination condition exceeds said selected termination threshold; is the well understood, routine, and conventional activity of “transmitting or receiving data over a network” (see MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
applying said inhibit signal to one or more processing elements of a layer thereby terminating processing for a particular layer before it would normally complete, thereby reducing computational resource usage, power consumption and latency of the neural network processor. uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 17 is subject-matter ineligible.
Regarding Claim 18:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 18 recites
wherein said ordered weights are generated by sorting weights by output and input features before inference in accordance with a metric including statistics related to the weights. (This is a mental process as it encompasses a human mentally sorting output and input feature to generate ordered weights.)
Therefore claim 18 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 18 has no additional elements that would integrate the abstract idea into a practical application. Therefore, Claim 18 is directed to the abstract idea.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements to provide significantly more than the abstract idea itself, taken alone and in combination, claim 18 is subject-matter ineligible.
Regarding claim 19, claim 19 recites substantially similar limitations to claim 5, and is therefore rejected under the same analysis.
Regarding claim 20, claim 20 recites substantially similar limitations to claim 7, and is therefore rejected under the same analysis.
Regarding claim 21, claim 21 recites substantially similar limitations to claim 8, and is therefore rejected under the same analysis.
Regarding claim 22, claim 22 recites substantially similar limitations to claim 9, and is therefore rejected under the same analysis.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1, 5-6, 8-10, 13, 15-19, and 21-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Behar et al. (US 2019/0370076 A1) (hereafter referred to as Behar) in view of Berestizshevsky et al. (“Dynamically Sacrificing Accuracy for Reduced Computation: Cascaded Inference Based on Softmax Confident) (hereafter referred to as Berest) in further view of Shah et al. (Using Saturation Detection to Shorten the Training Duration for Gaussian ANNs”) (hereafter referred to as Shah).
Regarding claim 1, Behar teaches
a method of early termination of layer processing in an artificial neural network (ANN) implemented on a neural network processor, the method comprising (Behar, page 31, paragraph 0171, “the processor platform 1100 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network)” where “the process 1000 begins at block 1002 when the host processor interface 608 obtains one or more workloads from a host processor” (Behar, page 30, paragraph 0164) and “the executable instructions of blocks 1002…may be executed on at least one processor such as the example processor 1110” (Behar, page 25, paragraph 0123) and where “the examples disclosed herein analyze the data dependences of a workload node and determine whether a workload node is a candidate for early termination to allow for the dynamic processing of a predefined amount of data” (Behar, page 14, paragraph 0027). Examiner notes that the ANN model is the workload and the neural network processor is the host processor interface run on the example processor.):
a compiler (Behar, page 16, paragraph 0046, “The graph compiler 302 receives the workload 306 and assigns various workload nodes of the workload 306 (e.g., a graph) to various CBBs (e.g., any of the convolution engine 214, the MMU 316, the RNN engine 318, and/or the DSP 320) of the accelerator 308.”);
loading an ANN model into the neural network processor (Behar, page 31, paragraph 0171, “the processor platform 1100 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network)” where “the process 1000 begins at block 1002 when the host processor interface 608 obtains one or more workloads from a host processor” (Behar, page 30, paragraph 0164) and where “the executable instructions of blocks 1002…may be executed on at least one processor such as the example processor 1110” (Behar, page 25, paragraph 0123). Examiner notes that the ANN model is the workload and the neural network processor is the host processor interface run on the example processor.)
performing ANN processing on input data … during inference on a plurality of processing elements (PEs) (Behar, page 32, paragraph 0186, “Example 1 includes an apparatus to enable dynamic processing of a predefined workload at one or more computational building blocks of an accelerator, the apparatus comprising an interface to obtain a workload node from a controller of the accelerator, the workload node associated with a first amount of data, the workload node to be executed at a first one of the one or more computational building blocks, an analyzer to determine whether the workload node is a candidate for early termination” where “In FIG. 1, the input 102 is an image to be processed by the accelerator (e.g., a VPU, another AI accelerator, etc.). The first workload node 104 is a layer of the mask R-CNN that, when executed, identifies one or more features in the input 102 (e.g., the image) by convolving the image with one or more matrices indicative of features in the image, such as edges, gradients, color, etc.” (Behar, page 13, paragraph 0022). Examiner notes that the computational building blocks are the plurality of processing elements (PEs).)
during inference, terminating processing for a particular layer (Behar, page 33, paragraph 0200, “in response to determining that the workload node is a candidate for early termination, set a flag associated with a tile of the first amount of data, and means for dispatching, the means for dispatching to, in response to the tile being transmitted from the first one of the one or more computation building blocks to a buffer, stop execution of the workload node at the first one of the one or more computation building blocks.” Examiner notes that the computation building blocks are layers, the first one of the one or more computation building blocks is the particular layer, and stopping execution is terminating processing.)
and wherein a hardware based layer control unit (LCU) comprising a state machine determines based on feedback from the plurality of PEs whether to terminate early a particular layer, thereby reducing computational resource usage, power consumption, and latency of the neural network (Behar, page 25, paragraph 0118, “In some examples, the example workload analyzer 604 implements example means for analyzing….The analyzing means is implemented by hardware logic, hardware implemented state machines, logic circuitry, and/or any other combination of hardware, software, and/or firmware” where “Example 1 includes an apparatus to enable dynamic processing of a predefined workload at one or more computational building blocks of an accelerator, the apparatus comprising an interface to obtain a workload node from a controller of the accelerator, the workload node associated with a first amount of data, the workload node to be executed at a first one of the one or more computational building blocks, an analyzer to determine whether the workload node is a candidate for early termination,” (Behar, page 32, paragraph 0186) and “the example methods, apparatus and articles of manufacture as disclosed herein reduce the number of computational cycles utilized by a processing device in order to process and/or otherwise execute a workload” (Behar, page 32, paragraph 0185) and where “an accelerator can execute an offloaded workload including a predefined data size dynamically by generating a composite result of each of the workload nodes of the workload prior to the completion of the entirety of the workload, when early termination is possible. This allows a dynamic processing of a predefined workload and reduces latencies and power consumption associated with processing the predefined workload” (Behar, page 14, paragraph 0027). Examiner notes that the layer control unit is the analyzing means which uses state machines. Examiner further notes that the workload node is the layer and the PEs are the computational building blocks.)
Behar does not explicitly disclose
for each layer of ANN, sorting before inference a plurality of weight tensors based on a plurality of ANN output and/or input features and calculating early termination statistics therefrom for each layer
Setting, during inference, a termination threshold for each layer in the ANN provided by a user or determined dynamically
Performing ANN processing on … said sorted plurality of weight tensors during inference on a plurality of processing elements (PEs), each PE comprising a plurality of multiply-accumulate (MAC) circuits and calculating activations for each layer
evaluating an early termination condition during inference in accordance with results of said calculations and the termination threshold
said evaluation comprising receiving feedback from said MAC circuits indicating saturation in MAC outputs
and during inference, …sending trigger indications to a next layer before processing would normally complete if said termination condition exceeds the termination threshold or continuing said processing and calculating otherwise
However, Berest discloses
for each layer of ANN, sorting before inference a plurality of weight tensors based on a plurality of ANN output and/or input features and calculating early termination statistics therefrom for each layer (Berest, page 5, last paragraph, "Let LM(outm, T ) denote a loss function of the cascade M with respect to the output of the m’th component, averaged over the labeled dataset T. In order to train the cascade M, we propose a backtrack-training (Algorithm 2) BT(M, T )…. We optimize the weights of the classifiers clfi, for 0 ≤ I ≤ D-2 (i.e., classifiers of intermediate components). Our approach differs from previous training procedures in which the loss functions associated with all the classifiers were jointly optimized". Examiner notes that this lines up with line 5 of Algorithm 2 (see below).
PNG
media_image1.png
276
936
media_image1.png
Greyscale
Examiner also notes that the early termination statistics are the loss functions where the loss function is calculated from optimizing the weights with respect to their outputs of the m’th layer. Examiner notes the sorting happens via the backtrack training while using the loss function.),
Setting, during inference, a termination threshold for each layer in the ANN provided by a user or determined dynamically (Berest, page 6, 2nd paragraph, “we present an automatic methodology for setting the confidence threshold … for every component Mm given an acceptable accuracy degradation….the important attribute of the automatic setting of the confidence thresholds is that one can change them on the fly during the inference stage.” Examiner notes that every component is each layer.)
Performing ANN processing on input data and said sorted plurality of weight tensors during inference on a plurality of processing elements (PEs) (Berest, page 5, last paragraph, "In order to train the cascade M, we propose a backtrack-training (Algorithm 2) BT(M,T). We emphasize that the training procedure first optimizes all the convolutional weights together with the weights of the last classifier. Only then, do we optimize the weights of the classifiers clf i, for 0 ≤ i ≤ D − 2 (i.e., classifiers of intermediate components)" where “the usage threshold for determining early termination in the cascade is listed as Algorithm 1. The algorithm applies the component DNNs one by one and stops as soon as the confidence measure reaches the confidence threshold of this component” (Berest, page 5, 3rd paragraph) (see algorithm below)
PNG
media_image2.png
403
1225
media_image2.png
Greyscale
And “Each component in a cascaded architecture consists of convolutional layers followed by a branching that leads to (1) a classifier and (2) the next component” (Berest, page 4, Section 3.1 Cascaded architecture, 1st paragraph). Examiner notes that the training procedure that optimizes all the convolutional weights together with the weights of the last classifier maps to sorted plurality of weight tensors, and lines 2-8 in Algorithm 1 is ANN processing on the input data and model which comprises the sorted weights. Examiner further notes that Algorithm 1 CI
(
M
,
δ
^
,
x
)
– Cascaded Inference is performed during inference. Examiner additionally notes that the cascaded architecture of algorithm 1 has layers, or a plurality of processing elements, on which to perform ANN processing)
Each PE comprising a plurality of multiply-accumulate (MAC) circuits (Berest, page 1, 1st paragraph, "State-of-the-art Deep Neural Networks (DNNs) usually consist of hundreds of layers [processing elements] and millions of trainable weights. At inference time, this translates into billions of multiply-accumulate operations (MACs) for a single input". Examiner notes that since each layer produces billions of MACs for a single input, each layer, or processing element, must have a MAC circuit to perform the MACs which are the output data for an input.)
and calculating activations for each layer (Berest, page 7, 3rd paragraph, “in order to examine the usefulness of cascaded inference we performed experiments on CIFAR-10, CIFAR-100 and SVHN datasets using ResNet-110 architecture” and Berest, page 7, Figure 2(c) (see figure below)
PNG
media_image3.png
733
471
media_image3.png
Greyscale
Examiner notes that the Figure 2(c) depicts calculating activations for each layer.
evaluating an early termination condition during inference in accordance with results of said calculations and the termination threshold (Berest, page 4, 2nd paragraph, "These classifiers attempt to classify the feature map and output a confidence measurement of their classification. If the confidence level is above a threshold, then execution terminates, and the classification of the intermediate feature map is output" where “As soon as the confidence threshold is met, inference terminates immediately without having to compute the output of the complete network” (Berest, abstract). Examiner notes that the early termination condition is if the confidence level is above a threshold, and results of said calculations is the confidence level/measurement.);
said evaluating comprising receiving feedback from said MAC circuits (Berest, page 1, 1st paragraph, "State-of-the-art Deep Neural Networks (DNNs) usually consist of hundreds of layers [processing elements] and millions of trainable weights. At inference time, this translates into billions of multiply-accumulate operations (MACs) for a single input" where “We trained the cascaded versions of ResNet-110 and ResNet-50-v2 models as described in Section 6. We evaluated the performance using various ϵ values. The tradeoff between test-accuracy and the number of MACs required for a single inference is shown in Figure 3. The MAC counts were obtained analytically by summing up the linear operations in the convolutional layers and the FC layers, excluding activations and batch normalization”. Examiner notes that since each layer produces billions of MACs for a single input, each layer, or processing element, must have a MAC circuit to perform the MACs which are the output data for an input. Examiner further notes that the feedback produced MACs is the number of MACs required for a single inference.)
and during inference, terminating processing for a particular layer and sending trigger indications to a next layer before processing would normally complete if said termination condition exceeds the termination threshold or continuing said processing and calculating otherwise (Berest, abstract "As soon as the confidence threshold is met, inference terminates immediately without having to compute the output of the complete network" where "The algorithm applies the component DNNs one by one and stops as soon as the confidence measure reaches the confidence threshold of this component” (Berest, page 5, 2nd paragraph). Examiner notes that this process lines up with Algorithm 1 [see image below]
PNG
media_image4.png
262
813
media_image4.png
Greyscale
Examiner notes that the confidence threshold being met is if said termination condition exceeds said selected threshold, and inference terminating immediately is terminating processing for a particular layer. Examiner also notes that since the algorithm stops when a layer’s confidence threshold is met, the algorithm is sending a trigger indication. Additionally, the for loop in Algorithm 1 is continuing processing and calculating otherwise).
Behar and Berest are analogous to the claimed invention because they both terminate processing of a machine learning model. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Behar to use the sorting, calculating, and evaluating from Berest because “using a softmax output as a confidence measure in a cascade of DNNs can provide a reduction of 15%-50% in the number of MAC operations while degrading the classification accuracy by roughly 1%” (Berest, page 11, 2nd paragraph).
Behar in view of Berest discloses MAC circuits on a neural network and receiving feedback from the MAC circuits as described above. Behar in view of Berest doesn’t explicitly disclose saturation. Thus, Behar in view of Berest does not explicitly disclose
Said evaluating comprising receiving feedback … indicating saturation in MAC outputs
Shah however discloses
Said evaluating comprising receiving feedback … indicating saturation in MAC outputs (Shah, page 2, 1st column, 1st paragraph, “Once saturation is detected, the adaption of center and radius is terminated and the learning rate for amplitude adjustment is increased to 0.06” where “the training process of gaussian networks is concerned with the determination of appropriate center, variance and amplitude values for each node. Our observations have shown that despite the fact that training is not yet complete, the center and radius parameters of several nodes appear as if they have reached their final value. In other words, the ability of the node to further learn a relationship through the adaptation of the center and radius parameters has become saturated” (Shah, page 1, 1st column, last paragraph). Examiner notes that the feedback is terminating the adaption of center and radius and increasing the learning rate. Examiner further notes that the MAC outputs are the center and radius. )
Behar, Berest, and Shah are analogous to the claimed invention because they terminate processing in machine learning models. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Behar in view of Berest to indicate saturation in MAC outputs. Doing so is advantageous because it “reduced training times without a degradation in the quality of the input-output mapping” (Shah, page 2, 2nd column, 1st paragraph).
Regarding claim 5, Behar in view of Berest and Shah teaches the method according to claim 1 (and thus the rejection of claim 1 is incorporated). Behar in view of Berest further teaches
wherein said termination threshold can be configured by a user (Berest, page 6, 2nd paragraph, "The important attribute of the automatic setting of the confidence thresholds is that one can change them on the fly during the inference stage." Examiner notes that because one can change the confidence thresholds during the inference stage, the termination threshold can be configured by a user.).
Behar and Berest are analogous to the claimed invention because they both terminate processing of a machine learning model. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Behar to configure the termination threshold by a user because “one can change them on the fly during the inference stage.” (Berest, page 6, 2nd paragraph).
Regarding claim 6, Behar in view of Berest and Shah teaches the method according to claim 1 (and thus the rejection of claim 1 is incorporated). Behar in view of Berest further teaches
wherein said termination threshold can be configured for each layer (Berest, page 6, 2nd paragraph, "we present an automatic methodology for setting the confidence threshold … for every component [layer] Mm given an acceptable accuracy degradation.").
Behar and Berest are analogous to the claimed invention because they both terminate processing of a machine learning model. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Behar to configure the termination threshold for each layer because “one can change them on the fly during the inference stage.” (Berest, page 6, 2nd paragraph).
Regarding claim 8, Behar in view of Berest and Shah teaches the method according to claim 1 (and thus the rejection of claim 1 is incorporated). Behar in view of Berest further teaches
wherein said evaluating said early termination condition is performed in accordance with a selected strategy of detecting no change to the output of a neuron over at least N cycles (Berest page 5, 2nd paragraph "The algorithm applies the component DNNs one by one and stops as soon as the confidence measure reaches the confidence threshold of this component” and see equation below (Berest, page 6, last paragraph):
PNG
media_image5.png
82
914
media_image5.png
Greyscale
Examiner notes that by using accuracy degradation to define the confidence threshold, Berest is performing in accordance with a selected strategy of detecting no change to a neuron. Because the algorithm applies component DNNs one by one and stops once the confidence measure reaches the threshold, Berest is evaluating said early termination condition over at least N cycles.)
Behar and Berest are analogous to the claimed invention because they both terminate processing of a machine learning model. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Behar to use technique of evaluating from Berest because “using a softmax output as a confidence measure in a cascade of DNNs can provide a reduction of 15%-50% in the number of MAC operations while degrading the classification accuracy by roughly 1%” (Berest, page 11, 2nd paragraph).
Regarding claim 9, Behar in view of Berest and Shah teaches the method according to claim 1 (and thus the rejection of claim 1 is incorporated). Behar in view of Berest further teaches
wherein said evaluating said early termination condition is performed in accordance with a selected strategy of the output of a neuron being either at or near zero or saturated over at least N cycles (Berest page 5, 2nd paragraph "The algorithm applies the component DNNs one by one and stops as soon as the confidence measure reaches the confidence threshold of this component” and see equation below (Berest, page 6, last paragraph)
PNG
media_image5.png
82
914
media_image5.png
Greyscale
Examiner notes that by using accuracy degradation to define the confidence threshold, Berest is performing in accordance with a selected strategy of the output of a neuron being…near zero. Because the algorithm applies component DNNs one by one and stops once the confidence measure reaches the threshold, Berest is evaluating said early termination condition…over at least N cycles.)
Behar and Berest are analogous to the claimed invention because they both terminate processing of a machine learning model. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Behar to use technique of evaluating from Berest because “using a softmax output as a confidence measure in a cascade of DNNs can provide a reduction of 15%-50% in the number of MAC operations while degrading the classification accuracy by roughly 1%” (Berest, page 11, 2nd paragraph).
Regarding claim 10, Behar teaches
a method of early termination of layer processing in an artificial neural network (ANN) implemented on a neural network processor, the method comprising (Behar, page 31, paragraph 0171, “the processor platform 1100 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network)” where “the process 1000 begins at block 1002 when the host processor interface 608 obtains one or more workloads from a host processor” (Behar, page 30, paragraph 0164) and “the executable instructions of blocks 1002…may be executed on at least one processor such as the example processor 1110” (Behar, page 25, paragraph 0123) and where “the examples disclosed herein analyze the data dependences of a workload node and determine whether a workload node is a candidate for early termination to allow for the dynamic processing of a predefined amount of data” (Behar, page 14, paragraph 0027). Examiner notes that the ANN model is the workload and the neural network processor is the host processor interface run on the example processor.):
a compiler (Behar, page 16, paragraph 0046, “The graph compiler 302 receives the workload 306 and assigns various workload nodes of the workload 306 (e.g., a graph) to various CBBs (e.g., any of the convolution engine 214, the MMU 316, the RNN engine 318, and/or the DSP 320) of the accelerator 308.”);
loading an ANN model into the neural network processor (Behar, page 31, paragraph 0171, “the processor platform 1100 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network)” where “the process 1000 begins at block 1002 when the host processor interface 608 obtains one or more workloads from a host processor” (Behar, page 30, paragraph 0164) and where “the executable instructions of blocks 1002…may be executed on at least one processor such as the example processor 1110” (Behar, page 25, paragraph 0123). Examiner notes that the ANN model is the workload and the neural network processor is the host processor interface run on the example processor.)
performing ANN processing on input data … during inference on a plurality of processing elements (PEs) (Behar, page 32, paragraph 0186, “Example 1 includes an apparatus to enable dynamic processing of a predefined workload at one or more computational building blocks of an accelerator, the apparatus comprising an interface to obtain a workload node from a controller of the accelerator, the workload node associated with a first amount of data, the workload node to be executed at a first one of the one or more computational building blocks, an analyzer to determine whether the workload node is a candidate for early termination” where “In FIG. 1, the input 102 is an image to be processed by the accelerator (e.g., a VPU, another AI accelerator, etc.). The first workload node 104 is a layer of the mask R-CNN that, when executed, identifies one or more features in the input 102 (e.g., the image) by convolving the image with one or more matrices indicative of features in the image, such as edges, gradients, color, etc.” (Behar, page 13, paragraph 0022). Examiner notes that the computational building blocks are the plurality of processing elements (PEs).)
during inference, terminating processing for a particular layer (Behar, page 33, paragraph 0200, “in response to determining that the workload node is a candidate for early termination, set a flag associated with a tile of the first amount of data, and means for dispatching, the means for dispatching to, in response to the tile being transmitted from the first one of the one or more computation building blocks to a buffer, stop execution of the workload node at the first one of the one or more computation building blocks.” Examiner notes that the computation building blocks are layers, the first one of the one or more computation building blocks is the particular layer, and stopping execution is terminating processing.)
wherein a hardware based layer control unit (LCU) comprising a state machine determines based on feedback from the plurality of PEs whether to terminate early a particular layer, thereby reducing computational resource usage, power consumption, and latency of the neural network (Behar, page 25, paragraph 0118, “In some examples, the example workload analyzer 604 implements example means for analyzing….The analyzing means is implemented by hardware logic, hardware implemented state machines, logic circuitry, and/or any other combination of hardware, software, and/or firmware” where “Example 1 includes an apparatus to enable dynamic processing of a predefined workload at one or more computational building blocks of an accelerator, the apparatus comprising an interface to obtain a workload node from a controller of the accelerator, the workload node associated with a first amount of data, the workload node to be executed at a first one of the one or more computational building blocks, an analyzer to determine whether the workload node is a candidate for early termination,” (Behar, page 32, paragraph 0186) and “the example methods, apparatus and articles of manufacture as disclosed herein reduce the number of computational cycles utilized by a processing device in order to process and/or otherwise execute a workload” (Behar, page 32, paragraph 0185) and where “an accelerator can execute an offloaded workload including a predefined data size dynamically by generating a composite result of each of the workload nodes of the workload prior to the completion of the entirety of the workload, when early termination is possible. This allows a dynamic processing of a predefined workload and reduces latencies and power consumption associated with processing the predefined workload” (Behar, page 14, paragraph 0027). Examiner notes that the layer control unit is the analyzing means which uses state machines. Examiner further notes that the workload node is the layer and the PEs are the computational building blocks.)
Behar does not explicitly disclose
for each layer of said ANN, calculating before inference a first metric including early termination statistics from a first plurality of weight tensors across a plurality of output features; …sorting said first plurality of weight tensors before inference in accordance with said first metric
for each layer of said ANN, calculating before inference a second metric from a second plurality of weight tensors across a plurality of input features; …sorting said second plurality of weight tensors before inference in accordance with said second metric
Setting, during inference, a termination threshold for each layer in the ANN provided by a user or determined dynamically
performing calculations utilizing said sorted first plurality of weight tensors and said sorted second plurality of weight tensors
Performing ANN processing on… said sorted first plurality of weight tensors, and said sorted second plurality of weight tensors during inference on a plurality of processing elements (PEs), each PE comprising a plurality of multiply-accumulate (MAC) circuits and calculating activations for each layer
evaluating an early termination condition during inference in accordance with results of said calculations and the termination threshold
said evaluation comprising receiving feedback from said MAC circuits indicating saturation in MAC outputs
and during inference, …sending trigger indications to a next layer before processing would normally complete if said termination condition exceeds the termination threshold or continuing said processing and calculating otherwise
However, Berest discloses
for each layer of said ANN, calculating before inference a first metric including early termination statistics from a first plurality of weight tensors across a plurality of output features; …sorting said first plurality of weight tensors before inference in accordance with said first metric (Berest, page 5, last paragraph, "Let LM(outm, T ) denote a loss function of the cascade M with respect to the output of the m’th component, averaged over the labeled dataset T. In order to train the cascade M, we propose a backtrack-training (Algorithm 2) BT(M, T )…. We optimize the weights of the classifiers clfi, for 0 ≤ I ≤ D-2 (i.e., classifiers of intermediate components). Our approach differs from previous training procedures in which the loss functions associated with all the classifiers were jointly optimized." Examiner notes that this lines up with line 5 of Algorithm 2 (see below).
PNG
media_image1.png
276
936
media_image1.png
Greyscale
Examiner also notes that a first metric including early termination statistics is the loss function where the loss function is calculated from optimizing the weights with respect to their outputs of the m’th layer [first plurality of weight tensors across a plurality of output features]. In addition, the sorting happens via the backtrack training while using the loss function [metric].);
for each layer of said ANN, calculating before inference a second metric from a second plurality of weight tensors across a plurality of input features; …sorting said second plurality of weight tensors before inference in accordance with said second metric (Berest, page 5, last paragraph, "Let LM(outm, T ) denote a loss function of the cascade M with respect to the output of the m’th component, averaged over the labeled dataset T. In order to train the cascade M, we propose a backtrack-training (Algorithm 2) BT(M, T )…. We optimize the weights of the classifiers clfi, for 0 ≤ I ≤ D-2 (i.e., classifiers of intermediate components). Our approach differs from previous training procedures in which the loss functions associated with all the classifiers were jointly optimized." Examiner notes that this lines up with line 3 of Algorithm 2 (see below).
PNG
media_image1.png
276
936
media_image1.png
Greyscale
Examiner also notes that a second metric is the loss function where the loss function is calculated from optimizing the weights with respect to outputs of the m’th layer [second plurality of weight tensors across a plurality of input features]. Because one layer is the input for the next layer, the output of a layer becomes the input into another layer. In addition, the sorting happens via the backtrack training while using the loss function [metric].);
Setting, during inference, a termination threshold for each layer in the ANN provided by a user or determined dynamically (Berest, page 6, 2nd paragraph, “we present an automatic methodology for setting the confidence threshold … for every component Mm given an acceptable accuracy degradation….the important attribute of the automatic setting of the confidence thresholds is that one can change them on the fly during the inference stage.” Examiner notes that every component is each layer.)
performing calculations utilizing said sorted first plurality of weight tensors and said sorted second plurality of weight tensors (Berest page 5, last paragraph, "We emphasize that the training procedure first optimizes all the convolutional weights together with the weights of the last classifier." Examiner notes that because the outputs of one layer become the inputs of another layer, optimizing [performing calculations] all the convolutional weights [second plurality of weight tensors] together with the weights of the last classifier [first plurality of weight tensors] reads on this invention.);
Performing ANN processing on input data, said sorted first plurality of weight tensors, and said sorted second plurality of weight tensors during inference on a plurality of processing elements (PEs) (Berest, page 5, last paragraph, " In order to train the cascade M, we propose a backtrack-training (Algorithm 2) BT(M,T). We emphasize that the training procedure first optimizes all the convolutional weights together with the weights of the last classifier. Only then, do we optimize the weights of the classifiers clf i, for 0 ≤ i ≤ D − 2 (i.e., classifiers of intermediate components)" where “the usage threshold for determining early termination in the cascade is listed as Algorithm 1. The algorithm applies the component DNNs one by one and stops as soon as the confidence measure reaches the confidence threshold of this component” (Berest, page 5, 3rd paragraph) (see algorithm below)
PNG
media_image2.png
403
1225
media_image2.png
Greyscale
And “Each component in a cascaded architecture consists of convolutional layers followed by a branching that leads to (1) a classifier and (2) the next component” (Berest, page 4, Section 3.1 Cascaded architecture, 1st paragraph). Examiner notes that the training procedure that optimizes all the convolutional weights [second plurality of weight tensors] together with the weights of the last classifier [first plurality of weight tensors] maps to sorted plurality of weight tensors, and lines 2-8 in Algorithm 1 is ANN processing on the input data and model which comprises the first and second plurality of weight tensors. Examiner further notes that Algorithm 1 CI
(
M
,
δ
^
,
x
)
– Cascaded Inference is performed during inference. Examiner additionally notes that the cascaded architecture of algorithm 1 has layers, or a plurality of processing elements, on which to perform ANN processing.);
Each PE comprising a plurality of multiply-accumulate (MAC) circuits (Berest, page 1, 1st paragraph, "State-of-the-art Deep Neural Networks (DNNs) usually consist of hundreds of layers [processing elements] and millions of trainable weights. At inference time, this translates into billions of multiply-accumulate operations (MACs) for a single input". Examiner notes that since each layer produces billions of MACs for a single input, each layer, or processing element, must have a MAC circuit to perform the MACs which are the output data for an input.)
and calculating activations for each layer (Berest, page 7, 3rd paragraph, “in order to examine the usefulness of cascaded inference we performed experiments on CIFAR-10, CIFAR-100 and SVHN datasets using ResNet-110 architecture” and Berest, page 7, Figure 2(c) (see figure below)
PNG
media_image3.png
733
471
media_image3.png
Greyscale
Examiner notes that the Figure 2(c) depicts calculating activations for each layer.
evaluating an early termination condition during inference in accordance with results of said calculations and the termination threshold (Berest, page 4, 2nd paragraph, "These classifiers attempt to classify the feature map and output a confidence measurement of their classification. If the confidence level is above a threshold, then execution terminates, and the classification of the intermediate feature map is output" where “As soon as the confidence threshold is met, inference terminates immediately without having to compute the output of the complete network” (Berest, abstract). Examiner notes that the early termination condition is if the confidence level is above a threshold, and results of said calculations is the confidence level/measurement.);
said evaluating comprising receiving feedback from said MAC circuits (Berest, page 1, 1st paragraph, "State-of-the-art Deep Neural Networks (DNNs) usually consist of hundreds of layers [processing elements] and millions of trainable weights. At inference time, this translates into billions of multiply-accumulate operations (MACs) for a single input" where “We trained the cascaded versions of ResNet-110 and ResNet-50-v2 models as described in Section 6. We evaluated the performance using various ϵ values. The tradeoff between test-accuracy and the number of MACs required for a single inference is shown in Figure 3. The MAC counts were obtained analytically by summing up the linear operations in the convolutional layers and the FC layers, excluding activations and batch normalization”. Examiner notes that since each layer produces billions of MACs for a single input, each layer, or processing element, must have a MAC circuit to perform the MACs which are the output data for an input. Examiner further notes that the feedback produced MACs is the number of MACs required for a single inference.)
during inference, terminating processing for a particular layer and sending trigger indications to a next layer before processing would normally complete if said termination condition exceeds the termination threshold or continuing said processing and calculating otherwise (Berest, abstract "As soon as the confidence threshold is met, inference terminates immediately without having to compute the output of the complete network" where "The algorithm applies the component DNNs one by one and stops as soon as the confidence measure reaches the confidence threshold of this component” (Berest, page 5, 2nd paragraph). Examiner notes that this process lines up with Algorithm 1 [see image below]
PNG
media_image4.png
262
813
media_image4.png
Greyscale
Examiner notes that the confidence threshold being met is if said termination condition exceeds said selected threshold, and inference terminating immediately is terminating processing for a particular layer. Examiner also notes that since the algorithm stops when a layer’s confidence threshold is met, the algorithm is sending a trigger indication. Additionally, the for loop in Algorithm 1 is continuing processing and calculating otherwise).
Behar and Berest are analogous to the claimed invention because they both terminate processing of a machine learning model. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Behar to use the sorting, calculating, and evaluating from Berest because “using a softmax output as a confidence measure in a cascade of DNNs can provide a reduction of 15%-50% in the number of MAC operations while degrading the classification accuracy by roughly 1%” (Berest, page 11, 2nd paragraph).
Behar in view of Berest discloses MAC circuits on a neural network and receiving feedback from the MAC circuits as described above. Behar in view of Berest doesn’t explicitly disclose saturation. Thus, Behar in view of Berest does not explicitly disclose
Said evaluating comprising receiving feedback … indicating saturation in MAC outputs
Shah however discloses
Said evaluating comprising receiving feedback … indicating saturation in MAC outputs (Shah, page 2, 1st column, 1st paragraph, “Once saturation is detected, the adaption of center and radius is terminated and the learning rate for amplitude adjustment is increased to 0.06” where “the training process of gaussian networks is concerned with the determination of appropriate center, variance and amplitude values for each node. Our observations have shown that despite the fact that training is not yet complete, the center and radius parameters of several nodes appear as if they have reached their final value. In other words, the ability of the node to further learn a relationship through the adaptation of the center and radius parameters has become saturated” (Shah, page 1, 1st column, last paragraph). Examiner notes that the feedback is terminating the adaption of center and radius and increasing the learning rate. Examiner further notes that the MAC outputs are the center and radius. )
Behar, Berest, and Shah are analogous to the claimed invention because they terminate processing in machine learning models. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Behar in view of Berest to indicate saturation in MAC outputs. Doing so is advantageous because it “reduced training times without a degradation in the quality of the input-output mapping” (Shah, page 2, 2nd column, 1st paragraph).
Regarding claim 13, Behar in view of Berest and Shah teaches the method according to claim 10 (and thus the rejection of claim 10 is incorporated). Behar in view of Berest further teaches
wherein said termination threshold is user configurable for each layer (Berest, page 6, 2nd paragraph, "In this section, we present an automatic methodology for setting the confidence threshold … for every component [layer] Mm given an acceptable accuracy degradation ϵ. We note that the hyper-parameter ϵ is a single parameter for the whole cascade, and the automatic methodology we present determines an individual confidence threshold for every component in the cascade. The important attribute of the automatic setting of the confidence thresholds is that one can change them on the fly during the inference stage." Examiner notes that because one can change the confidence thresholds during the inference stage, the termination threshold can be configured by a user.).
Behar and Berest are analogous to the claimed invention because they both terminate processing of a machine learning model. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Behar to configure the termination threshold by a user because “one can change them on the fly during the inference stage.” (Berest, page 6, 2nd paragraph).
Regarding claim 15, claim 15 recites substantially similar limitations to claim 8, and is therefore rejected under the same analysis.
Regarding claim 16, claim 16 recites substantially similar limitations to claim 9, and is therefore rejected under the same analysis.
Regarding claim 17, Behar teaches
an apparatus of early termination of layer processing in an artificial neural network (ANN), the model of which is loaded and implemented on a neural network processor, the method comprising (Behar, page 31, paragraph 0171, “the processor platform 1100 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network)” where “the process 1000 begins at block 1002 when the host processor interface 608 obtains one or more workloads from a host processor” (Behar, page 30, paragraph 0164) and “the executable instructions of blocks 1002…may be executed on at least one processor such as the example processor 1110” (Behar, page 25, paragraph 0123) and where “the examples disclosed herein analyze the data dependences of a workload node and determine whether a workload node is a candidate for early termination to allow for the dynamic processing of a predefined amount of data” (Behar, page 14, paragraph 0027). Examiner notes that the ANN model is the workload and the neural network processor is the host processor interface run on the example processor.):
a plurality of processing elements (PEs) (Behar, page 32, paragraph 0186, “Example 1 includes an apparatus to enable dynamic processing of a predefined workload at one or more computational building blocks of an accelerator, the apparatus comprising an interface to obtain a workload node from a controller of the accelerator, the workload node associated with a first amount of data, the workload node to be executed at a first one of the one or more computational building blocks, an analyzer to determine whether the workload node is a candidate for early termination” where “In FIG. 1, the input 102 is an image to be processed by the accelerator (e.g., a VPU, another AI accelerator, etc.). The first workload node 104 is a layer of the mask R-CNN that, when executed, identifies one or more features in the input 102 (e.g., the image) by convolving the image with one or more matrices indicative of features in the image, such as edges, gradients, color, etc.” (Behar, page 13, paragraph 0022). Examiner notes that the computational building blocks are the plurality of processing elements (PEs).)
a layer control unit (LCU) comprising a state machine (Behar, page 25, paragraph 0118, “In some examples, the example workload analyzer 604 implements example means for analyzing….The analyzing means is implemented by hardware logic, hardware implemented state machines, logic circuitry, and/or any other combination of hardware, software, and/or firmware” where “Example 1 includes an apparatus to enable dynamic processing of a predefined workload at one or more computational building blocks of an accelerator, the apparatus comprising an interface to obtain a workload node from a controller of the accelerator, the workload node associated with a first amount of data, the workload node to be executed at a first one of the one or more computational building blocks, an analyzer to determine whether the workload node is a candidate for early termination,” (Behar, page 32, paragraph 0186) and “the example methods, apparatus and articles of manufacture as disclosed herein reduce the number of computational cycles utilized by a processing device in order to process and/or otherwise execute a workload” (Behar, page 32, paragraph 0185). Examiner notes that the layer control unit is the analyzing means which uses state machines. Examiner further notes that the workload node is the layer.)
generating an inhibit signal (Behar, page 33, paragraph 0200, “in response to determining that the workload node is a candidate for early termination, set a flag associated with a tile of the first amount of data, and means for dispatching, the means for dispatching to, in response to the tile being transmitted from the first one of the one or more computation building blocks to a buffer, stop execution of the workload node at the first one of the one or more computation building blocks.” Examiner notes that the computation building blocks are layers, the first one of the one or more computation building blocks is the particular layer, and stopping execution is terminating processing. Examiner further notes that setting a flag is generating an inhibit signal.)
applying said inhibit signal to one or more processing elements of a layer thereby terminating processing for a particular layer before it would normally complete, thereby reducing computational resource usage, power consumption, and latency of the neural network (Behar, page 33, paragraph 0200, “in response to determining that the workload node is a candidate for early termination, set a flag associated with a tile of the first amount of data, and means for dispatching, the means for dispatching to, in response to the tile being transmitted from the first one of the one or more computation building blocks to a buffer, stop execution of the workload node at the first one of the one or more computation building blocks” and “the example methods, apparatus and articles of manufacture as disclosed herein reduce the number of computational cycles utilized by a processing device in order to process and/or otherwise execute a workload” (Behar, page 32, paragraph 0185) and where “an accelerator can execute an offloaded workload including a predefined data size dynamically by generating a composite result of each of the workload nodes of the workload prior to the completion of the entirety of the workload, when early termination is possible. This allows a dynamic processing of a predefined workload and reduces latencies and power consumption associated with processing the predefined workload” (Behar, page 14, paragraph 0027). Examiner notes that the computation building blocks are layers, the first one of the one or more computation building blocks is the particular layer, and stopping execution is terminating processing.)
Behar does not explicitly disclose
a plurality of processing elements, each having a multiply and accumulate (MAC) circuit operative to calculate an output and activations for each layer in accordance with input data and weights ordered before inference based on output features and/or input features
receive state information feedback from said processing elements during inference indicating a state of saturation of said MAC circuit
evaluate an early termination condition during inference in accordance with said state information and a termination threshold provided by a user or determined dynamically
sending trigger indications to a next layer if said termination condition exceeds said termination threshold or continuing to calculate an output and activations otherwise
However, Berest discloses
a plurality of processing elements, each having a multiply and accumulate (MAC) circuit operative to calculate an output and activations for each layer in accordance with input data and weights ordered before inference based on output features and/or input features (Berest, page 1, 1st paragraph, "State-of-the-art Deep Neural Networks (DNNs) usually consist of hundreds of layers [processing elements] and millions of trainable weights. At inference time, this translates into billions of multiply-accumulate operations (MACs) for a single input" and "We emphasize that the training procedure first optimizes all the convolutional weights together with the weights of the last classifier. Only then, do we optimize the weights of the classifiers clf i, for 0 ≤ i ≤ D − 2 (i.e., classifiers of intermediate components)" (Berest, page 5, last paragraph) where “in order to examine the usefulness of cascaded inference we performed experiments on CIFAR-10, CIFAR-100 and SVHN datasets using ResNet-110 architecture” (Berest, page 7, 3rd paragraph). Examiner notes that the image below depicts calculating activations for each layer.
PNG
media_image3.png
733
471
media_image3.png
Greyscale
Examiner notes that the training procedure that optimizes all the convolutional weights together with the weights of the last classifier maps to weights ordered before inference. Examiner also notes that since each layer produces billions of MACs for a single input, each layer [processing element] must have a MAC circuit to perform the MACs which are the output data for an input.),
and termination statistics calculated therefrom for each layer (Berest, page 5, last paragraph, "Let LM(outm, T ) denote a loss function of the cascade M with respect to the output of the m’th component, averaged over the labeled dataset T. In order to train the cascade M, we propose a backtrack-training (Algorithm 2) BT(M, T )…. We optimize the weights of the classifiers clfi, for 0 ≤ I ≤ D-2 (i.e., classifiers of intermediate components). Our approach differs from previous training procedures in which the loss functions associated with all the classifiers were jointly optimized" where “Our method can be easily incorporated into pre-trained non-cascaded architectures, as we exemplify on ResNet,” (Berest, page 1, abstract). Examiner notes that this lines up with line 5 of Algorithm 2 (see below).
PNG
media_image1.png
276
936
media_image1.png
Greyscale
Examiner also notes that the early termination statistics are the loss functions where the loss function is calculated from optimizing the weights with respect to their outputs of the m’th layer. Examiner notes the sorting happens via the backtrack training while using the loss function.)
receive state information feedback from said processing elements during inference indicating a state… of said MAC circuit (Berest, page 1, abstract, “As soon as the confidence threshold is met, inference terminates immediately without having to compute the output of the complete network” where “The usage of the threshold for determining early termination in the cascade is listed as Algorithm 1. The algorithm applies the component DNNs one by one and stops as soon as the confidence measure reaches the confidence threshold of this component” (Berest, page 5, 2nd paragraph). Examiner notes that because inference terminates after the threshold is met, this is receiving state information feedback from said processing elements [layers] indicating a state of said MAC circuit which the state of the MAC circuit in this case would be terminated since the layer is terminating.);
evaluate an early termination condition during inference in accordance with said state information and a termination threshold provided by a user or determined dynamically (Berest, page 4, 2nd paragraph, "If the confidence level is above a threshold, then execution terminates, and the classification of the intermediate feature map is output" where “we present an automatic methodology for setting the confidence threshold … for every component Mm given an acceptable accuracy degradation….the important attribute of the automatic setting of the confidence thresholds is that one can change them on the fly during the inference stage” (Berest, page 6, 2nd paragraph). Examiner notes that said state information is whether the confidence is above a threshold.);
generating an inhibit signal and sending trigger indications to a next layer if said termination condition exceeds said termination threshold or continuing to calculate an output and activations otherwise (Berest, page 4, 2nd paragraph, "These classifiers attempt to classify the feature map and output a confidence measurement of their classification. If the confidence level is above a threshold, then execution terminates, and the classification of the intermediate feature map is output" where “the algorithm applies the component DNNs one by one and stops as soon as the confidence measure reaches the confidence threshold of this component” (Berest, page 5, 2nd paragraph). Examiner notes that this process lines up with Algorithm 1 [see image below]
PNG
media_image4.png
262
813
media_image4.png
Greyscale
Examiner notes that the confidence threshold being met is if said termination condition exceeds said selected threshold, and inference terminating immediately is terminating processing for a particular layer. Examiner also notes that since the algorithm stops when a layer’s confidence threshold is met, the algorithm is sending a trigger indication. Additionally, the for loop in Algorithm 1 is continuing processing and calculating otherwise). Examiner notes that since the algorithm stops once a confidence measure reaches the threshold, an inhibit signal is sent to stop computation when execution terminates after the confidence level is above a threshold.);
Behar and Berest are analogous to the claimed invention because they both terminate processing of a machine learning model. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Behar to use the sorting, calculating, and evaluating from Berest because “using a softmax output as a confidence measure in a cascade of DNNs can provide a reduction of 15%-50% in the number of MAC operations while degrading the classification accuracy by roughly 1%” (Berest, page 11, 2nd paragraph).
Behar in view of Berest discloses MAC circuits on a neural network and receiving feedback from the MAC circuits as described above. Behar in view of Berest doesn’t explicitly disclose saturation. Thus, Behar in view of Berest does not explicitly disclose
receive state information feedback … indicating a state of saturation
Shah however discloses
receive state information feedback … indicating a state of saturation (Shah, page 2, 1st column, 1st paragraph, “Once saturation is detected, the adaption of center and radius is terminated and the learning rate for amplitude adjustment is increased to 0.06” where “the training process of gaussian networks is concerned with the determination of appropriate center, variance and amplitude values for each node. Our observations have shown that despite the fact that training is not yet complete, the center and radius parameters of several nodes appear as if they have reached their final value. In other words, the ability of the node to further learn a relationship through the adaptation of the center and radius parameters has become saturated” (Shah, page 1, 1st column, last paragraph). Examiner notes that the feedback is terminating the adaption of center and radius and increasing the learning rate.)
Behar, Berest, and Shah are analogous to the claimed invention because they terminate processing in machine learning models. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Behar in view of Berest to indicate saturation in MAC outputs. Doing so is advantageous because it “reduced training times without a degradation in the quality of the input-output mapping” (Shah, page 2, 2nd column, 1st paragraph).
Regarding claim 18, Behar in view of Berest and Shah teaches the apparatus according to claim 17 (and thus the rejection of claim 17 is incorporated). Behar in view of Berest further teaches
wherein said ordered weights are generated by sorting weights by output and input features before inference in accordance with a metric including statistics related to the weights (Berest, page 5, last paragraph, “In order to train the cascade M, we propose a backtrack-training (Algorithm 2) BT(M, T )…. We optimize the weights of the classifiers clfi, for 0 ≤ I ≤ D-2 (i.e., classifiers of intermediate components). Our approach differs from previous training procedures in which the loss functions associated with all the classifiers were jointly optimized." Examiner notes that the back track training uses outputs as inputs for the next layer in training the weights [sorting the weights]. Examiner further notes that the statistics related to the weights is the loss function.).
Behar and Berest are analogous to the claimed invention because they both terminate processing of a machine learning model. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Behar to use the sorting from Berest because “using a softmax output as a confidence measure in a cascade of DNNs can provide a reduction of 15%-50% in the number of MAC operations while degrading the classification accuracy by roughly 1%” (Berest, page 11, 2nd paragraph).
Regarding claim 19, claim 19 recites substantially similar limitations to claim 13, and is therefore rejected under the same analysis.
Regarding claim 21, Behar in view of Berest and Shah teaches the apparatus according to claim 17 (and thus the rejection of claim 17 is incorporated). Behar further teaches
wherein said LCU evaluates said early termination condition in accordance with a selected strategy (Behar, page 25, paragraph 0118, “In some examples, the example workload analyzer 604 implements example means for analyzing….The analyzing means is implemented by hardware logic, hardware implemented state machines, logic circuitry, and/or any other combination of hardware, software, and/or firmware” where “Example 1 includes an apparatus to enable dynamic processing of a predefined workload at one or more computational building blocks of an accelerator, the apparatus comprising an interface to obtain a workload node from a controller of the accelerator, the workload node associated with a first amount of data, the workload node to be executed at a first one of the one or more computational building blocks, an analyzer to determine whether the workload node is a candidate for early termination,” (Behar, page 32, paragraph 0186) and “the example methods, apparatus and articles of manufacture as disclosed herein reduce the number of computational cycles utilized by a processing device in order to process and/or otherwise execute a workload” (Behar, page 32, paragraph 0185). Examiner notes that the layer control unit is the analyzing means which uses state machines. Examiner further notes that the workload node is the layer.)
Behar does not teach, but Berest does teach
evaluates said early termination condition in accordance with a selected strategy of detecting no change to the output of said MAC circuit over at least N cycles (Berest page 5, 2nd paragraph "The algorithm applies the component DNNs one by one and stops as soon as the confidence measure reaches the confidence threshold of this component” and see equation below (Berest, page 6, last paragraph):
PNG
media_image5.png
82
914
media_image5.png
Greyscale
where "The algorithm applies the component DNNs one by one and stops as soon as the confidence measure reaches the confidence threshold of this component” (Berest, page 5, 2nd paragraph) and where "State-of-the-art Deep Neural Networks (DNNs) usually consist of hundreds of layers and millions of trainable weights. At inference time, this translates into billions of multiply-accumulate operations (MACs) for a single input." Examiner notes that the calculations done on the algorithm are done on a MAC circuit because MACs are used to train weights and must have a circuit to do so. Examiner further notes that by using accuracy degradation to define the confidence threshold, Berest is performing in accordance with a selected strategy of detecting no change to a neuron. Because the algorithm applies component DNNs one by one and stops once the confidence measure reaches the threshold, Berest is evaluating said early termination condition over at least N cycles.)
Behar and Berest are analogous to the claimed invention because they both terminate processing of a machine learning model. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Behar to use technique of evaluating from Berest because “using a softmax output as a confidence measure in a cascade of DNNs can provide a reduction of 15%-50% in the number of MAC operations while degrading the classification accuracy by roughly 1%” (Berest, page 11, 2nd paragraph).
Regarding claim 22, Behar in view of Berest and Shah teaches the apparatus according to claim 17 (and thus the rejection of claim 17 is incorporated). Behar further teaches
wherein said LCU evaluates said early termination condition in accordance with a selected strategy (Behar, page 25, paragraph 0118, “In some examples, the example workload analyzer 604 implements example means for analyzing….The analyzing means is implemented by hardware logic, hardware implemented state machines, logic circuitry, and/or any other combination of hardware, software, and/or firmware” where “Example 1 includes an apparatus to enable dynamic processing of a predefined workload at one or more computational building blocks of an accelerator, the apparatus comprising an interface to obtain a workload node from a controller of the accelerator, the workload node associated with a first amount of data, the workload node to be executed at a first one of the one or more computational building blocks, an analyzer to determine whether the workload node is a candidate for early termination,” (Behar, page 32, paragraph 0186) and “the example methods, apparatus and articles of manufacture as disclosed herein reduce the number of computational cycles utilized by a processing device in order to process and/or otherwise execute a workload” (Behar, page 32, paragraph 0185). Examiner notes that the layer control unit is the analyzing means which uses state machines. Examiner further notes that the workload node is the layer.)
Behar does not teach, but Berest does teach
evaluates said early termination condition in accordance with a selected strategy of the output of said MAC circuit being either at or near zero or saturated over at least N cycles (Berest page 5, 2nd paragraph "The algorithm applies the component DNNs one by one and stops as soon as the confidence measure reaches the confidence threshold of this component” and see equation below (Berest, page 6, last paragraph):
PNG
media_image5.png
82
914
media_image5.png
Greyscale
where "The algorithm applies the component DNNs one by one and stops as soon as the confidence measure reaches the confidence threshold of this component” (Berest, page 5, 2nd paragraph) and where "State-of-the-art Deep Neural Networks (DNNs) usually consist of hundreds of layers and millions of trainable weights. At inference time, this translates into billions of multiply-accumulate operations (MACs) for a single input." Examiner notes that the calculations done on the algorithm are done on a MAC circuit because MACs are used to train weights and must have a circuit to do so. Examiner notes that by using accuracy degradation to define the confidence threshold, Berest is performing in accordance with a selected strategy of the output of a neuron being near zero. Because the algorithm applies component DNNs one by one and stops once the confidence measure reaches the threshold, Berest is evaluating said early termination condition over at least N cycles.)
Behar and Berest are analogous to the claimed invention because they both terminate processing of a machine learning model. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Behar to use technique of evaluating from Berest because “using a softmax output as a confidence measure in a cascade of DNNs can provide a reduction of 15%-50% in the number of MAC operations while degrading the classification accuracy by roughly 1%” (Berest, page 11, 2nd paragraph).
Claim(s) 2-4 and 11-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Behar in view of Berest and Shah and in further view of Salimans et al. (“Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks”) (“Salimans”).
Regarding claim 2, Behar in view of Berest and Shah teaches the method according to claim 1 (and thus the rejection of claim 1 is incorporated). Behar in view of Berest and Shah does not teach, but Salimans does teach
wherein said early termination statistics comprises a mathematical norm (Salimans, page 2, 5th paragraph, “After associating a loss function to one or more neuron outputs, such a neural network is commonly trained by stochastic gradient descent in the parameters w, b of each neuron. In an effort to speed up the convergence of the optimization procedure, we propose to reparameterize each weight vector w in terms of a parameter vector v and a scalar parameter g…where “
w
=
g
v
v
” and “where
v
denotes the Euclidean norm of v.” Examiner notes that since each weight has a loss function [early termination statistics] and the weight vector is the Euclidean norm, the loss function is a part the Euclidean norm [mathematical norm].)
Behar in view of Berest and Shah and Salimans are analogous because they both teach deep neural networks that train using stochastic descent. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified Behar in view of Berest and Shah to use the weight normalization strategy in Salimans. Doing so would “improv[e] the optimizability of the weights of neural network models….In addition, the overhead imposed by [the] method is lower: no additional memory is required and the additional computation is negligible” (Salimans, page 2, 3rd paragraph).
Regarding claim 3, Behar in view of Berest and Shah teaches the method according to claim 1 (and thus the rejection of claim 1 is incorporated). Berest does not teach, but Salimans does teach
wherein said sorting comprising sorting either ascending or descending (Salimans, page 2, 2nd to last paragraph, “we propose to explicitly reparameterize the model and to perform stochastic gradient descent in the new parameters v, g directly.” Examiner notes that since the model is being trained using stochastic gradient descent, the weights are being sorted in a descending order.).
Behar in view of Berest and Shah and Salimans are analogous because they both teach deep neural networks that train using stochastic descent. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified Behar in view of Berest and Shah to use the sorting strategy in Salimans. Doing so “improves the conditioning of the gradient and leads to improved convergence of the optimization procedure: By decoupling the norm of the weight vector (g) from the direction of the weight vector … we speed up convergence of our stochastic gradient descent optimization" (Salimans, page 2, 6th paragraph).
Regarding claim 4, Behar in view of Berest and Shah teaches the method according to claim 1 (and thus the rejection of claim 1 is incorporated). Berest in view of Behar does not teach, but Salimans does teach
wherein said early termination statistics are selected from a group consisting of norm(weights), lambdamax/lambdamin, and variance (Salimans, page 2, 4th paragraph, "[they] consider standard artificial neural networks where the computation of each neuron consists in taking a weighted sum of input features, followed by an elementwise nonlinearity" and where the “reparameterization has the effect of fixing the Euclidean norm of the weight vector w" (Salimans, page 2, 5th paragraph). Examiner notes that since each layer has weights or early termination statistics, and each weight vector has the Euclidean norm, each layer has norm(weights).).
Behar in view of Berest and Shah and Salimans are analogous because they both teach deep neural networks that train using stochastic descent. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified Behar in view of Berest and Shah to use a normalization of the weights in Salimans. Doing so “improves the conditioning of the gradient and leads to improved convergence of the optimization procedure: By decoupling the norm of the weight vector (g) from the direction of the weight vector … we speed up convergence of our stochastic gradient descent optimization" (Salimans, page 2, 6th paragraph).
Regarding claim 11, Berest in view of Behar teaches the method according to claim 10 (and thus the rejection of claim 10 is incorporated). Berest in view of Behar does not teach, but Salimans does teach
wherein said first metric and second metric comprise a mathematical norm (Salimans, page 2, 5th paragraph, “After associating a loss function to one or more neuron outputs, such a neural network is commonly trained by stochastic gradient descent in the parameters w, b of each neuron. In an effort to speed up the convergence of the optimization procedure, we propose to reparameterize each weight vector w in terms of a parameter vector v and a scalar parameter g…where “
w
=
g
v
v
” and “where
v
denotes the Euclidean norm of v.” Examiner notes that since each weight has a loss function [metric] and the weight vector is the Euclidean norm, the loss function is a part the Euclidean norm [mathematical norm]. Examiner also notes that since there are multiple weights, there is a first and second metric.)
Behar in view of Berest and Shah and Salimans are analogous because they both teach deep neural networks that train using stochastic descent. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified Behar in view of Berest and Shah to use the weight normalization strategy in Salimans. Doing so would “improv[e] the optimizability of the weights of neural network models….In addition, the overhead imposed by [the] method is lower: no additional memory is required and the additional computation is negligible” (Salimans, page 2, 3rd paragraph).
Regarding claim 12, claim 12 recites substantially similar limitations to claim 4, and is therefore rejected under the same analysis.
Claim(s) 7, 14, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Berest in view Behar in further view of Hettinger et al. (“Forward Thinking: Building and Training Neural Networks One Layer at a Time”) (“Hettinger”).
Regarding claim 7, Behar in view of Berest and Shah teaches the method of claim 1 (and thus the rejection of claim 1 is incorporated). Behar in view of Berest and Shah further teaches
wherein said terminating processing for a particular layer comprises…a ‘done’ signal to a next layer (Berest, page 5, 2nd paragraph, "The algorithm applies the component DNNs one by one and stops as soon as the confidence measure reaches the confidence threshold of this component”. Examiner notes that since the algorithm stops when a layer’s confidence measure reaches the threshold, the algorithm of the layer is sending a done signal to the next layer.).
Behar and Berest are analogous to the claimed invention because they both terminate processing of a machine learning model. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Behar to use done signal because the processing “stops as soon as the confidence measure reaches the confidence threshold of this component” (Berest, page 5, 2nd paragraph).
Berest in view of Behar does not teach, but Hettinger does teach
wherein said terminating a layer comprises generating a ‘ready’ signal to a previous layer (Hettinger, page 4, 4th paragraph, "Once the first network is trained the weights coming into the first layer are frozen (and stored), and the training inputs…are pushed through the resulting layer to give new "synthetic" data… which is used to train the next layer." Examiner notes that before the next layer is trained, it sends a ‘ready’ signal to the previous layer so that the frozen and stored inputs can be the new inputs of the next layer.).
Behar in view of Berest and Shah and Hettinger are analogous because they both teach deep neural networks that use early exits. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified Behar in view of Berest and Shah to use the ‘ready’ signal in Hettinger. Doing so would provide the “advantages of…(i) speed: adding each new layer amounts to training a shallow network with only one hidden layer; and (ii) resilience to overfitting” (Hettinger, page 4, 5th paragraph).
Regarding claim 14, claim 14 recites substantially similar limitations to claim 7, and is therefore rejected under the same analysis.
Regarding claim 20, Behar in view of Berest and Shah teaches the apparatus according to claim 17 (and thus the rejection of claim 17 is incorporated). Behar further teaches
wherein said LCU is operative to generate… a ‘done’ signal (Behar, page 25, paragraph 0118, “In some examples, the example workload analyzer 604 implements example means for analyzing….The analyzing means is implemented by hardware logic, hardware implemented state machines, logic circuitry, and/or any other combination of hardware, software, and/or firmware” where “Example 1 includes an apparatus to enable dynamic processing of a predefined workload at one or more computational building blocks of an accelerator, the apparatus comprising an interface to obtain a workload node from a controller of the accelerator, the workload node associated with a first amount of data, the workload node to be executed at a first one of the one or more computational building blocks, an analyzer to determine whether the workload node is a candidate for early termination,” (Behar, page 32, paragraph 0186) and “the example methods, apparatus and articles of manufacture as disclosed herein reduce the number of computational cycles utilized by a processing device in order to process and/or otherwise execute a workload” (Behar, page 32, paragraph 0185) where “in response to determining that the workload node is a candidate for early termination, set a flag associated with a tile of the first amount of data, and means for dispatching, the means for dispatching to, in response to the tile being transmitted from the first one of the one or more computation building blocks to a buffer, stop execution of the workload node at the first one of the one or more computation building blocks” (Behar, page 33, paragraph 0200). Examiner notes that the layer control unit is the analyzing means which uses state machines. Examiner further notes that the workload node is the layer. Examiner additionally notes that the computation building blocks are layers, the first one of the one or more computation building blocks is the particular layer, and stopping execution is terminating processing. Examiner further notes that setting a flag is generating an ‘done’ signal.)
Behar does not explicitly teach, but Berest does teach
wherein said terminating processing for a particular layer comprises…a ‘done’ signal to a next layer if said termination condition exceeds the termination threshold (Berest, page 5, 2nd paragraph, "The algorithm applies the component DNNs one by one and stops as soon as the confidence measure reaches the confidence threshold of this component”. Examiner notes that since the algorithm stops when a layer’s confidence measure reaches the threshold, the algorithm of the layer is sending a done signal to the next layer.).
Behar and Berest are analogous to the claimed invention because they both terminate processing of a machine learning model. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Behar to use done signal because the processing “stops as soon as the confidence measure reaches the confidence threshold of this component” (Berest, page 5, 2nd paragraph).
Berest in view of Behar does not teach, but Hettinger does teach
wherein said terminating a layer comprises generating a ‘ready’ signal to a previous layer (Hettinger, page 4, 4th paragraph, "Once the first network is trained the weights coming into the first layer are frozen (and stored), and the training inputs…are pushed through the resulting layer to give new "synthetic" data… which is used to train the next layer." Examiner notes that before the next layer is trained, it sends a ‘ready’ signal to the previous layer so that the frozen and stored inputs can be the new inputs of the next layer.).
Behar in view of Berest and Shah and Hettinger are analogous because they both teach deep neural networks that use early exits. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified Behar in view of Berest and Shah to use the ‘ready’ signal in Hettinger. Doing so would provide the “advantages of…(i) speed: adding each new layer amounts to training a shallow network with only one hidden layer; and (ii) resilience to overfitting” (Hettinger, page 4, 5th paragraph).
Response to Arguments
The replacement drawing sheets have been examined.
The previous 112(a) rejections have been withdrawn in light of the instant amendments.
The previous 112(b) rejections have been withdrawn in light of the instant amendments.
Examiner respectfully notes that all of the 101 arguments were argued in the previous action. Examiner has included them here for completeness and has updated the responses accordingly.
Examiner additionally respectfully notes that many of the prior art rejections were also argued in the previous action. Examiner has included them here for completeness and has updated the responses accordingly.
On page 10-11, Applicant argues:
Considering representative independent claim 1, regarding Step 2A, Prong 1, the Examiner asserts that claim 1 recites mental processes, including calculating metrics, sorting weight tensors, performing calculations, and evaluating termination conditions. Applicant submits that the amended claim elements are not and cannot be performed in the human mind but are implemented on a compile and a neural network processor with specific hardware and preprocessing steps, as detailed below.
Regarding the Applicant’s argument that elements of claim 1 cannot be performed mentally, Examiner respectfully disagrees. Specifically, calculations can be performed in a human’s brain, sorting items can be performed mentally, and evaluating conditions can be mentally performed. Examiner further notes that the use of a compiler and a processor are generic computer components (see MPEP 2106.05(f)) which cannot provide significantly more. Examiner respectfully refers Applicant back to the 101 rejection of this action.
On page 11, Applicant argues:
The claim recites "on a compiler, for each layer of said ANN, sorting before inference a plurality of weight tensors based on a plurality of ANN output and/or input features and calculating early termination statistics therefrom for each layer." This step involves a compiler preprocessing weight tensors by sorting them based on ANN-specific features, enabling optimized inference without retraining. This is not a generic calculation but a tailored process for ANN layer optimization, reducing computational complexity before inference begins.
Regarding the Applicant’s argument that this step integrates the claim into a practical application, Examiner respectfully disagrees. Specifically, “for each layer of said ANN, sorting before inference a plurality of weight tensors based on a plurality of ANN output and/or input features and calculating early termination statistics therefrom for each layer” is a mental process as it encompasses a human mentally sorting weight tensors and calculating statistics. Examiner further notes that “on a compiler” amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).). Additionally, Examiner notes that integration of the claims into a practical application must come from additional elements and not abstract ideas (MPEP 2106.04(d)). Examiner respectfully refers Applicant back to the 101 rejection of this action.
On page 11, Applicant argues:
The claim also recites "wherein a hardware-based layer control unit (LCU) comprising a state machine determines based on feedback from the plurality of PEs whether to terminate early a particular layer, thereby reducing computational resource usage of the neural network processor." The LCU is a specific hardware component, not a generic computer, that uses a state machine to dynamically control layer processing based on real-time feedback from processing elements (PEs). This hardware-driven approach optimizes computational resource usage, reducing power consumption and processing time in neural network processors.
Regarding the Applicant’s argument that this limitation integrates the claim into a practical application, Examiner respectfully disagrees. Specifically, “determines based on feedback from the plurality of PEs whether to terminate early a particular layer, thereby reducing computational resource usage of the neural network processor” is a mental process as determining based on feedback to terminate processing can be performed in a human mind. Examiner also notes that “hardware-based layer control unit (LCU) comprising a state machine” is a generic computing component on which to perform the abstract idea (see MPEP 2106.05(f)), and therefore cannot provide significantly more. Examiner respectfully refers Applicant back to the 101 rejection of this action.
On page 12, Applicant argues:
The claim recites "setting, during inference, a termination threshold for each layer in the ANN provided by a user or determined dynamically" and "terminating processing for a particular layer and sending trigger indications to a next layer before processing would normally complete if said termination condition exceeds the termination threshold." This dynamic control mechanism, implemented via the LCU, enables real-time adaptation to input complexity, further enhancing efficiency.
Regarding the Applicant’s argument that this limitation integrates the claim into a practical application, Examiner respectfully disagrees. Specifically, “setting, during inference, a termination threshold for each layer in the ANN provided by a user or determined dynamically” is a mental process because a human can mentally set a threshold for each layer. Examiner further notes that “terminating processing for a particular layer” does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)) and “sending trigger indications to a next layer before processing would normally complete if said termination condition exceeds the termination threshold” does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g)). Examiner respectfully refers Applicant back to the 101 rejection of this action.
On page 11-12, Applicant argues:
These elements, taken together, integrate any alleged abstract idea into a practical
application by improving the technological process of ANN inference. The method reduces computational resource usage (e.g., multiply-accumulate operations) while maintaining accuracy, addressing a specific challenge in neural network processing, similar to McRO, Inc. v. Bandai Namco Games Am., 837 F.3d 1299 (Fed. Cir. 2016), where a specific improvement in computer animation was deemed patent-eligible. The claim's focus on a hardware-based LCU and preprocessing steps distinguishes it from merely applying mathematical operations on a generic computer (MPEP § 2106.0S(a)). The Examiner's assertion that the additional elements merely "apply it on a computer" (MPEP § 2106.0S(f)) overlooks the specific hardware implementation and preprocessing tailored to neural network processors, which provide a concrete technological improvement.
Regarding the Applicant’s argument that the additional elements of claim 1 integrate the abstract ideas into a practical application, the Examiner respectfully disagrees. Specifically, the sorting, setting a threshold, calculating, evaluating a condition, continuing processing and calculating, and determining whether to terminate a layer are all mental processes. A neural network processor, a compiler, terminating processing for a layer, performing ANN processing on input data and weight tensors during inference on a plurality of PEs and an LCU comprising a state machine are applying the abstract idea on generic computer components (see MPEP 2106.05(f)). Loading an ANN model into the processor and sending trigger indications are insignificant extra solution activity of data gathering (see MPEP 2106.05(g)). These elements taken alone and in combination cannot provide significantly more that the abstract idea itself. Examiner respectfully refers Applicant back to the 101 rejection of this action.
On page 12-13, Applicant argues:
Regarding Step 2B, even if the claim were directed to an abstract idea, it provides significantly more than the judicial exception. The combination of sorting weight tensors based on ANN features, loading a pre-trained model without modification, performing ANN processing on input data and the sorted plurality of weight tensors during inference, and using a hardware-based LCU with a state machine to control early termination is not a well-understood, routine, or conventional activity in ANN processing. See Berkheimer v. HP Inc., 881 F.3d 1360 (Fed. Cir. 2018). The Examiner's rejection cites MPEP § 2106.0S(f), asserting that the claim uses a computer as a tool, but fails to provide evidence that the claimed elements are conventional.
In response, the Applicant submits that sorting weight tensors based on ANN output/input features before inference is not a standard practice in ANN processing. It requires specific preprocessing to optimize layer computations, distinct from typical weight initialization or training methods. Further, the use of a hardware-based LCU with a state machine to dynamically terminate layer processing based on PE feedback is a nonconventional hardware solution. Most ANN systems rely on software-based control or fixed architectures (e.g., ResNet, as in prior art like Berestizshevsky), not dedicated hardware units for real-time termination decisions. Moreover, loading an ANN model without retraining or modifying weights is a non-conventional approach that enhances deployment efficiency, particularly for resource-constrained devices.
These elements, individually and in combination, provide an inventive concept by enabling efficient, hardware-accelerated ANN inference, reducing computational overhead in a manner not routinely practiced in the field. The Examiner's rejection does not address the specificity of these elements or provide evidence of their conventionality, as required by Berkheimer.
Claims 1-22 are thus patent-eligible under § 101 because they integrate any alleged abstract idea into a practical application by providing a technological improvement in ANN processing through a hardware-based LCU, weight tensor sorting, and efficient model deployment. Additionally, the claim provides significantly more than the judicial exception due to its non-conventional combination of elements. The Applicant respectfully requests withdrawal of the§ 101 rejection of Claim 1. If the Examiner believes further clarification is needed, the Applicant is available for an interview at the Examiner's convenience.
Regarding the Applicant’s argument that these elements provide significantly more than the abstract idea, the Examiner respectfully disagrees. Specifically, sorting weight tensors is a mental process, using an LCU with a state machine is a generic computing component on which to apply the abstract idea, and loading an ANN model is the well understood, routine, and conventional activity of “transmitting or receiving data over a network” (see MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
Regarding the Applicant’s argument that these elements provide an improvement, Examiner respectfully disagrees. Specifically, Examiner notes that the Applicant provides a bare assertion of an improvement without the detail necessary to be apparent to one of ordinary skill in the art and, thus, cannot provide an improvement (MPEP 2106.04(d)(1)). Claim 1 therefore remains rejected under 101 and thus claims 1-22 also remain rejected under 101. Examiner respectfully refers Applicant back to the 101 rejection of this action.
On page 16, Applicant argues:
Berestizshevsky describes a cascaded inference method where early termination is based on softmax confidence thresholds (Section 3 .2, Algorithm 1 ). The decision to terminate is made by comparing the softmax output of component DNNs to a confidence threshold (Page 5, Algorithm l; Page 6, Section 5). However, Berestizshevsky is silent on any hardware-based LCU or state machine for controlling termination. The reference focuses on software-based cascaded DNN architectures ( e.g., ResNet-110, ResNet-50-v2) and algorithmic processes (e.g., Algorithm 1, Algorithm 2) without describing specific hardware components or state machines. The mention of convolutional layers, classifiers, and softmax outputs (Page 4, Figure 1) pertains to software processing, not hardware control units.
Furthermore, Berestizshevsky's architecture applies to general multiclass classification architectures terminating with a softmax function (Page 2, Section 1. 1 ), with no indication of a hardware-specific implementation. The reference's experiments (Page 7, Section 6) use standard software frameworks (e.g., TensorFlow) and do not describe dedicated hardware like a neural network processor with an LCU. Inherent anticipation is not established, as a hardware-based LCU with a state machine is not a necessary feature of Berestizshevsky's software-based cascade. See Continental Can Co., 948 F.2d at 1268. Thus, Berestizshevsky fails to disclose this critical element of each of the independent claims.
Regarding the Applicant’s argument that the prior art does not disclose a hardware based LCU, the Examiner respectfully disagrees. Specifically, claim 1 is taught by a combination of Behar, Berestizshevsky, and Shah where Behar teaches the LCU as an analyzing means (Behar, page 25, paragraph 0118).
On page 17, Applicant argues:
Berestizshevsky does not disclose sorting weight tensors based on metrics derived from ANN output or input features. The reference describes a cascaded architecture where intermediate classifiers evaluate feature maps and terminate based on softmax confidence (Page 4, Section 3 .1; Page 5, Algorithm 1 ). While it mentions convolutional layers and weights, there is no discussion of sorting weight tensors or calculating metrics from them before inference. The confidence thresholds are derived from softmax outputs (Page 6, Section 5), not from weight tensor metrics. The reference's focus is on dynamic termination during inference, not on pre-inference sorting of weights (Page 2, Section 1. 1 ).
Additionally, Berestizshevsky's transformation of a pre-trained ResNet into a cascaded architecture (Page 7, Section 6) involves adding classifiers without modifying weights, but it does not describe sorting weights based on ANN features. The absence of this feature means Berestizshevsky does not anticipate this element of the independent claims.
Regarding the Applicant’s argument that the prior art does not disclose sorting weight tensors based on metrics derived from ANN output or input features, Examiner respectfully disagrees. Specifically, Examiner notes that since the weights are optimized during the back track training, the weight tensors are being sorted. Examiner further notes that since the classifiers containing the weights also optimized the loss functions, the sorting was based on ANN outputs or inputs. Examiner respectfully refers Applicant back to the 103 rejection of this action.
On page 18, Applicant argues:
The independent claims require "loading an ANN model into the neural network processor without requiring training data or modification to any training weights." This emphasizes that the ANN model is deployed for inference without altering pre-trained weights or needing additional training data.
Berestizshevsky describes transforming a pre-trained ResNet model into a cascaded architecture by adding classifiers and fine-tuning them (Page 7, Section 6; Page 6, algorithm 2). However, this process involves training the intermediate classifiers (Page 6, Algorithm 2, lines 4-5), which requires a training set T (Page 4, Table 1). The fine-tuning of classifiers (Page 8, Section 6) modifies weights of the fully connected layers, contradicting the claim's requirement of no modification to training weights. While the convolutional weights are frozen during fine-tuning (Page 8), the overall model requires training data to optimize the new classifiers, unlike the mechanism of the present invention, which avoids both training data and weight modification.
Regarding the Applicant’s argument the prior art does not disclose “loading an ANN model into the neural network processor without requiring training data or modification to any training weights,” Examiner respectfully disagrees. Firstly, Examiner respectfully notes that Applicant is arguing a limitation that has since been amended to exclude “without requiring training data or modification to any training weights”. Examiner will respectfully continue the response with the assumption that the limitation being argued is “loading the ANN model into the neural network processor”. Specifically, Behar in view of Berestizshevsky in further view of Shah disclose this limitation. Behar specifically teaches the ANN model into the neural network processor (Behar, page 31, paragraph 0171, “the processor platform 1100 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network)” where “the process 1000 begins at block 1002 when the host processor interface 608 obtains one or more workloads from a host processor” (Behar, page 30, paragraph 0164) and where “the executable instructions of blocks 1002…may be executed on at least one processor such as the example processor 1110” (Behar, page 25, paragraph 0123). Examiner notes that the ANN model is the workload and the neural network processor is the host processor interface run on the example processor.). Examiner respectfully refers Applicant back to the 103 rejection of this action.
On page 18-19, Applicant argues:
It is submitted that Berestizshevsky's teaching is purely algorithmic (i.e. no hardware, no sorting/compiler) while Behar's termination mechanism is dependency/tile-based (i.e. not activation/threshold-specific). The combination of the two references lacks explicit teaching for offline sorting, PE MAC circuit feedback, or exact LCU functionality. Neither reference teaches or mentions weight tensors and/or any sorting requirement.
Moreover, Berestizshevsky teaches an ad hoc (i.e. on the fly) criteria that is based on the actual softmax value. In contrast, the present invention is a sub-optimal approach that attempts to maximize the likelihood of gaining sufficient confidence about the output to allow early termination. The key difference here is that the present invention essentially determines upfront which elements contribute more (in relative terms) and ensures that these elements are calculated first. This approach is sub-optimal in the sense that it is not known what the actual input will be beforehand. It is implemented in the compiler during compile time without large hardware involvement.
It is also submitted that the cited references do not teach offline sorting of weight tensors and calculating statistics (compiler step). Neither reference teaches "on a compiler, for each layer...sorting before inference a plurality of weight tensors based on a plurality of ANN output and/or input features and calculating early termination statistics therefrom." Berestizshevsky lacks compiler or pre-inference sorting while cascades add classifiers to pre trained models, but weights are unchanged/fixed (see Abstract, section 1.1-1.2). Thresholds are calibrated offline, but not via weight sorting or feature-based statistics, i.e. mis classification section discusses likelihood and not sorting. Regarding Behar, the graph compiler assigns nodes to CBBs ([0046]), but lacks weight tensor sorting or statistics calculation. Further, termination candidacy is based on runtime data dependencies ( e.g., object count) and not pre-inference weights. This is a key difference in that the present invention teaches sorting optimizes for saturation-based termination (see specification [0260] with metrics like norm and variance based on MAC saturation feedback). The cited references lack any weight manipulation whatsoever, a limitation the Examiner completely ignores. The cited combination does not suggest this and the Examiner cannot use hindsight to suggest this (see In re Kubin: cannot use applicant's disclosure to fill gaps).
Regarding the Applicant’s argument that the prior art does not disclose sorting weight tensors based on metrics derived from ANN output or input features, Examiner respectfully disagrees. Specifically, Behar in view of Berest and Shah teach this limitation. More specifically, Examiner notes that under broadest reasonable interpretation of the claims, Behar discloses the compiler and Berest teaches “sorting before inference a plurality of weight tensors based on a plurality of ANN output and/or input features”, and Examiner notes that since the weights are optimized during the back track training, the weight tensors are being sorted. Examiner further notes that since the classifiers containing the weights also optimized the loss functions, the sorting was based on ANN outputs or inputs. Examiner respectfully refers Applicant back to the 103 rejection of this action.
Regarding the Applicant’s argument that the prior art does not disclose PE MAC circuit feedback, Examiner respectfully disagrees. Specifically, Examiner notes that Behar teaches the PEs and Berest teaches the MAC circuit feedback. Examiner further notes that Behar and Berest are analogous to the claimed invention because they both terminate processing of a machine learning model. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Behar to use the sorting, calculating, and evaluating from Berest because “using a softmax output as a confidence measure in a cascade of DNNs can provide a reduction of 15%-50% in the number of MAC operations while degrading the classification accuracy by roughly 1%” (Berest, page 11, 2nd paragraph).
Regarding the Applicant’s argument that the prior art does not disclose exact LCU functionality, Examiner respectfully disagrees. Specifically, Examiner notes that Behar discloses the LCU and Berest discloses the functionality. Examiner further notes that Behar and Berest are analogous to the claimed invention because they both terminate processing of a machine learning model. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Behar to use the sorting, calculating, and evaluating from Berest because “using a softmax output as a confidence measure in a cascade of DNNs can provide a reduction of 15%-50% in the number of MAC operations while degrading the classification accuracy by roughly 1%” (Berest, page 11, 2nd paragraph). Examiner respectfully points the Applicant to the above 103 rejections.
On page 19-20, Applicant argues:
It is further submitted the Examiner improperly maps the termination condition and evaluation. The present claims require "evaluating an early termination condition .. .in accordance with results of said calculations [activations] and the termination threshold." Regartding Berestizshevsky teaches confidence from softmax (i.e. activations), but threshold is used for misclassification risk (section 1.2, Algorithm 1) and not directly on activation calculations as claimed. There is no "calculating activations for each layer" explicitly tied to evaluation. Regarding Behar, termination if a candidate ([0 186]), but based on data amount/dependencies and not activations/thresholds. It is submitted that Berestizshevsky's confidence is post-classifier, not mid-layer activation-based while Behar lacks thresholds on calculations. Mapping "results of said calculations" to confidence is a stretch since the claim ties to PE activations and not high-level outputs.
Regarding the Applicant’s argument that the current mapping does not explicitly tie “calculating activations for each layer” to “evaluating an early termination condition…in accordance with results of said calculations and the termination threshold”, Examiner respectfully disagrees. Specifically, Examiner notes that by broadest reasonable interpretation, the calculated activations and the results of said calculations correspond to different results. Examiner notes that the results of said calculations are the outputs from Berest’s cascaded model that uses the activations. Examiner further notes that the broadest reasonable interpretation of “in accordance with” comprises using a model that calculates activations to generate results. The broadest reasonable interpretation of “in accordance with” also comprises using the model that calculates activations loosely with a threshold.
On page 20, Applicant argues:
It is also submitted that the Examiner improperly maps LCU hardware with state machine and PE feedback. The present invention claims a "hardware-based layer control unit (LCU) comprising a state machine determines based on feedback from the plurality of PEs whether to terminate early." In Behar, analyzer (see [0118]) uses state machines and determines candidacy ([0186]) but feedback is via tiles/credits and not direct PE feedback like saturation taught by the present invention. CBBs are PEs, but are layer-specific LCU, workloads are nodes and not necessarily ANN layers. Berestizshevsky does not teach any hardware whatsoever. The analyzer of Behar is not an LCU for ANN layers and "feedback from PEs" is not taught. Note that credits in Behar are buffer flow and not PE outputs. State machines are generic while any mapping ignores specificity. See In re Kotzab for references must teach elements, not mere capability. There is thus no motivation to adapt Behar's general accelerator to Berestizshevsky' s cascades without redesign.
Regarding the Applicant’s argument that the prior art doesn’t teach “a hardware based layer control unit (LCU) comprising a state machine determines based on feedback from the plurality of PEs whether to terminate early a particular layer”, the Examiner respectfully disagrees. Specifically, Behar teaches a hardware based layer control unit (LCU) comprising a state machine determines based on feedback from the plurality of PEs whether to terminate early a particular layer, thereby reducing computational resource usage, power consumption, and latency of the neural network (Behar, page 25, paragraph 0118, “In some examples, the example workload analyzer 604 implements example means for analyzing….The analyzing means is implemented by hardware logic, hardware implemented state machines, logic circuitry, and/or any other combination of hardware, software, and/or firmware” where “Example 1 includes an apparatus to enable dynamic processing of a predefined workload at one or more computational building blocks of an accelerator, the apparatus comprising an interface to obtain a workload node from a controller of the accelerator, the workload node associated with a first amount of data, the workload node to be executed at a first one of the one or more computational building blocks, an analyzer to determine whether the workload node is a candidate for early termination,” (Behar, page 32, paragraph 0186) and “the example methods, apparatus and articles of manufacture as disclosed herein reduce the number of computational cycles utilized by a processing device in order to process and/or otherwise execute a workload” (Behar, page 32, paragraph 0185) and where “an accelerator can execute an offloaded workload including a predefined data size dynamically by generating a composite result of each of the workload nodes of the workload prior to the completion of the entirety of the workload, when early termination is possible. This allows a dynamic processing of a predefined workload and reduces latencies and power consumption associated with processing the predefined workload” (Behar, page 14, paragraph 0027). Examiner notes that the layer control unit is the analyzing means which uses state machines. Examiner further notes that the workload node is the layer and the PEs are the computational building blocks.)
Examiner further notes that by broadest reasonable interpretation, a layer in an ANN comprises a node. Additionally, Examiner respectfully notes that Behar and Berest are analogous to the claimed invention because they both terminate processing of a machine learning model. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Behar to use the sorting, calculating, and evaluating from Berest because “using a softmax output as a confidence measure in a cascade of DNNs can provide a reduction of 15%-50% in the number of MAC operations while degrading the classification accuracy by roughly 1%” (Berest, page 11, 2nd paragraph). Examiner respectfully directs the Applicant to the above 103 rejection.
On page 20, Applicant argues:
It is further submitted that there is a lack of motivation to combine the references and there are no predictable results. Berestizshevsky teaches software cascades for accuracy computation trade-off while Behar teaches hardware for dynamic workloads ( e.g., vision pipelines) with early stop on data sufficiency. Thus, these two mechanisms differ greatly (i.e. confidence vs. dependency) and combining them requires non-obvious integration (e.g., mapping cascades to CBBs). There is no reason a POSITA would apply Berestizshevsky's softmax thresholds to Behar' s tile flags since the combination yields unpredictable hardware changes.
In response to applicant’s argument that there is no teaching, suggestion, or motivation to combine the references, the examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). In this case, Behar and Berest are analogous to the claimed invention because they both terminate processing of a machine learning model. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Behar to use the sorting, calculating, and evaluating from Berest because “using a softmax output as a confidence measure in a cascade of DNNs can provide a reduction of 15%-50% in the number of MAC operations while degrading the classification accuracy by roughly 1%” (Berest, page 11, 2nd paragraph).
On page 21-22, Applicant argues:
Salimans teaches a mechanism for weight normalization including a reparameterization of the weight vectors in a neural network that decouples the length of those weight vectors from their direction. By reparameterizing the weights in this way the conditioning of the optimization problem is improved and convergence of stochastic gradient descent is sped up. The reparameterization is inspired by batch normalization but does not introduce any dependencies between the examples in a minibatch. This means that the method can also be applied successfully to recurrent models such as LSTMs and to noise sensitive applications such as deep reinforcement learning or generative models, for which batch normalization is less well suited. Although the method is much simpler, it still provides much of the speed-up of full batch normalization. In addition, the computational overhead of the method is lower, permitting more optimization steps to be taken in the same amount of time.
Under 35 U.S.C. § 103, the Patent Office bears the burden of establishing a prima facie case of obviousness. In re Fine, 837 F.2d 1071, 1074 (Fed. Cir. 1988). There are four separate factual inquiries to consider in making an obviousness determination: ( 1) the scope and content of the prior art; (2) the level of ordinary skill in the field of the invention; (3) the differences between the claimed invention and the prior art; and (4) the existence of any objective evidence, or "secondary considerations," of non-obviousness. Graham v. John Deere Co., 383 U.S. 1, 17-18 (1966); see also KSR Int 'l Co. v. Teleflex Inc., 127 S. Ct. 1727 (2007). A claimed invention combining multiple known elements is not rendered obvious simply because each element was known independently in the prior art. KSR at 1741. Rather, there must still be some "reason that would have prompted" a person of ordinary skill in the art to combine the elements in the specific way that he or she did. Id. Also, modification of a prior art reference may be obvious only if there exists a reason that would have prompted a person of ordinary skill to make the change. Id. at 1740-41.
Thus, Berestizshevsky and Salimans, taken alone or in combination, fail to disclose the aforementioned elements of claims 2-4, 11-12. As expressed by the USPTO at MPEP § 2143.03, to establish prima facie obviousness of a claimed invention, all the claim limitations must be taught or suggested by the prior art. In re Royka, 490 F.2d 981 (CCPA 1974).
Regarding the Applicant’s argument that the prior art does not disclose claim 2-4, Examiner respectfully disagrees. Specifically, Examiner notes that claims 2-4 are taught by a combination of Berestizshevsky, Behar, Shah and Salimans. They are analogous because they teach deep neural networks that train using stochastic descent. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified Behar in view of Berest to use the weight normalization strategy in Salimans. Doing so would “improv[e] the optimizability of the weights of neural network models….In addition, the overhead imposed by [the] method is lower: no additional memory is required and the additional computation is negligible” (Salimans, page 2, 3rd paragraph). Examiner respectfully refers Applicant back to the 103 rejection of this action.
Therefore, claims 2-4 and claims 11-12 are remain rejected under 103.
On page 22-23, Applicant argues:
Hettinger teaches a general framework for training deep neural networks without backpropagation. This substantially decreases training time and also allows for construction of deep networks with many sorts of learners, including networks whose layers are defined by functions that are not easily differentiated, like decision trees. The main idea is that layers can be trained one at a time, and once they are trained, the input data are mapped forward through the layer to create a new learning problem. The process is repeated, transforming the data through multiple layers, one at a time, rendering a new data set, which is expected to be better behaved, and on which a final output layer can achieve good performance. This is called forward thinking and demonstrates a proof of concept by achieving state-of-the-art accuracy on the MNIST dataset for convolutional neural networks. A general mathematical formulation of forward thinking that allows for other types of deep learning problems to be considered is also provided.
Regarding claims 7, 14, 20, in light of the amendments and arguments regarding independent claim 1 discussed above, Applicant believes that claims 7, 14, and 20 dependent on claims 1, 10, 17, respectively, overcome the Examiner's § 103 rejection based on the Berestizshevsky and Hettinger references. The Examiner is respectfully requested to withdraw the rejection based on§ 103.
Regarding the Applicant’s argument that the dependent claims are allowable at least due in part to their dependency on the independent claims, the Examiner respectfully disagrees and notes the instant rejections and response to arguments regarding the independent claims above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Teerapittayanon et al. (“BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks”) also describes methods to implement early exits in deep neural networks.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KAITLYN R HAEFNER whose telephone number is (571)272-1429. The examiner can normally be reached Monday - Thursday: 7:15 am - 5:15 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached on (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.R.H./Examiner, Art Unit 2148 /MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148