DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This Action is non-final and is in response to the claims filed 02/07/2023. Claims 1-30 are currently pending, of which claims 1-30 are currently rejected.
Drawings
The drawings are objected to under 37 CFR 1.83(a). The drawings must show every feature of the invention specified in the claims. Therefore, the “max circuit configured to: receive output from the single first configurable nonlinear activation circuit as input” feature described in claim 6 must be shown or the feature(s) canceled from the claim(s). No new matter should be entered.
Closest figure showing this is fig. 7 showing the max circuit receiving data from the sequential circuit, but does not show the max circuit receiving the output of the CNLA circuit (interpreted as the single first configurable nonlinear activation circuit).
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“a first approximator” recited in claims 9, 16, 27, 30.
“a second approximator” recited in claims 9, 16, 27, 30.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 6, 9-16 and 27-30 rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 6 recites the limitation “a max circuit configured to: receive output from the single first configurable nonlinear activation circuit as input”. However, the written description fails to disclose the max circuit receiving data from the CNLA circuit (interpreted as the single first configurable nonlinear activation circuit). Paragraphs 0120-0122 disclose how the max circuit may obtain data from a buffer, or directly from the sequential circuit, but does not disclose the max circuit receiving data from the CNLA circuit.
Claims 9, 16, 27, and 30 recite the limitations “a first approximator” and “a second approximator”. Claims 10-15 and 28-29 recite the same limitation by reason of dependence. This limitation invokes 35 U.S.C 112 (f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to provide adequate written description of the corresponding structure, material, or acts for performing the entire claimed functions of this limitation. See rejection under 35 U.S.C 112 (b) below for further details as to the requirement for the written description.
Claims 10-15 and 28-29 inherit the same deficiency as Claims 9 and 27 by reason of dependence.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 9-16 and 27-30 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim limitations “a first approximator” and “a second approximator” as recited in claims 9, 16, 27, and 30 invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function.
The “first approximator” is included in CNLA function circuit 100, and is directly connected to bypass 105, and second approximator 104 as shown in figure 1, and is shown in figure 2 as block 206A as described in ¶0046 and directly connected to registers 219A and multiplexers 203A and 209A, and receives parameters retrieved from “a memory, a register, a look-up table, or the like” as described in ¶0033 of the specification, performs hardware-based mathematical function, such as a linear, quadratic, or cubic function as described in ¶0034, and may comprise a cubic approximator as described in ¶0035. These descriptions describe functional operations of the unit, and no algorithm could be found in the specification.
The “second approximator” is included in CNLA function circuit 100, and is directly connected to bypass 107, first approximator 102, and multiplier 108 as shown in figure 1, and is shown in figure 2 as block 206B as described in ¶0046 and directly connected to registers 219A and multiplexers 203A and 209A, and receives parameters retrieved from “a memory, a register, a look-up table, or the like” as described in ¶0033 and 0034 of the specification, performs hardware-based mathematical function, such as a linear, quadratic, or cubic function as described in ¶0034, and may comprise a cubic approximator as described in ¶0035. These descriptions describe functional operations of the unit, and no algorithm could be found in the specification.
Claims 10-15 and 28-29 inherit the same deficiency as Claims 9 and 27 by reason of dependence.
Claim 27 recites the limitations “wherein at least one of the first configurable nonlinear activation function circuits comprises: a first approximator configured to approximate a first function of the one or more approximation functions; a second approximator configured to approximate a second function of the one or more approximation functions; a first gain multiplier configured to multiply a first gain value based on the one or more gain parameters; and a constant adder configured to add a constant value based on the constant parameter.” This method claim is not directed to method steps, but rather to a system. A single claim which claims both an apparatus and the method steps of using the apparatus is indefinite, as it creates confusion as to when direct infringement occurs. Appropriate correction is required. See MPEP 2173.05(p).II.
Claim 28 inherits the same deficiency as claim 27 by reason of dependence. Also, claim 28 recites limitations similar to claim 27, which are directed to a system instead of method steps. It is rejected for the same reasons as claim 27.
Claim 29 inherits the same deficiency as claim 27 and 28 by reason of dependence and is rejected for the same reasons.
Claim 30 recites the limitations “the second configurable nonlinear activation function circuit comprises: a first approximator configured to approximate a first function using one or more first function parameters of the set of parameters; a second approximator configured to approximate a second function using one or more second function parameters of the set of parameters; a gain multiplier configured to multiply a gain value based on one or more gain parameters of the set of parameters; and a constant adder configured to add a constant value based on a constant parameter of the set of parameters, the gain parameters comprise a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is bypassed, and the second function is a natural logarithm look-up table.” Similar to claim 27, this method claim is not directed to method steps, but rather to a system. A single claim which claims both an apparatus and the method steps of using the apparatus is indefinite, as it creates confusion as to when direct infringement occurs. Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-2, 5, 8, 17, 18, 21, 22, and 24- 26 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1, at Step 1 the claim is directed to an apparatus, which is a statutory category of invention.
At Step 2A, Prong 1, Examiner notes that the claims are directed to mathematical concepts:
A processor, comprising:
one or more first configurable nonlinear activation function circuits configured to perform an exponential function on input data (mathematical calculations);
a summation circuit configured to receive output data of the one or more first configurable nonlinear activation function circuits; and
a second configurable nonlinear activation function circuit configured to receive output data of the summation circuit, perform a natural logarithm function, and output an approximated log softmax of the input data (mathematical calculations).
At Step 2A Prong 2, the additional elements are bolded above. These additional elements are merely an “apply it” scenario using generically recited computer components. See MPEP 2106.05 (f). In the “one or more first configurable nonlinear activation function circuits configured to” limitation, the claim is simply using generic circuitry to perform the mathematical calculations (perform an exponential function on input data). The “a second configurable nonlinear activation function circuit configured to” limitation, the claim simply uses generic circuitry to perform the mathematical calculations (perform a natural logarithm function, and output an approximated log softmax of the input data). The “a summation circuit configured to” limitation simply uses generic circuitry to receive the output data corresponding to the mathematical calculation. Alternatively, even if not considered as merely an “apply it” scenario, mere data transmitting recited at a high level in an insignificant extra solution activity. See Step 2B analysis below.
The italicized limitations above are describing insignificant extra-solution activity used for the processing of the mathematical concepts.
At Step 2B, there are no additional elements claimed that amount to significantly more than the recited judicial exception.
In regards to the insignificant extra-solution activity found in the Italicized limitations, the “receive output data” and “output an approximated log softmax of the input data” limitations describe mere data transmitting recited at a high level of generality. Per MPEP 2106.05(d)(II), the courts have recognized the following computer functions as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity: i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362. Therefore, data transmitting is considered as a computer function that is a well-understood, routine, and conventional function when claimed in a merely generic manner. These limitations therefore remain insignificant extra-solution activity even upon consideration. Thus, these limitations do not amount to significantly more.
Claim 2 further recites o the mathematical concept of performing computations resulting in an approximated log softmax of data (mathematical calculations). Moreover, under step 2A prong 2, the additional elements regarding the first and second cycle merely describe computing cycles recited at a high level, and are considered an insignificant extra solution activity. At Step 2B, the first and second cycles describe mere computing cycles recited at a high level of generality. Per the book Computer Organization and Design, Chapter 6 Parallel Processors from Client to Cloud, “Almost all computers are constructed using a clock that determines when events take place in the hardware. These discrete time intervals are called clock cycles (or ticks, clock ticks, clock periods, clocks, cycles).” (Page 33, Second paragraph). These limitations are well understood, routine, and conventional . Thus, these limitations do not amount to significantly more.
Claim 5 is merely describing the first configurable nonlinear activation circuit performing the mathematical calculations (exponential function). The additional element “sequential computing circuit” simply uses generic circuitry to transmit data to perform the mathematical concepts. As stated in claim 1 analysis, mere transmitting of data is a well-understood, routine, and conventional function when claimed in a merely generic manner.
Claim 8 further recites the mathematical concept of performing a nonlinear activation function (mathematical calculations). Under Steps 2A prong 2 and 2B, the claim does not recite any additional elements that integrate the abstract idea into a practical application nor do they amount to significantly more than the judicial exception.
Claims 17, 18, 21, and 24- 26 are method claims practiced by the apparatus of claims 1, 2, 5, and 8. Under Steps 2A prong 2 and 2B, the claims do not recite any additional elements that integrate the abstract idea into a practical application nor do they amount to significantly more than the judicial exception.
Claim 22 further recites the mathematical concept of determining a maximum value (mathematical relationships). Under Steps 2A prong 2 and 2B, the claim does not recite any additional elements that integrate the abstract idea into a practical application nor do they amount to significantly more than the judicial exception.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5, 8, 17, 21, 24 and 25 are rejected under 35 U.S.C. 103 as being unpatentable over He et al. (U.S. Patent Application Publication No.: US 20200394507 A1), hereinafter “He”, in view of Luschi et al. (U.S. Patent Application Publication No.: US 20220051095 A1), hereinafter “Luschi”.
Regarding Claim 1, He teaches:
A processor, comprising:
one or more first configurable nonlinear activation function circuits configured to perform an exponential function on input data (Fig. 5, e.g., shows first conversion circuit 410; ¶0079, e.g., first conversion circuit 410 performs exponential operation on first input data X);
a summation circuit configured to receive output data of the one or more first configurable nonlinear activation function circuits (Fig. 5, e.g., shows summation and comparison circuit 430 receiving output from first conversion circuit 410); and
a second configurable nonlinear activation function circuit configured to receive output data of the summation circuit, … (Fig. 5, e.g., shows second conversion circuit 440 receiving output from summation and comparison circuit 430).
He does not teach:
a second configurable nonlinear activation function circuit configured to receive output data of the summation circuit, perform a natural logarithm function, and output an approximated log softmax of the input data.
However, Luschi teaches an activation function that computes the natural logarithm of a Softmax value, where determining the natural logarithm comprises determining an approximation of the natural logarithm of the sum of exponential values. Luschi explains “the activation function implementation is configured to determine or compute the natural log of the Softmax value for that vector.” (Luschi: ¶0121), and “determining a natural logarithm of the sum of the exponential of each of the input values comprises determining an approximation of the natural logarithm of the sum of the exponential of each of the input values” (Luschi: ¶0055).
He teaches the second nonlinear operation circuit performing a logarithmic operation based on the data outputted from first nonlinear operation circuit, which performs an exponential operation. See He: ¶0020-0022 and Fig. 5. Therefore, it would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to which said subject matter pertains to modify the activation function of second nonlinear operation circuit performing a logarithmic operation as taught by He to perform a natural logarithm of a Softmax value to determine an approximation of the natural logarithm of the sum of exponential values as taught by Luschi. One would have been motivated to combine these references because both references disclose using logarithms for activation functions, and Luschi enhances the model of He because “The calculation of the natural log of the Softmax value eliminates the risk of overflow.” (Luschi: ¶0123)
Regarding Claim 5, He in view of Luschi teach:
The processor of Claim 1, wherein the one or more first configurable nonlinear activation function circuits comprise a single first configurable nonlinear activation circuit configured to receive the input data from a sequential computing circuit (He: Fig. 5, e.g., shows computing sub-circuit including single first nonlinear operation circuit; Fig. 3, e.g., shows computing sub-circuits in first computing circuit; Fig. 2, e.g., shows first computing circuit receiving data from input control block; ¶0067, e.g., neural network processor performs computations sequentially).
Regarding Claim 8, He in view of Luschi teach:
The processor of Claim 1, wherein at least one of the first configurable nonlinear activation function circuits is configured to:
determine a nonlinear activation function for application to the input data (He: ¶0020, e.g., first nonlinear operation circuit performs logarithmic operation or exponential operation on first input data);
determine, based on the determined nonlinear activation function, a set of parameters for the nonlinear activation function (He: ¶0079, e.g., first conversion circuit performs exponential operation or logarithmic operation on first input data x based on a control signal, or directly outputs x); and
generate output data based on application of the set of parameters for the nonlinear activation function (He: ¶0079, e.g., first conversion circuit performs exponential operation or logarithmic operation on first input data x based on a control signal, or directly outputs x).
Regarding 17, 21, and 24, they are method claims practiced by the apparatus of claims 1, 5, 8. They are rejected for the same reasons as claim 1, 5, 8.
Regarding Claim 25, He in view of Luschi teach:
The method of Claim 24, further comprising retrieving the set of parameters from a memory based on the determined nonlinear activation function (He: ¶0069, e.g., data storage circuit stores inputs for first computing circuit; Fig. 2).
Claims 2 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over He in view of Luschi, further in view of Jeongjun Lee in NPL: “Spike-Train Level Direct Feedback Alignment: Sidestepping Backpropagation for On-Chip Training of Spiking Neural Nets” (www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2020.00143/full), hereinafter “Lee”, further in view of Henry et al. (U.S. Patent Applicant Publication No.: US 20180267898 A1), hereinafter “Henry”.
Regarding Claim 2, He in view of Luschi teach the processor of Claim 1. They do not teach:
wherein the second configurable nonlinear activation function circuit is configured to output the approximated log softmax of the input data … the approximated log softmax of the input data is provided as input to the one or more first configurable nonlinear activation function circuits, and wherein the one or more first configurable nonlinear activation function circuits are configured to output an approximated softmax of the input data based on the approximated log softmax of the input data.
However, in the same field of endeavor, Lee teaches feedback of data to previous layers in a neural network. Lee explains “Backpropagation (BP) has been widely applied to train neural networks. It is based upon computing a global error at the output layer and then propagating the error signal to hidden neurons layer by layer.” (Lee: Page 3, Section 2.1.1. Direct Feedback Alignment, First Paragraph). Lee further shows in Fig. 2 the proposed direct feedback alignment, where data is fed back to previous layers.
Combination of the Direct Feedback Alignment method taught Lee with the computing sub-circuit taught by He would cause for the output data to be fed back to the inputs of the computing sub-circuit. Hence, the output of the second nonlinear operation circuit would input data to the first nonlinear operation circuit. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to which said subject matter pertains to perform this combination because these references both disclose neural networks and using activation functions, and Lee enhances the model of He in view of Luschi because “DFA can be performed for all hidden layers concurrently, reducing the backward phase latency.” (Lee: Page 2, Second column, Third paragraph)
He in view of Luschi in view of Lee do not teach:
The processor of Claim 1, wherein the second configurable nonlinear activation function circuit is configured to output the approximated log softmax of the input data during a first cycle, wherein during a second cycle subsequent to the first cycle, the approximated log softmax of the input data is provided as input to the one or more first configurable nonlinear activation function circuits, and wherein the one or more first configurable nonlinear activation function circuits are configured to output an approximated softmax of the input data based on the approximated log softmax of the input data.
However, Henry explains how activation functions are performed in a single clock cycle. Henry explains “circuitry of the AFU 212 performs the activation function in a single clock cycle.” (Henry: ¶0096).
Therefore, it would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to which said subject matter pertains to modify the computing sub-circuits taught by He in view of Luschi in view of Lee to perform activation functions in a single clock cycle as taught by Henry. One would have been motivated to combine these references because both references disclose activation functions in neural networks, and Henry enhances the model of He in view of Luschi in view of Lee by allowing for activation functions to process in a single clock cycle.
Regarding Claim 18, it is a method claim practiced by the apparatus of claim 2. It is rejected for the same reasons as claim 2.
Claims 3 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over He in view of Luschi, further in view of Yamazaki et al. (U.S. Patent No.: US 12033694 B2), hereinafter “Yamazaki”.
Regarding Claim 3, He in view of Luschi teach the processor of Claim 1. They do not teach:
wherein the one or more first configurable nonlinear activation function circuits comprise a plurality of first configurable nonlinear activation circuits, each associated with a corresponding output element from a parallelized computing array.
However, Yamazaki teaches using circuits ACTF in each column of a computing array, where each ACTF performs an activation function. Yamazaki explains “That is, the circuit ACTF[1] to the circuit ACTF[n] function as circuits that perform arithmetic operation of an activation function of the above-described neural network” (Yamazaki: Column 63 Line 65- Column 64 Line 64. See also Yamazaki: Fig. 25)
Therefore, it would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to which said subject matter pertains to combine the array ALP outputting results to ACTF circuits in each column as taught by Yamazaki with the N-1 computing sub-circuits as taught by He. One would have been motivated to combine these references because both references disclose performing nonlinear functions in neural networks, and Yamazaki enhances the model of He in view of Luschi by allowing for nonlinear functions to be applied to results from each multiplication-accumulation operation in each layer. See Yamazaki Column 59 Lines 46-58 and Column 60 Lines 31-61. Combination would cause for each computing sub-circuit taught by He to receive each output from the columns of the array taught by Yamazaki.
Regarding Claim 19, it is a method claim practiced by the apparatus of claim 3. It is rejected for the same reasons as claim 3.
Claims 4 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over He in view of Luschi, in view of Yamazaki, further in view of Ozaki et al. (U.S. Patent No.: US 10083153 B2), hereinafter “Ozaki”.
Regarding Claim 4, He in view of Luschi in view of Yamazaki teach the processor of Claim 3. They do not teach:
further comprising a max circuit configured to:
receive output elements from the parallelized computing array as input, and output a maximum value from the output elements to the one or more first configurable nonlinear activation function circuits.
However, Ozaki teaches:
a max circuit (Fig. 4, e.g., pooling processing portion; Column 5 Lines 17-20, e.g., activation portion receives max pooling results) configured to:
receive output elements from the parallelized computing array as input (Fig. 4, e.g., pooling portion receives outputs from systolic array), and output a maximum value from the output elements to the one or more first configurable nonlinear activation function circuits (Column 5 Lines 17-20, e.g., activation portion receives max pooling results).
Therefore, it would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to which said subject matter pertains to modify the computing sub-circuit including an array as taught by He in view of Luschi in view of Yamazaki to including a max polling portion between the array and the activation function as taught by Ozaki. One would have been motivated to combine these references because both references disclose activation functions in neural networks, and Ozaki enhances the model of He in view of Luschi in view of Yamazaki by allowing for larger size pixels to be represented using smaller pixels. See Ozaki: Column 3 Line 53 – Column 4 Line 6.
Regarding Claim 20, it is a method claim practiced by the apparatus of claim 4. It is rejected for the same reasons as claim 4.
Claims 6 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over He in view of Luschi, in view of Yamazaki, further in view of Brothers et al. (U.S. Patent Application Publication No.: US 20160358069 A1), hereinafter “Brothers”.
Regarding Claim 6, He in view of Luschi teach the processor of claim 5. They do not teach:
further comprising a max circuit configured to:
receive output from the single first configurable nonlinear activation circuit as input, and
output a maximum value from the sequential computing circuit.
However, Brothers teaches:
a max circuit (Fig. 2, e.g., Pooling and sub-sampling (PSS) circuit; ¶0042, e.g., PSS circuit takes maximum value at each 2x2 portion) configured to:
receive output from the single first configurable nonlinear activation circuit as input (Fig. 2, e.g., PSS circuit 135 receives output from Activation circuit 130), and
output a maximum value (¶0042, e.g., PSS circuit takes maximum value at each 2x2 portion) …
Therefore, it would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to which said subject matter pertains to modify the computing sub-circuit as taught by He to include the PSS circuit taking a maximum value from each portion of data. One would have been motivated to combine these references because both references disclose activation functions in neural networks, and Brothers enhances the model of He in view of Luschi because “PSS circuit 135 may perform threshold comparisons at low cost” (Brothers: ¶0043)
Regarding Claim 22, it is a method claim practiced by the apparatus of claim 6. It is rejected for the same reasons as claim 6.
Claims 7 and 23 are rejected under 35 U.S.C. 103 as being unpatentable over He in view of Luschi, further in view of Nestler et al. (U.S. Patent Application Publication No.: US 20190080231 A1), hereinafter “Nestler”.
Regarding Claim 7, He in view of Luschi teach:
The processor of Claim 5, further comprising a [data storage block] configured to [store] output from the sequential computing circuit (He: Fig. 2, e.g., shows data storage block receiving data from input control block 202).
He in view of Luschi do not teach:
The processor of Claim 5, further comprising a memory buffer configured to buffer output from the sequential computing circuit.
However, Nestler teaches using buffers for inputs for a neural network. Nestler explains “the input activation buffer 308 holds 3×3×d input activations and broadcasts the input activations over the neuron array 310” (Nestler: ¶0061)
Therefore, it would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to which said subject matter pertains to modify the data storage block as taught by He to function as data buffers as taught by Nestler. One would have been motivated to combine these references because both references disclose storing data for neural networks, and Nestler enhances the model of He in view of Luschi by allowing for the storage blocks to perform striding operations. See Nestler: ¶0061.
Regarding Claim 23, it is a method claim practiced by the apparatus of claim 7. It is rejected for the same reasons as claim 7.
Allowable Subject Matter
Claims 9-16 and 26-30 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112 or 35 U.S.C. 112 (pre-AIA ), and if claim 26 is rewritten to overcome the 35 U.S.C. 101 rejection set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
He teaches a neural network processor that includes a first computing circuit comprising a first nonlinear operation circuit for performing exponential operations on input data x, and a second nonlinear operation circuit performing logarithmic operations. See ¶0082-0084 and Figs. 4-5. He does not teach or suggest the first nonlinear operation circuit and second nonlinear operation circuit consisting of “a first approximator configured to approximate a first function using one or more first function parameters of the set of parameters; a second approximator configured to approximate a second function using one or more second function parameters of the set of parameters; a gain multiplier configured to multiply a gain value based on one or more gain parameters of the set of parameters; and a constant adder configured to add a constant value based on a constant parameter of the set of parameters” as described in claims 9, 16, 27, and 30, nor does it describe the first or second nonlinear operation circuits receiving a set of parameters including “a combination of one or more gain parameters, a constant parameter, and one or more approximation functions to apply to the input data via the configurable nonlinear activation function circuit” as described in claim 26. Therefore, He does not teach the combination of claims 9-16 and 26-30, including the limitations described above.
Luschi teaches an activation function generator inside each node of a neural network. The activation function generator receives weighed input values and generates an output value based on an activation function. Luschi further teaches an example of the activation function is the softmax function and taking the natural log of the softmax value. See ¶0009-0010, 0121-0124 and Fig. 1b. Luschi does not teach or suggest an activation function generator including “a first approximator configured to approximate a first function using one or more first function parameters of the set of parameters; a second approximator configured to approximate a second function using one or more second function parameters of the set of parameters; a gain multiplier configured to multiply a gain value based on one or more gain parameters of the set of parameters; and a constant adder configured to add a constant value based on a constant parameter of the set of parameters” as described in claims 9, 16, 27, and 30, nor does it describe receiving a set of parameters including “a combination of one or more gain parameters, a constant parameter, and one or more approximation functions to apply to the input data via the configurable nonlinear activation function circuit” as described in claim 26. Therefore, Luschi does not teach the combination of claims 9-16 and 26-30, including the limitations described above.
Zhigang Wei in NPL “Design Space Exploration for Softmax Implementations” (cited in IDS on 06/28/2024), hereinafter “Wei” - teaches a softmax architecture including EXP units, an adder tree, and a log block that calculates the natural logarithm of the result of the adder tree. See Figs. 1(b) and 2 and corresponding description. Wei does not teach or suggest the EXP unit or the log block to include a first approximator configured to approximate a first function using one or more first function parameters of the set of parameters; a second approximator configured to approximate a second function using one or more second function parameters of the set of parameters; a gain multiplier configured to multiply a gain value based on one or more gain parameters of the set of parameters; and a constant adder configured to add a constant value based on a constant parameter of the set of parameters” as described in claims 9, 16, 27, and 30, nor does it describe receiving a set of parameters including “a combination of one or more gain parameters, a constant parameter, and one or more approximation functions to apply to the input data via the configurable nonlinear activation function circuit” as described in claim 26. Therefore, Wei does not teach the combination of claims 9-16 and 26-30, including the limitations described above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CARLOS H DE LA GARZA whose telephone number is (571)272-0474. The examiner can normally be reached Monday-Friday 9:30AM-6PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Caldwell can be reached at (571) 272-3702. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.H.D./
Carlos H. De La GarzaExaminer, Art Unit 2182 (571)272-0474
/EMILY E LAROCQUE/Primary Examiner, Art Unit 2182
Prosecution Insights