DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is responsive to pending claims 1-5, 7-11 filed 6/26/2025.
Claim Objections
Claim 7 depends on cancelled claim 6.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4 are rejected under 35 U.S.C. 103 as being unpatentable over Kouretas ("Logarithmic number system for deep learning", published 2018) in view of Sun (US 20210019116 A1).
For claim 1, Kouretas discloses: a simplified sigmoid function circuit comprising (p.3 col.1 discloses piece-wise linear implementation of a sigmoid function, p.3 col.2-p.4 col.1, fig.4 discloses circuit architecture):
a first circuit configured to perform a computation on input data based on a simplified sigmoid function when a sign of a real region of the input data is positive (p.3 eq.14: shows a piecewise calculation of the sigmoid function over the log domain based on the sign (sx) bit, hence, a circuit for positive sign bit inputs);
a second circuit configured to perform the computation on the input data based on the simplified sigmoid function when the sign of the real region of the input data is negative (ibid: calculation of the sigmoid function based on negative sign (sx) bit for negative inputs); and
a first selector (p.3 ¶1, eq.14 discloses combinational logic used to disclose selection between cases when sign bit s-x is positive or negative, taking one input to rout it to various outputs via selectional logic via combinational logic, hence, a multiplexer) configured to select and output one of an output of the first circuit and an output of the second circuit, based on the sign of the input data (ibid: eq.14 shows piecewise linear function that selects between various circuits based on the sign of the input); and
wherein the simplified sigmoid function is obtained by transforming a sigmoid function of a real region into a sigmoid function of a logarithmic region (eq.14 shows sigmoid performed over logarithm of x, x-hat) and performing a variational transformation for the sigmoid function of the logarithmic region (ibid: piecewise linear approximation transforms the sigmoid function of the logarithmic region variationally based on input range, hence, variational transformation of the sigmoid function).
Kouretas does not disclose: wherein the selector is a multiplexer; at least one coefficient selection multiplexer configured to select a coefficient for use in the variational transformation in at least one of the first circuit and the second circuit.
Sun discloses: wherein the selector is a multiplexer (fig.4, 0056-63 discloses a circuit for computation of a piecewise linear function indexed by i (0056), with 430, 0060 disclosing the use of a sign multiplexer to performed sign-differentiated computations); at least one coefficient selection multiplexer configured to select a coefficient for use in the variational transformation in at least one of the first circuit and the second circuit (410-420, 0058-59: multiplexer for selection of the various constants, including coefficients, for the piecewise linear function).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the circuit of Kouretas by incorporating the multiplexing technique of Sun. Both concern the art of circuit approximations of piecewise-linear functions for machine learning applications, and the incorporation would have, according to Sun, improve calculation speed for deep-learning applications (0002).
For claim 2, Kouretas modified by Sun discloses the circuit of claim 1, as above. Kouretas modified by Sun further discloses: wherein the first circuit includes:
a second multiplexer (Kouretas p.3 ¶1, eq.14: combinational logic used to select between different circumstances based on conditions, with Sun disclosing a multiplexer) configured to select a first coefficient for the variational transformation (Kourets eq.14: selecting a coefficient for the piecewise-linear approximations (0, -1, -3/4, -17/12, etc.) based on range for positive values); and
a third multiplexer (ibid) configured to select a second coefficient for the variational transformation (ibid: likewise for additional coefficients), and
wherein the second circuit includes:
a fourth multiplexer (ibid) configured to select a third coefficient for the variational transformation (ibid: selecting a coefficient for the piecewise-linear approximations (0, 1/12, -1/6, -1, etc.) based on range for negative values); and
a fifth multiplexer (ibid) configured to select a fourth coefficient for the variational transformation (ibid: likewise for additional coefficients).
For claim 3, Kouretas modified by Sun discloses the circuit of claim 2, as above. Kouretas modified by Sun further discloses: wherein the first circuit further includes:
a first multiplier configured to multiply a magnitude of the input data and the first coefficient together (Kouretas eq.14, -3/4 * x-hat – 17/2 term, with Sun fig.5 disclosing a multiplier); and
a first adder configured to add a result of multiplying the magnitude of the input data and the first coefficient together and the second coefficient (ibid, with Sun fig.4 disclosing adders 440, 450), and
wherein the second circuit further includes:
a second multiplier configured to multiply the magnitude of the input data and the third coefficient together (Kouretas eq.14, 1/12 * x-hat – 1/6 term, Sun fig.5); and
a second adder configured to add a result of multiplying the magnitude of the input data and the third coefficient together and the fourth coefficient (ibid).
For claim 4, Kouretas modified by Sun discloses the circuit of claim 1, as above. Kouretas modified by Sun further discloses: wherein the variational transformation obtains a result approximated through the variational transformation for each section of the input data (eq.14: obtaining approximation through a variational transformation for each segment).
Claim(s) 5, 7-10 are rejected under 35 U.S.C. 103 as being unpatentable over Kouretas ("Logarithmic number system for deep learning", published 2018) in view of Yu (US 20210150319 A1) in view of Sun (US 20210019116 A1).
For claim 5, Kouretas discloses: a processor (fig.2, 4 shows LSTM neural architecture operation on the log domain) comprising:
a summation circuit configured to multiply input data and weights and add results of the multiplication (fig.2, p.1 col.2 describes implementation of recurrent networks showing neurons connected via weights to other neurons, hence, a summation circuit for multiplying and summing input data and weights; see also fig.4 showing multiplication and summing at the LSTM level); and
an activation function circuit configured to obtain an activation result from a processing result of the summation circuit through an activation function (fig.4, with p.3 eq.14 showing calculation of activation result obtained from processing a sum input x-hat),
wherein the activation function is obtained by transforming a sigmoid function of a real region into a sigmoid function of a logarithmic region (eq.14, performed on logarithmic region x-hat) and performing a variational transformation for the sigmoid function of the logarithmic region (eq.14: piecewise linear approximation constitutes a variational, e.g., varying for each segment, transformation of the sigmoid function of the logarithmic region);
wherein the activation function circuit includes at least one simplified sigmoid function circuit (p.3 col.1 discloses piece-wise linear implementation of a sigmoid function, p.3 col.2-p.4 col.1, fig.4 discloses circuit architecture),
wherein the at least one simplified sigmoid function circuit includes:
a first circuit configured to perform a computation on the input data based on a simplified sigmoid function when a sign of a real region of the input data is positive (p.3 eq.14: shows a piecewise calculation of the sigmoid function over the log domain based on the sign (sx) bit, hence, a circuit for positive sign bit inputs);
a second circuit configured to perform the computation on the input data based on the simplified sigmoid function when the sign of the real region of the input data is negative (ibid: calculation of the sigmoid function based on negative sign (sx) bit for negative inputs);
a first selector (p.3 ¶1, eq.14 discloses combinational logic used to disclose selection between cases when sign bit s-x is positive or negative, taking one input to rout it to various outputs via selectional logic via combinational logic, hence, a multiplexer) configured to select and output one of an output of the first circuit and an output of the second circuit, based on the sign of the input data (ibid: eq.14 shows piecewise linear function that selects between various circuits based on the sign of the input).
Kouretas does not disclose: wherein the processor is neuromorphic; comprising neuron-implemented array including a plurality of artificial neuron-implemented elements for performing computation of an artificial neural network; wherein each of the plurality of artificial neuron-implemented elements include the neural circuit; wherein the selector is a multiplexer, at least one coefficient selection multiplexer configured to select a coefficient for use in the variational transformation in at least one of the first circuit and the second circuit.
Yu discloses: wherein the processor is neuromorphic (fig.16 shows neuromorphic array architecture); comprising neuron-implemented array (fig.16) including a plurality of artificial neuron-implemented elements (fig.16:2120, figs. 2-4) for performing computation of an artificial neural network (fig.18 shows training and calculating branch 130-140); wherein each of the plurality of artificial neuron-implemented elements include the neural circuit (fig.16).
It would have been obvious before the effective filing date to one of ordinary skill to modify the processor of Kouretas with the neuromorphic architecture of Yu. Both concern the art of specialized neural network circuitry, and the incorporation would have, according to Yu, allow for high integration and low power for better implementation of neural networks (0004).
Kouretas modified by Yu does not disclose: wherein the selector is a multiplexer, at least one coefficient selection multiplexer configured to select a coefficient for use in the variational transformation in at least one of the first circuit and the second circuit.
Sun discloses: wherein the selector is a multiplexer (fig.4, 0056-63 discloses a circuit for computation of a piecewise linear function indexed by i (0056), with 430, 0060 disclosing the use of a sign multiplexer to performed sign-differentiated computations); at least one coefficient selection multiplexer configured to select a coefficient for use in the variational transformation in at least one of the first circuit and the second circuit (410-420, 0058-59: multiplexer for selection of the various constants, including coefficients, for the piecewise linear function).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the circuit of Kouretas modified by Yu by incorporating the multiplexing technique of Sun. Both concern the art of circuit approximations of piecewise-linear functions for machine learning applications, and the incorporation would have, according to Sun, improve calculation speed for deep-learning applications (0002).
For claim 7, Kouretas modified by Yu modified by Sun discloses the circuit of claim 5, as above. Kouretas modified by Yu modified by Sun further discloses: wherein the first circuit further includes:
a first multiplier configured to multiply a magnitude of the input data and a first coefficient together (Kouretas eq.14: selecting a coefficient for the piecewise-linear approximations (0, -1, -3/4, -17/12, etc.) based on range for positive values; -3/4 * x-hat – 17/2 term; with Sun fig.5 disclosing a multiplier); and
a first adder configured to add a result of multiplying the magnitude of the input data and the first coefficient together and a second coefficient (ibid, with Sun fig.4 disclosing adders 440, 450), and
wherein the second circuit further includes:
a second multiplier configured to multiply the magnitude of the input data and a third coefficient together (Kouretas ibid: selecting a coefficient for the piecewise-linear approximations (0, 1/12, -1/6, -1, etc.) based on range for negative values; eq.14, 1/12 * x-hat – 1/6 term, Sun fig.5); and
a second adder configured to add a result of multiplying the magnitude of the input data and the third coefficient together and a fourth coefficient (ibid).
For claim 8, Kouretas modified by Yu modified by Sun discloses the circuit of claim 7, as above. Kouretas modified by Yu modified by Sun further discloses: wherein the first circuit further includes:
a second multiplexer (Kouretas p.3 ¶1, eq.14: combinational logic used to select between different circumstances based on conditions, with Sun fig.4 disclosing the use of multiplexers) configured to select the first coefficient for the variational transformation (Kouretas eq.14: selecting a coefficient for the piecewise-linear approximations (0, -1, -3/4, -17/12, etc.) based on range for positive values); and
a third multiplexer (ibid) configured to select the second coefficient for the variational transformation (ibid: likewise for additional coefficients), and
wherein the second circuit further includes:
a fourth multiplexer (ibid) configured to select the third coefficient for the variational transformation (ibid: selecting a coefficient for the piecewise-linear approximations (0, 1/12, -1/6, -1, etc.) based on range for negative values); and
a fifth multiplexer (ibid) configured to select the fourth coefficient for the variational transformation (ibid: likewise for additional coefficients).
For claim 9, Kouretas modified by Yu modified by Sun discloses the circuit of claim 5, as above. Kouretas modified by Yu modified by Sun further discloses: wherein the variational transformation obtains a result approximated through the variational transformation for each section of the input data (Kouretas eq.14: piecewise linear variational transformation, hence, obtaining a result approximated through the transformation for each section of the log-domain input data).
For claim 10, Kouretas modified by Yu modified by Sun discloses the circuit of claim 5, as above. Kouretas modified by Yu further discloses: an input/output unit configured to receive the input data from the outside and to output a computation result of the artificial neural network associated with the input data to the outside (Kouretas figs. 1-2 shows input-output unit for receiving data into the layers and submitting output data from the output layer; );
a control logic unit configured to receive the input data from the input/output unit and to transfer the input data (Kouretas figs.1-2: logic for transferring the input data to the neurons and additional neurons in further layers);
a word line bias unit configured to transfer the input data provided from the control logic unit to the artificial neuron-implemented element array (Yu fig.18:S140-S150, 0111 gives overview of neural network operation, including receiving address of memory bank (see fig.16:2100), the memory bank including word line selectors, see fig.4:1600, WL1-WL4, hence, word line bias selection unit of fig.4 being transmitted to artificial neuron memory array for accumulation and activation, see fig.4:1200); and
a bit line bias and detect unit configured to detect the computation result associated with the input data from the artificial neuron-implemented element array (Yu fig.4 shows bit-lines being selected as output to feed into the neuron circuit of fig.4:1200, fig.16:2120; see also bit line sense amplifier fig.4:1111_1, 0043, hence, detecting bit line voltage, bit line selection switch fig.4:1112_1, 0042 for selecting a bit line based on bias input; the detected bit-line results are then fed into the integrator 1300 and activation 1400 of the neuron array element fig.14:1200, fig.16:2100-2120 and the output routed, see fig.18, 0111, hence, ).
Claim(s) 11 are rejected under 35 U.S.C. 103 as being unpatentable over Kouretas ("Logarithmic number system for deep learning", published 2018) in view of Yu (US 20210150319 A1) in view of Sun (US 20210019116 A1) in view of Franca-neto (US 20200310674 A1).
For claim 11, Kouretas modified by Yu modified by Sun discloses the circuit of claim 5, as above. Kouretas modified by Yu modified by Sun further discloses: a nonvolatile memory configured to store information about a connection relationship of the plurality of artificial neuron-implemented elements included in the artificial neuron-implemented element array (0120 contemplates implementation of a program of instructions to implement the functionality of neural network processing on a solid state drive (SSD), such as implementation of the processing of neural networks (e.g., fig.17) according to the instructions of fig.18, hence, containing information about connections between nodes and layers implemented by the neuron-implemented elements via memory arrays; see also 0033 disclosing storing memory cells as non-volatile flash memory, the memory cells containing weights and biases arranged in a way that indicates connection relationships of the neural network).
Kouretas modified by Yu modified by Sun does not disclose: a volatile memory configured to store the computation result detected from the artificial neuron-implemented element array.
Franca-neto discloses: : a volatile memory configured to store the computation result detected from the artificial neuron-implemented element array (0033, 0048, 0065).
It would have been obvious before the effective filing date to one of ordinary skill to modify the processor of Kouretas modified by Yu with the volatile memory of Franca-neto. Both concern the art of specialized neural network circuitry, and the incorporation would have, according to Franca-neto, allow for readily accessible volatile memory such as DRAM for the CPU to operate on, hence allowing for convenience, speed, parallelizability advantages (0048).
Response to Arguments
In the remarks, Applicant argued:
1. Kouretas fails to disclose the claimed coefficient multiplexers, but rather merely discloses a mathematical model that performs piecewise approximation.
Applicant’s arguments are moot in view of newly applied art.
2. Kouretas eq. 14 merely describes a mathematical function ad does not disclose explicit multiplexer circuit elements.
Applicant’s arguments are moot in view of newly applied art.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Pillai (US 11775805 B2) discloses the use of piecewise approximations for activation functions.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LIANG LI whose telephone number is (303)297-4263. The examiner can normally be reached Mon-Fri 9-12p, 3-11p MT (11-2p, 5-1a ET).
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Jennifer Welch can be reached on (571)272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center or Private PAIR to authorized users only. Should you have questions about access to Patent Center or the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. The examiner is available for interviews Mon-Fri 6-11a, 2-7p MT (8-1p, 4-9p ET).
/LIANG LI/
Primary examiner AU 2143