DETAILED ACTION
This correspondence is responsive to the Application filed on November 22, 2023. Claims 1-12 are pending in the case, with claims 1 and 7 in independent form.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Summary of Detailed Action
Claims 7-12 are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claims 7-12 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite.
Claims 7-12 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter as the claim(s) do not fall within at least one of the four categories of patent eligible subject matter.
Claims 1-6 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: softmax function computing module, an exponential function approximation computing unit, an addition computing unit, a division computing unit in claims 7-12.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
This application includes one or more claim limitations that use the word “means” or “step” but are nonetheless not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph because the claim limitation(s) recite(s) sufficient structure, materials, or acts to entirely perform the recited function. Such claim limitation(s) is/are: an exponential function approximation computing step, an addition computing step, a division computing step in claims 1-3 and 5.
Because this/these claim limitation(s) is/are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are not being interpreted to cover only the corresponding structure, material, or acts described in the specification as performing the claimed function, and equivalents thereof.
If applicant intends to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to remove the structure, materials, or acts that performs the claimed function; or (2) present a sufficient showing that the claim limitation(s) does/do not recite sufficient structure, materials, or acts to perform the claimed function.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 7-12 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. As similarly discussed above, claim 7 recites a neural network having a classifier which has a softmax function computing module, an exponential function approximation computing unit, an addition computing unit, a division computing unit, which are interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitations uses a generic placeholder (module, unit) that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. It is not clear what the module and units are or are not. It is further unclear what the module and units include or do not include. The originally filed specification and drawings do not disclose any specific structure or hardware for the module and units. For purposes of examination, the neural network, module and units are interpreted as being software. Therefore, claim 7 is indefinite. Claims 8-12 depend directly or indirectly from claim 7 and are rejected based on substantially the same rationale as set forth above with respect to claim 7. Applicant may cancel claims 7-12 or amend claims 7-12 to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 7-12 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because independent claim 7 recites “A neural network” having a classifier which has a softmax function computing module that includes an exponential function approximation computing unit, an addition computing unit, and a division computing unit. Claim 7 fails to provide any structure or hardware for the neural network. The specification also fails to provide any neural network structure or hardware. Therefore, the neural network of claim 7 can reasonably be interpreted as being software per se and is not a process, machine, manufacture or composition of matter as defined in 35 U.S.C. 101. Claims 8-12 depend directly or indirectly from claim 7 and are rejected for the same reasons discussed above with respect to claim 7.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-6 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim(s) recite(s) An approximation method of softmax function, converting one or more input values of a k-dimensional vector into one or more output values of an m-dimensional vector, comprising: an exponential function approximation computing step performing a Leaky Rectified Linear Unit (Leaky ReLU) computation on one of the input values of the k-dimensional vector to obtain a Leaky ReLU computation value and performing a polynomial function computation of a certain order based on the Leaky ReLU computation value to obtain an exponential approximation value, wherein the exponential function approximation computing step is repeated for another one of the input values of the k-dimensional vector to obtain another exponential approximation value; an addition computing step adding the exponential approximation value and the another exponential approximation value to obtain a sum value; and a division computing step dividing at least one of the exponential approximation values obtained in the exponential function approximation computing step by the sum value to obtain one of the output values of the in-dimensional vector, which are mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I). This judicial exception is not integrated into a practical application and the claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Claims 1-6 recite one of the four statutory categories of patent able subject matter and belong to the statutory class(es) of a process (method claims 1-7), a machine (system/apparatus claims ), and an article of manufacture (non-transitory computer readable media claims ).
The examiner notes that the neural network of claims 7-12 are software per se and are not a process, machine, manufacture or composition of matter as defined in 35 U.S.C. 101. However, if claims 7-12 are amended only to fall within at least one of the four categories of patentable subject matter, then claims 7-12 would be rejected as being directed to an abstract idea without significantly more similar to claims 1-6 as shown below.
Claim 1 recites a method, thus a process and one of the four statutory categories of patentable subject matter. However, claim 1 further recites an approximation method of softmax function, converting one or more input values of a k-dimensional vector into one or more output values of an m-dimensional vector, comprising: an exponential function approximation computing step performing a Leaky Rectified Linear Unit (Leaky ReLU) computation on one of the input values of the k-dimensional vector to obtain a Leaky ReLU computation value and performing a polynomial function computation of a certain order based on the Leaky ReLU computation value to obtain an exponential approximation value, wherein the exponential function approximation computing step is repeated for another one of the input values of the k-dimensional vector to obtain another exponential approximation value; an addition computing step adding the exponential approximation value and the another exponential approximation value to obtain a sum value; and a division computing step dividing at least one of the exponential approximation values obtained in the exponential function approximation computing step by the sum value to obtain one of the output values of the in-dimensional vector, which are mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I). The claim does not include any additional elements which integrate the abstract idea into a practical application. Thus, the claim is directed to the abstract idea.
Claim 2, dependent on claim 1, recites only additional mathematical concepts for wherein in the exponential function approximation computing step, a Clamp function computation is performed on the input value before the exponential function approximation computing step is performed, which are mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
Claim 3, dependent on claim 2, recites only additional mathematical concepts for wherein in the addition computing step, the sum value is further added with a protection value to ensure that an absolute value of the sum value is greater than zero, which are mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
Claim 4, dependent on claim 1, recites only additional mathematical concepts for wherein the polynomial function computation of a certain order is a polynomial function computation of second-order to fifth-order, which are mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
Claim 5, dependent on claim 1, recites only additional mathematical concepts for wherein the exponential function approximation computing step is repeated until the Leaky Rectified Linear Unit (Leaky ReLU) computation is performed on each of the input values to obtain the corresponding Leaky ReLU computation value, and the polynomial function computation of a certain order is performed based on the corresponding Leaky ReLU computation value to obtain the corresponding exponential approximation value, the addition computing step adds up all the corresponding exponential approximation values to obtain the sum value, and the division computing step divides each of the corresponding exponential approximation values by the sum value to obtain a plurality of the output values of the m-dimensional vector corresponding to the k- dimensional vector, which are mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
Claim 6, dependent on claim 1, recites only additional mathematical concepts for wherein the input value is a numerical value in the form of an integer, which are mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
As the examiner noted above, claims 7-12 do not fall within the four categories of patentable subject matter. However, if claims 7-12 are amended only to fall within at least one of the four categories of patentable subject matter, then claims 7-12 would be rejected as being directed to an abstract idea without significantly more as shown below.
For Example Only:
Claim 7, a softmax function, the softmax function converting one or more input values of a k-dimensional vector into one or more output values of a m-dimensional vector, the softmax function including: an exponential function approximation a Leaky Rectified Linear Unit (Leaky ReLU) computation on one of the input values of the k-dimensional vector to obtain a Leaky ReLU computation value and performing a polynomial function computation of a certain order based on the Leaky ReLU computation value to obtain an exponential approximation value, wherein the exponential function approximation further repeatedly processes another one of the input values of the k-dimensional vector to obtain another exponential approximation value; an addition adding the exponential approximation value and the another exponential approximation value to obtain a sum value; and a division dividing at least one of the exponential approximation values by the sum value to obtain one of the output values of the m-dimensional vector, which are mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
A neural network, the neural network having a classifier which has (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
computing module (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
computing unit performing (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Thus, the claim is directed to the abstract idea.
Further, the additional elements, alone or in combination, do not provide significantly more than the abstract idea itself, because implementation on a computer (MPEP 2106.05(f)) cannot provide significantly more and generally linking the use of the judicial exception to a particular technological field of use does not meaningfully limit the claims (MPEP 2106.04(d)) and the combination of additional elements does not provide an inventive concept. Thus, the claim is ineligible.
For example only, dependent claims 8-12 would be comparably rejected as set forth above with respect to claim 7 and dependent claims 2-6.
Allowable Subject Matter
Claims 1-12 would be allowable if rejections under 35 U.S.C. 112(b) as being indefinite are overcome and if rejections under 35 U.S.C. 101 as not falling within at least one of the four categories of patent eligible subject matter are overcome and if rejections under 35 U.S.C. 101 as being directed to an abstract idea without significantly more are overcome.
Closest Prior Art of Record
Solovyev et al. (Pub. No. US 2025/008128) discloses A neural network including at least one neural network layer and an activation function connected to an output of the at least one neural network layer. The activation function is implemented as an approximation function of a mathematically defined real valued non-linear activation function, wherein the approximation function allows for integer-only processing of fixed-point representations of input values of the approximation function. Abstract.
Li et al. (Pub. No. US 2023/0185533) discloses a method for processing input data by a set of configurable nonlinear activation function circuits, including generating an exponent output by processing input data using one or more first configurable nonlinear activation function circuits configured to perform an exponential function, summing the exponent output of the one or more first configurable nonlinear activation function circuits, and generating an approximated log softmax output by processing the summed exponent output using a second configurable nonlinear activation function circuit configured to perform a natural logarithm function. Abstract.
Gomez et al. (Pub. No. US 2020/0036510) discloses Systems and methods are provided for receiving input data to be processed by an encrypted neural network (NN) model, and encrypting the input data using a fully homomorphic encryption (FHE) public key associated with the encrypted NN model to generate encrypted input data. The systems and methods further provided for processing the encrypted input data to generate an encrypted inference output, using the encrypted NN model by, for each layer of a plurality of layers of the encrypted NN model, computing an encrypted weighted sum using encrypted parameters and a previous encrypted layer, the encrypted parameters comprising at least an encrypted weight and an encrypted bias, approximating an activation function for the level into a polynomial, and computing the approximated activation function on the encrypted weighted sum to generate an encrypted layer. The generated encrypted inference output is sent to a server system for decryption. Abstract.
Cheema et al. (Pub. No. US 2023/0351181) discloses An activation function unit can compute activation functions approximated by Taylor series. The activation function unit may include a plurality of compute elements. Each compute element may include two multipliers and an accumulator. The first multiplier may compute intermediate products using an activation, such as an output activation of a DNN layer. The second multiplier may compute terms of Taylor series approximating an activation function based on the intermediate products from the first multiplier and coefficients of the Taylor series. The accumulator may compute a partial sum of the terms as an output of the activation function. The number of the terms may be determined based on a predetermined accuracy of the output of the activation function. The activation function unit may process multiple activations. Different activations may be input into different compute elements in different clock cycles. The activation function unit may compute activation functions with different accuracies. Abstract.
Patwari et al. (Pub. No. US 2023/0297824) discloses A programmable, non-linear (PNL) activation engine for a neural network is capable of receiving input data within a circuit. In response to receiving an instruction corresponding to the input data, the PNL activation engine is capable of selecting a first non-linear activation function from a plurality of non-linear activation functions by decoding the instruction. The PNL activation engine is capable of fetching a first set of coefficients corresponding to the first non-linear activation function from a memory. The PNL activation engine is capable of performing a polynomial approximation of the first non-linear activation function on the input data using the first set of coefficients. The PNL activation engine is capable of outputting a result from the polynomial approximation of the first non-linear activation function. Abstract.
KIM et al. (Pub. No. US 2024/0403615) discloses A method of programming an activation function is provided. The method includes generating a segment data for segmenting the activation function; segmenting the activation function into a plurality of segments using the segment data; and approximating at least one segment of the plurality of segments to a programmable segment. Abstract.
Duong et al. (Patent No. US 12,579,416) discloses a neural network inference circuit for executing a neural network that includes computation nodes. Each respective computation node of a set of the computation nodes includes (i) a respective linear function that includes a respective dot product of input values for the computation node and weight values for the computation node and (ii) a respective non-linear activation function. The neural network inference circuit includes a set of dot product circuits to compute the dot product for a computation node and a post-processing circuit to compute (i) a result of the linear function for the computation node based on the dot product for the computation node and (ii) an output for the computation node by applying a piecewise linear function to the result of the linear function for the computation node to apply the non-linear activation function for the computation node. Abstract.
Sehoon Kim, et al., (I-BERT: Integer-only BERT Quantization, arXiv:2101.01321v3 [cs.CL]) discloses Transformer based models, like BERT and RoBERTa, have achieved state-of-the-art results in many Natural Language Processing tasks. How ever, their memory footprint, inference latency, and power consumption are prohibitive for efficient inference at the edge, and even at the data center. While quantization can be a viable solution for this, previous work on quantizing Trans former based models use floating-point arithmetic during inference, which cannot efficiently utilize integer-only logical units such as the recent Turing Tensor Cores, or traditional integer-only ARM processors. In this work, we propose I-BERT, a novel quantization scheme for Transformer based models that quantizes the entire inference with integer-only arithmetic. Based on lightweight integer-only approximation methods for nonlinear operations, e.g., GELU, Softmax, and Layer Normalization, I-BERT performs an end-to-end integer-only BERT inference without any floating point calculation. We evaluate our approach on GLUE downstream tasks using RoBERTa Base/Large. We show that for both cases, I-BERT achieves similar (and slightly higher) accuracy as compared to the full-precision baseline. Abstract.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. US-20200036510-A1, US-20220180644-A1, US-20230185533-A1, US-20230297824-A1, US-20230351181-A1, US-20240403615-A1, US-20250008128-A1, US-12579416-B1, US-20230306242-A1.
Sehoon Kim, et al., I-BERT: Integer-only BERT Quantization, arXiv:2101.01321v3 [cs.CL] https://doi.org/10.48550/arXiv.2101.0132, June 8, 2021.
CAI, Jun CN 115051613A, published 2022-09-13, Multi-Dimensional Taylor Network Estimator of Switch Reluctance Motor Torque.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BARBARA LEVEL whose telephone number is (303)297-4748. The examiner can normally be reached Monday through Friday 8:00 AM - 5:00 PM MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BARBARA M LEVEL/ Examiner, Art Unit 2142
Prosecution Insights