DETAILED ACTION
1. This office action is in response to the Application No. 17870038 filed on 01/21/2026. Claim 21 has been cancelled. Claims 1-20 and 22-25 are presented for examination and are currently pending.
Notice of Pre-AIA or AIA Status
2. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
3. A request for continued examination under 37 CFR 1.114, including the fee set
forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this
application is eligible for continued examination under 37 CFR 1.114, and the fee set
forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action
has been withdrawn pursuant to 37 CFR 1.114. Applicant’s submission filed on
04/21/2026 has been entered.
Response to Arguments
4. Applicant’s arguments with respect to the amended Independent claims 1, 24
and 25 have been considered but are moot because a new ground of rejection has
been issued. A new secondary reference has been added to address the newly added
limitations.
On page 7-8 of the remarks, the Applicant argued that “The claims, particularly as amended, are not directed to an abstract idea, but to a specific improvement in computer hardware functionality that addresses and solves a technical problem rooted in computer architecture. As explained in the specification, certain existing computer hardware architectures are "unable to multiply together two matrices which are both dynamically generated at runtime" (paragraph [0016]), forcing a neural processing unit (NPU) to offload such operations to a separate processing unit, such as a CPU. This offloading introduces a technical problem, resulting in a "relative slowdown of the inference process" (paragraph [0017]), and is "not” optimal" because it "requires multiple writes and reads to memory" and consumes excess "memory bandwidth and electrical power" (paragraph [0030]). Claim 1 solves this problem by reciting a specific hardware configuration: a neural processing unit comprising a multiplication accumulation engine and a shared buffer which are specifically configured to perform the calculations of the first score matrix, the second score matrix, and the similarity matrix using data from the shared buffer. As a result, the claimed neural processing unit avoids the need to transfer data to a separate processing unit, thereby reducing memory accesses and associated bandwidth usage.
On page 8 of the remarks, the Applicant argued that “This new hardware architecture for a neural processing unit is expressly supported by the specification, which explains that the calculations can be performed "without having to offload any steps. to another processing unit" (paragraph [0059]), and that this results in operation "more quickly, with less memory usage and requiring less electrical power" (paragraph [0105]). Accordingly, the claimed neural processing unit hardware architecture provides a concrete technical solution that improves the functioning of the neural processing unit itself, increasing its speed and efficiency. Furthermore, the claims are directed to a specific technological context, namely machine learning inference. When considered as a whole, the claims recite a specific hardware improvement that solves a known technical problem, applied within a defined technological field achieves improved operation of a neural processing unit, and therefore, provides significantly more than the abstract mathematical concept. The claims are patent-eligible under 35 U.S.C. § 101”
The above argument that the claimed invention aim to solve the slowing down of the inference process is not persuasive. This is because the claimed invention does not reflect the detailed use of the abstract ideas to perform inferencing. For example, the claim recites “neural processing unit for machine learning inference”. This is an intended use limitation; the claimed invention lack the details of how the abstract ideas of “calculate a first score matrix based on differences between a query matrix and a key matrix” or “calculate a second score matrix based on differences between the key matrix and a learned key matrix” and so on are used for machine learning inferencing.
Also, the claimed invention does not include details of how the multiplication accumulation engine and the shared buffer in the neural processing unit is implemented in the invention such that the neural processing unit avoids the need to transfer data to a separate processing unit, thereby reducing memory accesses and associated bandwidth usage as argued by the Applicant.
In addition, the arguments of the Applicant are directed to the abstract ideas and as a result, the abstract ideas are not integrated into practical application. For example, the Applicant’s argument does not include detailed information of how the abstract ideas such as “calculate a first score matrix based on differences between a query matrix and a key matrix” or “calculate a second score matrix based on differences between the key matrix and a learned key matrix” and so on is integrated into practical application. According to MPEP 2106.05(a), it is important to note, the judicial exception alone cannot provide the improvement. The improvement can be provided by one or more additional elements.
Therefore, the claims are ineligible under 35 U.S.C. § 101.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
5. Claims 1-25 are rejected under 35 U.S.C 101 because the claimed invention is directed towards an abstract idea without significantly more.
Step 1
Independent claim 1 is directed to a device, and falls into one of the four statutory categories.
Step 2A, Prong 1
Claim 1 recites the following abstract ideas:
calculate a first score matrix based on differences between a query matrix and a key matrix (Mathematical concepts directed to the calculation of first score matrix using the difference between query matrix and key matrix);
calculate a second score matrix based on differences between the key matrix and a learned key matrix (Mathematical concepts directed to the calculation of second score matrix using the difference between query matrix and key matrix);
calculate a similarity matrix based on a combination of the first score matrix and second score matrix (Mathematical concepts directed to the calculation of similarity matrix using the combination of first score matrix and second score matrix) and
calculate an attention matrix comprising applying a normalisation function to the similarity matrix (Mathematical concepts directed to the calculation of attention matrix by applying normalisation function).
to calculate at least one of the first score matrix and the second score matrix (Mathematical concepts directed to the calculation of the first and second score matrix).
Step 2A, Prong 2
Claim 1 recites the following additional elements:
neural processing unit for machine learning inference, (this limitation is directed to merely using a computer (neural processing unit) as a tool to perform an abstract idea. This does not integrate the abstract idea into a practical application. See MPEP 2016.05(f)):
the neural processing unit comprising a multiplication accumulation engine (this
limitation is directed to generally linking the use of a judicial exception to a particular technological environment. This limitation does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)) and
a shared buffer (this limitation is directed to generally linking the use of a judicial
exception to a particular technological environment. This limitation does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)),
the neural processing unit being configured to (this limitation is directed to merely using a computer (neural processing unit) as a tool to perform an abstract idea. This does not integrate the abstract idea into a practical application. See MPEP 2016.05(f)):
comprising a multiplication accumulation engine configured (this limitation is directed to using a computer (accumulation engine) as a tool to perform the abstract idea. This does not integrate the abstract idea into practical application. See MPEP 2106.05(f))
wherein the multiplication accumulation engine is configured to calculate the first score matrix, the second score matrix, and the similarity matrix using data from the shared buffer (this limitation is directed to using computer (accumulation engine) as a tool to perform the abstract idea. This does not integrate the abstract idea into practical application. See MPEP 2106.05(f)).
Step 2B
Claim 1 recites the following additional elements:
neural processing unit for machine learning inference, (this limitation is directed to merely using a computer (neural processing unit) as a tool to perform an abstract idea. This does not amount to significantly more than judicial exception. See MPEP 2016.05(f)):
the neural processing unit comprising a multiplication accumulation engine (this
limitation is directed to generally linking the use of a judicial exception to a particular technological environment. This does not amount to significantly more than judicial exception. See MPEP 2106.05(h)) and
a shared buffer (this limitation is directed to generally linking the use of a judicial
exception to a particular technological environment. This does not amount to significantly more than judicial exception. See MPEP 2106.05(h)),
the neural processing unit being configured to (this limitation is directed to merely using a computer (neural processing unit) as a tool to perform an abstract idea. This does not amount to significantly more than judicial exception. See MPEP 2016.05(f)):
comprising a multiplication accumulation engine configured (this limitation is directed to using a computer (accumulation engine) as a tool to perform the abstract idea. This does not amount to significantly more than judicial exception. See MPEP 2106.05(f))
wherein the multiplication accumulation engine is configured to calculate the first score matrix, the second score matrix, and the similarity matrix using data from the shared buffer (this limitation is directed to using computer (accumulation engine) as a tool to perform the abstract idea. This does not amount to significantly more than judicial exception. See MPEP 2106.05(f)).
6. Dependent claim 2 is directed to a device, and falls into one of the four statutory categories.
Claim 2 recites the following abstract ideas:
to calculate at least one input to a layer of a neural network by multiplying together at least one element of the attention matrix and at least one element of a learned value matrix (Mathematical concepts directed to the calculation of input layer using the multiplication).
Claim 2 do not recite any additional elements.
7. Dependent claim 3 is directed to a device, and falls into one of the four statutory categories.
Claim 3 do not recite any abstract ideas.
Claim 3 recite the following additional elements:
wherein the learned value matrix is identical to the learned key matrix (this limitation is directed to a particular type or source of data, which is field of use and it does not integrate the abstract ideas into practical application. See MPEP 2106.05(h)).
Claim 3 recite the following additional elements:
wherein the learned value matrix is identical to the learned key matrix (this limitation is directed to a particular type or source of data, which is field of use and it does not amount to significantly more than judicial exception. See MPEP 2106.05(h)).
8. Dependent claim 4 is directed to a device, and falls into one of the four statutory categories.
Claim 4 recites the following abstract ideas:
wherein the combination comprises calculating weighted values (Mathematical concepts directed to the combination of first score matrix and second score matrix that also include calculating weighted values).
Claim 4 do not recite any additional elements.
9. Dependent claim 5 is directed to a device, and falls into one of the four statutory categories.
Claim 5 recites the following abstract ideas:
wherein the combination comprises maximization (Mathematical concepts directed to the combination of first score matrix and second score matrix that also include maximization).
Claim 5 do not recite any additional elements.
10. Dependent claim 6 is directed to a device, and falls into one of the four statutory categories.
Claim 6 recites the following abstract ideas:
multiply together an input matrix and a query projection matrix to obtain the query matrix (Mathematical concepts directed to multiplying an input matrix with query projection matrix).
Claim 6 do not recite any additional elements.
11. Dependent claim 7 is directed to a device, and falls into one of the four statutory categories.
Claim 7 recites the following abstract ideas:
to multiply together an input matrix and a key projection matrix to obtain the key matrix (Mathematical concepts directed to multiplication of input matrix and key projection matrix).
Claim 7 do not recite any additional elements.
12. Dependent claim 8 is directed to a device, and falls into one of the four statutory categories.
Claim 8 recites the following abstract ideas:
wherein calculating the first score matrix comprises calculating at least one sum of absolute values of differences between elements of the query matrix and elements of the key matrix (Mathematical concepts directed to the calculation of the absolute values of the differences between query matrix and key matrix).
Claim 8 do not recite any additional elements.
13. Dependent claim 9 is directed to a device, and falls into one of the four statutory categories.
Claim 9 recites the following abstract ideas:
wherein calculating the second score matrix comprises calculating at least one sum of absolute values of differences between elements of the key matrix and elements of the learned key matrix (Mathematical concepts directed to the calculation of a second score matrix using the sum of the absolutes of the differences between key matrix and learned key matrix).
Claim 9 do not recite any additional elements.
14. Dependent claim 10 is directed to a device, and falls into one of the four statutory categories.
Claim 10 recites the following abstract ideas:
wherein calculating the similarity matrix comprises calculating maxima of a sum of the first score matrix and the second score matrix (Mathematical concepts directed to the calculation of similarity matrix by calculating the maxima of the sum of the first and second score matrix).
Claim 10 do not recite any additional elements.
15. Dependent claim 11 is directed to a device, and falls into one of the four statutory categories.
Claim 11 recites the following abstract ideas:
wherein calculating the similarity matrix comprises calculating maxima of a reciprocal of a sum of the first score matrix and the second score matrix (Mathematical concepts directed to the calculation of similarity matrix by calculating the maxima of the reciprocal of the sum of the first and second score matrix).
Claim 11 do not recite any additional elements.
16. Dependent claim 12 is directed to a device, and falls into one of the four statutory categories.
Claim 12 recites the following abstract ideas:
wherein the normalisation function comprises at least one of: a softmax function; a normalization by subtracting a mean and dividing by a standard deviation; a hyperbolic tangent function; and a sigmoid function (Mathematical concepts directed to the calculation of the normalisation function using softmax function, normalization, hyperbolic tangent function and sigmoid function).
Claim 12 do not recite any additional elements.
17. Dependent claim 13 is directed to a device, and falls into one of the four statutory categories.
Claim 13 recites the following abstract ideas:
further configured to apply a scaling function to the similarity matrix based on one or more dimensions of the similarity matrix (Mathematical concepts is directed to applying a scaling function to the similarity matrix).
Claim 13 do not recite any additional elements.
18. Dependent claim 14 is directed to a device, and falls into one of the four statutory categories.
Claim 14 do not recite any abstract ideas.
Claim 14 recites the following additional elements:
wherein the neural processing unit comprises an Ethos-U processor (this limitation is directed to generally linking the use of a judicial exception to a particular technological environment. This limitation does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)).
Claim 14 recites the following additional elements:
wherein the neural processing unit comprises an Ethos-U processor (this limitation is directed to generally linking the use of a judicial exception to a particular technological environment. This does not amount to significantly more than judicial exception. See MPEP 2106.05(h)).
19. Dependent claim 15 is directed to a device, and falls into one of the four statutory categories.
Claim 15 do not recite any abstract ideas.
Claim 15 recites the following additional elements:
comprising a direct memory access element configured to fetch the learned key matrix and/or the learned value matrix from a memory external to the neural processing unit (This limitation is directed to insignificant extra-solution activity of mere data gathering. This limitation does not integrate the abstract idea into practical application. See MPEP 2106.05(g)).
Claim 15 recites the following additional elements:
comprising a direct memory access element configured to fetch the learned key matrix and/or the learned value matrix from a memory external to the neural processing unit (This limitation is directed to transmission of data and it is a well understood routine and conventional activity. This limitation does not amount to significantly more. See MPEP 2106.05(d)(II), example i).
20. Dependent claim 16 is directed to a device, and falls into one of the four statutory categories.
Claim 16 do not recite any abstract ideas.
Claim 16 recites the following additional elements:
wherein the direct memory access element is configured to prefetch the learned key matrix and/or learned value matrix from the memory to a buffer (This limitation is directed to insignificant extra-solution activity of mere data gathering. This limitation does not integrate the abstract idea into practical application. See MPEP 2106.05(g)).
Claim 16 recites the following additional elements:
wherein the direct memory access element is configured to prefetch the learned key matrix and/or learned value matrix from the memory to a buffer (This limitation is directed to transmission of data and it is a well understood routine and conventional activity. This limitation does not amount to significantly more. See MPEP 2106.05(d)(II), example i)
21. Dependent claim 17 is directed to a device, and falls into one of the four statutory categories.
Claim 17 do not recite any abstract ideas.
Claim 17 recites the following additional elements:
wherein the buffer is a further memory external to the neural processing unit (this limitation is directed to generally linking the use of a judicial exception to a particular technological environment. This does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)).
Claim 17 recites the following additional elements:
wherein the buffer is a further memory external to the neural processing unit (this limitation is directed to generally linking the use of a judicial exception to a particular technological environment. This does not amount to significantly more than judicial exception. See MPEP 2106.05(h)).
22. Dependent claim 18 is directed to a device, and falls into one of the four statutory categories.
Claim 18 do not recite any abstract ideas.
Claim 18 recites the following additional elements:
wherein the buffer is a scratch buffer (this limitation is directed to generally linking the use of a judicial exception to a particular technological environment. This does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)).
Claim 18 recites the following additional elements:
wherein the buffer is a scratch buffer (this limitation is directed to generally linking the use of a judicial exception to a particular technological environment. This does not amount to significantly more than judicial exception. See MPEP 2106.05(h)).
23. Dependent claim 19 is directed to a device, and falls into one of the four statutory categories.
Claim 19 do not recite any abstract ideas.
Claim 19 recites the following additional elements:
wherein the neural processing unit comprises a shared buffer, and wherein the buffer is the shared buffer (this limitation is directed to generally linking the use of a judicial exception to a particular technological environment. This does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)).
Claim 19 recites the following additional elements:
wherein the neural processing unit comprises a shared buffer, and wherein the buffer is the shared buffer (this limitation is directed to generally linking the use of a judicial exception to a particular technological environment. This does not amount to significantly more than judicial exception. See MPEP 2106.05(h)).
24. Dependent claim 20 is directed to a device, and falls into one of the four statutory categories.
Claim 20 recites the following abstract ideas:
to calculate at least one input to a layer of a neural network by multiplying together at least one element of the attention matrix and at least one element of the learned value matrix (Mathematical concepts directed to the calculation of one input layer by the multiplication of attention matrix and learned value matrix), and
Claim 20 recites the following additional elements:
wherein the neural processing unit is further configured (this limitation is directed to using a computer (processing unit) as a tool to perform the abstract idea. This does not integrate the abstract idea into practical application. See MPEP 2106.05(f))
wherein the direct memory access element is configured to write the calculated at least one input to the memory external to the neural processing unit (this limitation is directed to insignificant extra-solution activity of the transfer of data. This does not integrate the abstract idea into practical application. See MPEP 2106.05(g)).
Claim 20 recites the following additional elements:
wherein the neural processing unit is further configured (this limitation is directed to using a computer (neural processing unit) as a tool to perform the abstract idea. This does not amount to significantly more than judicial exception. See MPEP 2106.05(f))
wherein the direct memory access element is configured to write the calculated at least one input to the memory external to the neural processing unit (This limitation is directed insignificant extra-solution activity of the transfer of data and it is a well understood routine and conventional activity. This limitation does not amount to significantly more. See MPEP 2106.05(d)(II), example i).
25. Dependent claim 22 is directed to a device, and falls into one of the four statutory categories.
Claim 22 recites the following abstract ideas:
to calculate at least one input to a layer of a neural network by multiplying together at least one element of the attention matrix and at least one element of the learned value matrix (Mathematical concepts directed to the calculation of one input layer by the multiplication of attention matrix and learned value matrix).
Claim 22 recites the following additional elements:
wherein the multiplication accumulation engine is configured (this limitation is directed to using a computer (accumulation engine) as a tool to perform the abstract idea. This does not integrate the abstract idea into practical application. See MPEP 2106.05(f))
Claim 22 recites the following additional elements:
wherein the multiplication accumulation engine is configured (this limitation is directed to using a computer (accumulation engine) as a tool to perform the abstract idea. This does not amount to significantly more than judicial exception. See MPEP 2106.05(f))
26. Dependent claim 23 is directed to a device, and falls into one of the four statutory categories.
Claim 23 do not recite any abstract ideas.
Claim 23 recites the following additional element:
an activation output element configured to, together with the multiplication accumulation engine, calculate the attention matrix (This limitation is directed to using a computer (activation output element) as a tool to implement the abstract idea. This does not integrate the abstract idea into practical application. See MPEP 2106.05(f)).
Claim 23 recites the following additional element:
an activation output element configured to, together with the multiplication accumulation engine, calculate the attention matrix (This limitation is directed to using a computer (activation output element) as a tool to implement the abstract idea. This does not amount to significantly more than judicial exception. See MPEP 2106.05(f)).
27. Independent claim 24 is directed to a device, and falls into one of the four statutory categories.
With regards to claim 24, it is substantially similar to claim 1, and is rejected in the same manner and reasoning applying.
Claim 24 further recites “an apparatus comprising at least one neural processing unit and at least one memory, the memory configured to pass, on demand, a learned key matrix to the neural processing unit” these limitations are directed to using a computer (the processing unit of the apparatus) as a tool to perform the implementation of the abstract idea. These limitations do not integrate the abstract idea into a practical application and do not amount to significantly more. See MPEP 2106.05(f).
28. Independent claim 25 is directed to a device, and falls into one of the four statutory categories.
With regards to claim 25, it is substantially similar to claim 1, and is rejected in the same manner and reasoning applying.
Claim 25 further recites “a computer program product comprising a non-transitory computer readable medium having computer readable program code stored thereon which, when executed by a neural processing unit” these limitations are directed to using a computer (the processing unit of the apparatus) as a tool to perform the implementation of the abstract idea. These limitations do not integrate the abstract idea into a practical application and do not amount to significantly more. See MPEP 2106.05(f).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
29. Claims 1, 2, 4, 5, 8, 9, 11-13, 15, 18, 19 and 25 is rejected under 35 U.S.C. 103 as being unpatentable over Tu et al. (US20210027165) in view of Delp, III et al. (US20230222821 PCT filed 04/28/2021 hereinafter “Delp”) and further in view of
Chen et al. ("Enabling Energy-Efficient Inference for Self-Attention Mechanisms in Neural Networks," 2022 IEEE 4th International Conference on Artificial Intelligence Circuits and Systems (AICAS), Date of Conference: 13-15 June 2022).
Regarding claim 1, Tu teaches a neural processing unit (As shown in FIG. 16, the computer device includes a processor [0174]; a neural network training apparatus may be implemented in a form of a computer program, and the computer program may be run on the computer device shown in FIG. 16 [0176]; FIG. 9 is a schematic flowchart of a step of calculating an attention matrix difference degree according to attention matrices corresponding to adjacent subspaces[0017]) for machine learning inference (so that a translated sentence can be determined according to the outputted target network representation sequence [0094]. The Examiner notes that translated sentence is an inference),
the neural processing unit configured to: calculate a first score matrix based on differences between a query matrix and a key matrix (Step 302: Calculate a logical similarity degree between a query vector sequence and a key vector sequence in a current subspace [0059]; The query vector sequence Q, the key vector sequence K, and the value vector sequence V are matrices of I×d [0041]; calculating the logical similarity degree between the query vector sequence and the key vector sequence in the current subspace by using a Euclidean distance [0060]);
calculate a second score matrix based on differences between the key matrix and a learned key matrix (Step 206: Calculate a space difference degree between the subspaces by using the neural network model [0046]; The space difference degree is used for measuring a difference between the subspaces [0047]; … n subspaces including a key vector sequence (abstract); the key vector sequence K … is matrix of I×d [0041]; The learnable parameter matrices Wi K of the ith subspace are matrices of d×d [0084]. The Examiner notes that space difference degree between the subspaces include a key vector sequence (key matrix) and learnable parameter matrices Wi K (learned key matrix) and the instant specification discloses key projection matrix Wk comprise elements which are learned [0034]);
and calculate an attention matrix comprising applying a normalisation function to the similarity matrix (Specifically, after logical similarity degrees corresponding to the subspaces are obtained, the logical similarity degrees corresponding to the subspaces are normalized, and the attention matrix corresponding to the current subspace is finally obtained [0066]).
Tu does not explicitly teach the neural processing unit comprising a multiplication accumulation engine and a shared buffer, calculate a similarity matrix based on a combination of the first score matrix and second score matrix; wherein the multiplication accumulation engine is configured to calculate the first score matrix, the second score matrix, and the similarity matrix using data from the shared buffer.
Delp teaches calculate a similarity matrix based on a combination of the first score matrix and second score matrix (The similarity scores in the…similarity matrix 1332 can be combined with the similarity scores in the…similarity matrix 1334 [0117], Figure 13B);
Since TU desires a neural network training method applied in an image annotation application scenario [00134] to improve accuracy of an output result [0005], and Delp teaches an output can include an annotated image [0046] using neural networks [0081] to improve the efficiency and accuracy [0045], then, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Tu to incorporate the teachings of Delp for the benefit of improving efficiency and accuracy [0045] using neural networks (Delp [0081])
Tu and Delp does not explicitly teach the neural processing unit comprising a multiplication accumulation engine and a shared buffer, wherein the multiplication accumulation engine is configured to calculate the first score matrix, the second score matrix, and the similarity matrix using data from the shared buffer.
Chen teaches a neural processing unit (The proposed accelerator: (a) block diagram, (b) instruction flow, Fig. 4, pg. 3, right col.) for machine learning inference (Enabling Energy-Efficient Inference for Self-Attention Mechanisms in Neural Networks (title); Therefore, in this work, we propose an software-hardware co-design solution for energy-efficient self-attention inference (abstract)), the neural processing unit (The proposed accelerator: (a) block diagram, (b) instruction flow, Fig. 4, pg. 3, right col.) comprising a multiplication accumulation engine (MAC is responsible for processing a vector-matrix multiplication, pg. 3, right col., last para.) and a shared buffer (Memory Module comprises Value Buffer, Key Buffer, Query Buffer and Output Buffer, Fig. 4),
wherein the multiplication accumulation engine (MACs for query-matrix-key-matrix multiplication, pg. 3, left col., second para.) is configured to calculate the first score matrix, the second score matrix, and the similarity matrix using data (For n input entities, three different dense matrix with n×d dimensions are provided: the query matrix (Q), the key matrix (K) and the value matrix (V) (pg. 1, right col., last para.); The self-attention mechanism is composed of similarity computation, softmax normalization ... Similarity computation as an important component of the self attention mechanism is to compute scores by calculating the inner product between a query vector and a key vector, pg. 2, left col., second to the last para.) from the shared buffer (Memory Module comprises Value Buffer, Key Buffer, Query Buffer and Output Buffer, Fig. 4; The memory prepares the input matrix
consumed by computing resources and stores the generated outputs, pg. 3, left col., second to the last para.).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Tu and Delp to incorporate the teachings of Chen for the benefit of prediction-based approximation method with low overhead for computing similarity is proposed, substantially reducing the computational cost (Chen, pg. 1, right col., second para.)
Regarding claim 2, Modified Tu teaches the neural processing unit of claim 1, Tu teaches further configured to calculate at least one input to a layer of a neural network (Referring to FIG. 6, inputs are the same for each layer, and each of the inputs is an output of an upper layer. Subsequently, the input is divided into a plurality of sub-inputs, and the same transformation is performed on the sub-inputs by using respective network parameters of a plurality of subspaces (also referred to as heads), to obtain outputs of all the subspaces [0093]) by multiplying together at least one element of the attention matrix and at least one element of a learned value matrix (Alternatively, similarity degrees between the attention matrices of the adjacent subspaces may be measured by multiplying the attention matrices corresponding to the adjacent subspaces in the neural network model according to an element matrix [0099]).
Regarding claim 4, Modified Tu teaches the neural processing unit of claim 1, Tu teaches wherein the combination comprises calculating weighted values (performing weighted summation on the attention matrices corresponding to the adjacent subspaces to obtain the attention matrix difference degree [0099]).
Regarding claim 5, Modified Tu teaches the neural processing unit of claim 1, Tu teaches wherein the combination comprises maximization (The convergence condition may be that both the space difference degree and the output similarity degree are maximized [0054]).
Regarding claim 8, Modified Tu teaches the neural processing unit of claim 1, Tu teaches wherein calculating the first score matrix (Step 302: Calculate a logical similarity degree between a query vector sequence and a key vector sequence in a current subspace [0059]; The query vector sequence Q, the key vector sequence K, and the value vector sequence V are matrices of I×d [0041] comprises calculating at least one sum of absolute values of differences between elements of the query matrix and elements of the key matrix (The calculation manner may be customized by: … a Manhattan distance similarity degree calculation manner [0108]. The Examiner notes Manhattan Distance is the sum of absolute values differences between points).
Regarding claim 9, Modified Tu teaches the neural processing unit of claim 1, Tu teaches wherein calculating the second score matrix comprises (Step 206: Calculate a space difference degree between the subspaces by using the neural network model [0046]; The space difference degree is used for measuring a difference between the subspaces [0047]; n subspaces including … a key vector sequence (abstract); the key vector sequence K … is matrix of I×d [0041]; The learnable parameter matrix Wi K of the ith subspace is matrix of d×d [0084]. The Examiner notes instant specification discloses key projection matrix W K comprise elements which are learned) calculating at least one sum of absolute values of differences between elements of the key matrix and elements of the learned key matrix (The calculation manner may be customized by: … a Manhattan distance similarity degree calculation manner [0108]. The Examiner notes Manhattan Distance is the sum of absolute values differences between points).
Regarding claim 11, Modified Tu teaches the neural processing unit of claim 1, Tu teaches wherein calculating the similarity matrix (Here, J(θ) is the target function, likelihood is the output similarity degree, … and arg max is an arguments of the maxima for obtaining a maximized value [0056]) comprises
calculating maxima of a reciprocal of a sum of the first score matrix and the second score matrix (
PNG
media_image1.png
86
488
media_image1.png
Greyscale
[0056]).
Regarding claim 12, Modified Tu teaches the neural processing unit of claim 1, Tu teaches wherein the normalisation function comprises at least one of: a softmax function; a normalization by subtracting a mean and dividing by a standard deviation; a hyperbolic tangent function; and a sigmoid function (Subsequently, in the hth subspace, non-linear transformation is performed on the logical similarity degree Eh by using the softmax function to obtain an attention matrix Ah corresponding to the hth subspace: Ah=soft max(E h)(15) [0090]).
Regarding claim 13, Modified Tu teaches the neural processing unit of claim 1, Tu teaches further configured to apply a scaling function to the similarity matrix (A calculation process of the logical similarity degree matrix E is described below through specific calculation: [0063]) based on one or more dimensions of the similarity matrix (Q=(q1, q2, . . . , qi, . . . , qI) and K=(k1, k2, . . . , ki, . . . , kI). qi and ki are d-dimensional column vectors, and are respectively a query vector and a key vector that correspond to the source vector representation zi. In the logical similarity degree matrix E=(e1, e2, . . . , ei, . . . , eI), the element ei is logical similarity degrees between the query vector qi corresponding to the source vector representation zi and key vectors k1, k2, . . . , ki, . . . , kI corresponding to all the elements in the training sample. ei is an element in an ith column of E, ei is an I-dimensional column vector, and a calculation formula is
PNG
media_image2.png
52
320
media_image2.png
Greyscale
[0064]. The Examiner notes (1/√d) is called attention scaling).
Regarding claim 18, Modified Tu teaches the neural processing unit of claim 15, Yang teaches wherein the buffer is a scratch buffer (In each layer, the Q, K, and V vectors of valid token (4-bit tokens, 8-bit tokens, and a presentative token) are sent and stored in the corresponding line buffers. Then, the two matrices Q and KT are fed into the systolic PE array in a stepwise style as shown in Fig. 10(a) to meet the dataflow requirement in the systolic array, pg. 514, right col., second para. The Examiner notes the line buffer is a scratch buffer because it is used for temporary storage during operations).
The same motivation to combine dependent claim 15 applies here.
Regarding claim 19, Modified Tu teaches the neural processing unit of claim 15, Yang teaches wherein the neural processing unit comprises a shared buffer, and wherein the buffer is the shared buffer (line buffer, Fig. 9, pg. 514 is a shared buffer because multiple processes access the line buffer memory space).
The same motivation to combine dependent claim 15 applies here.
Regarding claim 25, claim 25 is similar to claim 1. It is rejected in the same manner and reasoning applying.
Further, Tu teaches a computer program product comprising a computer readable medium having computer readable program code stored thereon which, when executed by a neural processing unit for calculating an attention mechanism comprising an attention matrix during machine learning inference, causes the neural processing unit to (According to an embodiment, there is provided a non-transitory computer-readable storage medium storing computer program code to cause at least one processor to: [0008]; FIG. 9 is a schematic flowchart of a step of calculating an attention matrix difference degree according to attention matrices corresponding to adjacent subspaces[0017]):
30. Claims 24 and is rejected under 35 U.S.C. 103 as being unpatentable over Tu et al. (US20210027165) in view of Delp, III et al. (US20230222821 PCT filed 04/28/2021 hereinafter “Delp”) in view of Chen et al. ("Enabling Energy-Efficient Inference for Self-Attention Mechanisms in Neural Networks," 2022 IEEE 4th International Conference on Artificial Intelligence Circuits and Systems (AICAS), Date of Conference: 13-15 June 2022) and further in view of Yang et al. ("DTATrans: Leveraging dynamic token-based quantization with accuracy compensation mechanism for efficient transformer architecture." IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 42.2 (2022): 509-520.)
Regarding claim 24, claim 24 is similar to claim 1. It is rejected in the same manner and reasoning applying.
Further, Tu teaches an apparatus for machine learning inference, the apparatus comprising at least one neural processing unit and at least one memory, (Fig. 16 is an apparatus that includes a processor and a memory; so that a translated sentence can be determined according to the outputted target network representation sequence [0094]. The Examiner notes that translated sentence is an inference) but does not explicitly teach the memory configured to pass, on demand, a learned key matrix to the neural processing unit,
Yang teaches the memory configured to pass, on demand, a learned key matrix to the neural processing unit (The output of the multihead attention and the input tokens will be added up and then executing the Layer_Norm operation in the first residual module ❸. After that, the output of ❸ buffered in R1 buffer will be sent to the FFN module ❹ and be reused in the second residual module ❺ to generate the transformer output (output tokens), pg. 514, left col., first para. para. The Examiner notes that the output of ❸ buffered in R1 buffer includes learned key matrix),
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Modified Tu to incorporate the teachings of Yang for the benefit of designing the DTATrans model which has much less computation amount in attention operation, causing a 1.37× speedup averagely, low precision PEs (processing elements) and lower inference latency (Yang, pg. 519, right col., first para.)
31. Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Tu et al. (US20210027165) in view of Delp, III et al. (US20230222821 PCT filed 04/28/2021 hereinafter “Delp”) in view of Chen et al. ("Enabling Energy-Efficient Inference for Self-Attention Mechanisms in Neural Networks," 2022 IEEE 4th International Conference on Artificial Intelligence Circuits and Systems (AICAS), Date of Conference: 13-15 June 2022) and further in view of Tomkins et al. (US20210295822)
Regarding claim 3, Modified Tu teaches the neural processing unit of claim 2, Modified Tu does not explicitly teach wherein the learned value matrix is identical to the learned key matrix.
Tomkins teaches wherein the learned value matrix is identical to the learned key matrix (Some embodiments may use a single index that includes one or more keys based on an n-gram …or a domain category value [0210]; wherein the first n-gram and the second n-gram are identical [0372])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Modified Tu to incorporate the teachings of Tomkins for the benefit of improving the accuracy of output recommendations (Tomkins [0126])
32. Claims 6 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Tu et al. (US20210027165) in view of Delp, III et al. (US20230222821 PCT filed 04/28/2021 hereinafter “Delp”) in view of Chen et al. ("Enabling Energy-Efficient Inference for Self-Attention Mechanisms in Neural Networks," 2022 IEEE 4th International Conference on Artificial Intelligence Circuits and Systems (AICAS), Date of Conference: 13-15 June 2022) and further in view Huang et al. (US20250202679 PCT filed 03/30/2022)
Regarding claim 6, Modified Tu teaches the neural processing unit of claim 1, Modified Tu does not explicitly teach further configured to multiply together an input matrix and a query projection matrix to obtain the query matrix.
Huang teaches further configured to multiply together an input matrix and a query projection matrix to obtain the query matrix (The query vector Q … may be computed by multiplying the homomorphically encrypted input embedding vector 24 by a query projection layer WQ… The projection layers WQ … may include matrix elements that are parameters of the transformer network 30 [0036]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Modified Tu to incorporate the teachings of Huang for the benefit of protecting privacy of the user's data when performing inferencing at a transformer network (Huang [0023]).
Regarding claim 7, Modified Tu teaches the neural processing unit of claim 1, Modified Tu does not explicitly teach further configured to multiply together an input matrix and a key projection matrix to obtain the key matrix.
Huang teaches further configured to multiply together an input matrix and a key projection matrix to obtain the key matrix (the key vector K … may be computed by multiplying the homomorphically encrypted input embedding vector 24 by … a key projection layer WK … The projection layers … WK … may include matrix elements that are parameters of the transformer network 30 [0036]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Modified Tu to incorporate the teachings of Huang for the benefit of protecting privacy of the user's data when performing inferencing at a transformer network (Huang [0023]).
33. Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Tu et al. (US20210027165) in view of Delp, III et al. (US20230222821 PCT filed 04/28/2021 hereinafter “Delp”) in view of Chen et al. ("Enabling Energy-Efficient Inference for Self-Attention Mechanisms in Neural Networks," 2022 IEEE 4th International Conference on Artificial Intelligence Circuits and Systems (AICAS), Date of Conference: 13-15 June 2022) and further in view of Kim et al. (US20210183074)
Regarding claim 10, Modified Tu teaches the neural processing unit of claim 1, Tu teaches calculating maxima of a result (in a case that the model adjustment reference result is maximized, that the neural network model meets the convergence condition may specifically be performed according to the following formula: J=arg max{L+D} (20) [0127]; Here, J represents the model adjustment reference result, arg max represents arguments of the maxima in which the model adjustment reference result is maximized [0128]) but does not explicitly teach calculating the similarity matrix comprises calculating maxima of a sum of the first score matrix and the second score matrix.
Kim teaches wherein calculating the similarity matrix comprises calculating maxima of a sum of the first score matrix and the second score matrix (… to maximize the sum of similarity values in the matrix).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Modified Tu to incorporate the teachings of Kim for the benefit of training an integrated similarity neural network to increase training efficiency (Kim [0122])
34. Claims 14 is rejected under 35 U.S.C. 103 as being unpatentable over Tu et al. (US20210027165) in view of Delp, III et al. (US20230222821 PCT filed 04/28/2021 hereinafter “Delp”) in view of Chen et al. ("Enabling Energy-Efficient Inference for Self-Attention Mechanisms in Neural Networks," 2022 IEEE 4th International Conference on Artificial Intelligence Circuits and Systems (AICAS), Date of Conference: 13-15 June 2022) and further in view of Sung et al. (US20240069500 filed 05/13/2022)
Regarding claim 14, Modified Tu teaches the neural processing unit of claim 1, Modified Tu does not explicitly teach wherein the neural processing unit comprises an Ethos-U processor.
Sung teaches wherein the neural processing unit comprises an Ethos-U processor (ARM's two new processors with AI capabilities-Arm Cortex-M55 and Ethos-U55, which are a neural processing unit (NPU)-specifically designed for Internet of Things (IoT) endpoint devices [0182]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Modified Tu to incorporate the teachings of Sung for the benefit of providing up to 480 times the machine learning performance improvement (Sung [0182])
35. Claims 16, 20, 22 and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Tu et al. (US20210027165) in view of Delp, III et al. (US20230222821 PCT filed 04/28/2021 hereinafter “Delp”) in view of Chen et al. ("Enabling Energy-Efficient Inference for Self-Attention Mechanisms in Neural Networks," 2022 IEEE 4th International Conference on Artificial Intelligence Circuits and Systems (AICAS), Date of Conference: 13-15 June 2022) in view of Yang et al. ("DTATrans: Leveraging dynamic token-based quantization with accuracy compensation mechanism for efficient transformer architecture." IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 42.2 (2022): 509-520.) and further in view of Sung et al. (US20240069500 filed 05/13/2022)
Regarding claim 16, Modified Tu teaches the neural processing unit of claim 14,
Yang teaches wherein the direct memory access element (DMA, Fig. 8, pg. 514) is configured to prefetch the learned key matrix and/or learned value matrix from the memory (DRAM, Fig. 8, pg. 514) to a buffer (The DMA can then fetch the tokens to on-chip buffers with the address and data length (pg. 513, right col., last para.); Our quantization method dynamically tracks the token tolerance and adjusts the precision of token feature vectors, i.e., query (Q), key (K), and value (V) vectors in each block of attention-based NLP models, pg. 510, left col., second para.).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Tu, Delp and Sung to incorporate the teachings of Yang for the benefit of designing the DTATrans model which has much less computation amount in attention operation, causing a 1.37× speedup averagely, low precision PEs (processing elements) and lower inference latency (Yang, pg. 519, right col., first para.)
Regarding claim 20, Modified Tu teaches the neural processing unit of claim 14, Yang teaches wherein the neural processing unit (Neural processing unit refers to all components in Fig. 8 except DRAM) is further configured to calculate at least one input to a layer of a neural network by multiplying together at least one element of the attention matrix and at least one element of the learned value matrix (In the attention layer, the attention probabilities (attention_prob) are produced by employing softmax on Q × KT. The attention output (attention_out) is obtained by multiplying attention_prob with V, pg. 510, right col., second to the last paragraph), and
wherein the direct memory access element is configured to write the calculated at least one input to the memory external to the neural processing unit (Thus, we can sequentially store the block results of ordered input tokens on DRAM, pg. 516, left col., first para.).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Modified Tu to incorporate the teachings of Yang for the benefit of designing the DTATrans model which has much less computation amount in attention operation, causing a 1.37× speedup averagely, low precision PEs (processing elements) and lower inference latency (Yang, pg. 519, right col., first para.)
Regarding claim 22, Modified Tu teaches the neural processing unit of claim 20, Yang teaches wherein the multiplication accumulation engine (Systolic-base design, Systolic-optimal design, and our design have 3168 (≈ 16 × 18 × 11) 4-bit MAC units, pg. 516, left col., last para.) is configured to calculate at least one input to a layer of a neural network (We can draw that reordering the input tokens results in a reordered Attention_out without impacting the output values. The following FFN will not change the order of Attention_out, pg. 515, right col., last para.) by multiplying together at least one element of the attention matrix and at least one element of the learned value matrix (Module ❶ is the systolic array for Attention_prob × V, which can produce the final results of the attention, pg. 514, right col., last para.).
The same motivation to combine dependent claim 20 applies here.
Regarding claim 23, Modified Tu teaches the neural processing unit of claim 20, Yang teaches comprising an activation output element configured to, together with the multiplication accumulation engine, calculate the attention matrix (Then, the two matrices Q and KT are fed into the systolic PE array in a stepwise style as shown in Fig. 10(a) to meet the dataflow requirement in the systolic array. A softmax unit then processes the result of Q×KT to get attention probabilities (pg. 514, right col., last para.); Systolic-base design, Systolic-optimal design, and our design have 3168 (≈ 16 × 18 × 11) 4-bit MAC units, pg. 516, left col., last para.).
The same motivation to combine dependent claim 20 applies here.
36. Claims 15 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Tu et al. (US20210027165) in view of Delp, III et al. (US20230222821 PCT filed 04/28/2021 hereinafter “Delp”) in view of Chen et al. ("Enabling Energy-Efficient Inference for Self-Attention Mechanisms in Neural Networks," 2022 IEEE 4th International Conference on Artificial Intelligence Circuits and Systems (AICAS), Date of Conference: 13-15 June 2022) in view of Yang et al. ("DTATrans: Leveraging dynamic token-based quantization with accuracy compensation mechanism for efficient transformer architecture." IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 42.2 (2022): 509-520.) and further in view of Shin et al. (US20220156569 filed 11/08/2021)
Regarding claim 15, Modified Tu teaches the neural processing unit of claim 1, Yang teaches comprising a direct memory access element configured to fetch the learned key matrix and/or the learned value matrix (DMA directly accesses/fetches data from DRAM as shown in Fig. 8, pg. 514. The Examiner notes that DMA can fetch data like learned key matrix) from a memory (DRAM, Fig. 8, pg. 514) external to the neural processing unit (Neural processing unit refers to all components in Fig. 8 except DRAM).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Modified Tu to incorporate the teachings of Yang for the benefit of designing the DTATrans model which has much less computation amount in attention operation, causing a 1.37× speedup averagely, low precision PEs (processing elements) and lower inference latency (Yang, pg. 519, right col., first para.)
Regarding claim 17, Modified Tu teaches the neural processing unit of claim 15, Modified Tu does not explicitly teach wherein the buffer is a further memory external to the neural processing unit.
Shin teaches wherein the buffer (The activation buffer registers 207 may be configured as RAM, such as static random-access memory (SRAM) or dynamic random-access memory (DRAM) [0041], Fig. 2A) is a further memory external to the neural processing unit (an array 201 of processing elements (PEs) … Each PE may include K0 multipliers 204 (of which only one multiplier 204 is indicated), and an accumulator (adder tree) 205 connected as shown [0040], Fig. 2A).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Modified Tu to incorporate the teachings of Shin for the benefit of an accelerator core for a neural network [0002] to perform checking for similarity between all the generated query and key vectors (Shin [0033])
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MORIAM MOSUNMOLA GODO whose telephone number is (571)272-8670. The examiner can normally be reached Monday-Friday 8:00am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T. Bechtold can be reached on (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.G./
Examiner, Art Unit 2148
/MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148
Prosecution Insights