DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on February 20, 2025 was considered by the examiner. The submission is in compliance with the provisions of 37 CFR 1.97.
Claim Objections
The following limitation “the attention neural network comprising a plurality of attention blocks, each block comprising an attention block and a binarized feedforward block” in claim 1 is objected to because of the following informalities: it is unclear what “each block” is referring to since the preceding element mentioned “attention blocks” and not a general block. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 1 recites the limitation “the attention block configured to: receive an input sequence for the block comprising a respective layer input at each of one or more positions;” However, it is unclear whether “the block” refers to an ‘attention block’ or a ‘binarized feedforward block’ or both as mentioned in the limitation “an attention neural network configured to perform the machine learning task, the attention neural network comprising a plurality of attention blocks, each block comprising an attention block and a binarized feedforward block,” before. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea (mental process or math concept) without significantly more.
Claim 1:
Regarding claim 1, in step 1 of the 101-analysis set forth in MPEP 2106, the claim recites
“1. A system for performing a machine learning task on a network input to generate a network output, the system comprising …”, and a system or machine is one of the four statutory categories of invention.
In step 2A prong 1 of the 101-analysis set forth in the MPEP 2106, the examiner has determined that the following limitations recite a process that, under the broadest reasonable interpretation, covers a mental process but for recitation of generic computer components:
and generate an attended input sequence…, (mental process, a person can mentally evaluate and generate an input sequence which is words or text, see MPEP 2106.04(a)(2)(III)),
generate an initial binarized output for the position, comprising computing a binarized matrix multiplication between the binarized input and a binarized weight matrix for the binarized feedforward layer; (This is considered a mathematical relationship, mathematical formula or equation, or mathematical calculation, see in US PG Pub. US20240256966A1 in paragraphs [0072-0073 ] or page 14 from the specification describe “For this matrix multiplication, the system 100 can apply binarization bound BW and BA for weights and activations, respectively:
PNG
media_image1.png
97
412
media_image1.png
Greyscale
where “axis” is the dimension along which max is taken” , see MPEP 2106.04(a)(2), subsection I),
and scale each element of the initial binarized output to generate a final output of the binarized feedforward layer. (This is considered a mathematical relationship, mathematical formula or equation, or mathematical calculation, see in paragraphs [0071, 0090] from the specification state in page 13 or [0071] “The bound B therefore serves as a hyperparameter that controls the range of the input values that will have non-zero gradients. B also serves as a scaling factor for the outputs since the binarization function maps each element of x to either −B/2 or +B/2”, and see in paragraph [0090] or page 16 states “As another example, the layer 210 can perform the scaling 240 by applying a LayerNorm operation to the initial binarized output. In particular, while the scaling factor s enables the binarization of FFNs, it requires hyperparameter tuning, which can be challenging for large models. Instead, the system can perform the scaling 240 by replacing the operation of dividing by the scaling factor with applying a LayerNorm operation. A LayerNorm operation performed on a tensor x satisfies:
PNG
media_image2.png
95
462
media_image2.png
Greyscale
where the operations are performed element-wise, β and γ are learnable parameters, E[x] is the mean of the elements of x, Var(x) is the variance of x, and ϵ is a small floating-point number for numerical stability,” see MPEP 2106.04(a)(2), subsection I),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process or math concept but for the recitation of generic computer components, then it falls within the mental process or math concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
In step 2A prong 2 of the 101-analysis set forth in MPEP 2106, the examiner has determined that the following additional elements do not integrate this judicial exception into a practical application:
A system for performing a machine learning task on a network input to generate a network output, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers, (Mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
at least in part by applying an attention mechanism to the input sequence for the block, the attended input sequence comprising a respective attended layer input at each of the one or more positions, (Mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
to implement: an attention neural network configured to perform the machine learning task, the attention neural network comprising a plurality of attention blocks, each block comprising an attention block and a binarized feedforward block, (Mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
the attention block configured to: receive an input sequence for the block comprising a respective layer input at each of one or more positions, (In step 2A, prong 2, this recites mere data gathering, which is considered insignificant extra-solution activity – see MPEP 2106.05(g)),
the binarized feedforward block comprising a plurality of binarized feedforward layers that are each configured to, for each of the one or more positions: (Mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
receive a binarized input derived from the attended layer input at the position; (In step 2A, prong 2, this recites mere data gathering, which is considered insignificant extra-solution activity – see MPEP 2106.05(g)),
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is “directed” to an abstract idea.
In step 2B of the 101-analysis set forth in the 2019 PEG, the examiner has determined that the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, additional elements iv, v, vi, and viii recite mere instructions to apply the judicial exception using generic computer components, which are not indicative of significantly more. The additional elements vii and ix recite mere data gathering, and are considered insignificant extra-solution activities. In step 2B, these insignificant extra-solution activities are well understood routine and conventional activities, which include receiving or transmitting data over a network from court case Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016), – see MPEP 2106.05(d) (II)(i)),
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claim 2:
Regarding claim 2, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 2 recites the following abstract idea:
The system of claim 1, wherein scaling each element of the initial binarized output comprises: dividing each element of the initial binarized output by a scaling hyperparameter. (This is considered a mathematical relationship, mathematical formula or equation, or mathematical calculation, see in paragraphs [0071, 0090] from the specification state in page 13 or [0071] “The bound B therefore serves as a hyperparameter that controls the range of the input values that will have non-zero gradients. B also serves as a scaling factor for the outputs since the binarization function maps each element of x to either −B/2 or +B/2”, and see in paragraph [0090] or page 16 states “As another example, the layer 210 can perform the scaling 240 by applying a LayerNorm operation to the initial binarized output. In particular, while the scaling factor s enables the binarization of FFNs, it requires hyperparameter tuning, which can be challenging for large models. Instead, the system can perform the scaling 240 by replacing the operation of dividing by the scaling factor with applying a LayerNorm operation. A LayerNorm operation performed on a tensor x satisfies:
PNG
media_image2.png
95
462
media_image2.png
Greyscale
where the operations are performed element-wise, β and γ are learnable parameters, E[x] is the mean of the elements of x, Var(x) is the variance of x, and ϵ is a small floating-point number for numerical stability,” see MPEP 2106.04(a)(2), subsection I),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 3:
Regarding claim 3, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 3 recites the following abstract idea:
The system of claim 1, wherein scaling each element of the initial binarized output comprises: applying a LayerNorm operation to the initial binarized output. (This is considered a mathematical relationship, mathematical formula or equation, or mathematical calculation, see in paragraph [0090] or page 16 states “As another example, the layer 210 can perform the scaling 240 by applying a LayerNorm operation to the initial binarized output. In particular, while the scaling factor s enables the binarization of FFNs, it requires hyperparameter tuning, which can be challenging for large models. Instead, the system can perform the scaling 240 by replacing the operation of dividing by the scaling factor with applying a LayerNorm operation. A LayerNorm operation performed on a tensor x satisfies:
PNG
media_image2.png
95
462
media_image2.png
Greyscale
where the operations are performed element-wise, β and γ are learnable parameters, E[x] is the mean of the elements of x, Var(x) is the variance of x, and ϵ is a small floating-point number for numerical stability,” see MPEP 2106.04(a)(2), subsection I),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 4:
Regarding claim 4, it is dependent upon claim 3, and thereby incorporates the limitations of, and corresponding analysis applied to claim 3. Further, claim 4 recites the following abstract ideas:
and, for each position: generate a first initial binarized output for the position (This recites a mental process, a person can mentally evaluate and generate a binarized output (takes two values) for the position, see MPEP 2106.04(a)(2)(III)),
comprising: computing a binarized matrix multiplication between the binarized attended layer input and a binarized weight matrix for the first binarized feedforward layer; (This is considered a mathematical relationship, mathematical formula or equation, or mathematical calculation, see in US PG Pub. US20240256966A1 in paragraphs [0072-0073 ] or page 14 from the specification describe “For this matrix multiplication, the system 100 can apply binarization bound BW and BA for weights and activations, respectively:
PNG
media_image1.png
97
412
media_image1.png
Greyscale
where “axis” is the dimension along which max is taken” , see MPEP 2106.04(a)(2), subsection I),
adding a binarized bias to a product of the binarized matrix multiplication to generate a binarized sum (This is considered a mathematical relationship, mathematical formula or equation, or mathematical calculation, see in US PG Pub. US20240256966A1 in paragraphs [ 0085, 0089 ] from the specification stating in [0085] “The system can also perform other operations as part of generating the initial binarized output, e.g., adding a bias, applying a non-linear element-wise activation function”, and in paragraph [0089] “Thus, in this example, the binarized matrix multiplication that replaces A·W becomes Ab·Wb/s instead of Ab·Wb”, see MPEP 2106.04(a)(2), subsection I),
applying a binarized Rectified Linear Unit (ReLU) activation function to the binarized sum to generate the first initial binarized output; (This is considered a mathematical relationship, mathematical formula or equation, or mathematical calculation, see in paragraph [ 0097 ] from the specification state “As can be seen from the above, as a result of the binarized ReLU (max(0, Ab W1b+b1)) the activations matrix is all positive values”, see MPEP 2106.04(a)(2), subsection I),
and scale each element of the first initial binarized output by applying a LayerNorm operation, (This is considered a mathematical relationship, mathematical formula or equation, or mathematical calculation, see in paragraph [0090] or page 16 states “As another example, the layer 210 can perform the scaling 240 by applying a LayerNorm operation to the initial binarized output. In particular, while the scaling factor s enables the binarization of FFNs, it requires hyperparameter tuning, which can be challenging for large models. Instead, the system can perform the scaling 240 by replacing the operation of dividing by the scaling factor with applying a LayerNorm operation. A LayerNorm operation performed on a tensor x satisfies:
PNG
media_image2.png
95
462
media_image2.png
Greyscale
where the operations are performed element-wise, β and γ are learnable parameters, E[x] is the mean of the elements of x, Var(x) is the variance of x, and ϵ is a small floating-point number for numerical stability,” see MPEP 2106.04(a)(2), subsection I),
to the first initial binarized output to generate a first binarized output for the position. (This is considered a mathematical relationship, mathematical formula or equation, or mathematical calculation, see in paragraphs [0089, 0094-0096] from the specification, view [0094] stating “the first feedforward layer is configured to receive a respective binarized attended layer input for each position, and, for each position, compute a binarized matrix multiplication between the binarized attended layer input and a binarized weight matrix for the first binarized feedforward layer, add a binarized bias to a product of the binarized matrix multiplication to generate a binarized sum and apply a binarized Rectified Linear Unit (ReLU) activation function to the binarized sum to generate the first initial binarized output. The first feedforward layer can then scale each element of the first initial binarized output by applying a LayerNorm operation to the first initial binarized output to generate a first binarized output for the position,” see MPEP 2106.04(a)(2), subsection I),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process or mathematical concept but for the recitation of generic computer components, then it falls within the mental process or mathematical concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
Further, claim 4 recites the following additional elements:
The system of claim 3, wherein the binarized feedforward block comprises: a first feedforward layer followed by a second feedforward layer, (In step 2A, prong 2, this is considered mere instructions to implement an abstract idea using generic computer – see MPEP 2106.05(f)). (In step 2B, this is also considered mere instructions to implement an abstract idea using generic computer – see MPEP 2106.05(f)).
wherein: the first feedforward layer is configured to receive a respective binarized attended layer input for each position, (In step 2A, prong2, this recites mere data receiving and gathering, which is considered insignificant extra-solution activity – see MPEP 2106.05(g),). In step 2B, this insignificant extra-solution activity is well understood routine and conventional activity which includes receiving or transmitting data over a network from court case Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016), – see MPEP 2106.05(d) (II)(i)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 5:
Regarding claim 5, it is dependent upon claim 4, and thereby incorporates the limitations of, and corresponding analysis applied to claim 4. Further, claim 5 recites the following abstract ideas:
The system of claim 4, wherein the second feedforward layer is configured to receive the respective first binarized output for each position, and, for each position: generate a second initial binarized output for the position, (This is considered a mathematical relationship, mathematical formula or equation, or mathematical calculation, see in US PG Pub. US20240256966A1 in paragraphs [0095 -0096] state “The layer then scales each element of the second initial binarized output by applying a LayerNorm operation to the second initial binarized output to generate a second binarized output for the position. These operations can be represented as:
PNG
media_image3.png
48
740
media_image3.png
Greyscale
” , see MPEP 2106.04(a)(2), subsection I),
comprising: computing a second binarized matrix multiplication between the first binarized output and a binarized weight matrix for the second binarized feedforward layer; (This is considered a mathematical relationship, mathematical formula or equation, or mathematical calculation, see in US PG Pub. US20240256966A1 in paragraphs [0072-0073 ] or page 14 from the specification describe “For this matrix multiplication, the system 100 can apply binarization bound BW and BA for weights and activations, respectively:
PNG
media_image1.png
97
412
media_image1.png
Greyscale
where “axis” is the dimension along which max is taken” , see MPEP 2106.04(a)(2), subsection I),
and adding a second binarized bias to a product of the second binarized matrix multiplication to generate a binarized sum This is considered a mathematical relationship, mathematical formula or equation, or mathematical calculation, see in US PG Pub. US20240256966A1 in paragraphs [0095 -0096] state “The layer then scales each element of the second initial binarized output by applying a LayerNorm operation to the second initial binarized output to generate a second binarized output for the position. These operations can be represented as:
PNG
media_image3.png
48
740
media_image3.png
Greyscale
” , see MPEP 2106.04(a)(2), subsection I),
and scale each element of the second initial binarized output by applying a LayerNorm operation to the second initial binarized output to generate a second binarized output for the position, (This is considered a mathematical relationship, mathematical formula or equation, or mathematical calculation, see in paragraphs [0095-0096] and [0090] or page 16 states “As another example, the layer 210 can perform the scaling 240 by applying a LayerNorm operation to the initial binarized output. In particular, while the scaling factor s enables the binarization of FFNs, it requires hyperparameter tuning, which can be challenging for large models. Instead, the system can perform the scaling 240 by replacing the operation of dividing by the scaling factor with applying a LayerNorm operation. A LayerNorm operation performed on a tensor x satisfies:
PNG
media_image2.png
95
462
media_image2.png
Greyscale
where the operations are performed element-wise, β and γ are learnable parameters, E[x] is the mean of the elements of x, Var(x) is the variance of x, and ϵ is a small floating-point number for numerical stability,” see MPEP 2106.04(a)(2), subsection I),
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 6:
Regarding claim 6, it is dependent upon claim 4, and thereby incorporates the limitations of, and corresponding analysis applied to claim 4. Further, claim 6 recites the following additional elements:
The system of claim 4, wherein the second feedforward layer is binarized using bipolar binarization, (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)), (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 7:
Regarding claim 7, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 7 recites the following abstract idea:
and generating the attended input sequence from the attention outputs for the one or more attention heads, ( mental process, a person can mentally evaluate and generate an input sequence which is words or text from outputs for one or more heads where each head has its own process, see MPEP 2106.04(a)(2)(III)),
Further, claim 7 recites the following additional elements:
The system of claim 1, wherein generating an attended input sequence at least in part by applying an attention mechanism to the input sequence for the layer comprises: for each one or more attention heads:, (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)), (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
applying query-key-value attention to a set of queries for the attention head, a set of keys for the attention head, and a set of values for the attention head to generate an attention output; (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)), (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 8:
Regarding claim 8, it is dependent upon claim 7, and thereby incorporates the limitations of, and corresponding analysis applied to claim 7. Further, claim 8 recites the following additional elements:
The system of claim 7, wherein for at least one of the blocks, the set of queries, the set of keys, and the set of values for each attention head are derived from the input sequence to the block, (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)), (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 9:
Regarding claim 9, it is dependent upon claim 8, and thereby incorporates the limitations of, and corresponding analysis applied to claim 8. Further, claim 9 recites the following additional element:
The system of claim 8, wherein for at least one of the blocks for which the set of queries, the set of keys, and the set of values for each attention head are derived from the input sequence to the block: the query-key-value attention is causal query-key-value attention. (In step 2A, prong 2, this is considered mere instructions to implement an abstract idea using generic computer – see MPEP 2106.05(f)). (In step 2B, this is also considered mere instructions to implement an abstract idea using generic computer – see MPEP 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 10:
Regarding claim 10, it is dependent upon claim 7, and thereby incorporates the limitations of, and corresponding analysis applied to claim 7. Further, claim 10 recites the following additional element:
The system of claim 7, wherein for at least one of the blocks, the set of queries are derived from the input sequence to the block, and the set of keys and the set of values are derived from an encoded representation of the network input generated by an encoder neural network. (In step 2A, prong 2, this is considered mere instructions to implement an abstract idea using generic computer – see MPEP 2106.05(f)). (In step 2B, this is also considered mere instructions to implement an abstract idea using generic computer – see MPEP 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 11:
Regarding claim 11, it is dependent upon claim 10, and thereby incorporates the limitations of, and corresponding analysis applied to claim 10. Further, claim 11 recites the following additional element:
The system of claim 10, wherein the attention neural network comprises the encoder neural network and a subset of the plurality of blocks are in the encoder neural network, (In step 2A, prong 2, this is considered mere instructions to implement an abstract idea using generic computer – see MPEP 2106.05(f)). (In step 2B, this is also considered mere instructions to implement an abstract idea using generic computer – see MPEP 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 12:
Regarding claim 12, it is dependent upon claim 7, and thereby incorporates the limitations of, and corresponding analysis applied to claim 7. Further, claim 12 recites the following abstract ideas:
comprises: scaling the set of queries for the attention head, the set of keys for the attention head, and the set of values for the attention head by applying LayerNorm to the set of queries for the attention head, a set of keys for the attention head, and a set of values, (This is considered a mathematical relationship, mathematical formula or equation, or mathematical calculation, see in paragraphs [0071, 0090] from the specification state in page 13 or [0071] “The bound B therefore serves as a hyperparameter that controls the range of the input values that will have non-zero gradients. B also serves as a scaling factor for the outputs since the binarization function maps each element of x to either −B/2 or +B/2”, and see in paragraph [0090] or page 16 states “As another example, the layer 210 can perform the scaling 240 by applying a LayerNorm operation to the initial binarized output. In particular, while the scaling factor s enables the binarization of FFNs, it requires hyperparameter tuning, which can be challenging for large models. Instead, the system can perform the scaling 240 by replacing the operation of dividing by the scaling factor with applying a LayerNorm operation. A LayerNorm operation performed on a tensor x satisfies:
PNG
media_image2.png
95
462
media_image2.png
Greyscale
where the operations are performed element-wise, β and γ are learnable parameters, E[x] is the mean of the elements of x, Var(x) is the variance of x, and ϵ is a small floating-point number for numerical stability,” see MPEP 2106.04(a)(2), subsection I),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
Further, claim 12 recites the following additional element:
The system of claim 7, wherein the attention block is a binarized attention block and wherein applying query-key-value attention (In step 2A, prong 2, this is considered mere instructions to implement an abstract idea using generic computer – see MPEP 2106.05(f)). (In step 2B, this is also considered mere instructions to implement an abstract idea using generic computer – see MPEP 2106.05(f)).
and applying a binarized query-key-value attention to the scaled sets of queries, keys, and values for the attention head to generate a binarized attention output. (In step 2A, prong 2, this is considered mere instructions to implement an abstract idea using generic computer – see MPEP 2106.05(f)). (In step 2B, this is also considered mere instructions to implement an abstract idea using generic computer – see MPEP 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 13:
Regarding claim 13, it is dependent upon claim 12, and thereby incorporates the limitations of, and corresponding analysis applied to claim 12. Further, claim 13 recites the following abstract idea:
scaling the binarized projected output by applying LayerNorm to the binarized projected output to generate a scaled binarized projected output; (This is considered a mathematical relationship, mathematical formula or equation, or mathematical calculation, see in paragraphs [0071, 0090] from the specification state in page 13 or [0071] “The bound B therefore serves as a hyperparameter that controls the range of the input values that will have non-zero gradients. B also serves as a scaling factor for the outputs since the binarization function maps each element of x to either −B/2 or +B/2”, and see in paragraph [0090] or page 16 states “As another example, the layer 210 can perform the scaling 240 by applying a LayerNorm operation to the initial binarized output. In particular, while the scaling factor s enables the binarization of FFNs, it requires hyperparameter tuning, which can be challenging for large models. Instead, the system can perform the scaling 240 by replacing the operation of dividing by the scaling factor with applying a LayerNorm operation. A LayerNorm operation performed on a tensor x satisfies:
PNG
media_image2.png
95
462
media_image2.png
Greyscale
where the operations are performed element-wise, β and γ are learnable parameters, E[x] is the mean of the elements of x, Var(x) is the variance of x, and ϵ is a small floating-point number for numerical stability,” see MPEP 2106.04(a)(2), subsection I),
and summing the binarized attention output and the scaled binarized projected output to generate a binarized output of the attention head. (This is considered a mathematical relationship, mathematical formula or equation, or mathematical calculation, see in page 17, or in US PG Pub. paragraphs [0089, 0094-0096] from the specification, view [0094] stating “the first feedforward layer is configured to receive a respective binarized attended layer input for each position, and, for each position, compute a binarized matrix multiplication between the binarized attended layer input and a binarized weight matrix for the first binarized feedforward layer, add a binarized bias to a product of the binarized matrix multiplication to generate a binarized sum and apply a binarized Rectified Linear Unit (ReLU) activation function to the binarized sum to generate the first initial binarized output. The first feedforward layer can then scale each element of the first initial binarized output by applying a LayerNorm operation to the first initial binarized output to generate a first binarized output for the position,” see MPEP 2106.04(a)(2), subsection I),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
Further, claim 13 recites the following additional element:
The system of claim 12, wherein generating an attended input sequence at least in part by applying an attention mechanism to the input sequence for the layer comprises: for each of the one or more attention heads: (In step 2A, prong 2, this is considered mere instructions to implement an abstract idea using generic computer – see MPEP 2106.05(f)). (In step 2B, this is also considered mere instructions to implement an abstract idea using generic computer – see MPEP 2106.05(f)).
applying a binarized linear projection to the binarized attention output to generate a binarized projected output; (In step 2A, prong 2, this is considered mere instructions to implement an abstract idea using generic computer – see MPEP 2106.05(f)). (In step 2B, this is also considered mere instructions to implement an abstract idea using generic computer – see MPEP 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 14:
Regarding claim 14, it is dependent upon claim 12, and thereby incorporates the limitations of, and corresponding analysis applied to claim 12. Further, claim 14 recites the following additional element:
The system of claim 12, wherein each attention head is configured to generate the set of queries for the attention head by applying a first binarized linear projection to a corresponding input, the set of keys for the attention head by applying a second binarized linear projection to a corresponding input, and the set of values for the attention head by applying a third binarized linear projection to a corresponding input, (In step 2A, prong 2, this is considered mere instructions to implement an abstract idea using generic computer – see MPEP 2106.05(f)). (In step 2B, this is also considered mere instructions to implement an abstract idea using generic computer – see MPEP 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 15:
Regarding claim 15, this claim recites similar limitations as corresponding independent claim 1 listed above, and is rejected for similar reasons under 35 U.S.C. 101.
Claims 16 -19:
All of claim 15’s dependent claims follow the deficiencies of their parent claim. Since claims 16-19 recite similar limitations as corresponding claims 2-5 listed above, and are rejected for similar reasons under 35 U.S.C. 101.
Claim 20:
Regarding claim 20, in step 1 of the 101-analysis set forth in MPEP 2106, the claim recites
“One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to implement:…” , and a non-transitory computer-readable storage media or machine is one of the four statutory categories of invention.
In step 2A prong 1 of the 101-analysis set forth in the MPEP 2106, the examiner has determined that the following limitations recite a process that, under the broadest reasonable interpretation, covers a mental process but for recitation of generic computer components:
and generate an attended input sequence…, (mental process, a person can mentally evaluate and generate an input sequence which is words or text , see MPEP 2106.04(a)(2)(III)),
generate an initial binarized output for the position, comprising computing a binarized matrix multiplication between the binarized input and a binarized weight matrix for the binarized feedforward layer; (This is considered a mathematical relationship, mathematical formula or equation, or mathematical calculation, see in US PG Pub. US20240256966A1 in paragraphs [0072-0073 ] or page 14 from the specification describe “For this matrix multiplication, the system 100 can apply binarization bound BW and BA for weights and activations, respectively:
PNG
media_image1.png
97
412
media_image1.png
Greyscale
where “axis” is the dimension along which max is taken” , see MPEP 2106.04(a)(2), subsection I),
and scale each element of the initial binarized output to generate a final output of the binarized feedforward layer. (This is considered a mathematical relationship, mathematical formula or equation, or mathematical calculation, see in paragraphs [0071, 0090] from the specification state in page 13 or [0071] “The bound B therefore serves as a hyperparameter that controls the range of the input values that will have non-zero gradients. B also serves as a scaling factor for the outputs since the binarization function maps each element of x to either −B/2 or +B/2”, and see in paragraph [0090] or page 16 states “As another example, the layer 210 can perform the scaling 240 by applying a LayerNorm operation to the initial binarized output. In particular, while the scaling factor s enables the binarization of FFNs, it requires hyperparameter tuning, which can be challenging for large models. Instead, the system can perform the scaling 240 by replacing the operation of dividing by the scaling factor with applying a LayerNorm operation. A LayerNorm operation performed on a tensor x satisfies:
PNG
media_image2.png
95
462
media_image2.png
Greyscale
where the operations are performed element-wise, β and γ are learnable parameters, E[x] is the mean of the elements of x, Var(x) is the variance of x, and ϵ is a small floating-point number for numerical stability,” see MPEP 2106.04(a)(2), subsection I),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process or math concept but for the recitation of generic computer components, then it falls within the mental process or math concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
In step 2A prong 2 of the 101-analysis set forth in MPEP 2106, the examiner has determined that the following additional elements do not integrate this judicial exception into a practical application:
One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers, (Mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
at least in part by applying an attention mechanism to the input sequence for the block, the attended input sequence comprising a respective attended layer input at each of the one or more positions, (Mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
to implement: an attention neural network configured to perform the machine learning task, the attention neural network comprising a plurality of attention blocks, each block comprising an attention block and a binarized feedforward block, (Mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
the attention block configured to: receive an input sequence for the block comprising a respective layer input at each of one or more positions, (In step 2A, prong 2, this recites mere data gathering, which is considered insignificant extra-solution activity – see MPEP 2106.05(g)),
the binarized feedforward block comprising a plurality of binarized feedforward layers that are each configured to, for each of the one or more positions: (Mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
receive a binarized input derived from the attended layer input at the position; (In step 2A, prong 2, this recites mere data gathering, which is considered insignificant extra-solution activity – see MPEP 2106.05(g)),
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is “directed” to an abstract idea.
In step 2B of the 101-analysis set forth in the 2019 PEG, the examiner has determined that the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, additional elements iv, v, vi, and viii recite mere instructions to apply the judicial exception using generic computer components, which are not indicative of significantly more. The additional elements vii and ix recite mere data gathering, and are considered insignificant extra-solution activities. In step 2B, these insignificant extra-solution activities are well understood routine and conventional activities, which include receiving or transmitting data over a network from court case Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016), – see MPEP 2106.05(d) (II)(i)),
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 15, and 20 are rejected under 35 U.S.C. 103 over Liu, X. et al., (Pub. No. CN114282521A), published on April 5, 2022, (hereafter, Liu_X), in view of Qian, Y. et al., in “Binary neural networks for speech recognition”, published on May 2019, available at https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=11289869 , (hereafter, Qian), and further in view of Fraser, N. et al., in “Scaling binarized neural networks on reconfigurable logic,” published on 25th of January 2017, available at https://dl.acm.org/doi/pdf/10.1145/3029580.3029586 , (hereafter, Fraser).
Claim 1:
Regarding claim 1, Liu_X teaches “A system for performing a machine learning task on a network input to generate a network output, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement: an attention neural network configured to perform the machine learning task,”
See Liu_X in [n0084 – n0085] describe “Based on the aforementioned neural network binarization quantization method for the BERT model, this invention further provides a quantization device (also known as a natural language processing device) that adopts this neural network binarization quantization method to meet the needs of sentiment analysis scenarios in natural language processing tasks. [n0085] As shown in Figure 4, the natural language processing device includes a memory and a processor, and may further include communication components, sensor components, power supply components, multimedia components and input/output interfaces as needed.” Here, Liu_X describes a neural network binarization method that applies to a BERT transformer model as well as a system with memory and a processor.
Also, see Liu_X in [n0023-n0024] describe “In the binarization stage of the BERT model, the input data first passes through the binarization embedding layer and is then fed into the converter module; the converter module includes a multi-head attention module and a feedforward network. [n0024] A bidirectional attention mechanism based on information theory is introduced into the multi-head attention module, and a binary representation with maximized information entropy is adopted to enable the binary model to restore its perception of the input data.” Here, Liu_X describes neural network that includes an attention module that in [n0084] performs natural language processing tasks (i.e. machine learning task).
Further, Liu_X teaches “the attention neural network comprising a plurality of attention blocks, each block comprising an attention block and a binarized feedforward block”
See Liu_X in [n0043-n0045] As shown in Figure 1, the input data first passes through a binarization embedding layer and is then fed into the transformer block. Each converter module consists of two parts: a multi-head attention (MHA) module and a feed-forward network (FFN). The computation of the MHA module depends on the query Q, the key K, and the value V, all of which are derived from the hidden state. Here, N represents the length of the sequence, and D represents the dimension of the feature... For a specific transformation layer, the computation in the attention head can be represented as: [n0044] Q＝bi-linearQ(H), K＝bi-linearK(H), V＝bi linearV(H) (2) [n0045] Among them, bi-linearQ, bi-linearK, and bi linearV represent three different binary linear layers.” Here, Liu_X shows a converter module, part of the neural network, that includes both a multi-head attention module and a feed-forward network, and contain multiple attention layers (i.e. blocks) one each for query, key, and value. Under broadest reasonable interpretation, the examiner construes each block to mean a layer from the attention model.
Also, see Liu_X in [n0079-n0080] also describe “The extraction loss can be expressed as: [n0080] Where L represents the number of layers in the converter.” Here, Liu_X mentions the neural network contains various layers or a plurality of blocks.
Further, Liu_X teaches “the attention block configured to: receive an input sequence for the block comprising a respective layer input at each of one or more positions;”
See Liu_X mention in [n0023 – n0024] “In the binarization stage of the BERT model, the input data first passes through the binarization embedding layer and is then fed into the converter module; the converter module includes a multi-head attention module and a feedforward network. [n0024] A bidirectional attention mechanism based on information theory is introduced into the multi-head attention module, and a binary representation with maximized information entropy is adopted to enable the binary model to restore its perception of the input data”. Here, Liu_X shows receiving input data that later gets through a multi-head attention module.
Further, see Liu in [n0071 - n0073] describe “Formula (5) above shows that the attention score A can be represented as MSE(A,AT) in a specific layer, where A and AT are the attention scores in the binarized BERT model and the fully accurate BERT model, respectively. Since the attention score in the binarized BERT model is the result of multiplying the binarized query BQ and the key BK, the loss can be expressed as: [n0073] Since the attention score is obtained by directly multiplying two binary activations (binary query BQ and key BK). Its extraction can also be misled by direction mismatch. As shown in the upper part (a) of Figure 3, directional mismatch in attention score extraction is common and severe, and may even lead to a higher degree of optimization in the mismatch direction.” Here, Liu_X shows that after receiving an input, the model calculating attention scores in a specific layer by multiplying a binarized query and a key. The examiner construes the term position to mean a specific index, location, or "time step" within a sequence of data (such as a word in a sentence). Position also includes any item or element where the model evaluates each discrete element in the sequence independently, applying the exact same computations on each item.
See Liu_X in [n0062] describe "First, the inventors noted that the softmax function is order-preserving, which means that there exists a fixed threshold φ(τ,A) that maximizes the information entropy (A-φ(τ,A) of the binary representation.” Also, see Liu_X in [n0043 - n0047] describe " For a specific transformation layer, the computation in the attention head can be represented as: Q＝bi-linearQ(H), K＝bi-linearK(H), V＝bi linearV(H) (2)Among them, bi-linearQ, bi-linearK, and bi linearV represent three different binary linear layers. The inventors then calculated the attention score A as follows: [n0047] Among them, BQ and BK are the binarized query and key, respectively. It is important to note that the resulting attention weights are truncated by the attention mask, and each row in A can be viewed as a k-dim vector, where k is the number of unmasked elements.” By order-preserving, this shows the attention module Liu_X mentions incorporates order or position within the elements processed by a model into calculating an attention score described in [n0043 - n0047] . By mentioning unmasked elements where each row in an attention score is viewed as a k-dimensional vector, Liu_X describes input elements from the layer input of the attention model into one or more elements that has an ordered direction or position.
Also, see Liu_X also in [n0075] describe “First, extract the upstream query Q and key K instead of the attention score to leverage its knowledge while mitigating orientation mismatch. In addition, the inventors extracted the value V to further cover all inputs of the MHA module.” This further let Liu_X elaborates this attention score calculation applies to all input data that go into the multi-head attention module.
Also, see Liu_X in [n0068] describe “Here, BV is the binarized value sign(V), BA is the binarized attention weight, and is a carefully designed Bitwise-Affine Matrix Multiplication (BAMM) operator consisting of and shifts, used to align training and inference representations and perform efficient bit computation. [n0069] In short, in the Bi-Attention mechanism provided in this embodiment of the invention, the information entropy of the binarized attention weights is maximized to alleviate the huge information degradation caused by binarization and restore the bi-attention mechanism. By excluding the softmax function, this Bi-Attention mechanism also achieves higher efficiency. 12 [n0070] To address the direction mismatch issue that occurs in the backpropagation of the fully binarized BERT model, the inventors further proposed a Direction Matching Extraction (DMD) method. This method effectively utilizes the knowledge of the teacher network by using appropriate extraction selection and a carefully constructed similarity matrix, thereby optimizing the fully binarized BERT model more accurately.” Here, Liu_X elaborates in [n0068-n0070] by using a direction matching extraction method, in the attention mechanism, this shows applying direction or position information for each layer of all inputs from [n0075] processed by the attention model.
Further, Liu_X teaches “and generate an attended input sequence at least in part by applying an attention mechanism to the input sequence for the block, the attended input sequence comprising a respective attended layer input at each of the one or more positions,”
See Liu_X also in [n0075] describe “First, extract the upstream query Q and key K instead of the attention score to leverage its knowledge while mitigating orientation mismatch. In addition, the inventors extracted the value V to further cover all inputs of the MHA module.” This further let Liu_X elaborates this attention score calculation applies to all input data that go into the multi-head attention module.
See Liu_X in [n0062] describe "First, the inventors noted that the softmax function is order-preserving, which means that there exists a fixed threshold φ(τ,A) that maximizes the information entropy (A-φ(τ,A) of the binary representation.” Also, see Liu_X in [n0045 - n0047] describe " Among them, bi-linearQ, bi-linearK, and bi linearV represent three different binary linear layers. The inventors then calculated the attention score A as follows: [n0047] Among them, BQ and BK are the binarized query and key, respectively. It is important to note that the resulting attention weights are truncated by the attention mask, and each row in A can be viewed as a k-dim vector, where k is the number of unmasked elements.” By order-preserving, this shows the attention module Liu_X mentions incorporates order or position within the elements processed by a model into calculating an attention score described in [n0045 - n0047] .
Also, see Liu_X in [n0068] describe “Here, BV is the binarized value sign(V), BA is the binarized attention weight, and is a carefully designed Bitwise-Affine Matrix Multiplication (BAMM) operator consisting of and shifts, used to align training and inference representations and perform efficient bit computation. [n0069] In short, in the Bi-Attention mechanism provided in this embodiment of the invention, the information entropy of the binarized attention weights is maximized to alleviate the huge information degradation caused by binarization and restore the bi-attention mechanism. By excluding the softmax function, this Bi-Attention mechanism also achieves higher efficiency. 12 [n0070] To address the direction mismatch issue that occurs in the backpropagation of the fully binarized BERT model, the inventors further proposed a Direction Matching Extraction (DMD) method. This method effectively utilizes the knowledge of the teacher network by using appropriate extraction selection and a carefully constructed similarity matrix, thereby optimizing the fully binarized BERT model more accurately.” Here, Liu_X elaborates in [n0068-n0070] by using a direction matching extraction method, in the attention mechanism, this shows applying direction or position information for each layer of all inputs from [n0075] processed by the attention model.
Also, see Liu_X in [n0050] describe “The extraction step can alleviate the performance degradation of quantized BERT models under ultra-low bit width settings, and it can be applied to any architecture without hindrance. One feasible approach is to extract the attention score ATl, the output MTl of the MHA module, and the hidden state HTl from the full-precision teacher network in a hierarchical manner, and then transfer them to the binarized student counterparts network.” Liu_X shows using the multi-head attention module to generate an attention score (i.e. generate an attended input sequence at least in part by applying an attention mechanism to the input sequence for the block).
Further, see Liu_X also in [n0061 – n0065] describe “Furthermore, it is fixed during the reasoning process. [n0061] To mitigate the information degradation caused by binarization in the bi attention mechanism, the inventors introduced an efficient Bi-Attention mechanism for the fully binarized BERT model. This mechanism statistically maximizes the information entropy of the binary representation and applies bitwise operations for fast reasoning. The following is a detailed explanation of this. [n0062] First, the inventors noted that the softmax function is order-preserving, which means that there exists a fixed threshold φ(τ,A) that maximizes the information entropy (A-φ(τ,A) of the binary representation. [n0063] In order to restore the bidirectional attention mechanism to capture key factors, the inventors here binarize the attention weights into Boolean values, and the inventors' design is driven by maximizing information entropy. In the Bi-Attention mechanism, the inventors used a boolean function to binarize the attention score A, which is defined as: [n0065] By applying the boolean function, the elements with lower values in the attention weights are binarized to 0, so the attention weight with the largest entropy value can filter out the key elements.” Liu_X describes a method for binarizing a BERT model, where continuous attention weights are converted into discrete Boolean values described in [n0061]. Here, Liu_X shows by calculating an attention score matrix, then filter this to generate a binarized attended representation, this relates to generating an attended sequence since this was processed by the attention module.
Further, Liu_X describes filter out the key elements in [n0065], which relates to attended layer input at each position. This determines which preceding tokens or positions get routed through the block.
Further, Liu_X teaches “receive a binarized input derived from the attended layer input at the position;”
See Liu_X in [n0013-n0019] describe “Preferably, the bidirectional attention mechanism is implemented using the following formula: [n0015] Where BA is the binarized attention weight, BV is the binarized value sign(V), and is a bitwise-Affine matrix multiplier consisting of and shifts. [n0016] Preferably, the extracted parameters are query Q, key K, and value V; wherein... [n0017] Q=bi-linearQ(H), K=bi-linearK(H), V=bi linearV(H); [n0018] bi-linearQ, bi-linearK, and bi-linearV represent three different binary linear layers. [n0019] Preferably, in the binarized linear layer, a pre-binarization weight with zero mean is applied to maximize the binarization weight and activation information.” Here, Liu_X shows extracting parameters of query, key, and value, and then using binarization operations on each.
Later, see Liu_X in [n0043-n0047] describe “For a specific transformation layer, the computation in the attention head can be represented as: [n0044] Q＝bi-linearQ(H), K＝bi-linearK(H), V＝bi linearV(H) (2) [n0045] Among them, bi-linearQ, bi-linearK, and bi linearV represent three different binary linear layers. The inventors then calculated the attention score A as follows: [n0047] Among them, BQ and BK are the binarized query and key, respectively.” Here, Liu_X shows receiving a binarized query and binarized key (i.e. binarized input) from the attention scores (i.e. attended layer input ).
Further, Liu_X teaches “generate an initial binarized output for the position, comprising computing a binarized matrix multiplication between the binarized input and a binarized weight matrix for the binarized feedforward layer; “
See Liu_X in [n0068] describe “Here, BV is the binarized value sign(V), BA is the binarized attention weight, and is a carefully designed Bitwise-Affine Matrix Multiplication (BAMM) operator consisting of and shifts, used to align training and inference representations and perform efficient bit computation.”
Also, see Liu_X in [n0020] describe “Preferably, the attention score ATl, the output MTl of the multi-head attention module, and the hidden state HTl are extracted from the full-precision teacher network in a hierarchical manner and then transferred to the binarized student network.” Here, Liu_X shows taking an attention score and an output of the multi-head attention module, and then providing them to a binarized model to later create a binarized output.
Also, see Liu_X in [n0068-n0069] describe “ Here, BV is the binarized value sign(V), BA is the binarized attention weight, and is a carefully designed Bitwise-Affine Matrix Multiplication (BAMM) operator consisting of and shifts, used to align training and inference representations and perform efficient bit computation. [n0069] In short, in the Bi-Attention mechanism provided in this embodiment of the invention, the information entropy of the binarized attention weights is maximized to alleviate the huge information degradation caused by binarization and restore the bi-attention mechanism.” Liu_X mentions using matrix multiplication to multiply a binarized weight and the binarized value sign (which is an input ) to perform operations in an attention mechanism.
However, Liu_X did not teach “and the binarized feedforward block comprising a plurality of binarized feedforward layers that are each configured to, for each of the one or more positions,” or “… a binarized weight matrix for the binarized feedforward layer; “ or “and scale each element of the initial binarized output to generate a final output of the binarized feedforward layer”
In an analogous art, Qian teaches “and the binarized feedforward block comprising a plurality of binarized feedforward layers that are each configured to, for each of the one or more positions,”
See Qian in page 705 section 4. Binary neural networks for speech recognition describe “ Binarization on a feed-forward DNN is first introduced for speech recognition. In a DNN with L layers, let the activation in layer l be al, the weight matrix between layers l and l+1 be Wl+1, l, and the bias in layer l+1 be b l+1,where 1 ≤l ≤ L−1. The binarized weight matrices and activations are denoted by ˆWl+1,l and ˆal, respectively.” Here, Qian describes the process of binarizing a deep neural network (DNN) feed forward layers by using weight matrices and activations.
Further, see Qian in page 703, section 2 Neural network based acoustic modeling describe “We take a DNN based acoustic model as an example of neural network based acoustic modeling. A DNN is a feed-forward multi-layer perceptron with several hidden layers. The output of the previous layer is first linearly transformed with a weight matrix and a bias vector, and then nonlinearly transformed element-wise by an activation function. Since the DNN is used to model p(st|xt), the output layer of the DNN is typically a softmax layer. " Here, Qian shows that a deep neural network (DNN) contains a feed-forward network with multiple layers (i.e. plurality of binarized feedforward layers). Qian shows each layer is later “nonlinearly transformed element-wise by an activation function”, where element-wise relates to configuring for a position.
Further, Qian teaches “… a binarized weight matrix for the binarized feedforward layer;”
See Qian in page 705 section 4. Binary neural networks for speech recognition describe “ Binarization on a feed-forward DNN is first introduced for speech recognition. In a DNN with L layers, let the activation in layer l be al, the weight matrix between layers l and l+1 be Wl+1, l, and the bias in layer l+1 be b l+1,where 1 ≤l ≤ L−1.
The binarized weight matrices and activations are denoted by ˆWl+1,l and ˆal, respectively.” Here, Qian describes the process of binarizing a deep neural network (DNN) feed forward layers by using weight matrices and activations.
Further, see Qian in page 713, section 7. Conclusions and future work, describe " With binary weights and activations during the inference, binary matrix multiplication provides a 5–7 times speedup over highly optimized floating point matrix multiplication on modern CPUs and GPUs. This benefit results in a 3–4 times speedup for model inference in real scenarios." Here, Qian talks about using matrix multiplication with binarized weights and activation calculated from the feed-forward neural network mentioned in page 705.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Liu_X and incorporate into the teachings of Qian because both references teach a binarized model with attention network and a feedforward network.
One of ordinary skill in the art would be motivated to do so because this would bring a method whose “ frame work can be used for knowledge distillation from the full precision floating-point deep model to the binary model. Experiments show that with this knowledge distillation approach, the performance of a binary neural network can be significantly improved”, (see Qian in page 708, section 5).
However, Liu_X in view of Qian did not teach “and scale each element of the initial binarized output to generate a final output of the binarized feedforward layer”.
In an analogous art, Fraser teaches “and scale each element of the initial binarized output to generate a final output of the binarized feedforward layer,”
See Fraser in page 2, first half paragraph discusses " Significant research investigates binarization of neural networks whereby either input activations, synapse weights or output activations or a combination thereof are binarized. If all three components are binary, we refer to this as full binarization."
See also Fraser in page 4, in section 5. Evaluation, in 5.1.1 BNN Topologies, describe "To explore how Finn performs on a range of network sizes , we introduce a scaling factor , σ, to scale the width of each layer, and denote the resulting topology as cnn(σ). In terms of convolutional networks, [25] only evaluated a single non-padded BNN topology (cnnNoPad(1/2))." Here, Fraser describes using a scaling factor to scale each layer or element (examiner construes element to mean any item that is part of a model) of the binarized neural network, to create a final output see figure 1 in Fraser on page 2.
PNG
media_image4.png
835
669
media_image4.png
Greyscale
See Fraser in section 4.1 page 3 describe “In fact, the original BinaryNet[6] paper uses ternary values{−1,0,+1} for the forward pass, with zeros used for padding. .. Since Finn focuses on BNNs that fit entirely into on-chip memory of a single FPGA, minimizing the resource footprint is essential. Thus, a padding solution that avoids ternary values is preferable." Since the model FINN is a framework for Binarized Neural Networks (BNNs), and BNNs rely exclusively on a one-way flow of data from inputs through hidden layers to outputs without memory loops, they inherently function as feedforward networks. In a feedforward network, information moves in one direction—from the input layers, through any hidden layers, to the output layers—without any feedback loops. The mention of a "forward pass" confirms this architecture, as it refers to the standard process of passing input data forward through the network layers to calculate an output.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Liu_X and Qian, and incorporate with the teachings of Fraser by using the teachings of Liu_X and Qian, with Fraser’s teaching of scaling each element of the binarized output.
One of ordinary skill in the art would be motivated to do so because by integrating Fraser’s framework into the methods of Liu_X and Qian, one with ordinary skill in the art would achieve the goal of providing “We show that a small modification to padding (padding with-1 values) improves accuracy over no-padding and is comparable to 0-padding, while still allowing networks to maintain a binary data path. We found that high performance for large networks can be attained, with our highest demonstrated performance achieving 12 kFPS at less than 41 W of board power and 14.8 TOPS of raw computational performance,” (see Fraser in page 29, section 6. Conclusion), and “Finn utilized a novel set of optimizations that enable efficient mapping of BNNs to hardware and implemented fully connected, non-padded convolutional and pooling layers,” (see Fraser in page 25, abstract).
Claim 15:
Regarding claim 15, the claim recites similar limitations as corresponding independent claim 1 and is rejected for similar reasons as claim 1 using similar teachings and rationale.
Claim 20:
Regarding claim 20, Liu_X teaches “One or more non-transitory computer-readable storage media storing instructions …”
See Liu_X in [n0086] describe “In the above-mentioned natural language processing device, the memory stores the program or instructions of the above-mentioned neural network binarization and quantization method; the processor is coupled to the memory 15 and is used to execute the program or instructions in the memory so that the electronic device executes the neural network binarization and quantization method in the above embodiment to perform downstream tasks of natural language processing, including sequence classification,” and see Liu_X in [n0022] describe “According to a second aspect of the present invention, a neural network binarization and quantization apparatus for BERT models is provided for performing natural language processing tasks. The apparatus includes a processor and a memory, wherein the processor reads a computer program from the memory to perform the following operations:” From [n0086] and [n0022], Liu_X mentions a memory storing a computer program with instructions to run the methods.
Regarding claim 20, the claim recites similar limitations as corresponding independent claim 1 and is rejected for similar reasons as claim 1 using similar teachings and rationale.
Claims 2 and 16; 4 and 18 are rejected under 35 U.S.C. 103 over Liu_X in view of Qian, further in view of Fraser, further in view of Hou, L., et al. in “Normalization helps training of quantized LSTM,” published on December 2019 for a conference, available at https://proceedings.neurips.cc/paper/2019/file/f8eb278a8bce873ef365b45e939da38a-Paper.pdf , (hereafter, Hou), and further in view of Vaswani, A., et al., in “Attention is all you need”, published on December 2017, and cited in the IDS, (hereafter, Vaswani).
Claim 2:
Regarding claim 2, Liu_X in view of Qian, further in view of Fraser, teach the limitations in claim 1. However, Liu_X in view of Qian, further in view of Fraser, did not teach “2. The system of claim 1, wherein scaling each element of the initial binarized output comprises: dividing each element of the initial binarized output by a scaling hyperparameter.”
See Hou in page 4, section 3.2 Layer Normalization (LN) mentions " The output from layer normalization is y=LN(x)=g z+b, where g and b are the scaling and bias parameters. For layer normalization applied to Wh∗ht−1, let g∗=gk, σ∗=σk, where k=argmax1≤j≤dgj." Here, Hou mentions g as a scaling parameter, which scales each layer or element with layer norm.
See also Hou in page 1, Introduction, paragraphs 1-2 where Hou mentions " The long-short-term memory (LSTM) [10] has achieved remarkable performance in various sequence modeling tasks [26, 28, 14]. Though powerful, the high-dimensional input/hidden/output and recursive computation across long time steps lead to space and time inefficiencies... In BinaryConnect [5], each weight is binarized. By introducing a scaling parameter, the binary weight network [23], ternary weight network [17], and loss-aware binarized/ternarized network [11, 12] often report performance that are even comparable with the full-precision network. " Hou mentions in sequence modeling tasks and binarizing the weights of the sequence of the binary network, which relates to creating an initial binarized output, using a scaling parameter helps improve model performance.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Liu_X , Qian, and Fraser and incorporate with the teachings of Hou by using the teachings of Liu_X, Qian, and Fraser with the teaching of Hou of using a scaling hyperparameter.
One of ordinary skill in the art would be motivated to do so because by integrating Hou’s framework into the methods of Liu_X, Qian, and Fraser, one with ordinary skill in the art would achieve the goal of providing “With only one or two bits, the normalized quantized LSTMs achieve comparable performance with the full-precision baseline. Moreover, weight/layer normalization perform as well as batch normalization (with separate statistics), but are more memory efficient,” (see Hou in page 2, part of Introduction, first half paragraph on page).
However, Liu_X in view of Qian, further in view of Fraser, and further in view of Hou did not teach “2. The system of claim 1, wherein scaling each element of the initial binarized output comprises: dividing each element of the initial binarized output by a scaling hyperparameter,”
In an analogous field, Vaswani teaches “2. The system of claim 1, wherein scaling each element of the initial binarized output comprises: dividing each element of the initial binarized output by a scaling hyperparameter,”
See Vaswani in pages 3-4, in section 3.2.1 Scaled Dot-Product Attention describe “We call our particular attention "Scaled Dot-Product Attention" (Figure 2). The input consists of queries and keys of dimension dk, and values of dimension dv. We compute the dot products of the query with all keys, divide each by √dk, and apply a softmax function to obtain the weights on the values. In practice, we compute the attention function on a set of queries simultaneously, packed together into a matrix Q. The keys and values are also packed together into matrices K and V . We compute the matrix of outputs as:
PNG
media_image5.png
108
799
media_image5.png
Greyscale
… We suspect that for large values of dk, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients. To counteract this effect, we scale the dot products by 1 / √ dk .” Vaswani mentions dividing each element of an output dot products by √dk , where √dk is a scaling hyperparameter.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Liu_X , Qian, Fraser, and Hou, and incorporate with the teachings of Vaswani by using the teachings of Liu_X, Qian, Fraser, and Hou, with the teaching of Vaswani of using division by a scaling hyperparameter.
One of ordinary skill in the art would be motivated to do so because by integrating Vaswani’s framework into the methods of Liu_X, Qian, Fraser, and Hou, one with ordinary skill in the art would provide “models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU,” (see Vaswani in page 1, abstract).
Claim 4:
Regarding claim 4, Liu_X in view of Qian, further in view of Fraser, further in view of Hou, teach the limitations in claim 3.
Further, Liu_X teaches “4. The system of claim 3, wherein the binarized feedforward block comprises: a first feedforward layer followed by a second feedforward layer, wherein: the first feedforward layer is configured to receive a respective binarized attended layer input for each position, and, for each position:”
See Liu_X in [n0013-n0019] describe “Preferably, the bidirectional attention mechanism is implemented using the following formula: [n0015] Where BA is the binarized attention weight, BV is the binarized value sign(V), and is a bitwise-Affine matrix multiplier consisting of and shifts. [n0016] Preferably, the extracted parameters are query Q, key K, and value V; wherein... [n0017] Q=bi-linearQ(H), K=bi-linearK(H), V=bi linearV(H); [n0018] bi-linearQ, bi-linearK, and bi-linearV represent three different binary linear layers. [n0019] Preferably, in the binarized linear layer, a pre-binarization weight with zero mean is applied to maximize the binarization weight and activation information.” Here, Liu_X shows extracting parameters of query, key, and value from the attention mechanism, and then using binarization operations on each, and shows “three different binary linear layers”.
Further, Liu_X teaches “and applying a binarized Rectified Linear Unit (ReLU) activation function to the binarized sum to generate the first initial binarized output; and scale each element of the first initial binarized output by applying a LayerNorm operation to the first initial binarized output to generate a first binarized output for the position.”
See Liu_X in [n0071- n0073] describe “Since the attention score in the binarized BERT model is the result of multiplying the binarized query BQ and the key BK, the loss can be expressed as:
PNG
media_image6.png
86
851
media_image6.png
Greyscale
[n0073] Since the attention score is obtained by directly multiplying two binary activations (binary query BQ and key BK).” Liu_X here shows combining two binary activations.
Also, see Liu_X in [n0052] describe “Then, the prediction layer extraction loss is obtained by minimizing the soft cross-entropy (SCE) between the teacher's logical value (yT) and the student's logical value (y). 10 The objective function is expressed as:
PNG
media_image7.png
80
866
media_image7.png
Greyscale
”, where Liu_X mentions a sum from the binarized attention calculation above.
See also Liu_X in [n0019-n0020] describe “Preferably, in the binarized linear layer, a pre-binarization weight with zero mean is applied to maximize the binarization weight and activation information. [n0020] Preferably, the attention score ATl, the output MTl of the multi-head attention module, and the hidden state HTl are extracted from the full-precision teacher network in a hierarchical manner and then transferred to the binarized student network.” Liu_X continues to describe the outputs of the binarized layer calculated from the attention module.
Further, see Liu_X in [n0043 -n0047] describe “Each converter module consists of two parts: a multi-head attention (MHA) module and a feed-forward network (FFN). The computation of the MHA module depends on the query Q, the key K, and the value V, all of which are derived from the hidden state. Here, N represents the length of the sequence, and D represents the dimension of the feature. 9 For a specific transformation layer, the computation in the attention head can be represented as: [n0044] Q＝bi-linearQ(H), K＝bi-linearK(H), V＝bi linearV (H) (2) [n0045] Among them, bi-linearQ, bi-linearK, and bi linearV represent three different binary linear layers. The inventors then calculated the attention score A as follows: [n0047] Among them, BQ and BK are the binarized query and key, respectively.” Liu_X in [n0043] describes a multi-head attention “MHA module that depends on the query Q, the key K, and the value V”, and in [n0044], Liu_X shows applying a binarized attention operation on bi-linear () to the set of queries, keys, and values. In [n0045], Liu_X describes a calculated attention score, which includes a binarized attention output, which is considered an initial binarized output.
Further, Qian teaches “generate a first initial binarized output for the position, comprising: computing a binarized matrix multiplication between the binarized attended layer input and a binarized weight matrix for the first binarized feedforward layer; adding a binarized bias to a product of the binarized matrix multiplication to generate a binarized sum;”
See Qian in page 5, section 4. Binary neural networks for speech recognition Binarization describe “Binarization on a feed-forward DNN is first introduced for speech recognition. In a DNN with L layers, let the activation in layer l be al, the weight matrix between layers l and l+1 be Wl+1,l, and the bias in layer l+1 be bl+1, where 1 ≤l≤L−1. The binarized weight matrices and activations are denoted by W^l+1,l and a^l, respectively.” Qian describes incorporating bias addition into the weight matrix for binarizing a DNN model, where binarized bias relates to adding a bias term to matrix multiplication .
See Qian in page 3, section 3 Binary Matrix Multiplication, subsection 3.1 Population Count Based Binary Matrix Multiplication describe “In practical applications, the multiplication of two matrices A∈Rm×k and B∈Rk×n requires 2×m×n×k floating-point arithmetic operations (multiplications and additions).” Later, Qian describes this matrix multiplication used in algorithm 1.
PNG
media_image8.png
928
856
media_image8.png
Greyscale
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Liu_X and incorporate into the teachings of Qian because both references teach a binarized model with attention network and a feedforward network.
One of ordinary skill in the art would be motivated to do so because this would bring a method whose “ frame work can be used for knowledge distillation from the full precision floating-point deep model to the binary model. Experiments show that with this knowledge distillation approach, the performance of a binary neural network can be significantly improved”, (see Qian in page 708, section 5).
However, Liu_X in view of Qian, further in view of Fraser, and further in view of Hou, did not teach “ 4. The system of claim 3, wherein the binarized feedforward block comprises: a first feedforward layer followed by a second feedforward layer, wherein: the first feedforward layer is configured to receive a respective binarized attended layer input for each position, and, for each position:” or “and applying a binarized Rectified Linear Unit (ReLU) activation function to the binarized sum to generate the first initial binarized output; and scale each element of the first initial binarized output by applying a LayerNorm operation to the first initial binarized output to generate a first binarized output for the position.”
In an analogous art, Vaswani teaches“ 4. The system of claim 3, wherein the binarized feedforward block comprises: a first feedforward layer followed by a second feedforward layer, wherein: the first feedforward layer is configured to receive a respective binarized attended layer input for each position, and, for each position:”
See Vaswani in pages 2-3, section 3.1 Encoder and Decoder Stacks Encoder, describe “The encoder is composed of a stack of N = 6 identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network. We employ a residual connection [10] around each of the two sub-layers, followed by layer normalization [1]. That is, the output of each sub-layer is LayerNorm(x + Sublayer(x)), where Sublayer(x) is the function implemented by the sub-layer itself.” Vaswani describes two layers, one attention mechanism layer, and the second is a feedforward layer. Also, see Vaswani in page 5, section 3.2.3, note “In "encoder-decoder attention" layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder. This allows every position in the decoder to attend over all positions in the input sequence,” where Vaswani describes an input sequence attended by attention model’s decoder to every position.
Later, see Vaswani in page 4, section 3.2.2. mention “multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions.” Here, Vaswani shows the model attend to information at different positions and relates to receive a respective attended layer input for each position.
Further, Vaswani teaches “and applying a binarized Rectified Linear Unit (ReLU) activation function to the binarized sum to generate the first initial binarized output; and scale each element of the first initial binarized output by applying a LayerNorm operation to the first initial binarized output to generate a first binarized output for the position.”
See Vaswani in page 5, section 3.3 Position-wise Feed-Forward Networks describe “In addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully connected feed-forward network, which is applied to each position separately and identically. This consists of two linear transformations with a ReLU activation in between.” Here, Vaswani shows a feed-forward network with multiple layers that is applied to each position individually. Vaswani shows that there are more than one feedforward network layers. Vaswani also applies a RELU activation in between layers. See Vaswani in pages 2-3, section 3.1 Encoder and Decoder Stacks Encoder, describe LayerNorm operation.
Also, see Vaswani in pages 4-5, section 3.2.2. mention “multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions… in this work we employ h = 8 parallel attention layers, or heads. For each of these we use dk = dv = dmodel/h = 64.” This shows the attention model can attend on layer inputs at different positions.
Later, see Vaswani describe in page 5, section 3.2.3 Applications of Attention in our Model “the encoder contains self-attention layers. In a self-attention layer all of the keys, values and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder. Similarly, self-attention layers in the decoder allow each position in the decoder to attend to all positions in the decoder up to and including that position. We need to prevent leftward information flow in the decoder to preserve the auto-regressive property*”. Here, Vaswani shows a layer input that has been attended to, where "Each position in the encoder can attend to all positions in the previous layer" showing receive an attended layer input for each position.
Further, see Vaswani in page 3, figure 1 where Vaswani mentions using a feedforward network that receives an input already processed by the multi-head attention function, and then further output results.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Liu_X , Qian, Fraser, and Hou, and incorporate with the teachings of Vaswani by using the teachings of Liu_X, Qian, Fraser, and Hou with the teaching of Vaswani of using ReLU and LayerNorm.
One of ordinary skill in the art would be motivated to do so because by integrating Vaswani’s framework into the methods of Liu_X, Qian, Fraser, and Hou, one with ordinary skill in the art would provide “models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU,” (see Vaswani in page 1, abstract).
Claim 16:
Regarding claim 16, the claim recites similar limitations as corresponding claim 2, and is rejected for similar reasons as claim 2 using similar teachings and rationale.
Claim 18:
Regarding claim 18, the claim recites similar limitations as corresponding claim 4 and is rejected for similar reasons as claim 4 using similar teachings and rationale.
Claims 3 and 17 are rejected under 35 U.S.C. 103 over Liu_X in view of Qian, further in view of Fraser, further in view of Hou.
Claim 3:
Regarding claim 3, Liu_X in view of Qian, further in view of Fraser, teach the limitations in claim 1. However, Liu_X in view of Qian, further in view of Fraser, did not teach “The system of claim 1, wherein scaling each element of the initial binarized output comprises: applying a LayerNorm operation to the initial binarized output.”
However, in an analogous field, Hou teaches “The system of claim 1, wherein scaling each element of the initial binarized output comprises: applying a LayerNorm operation to the initial binarized output.”
See Hou in pages 3-4, in section 3, Normalization in LSTM, describe “study the properties of (full-precision and quantized) LSTMs with weight normalization [24], layer normalization [2], and batch normalization [13], and how these properties help optimization of the quantized LSTMs.” Further, see Hou in page 1, Introduction, mention “the long-short-term memory (LSTM) [10] has achieved remarkable performance in various sequence modeling tasks [26, 28, 14]. Though powerful, the high-dimensional input/hidden/output and recursive computation across long time steps…” Here, Hou describes how layer normalization is used on binarized LSTMs and its inputs and output. Examiner construes initial binarized outputs to be any result from a model that performs binarization methods.
See Hou in page 5, section 4, Experiments describe “We compare with the full-precision LSTM, and popular state-of-the-art quantized LSTMs including(i)1-bitLSTMs with/without normalization: binarized using BinaryConnect(BC) [5],binary weight network (BWN)[23],and loss-aware binarization(LAB)[11]” , where Hou shows using normalization on binarized data.
See Hou in page 6, section 4.1, Character-level Language Model in subsection Normalization, describe "with weight / layer /batch(shared) normalization, BinaryConnect achieves comparable or even better results than the full-precision LSTM". This shows that Hou describes using layer normalization on binarized networks achieve improved results. See Hou in page 7, in Hou for details about results performed with and without normalization.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Liu_X , Qian, and Fraser and incorporate with the teachings of Hou by using the teachings of Liu_X, Qian, and Fraser with the teaching of Hou of applying LayerNorm to binarized output.
One of ordinary skill in the art would be motivated to do so because by integrating Hou’s framework into the methods of Liu_X, Qian, and Fraser, one with ordinary skill in the art would bring “normalization techniques, such as weight normalization [24], layer normalization [2] and batch normalization [13], have been found useful in improving deep network training and performance… Moreover, weight/layer normalization perform as well as batch normalization (with separate statistics), but are more memory efficient,” (see Hou pages 1-2, of Introduction).
Claim 17:
Regarding claim 17, the claim recites similar limitations as corresponding claim 3 and is rejected for similar reasons as claim 3 using similar teachings and rationale.
Claims 5 and 19 are rejected under 35 U.S.C. 103 over Liu_X in view of Qian, further in view of Fraser, further in view of Hou, further in view of Vaswani, and further in view of Umuroglu Y. et al. in “FINN: A Framework for Fast Scalable Binarized Neural Network Inference,” published on February 22, 2017, available at: https://dl.acm.org/doi/pdf/10.1145/3020078.3021744 , (hereafter, Umuroglu).
Claim 5:
Regarding claim 5, Liu_X in view of Qian, further in view of Fraser, further in view of Hou, and further in view of Vaswani, teach the limitations in claim 4.
Further, Qian teaches “comprising: computing a second binarized matrix multiplication between the first binarized output and a binarized weight matrix for the second binarized feedforward layer,”
See Qian in page 5, section 4. Binary neural networks for speech recognition Binarization describe “Binarization on a feed-forward DNN is first introduced for speech recognition. In a DNN with L layers, let the activation in layer l be al, the weight matrix between layers l and l+1 be Wl+1,l, and the bias in layer l+1 be bl+1, where 1 ≤l≤L−1. The binarized weight matrices and activations are denoted by W^l+1,l and a^l, respectively.” Qian describes incorporating bias addition into the weight matrix for binarizing a DNN model.
See Qian in page 3, section 3 Binary Matrix Multiplication, subsection 3.1 Population Count Based Binary Matrix Multiplication describe “In practical applications, the multiplication of two matrices A∈Rm×k and B∈Rk×n requires 2×m×n×k floating-point arithmetic operations (multiplications and additions).” Later, Qian describes this matrix multiplication used in algorithm 1"
Further, Qian shows “5. The system of claim 4, wherein the second feedforward layer is configured to receive the respective first binarized output for each position, … adding a second binarized bias to a product of the second binarized matrix multiplication to generate a binarized sum;”
See Qian in page 705 section 4. Binary neural networks for speech recognition describe “ Binarization on a feed-forward DNN is first introduced for speech recognition. In a DNN with L layers, let the activation in layer l be al, the weight matrix between layers l and l+1 be Wl+1, l, and the bias in layer l+1 be b l+1,where 1 ≤l ≤ L−1. The binarized weight matrices and activations are denoted by ˆWl+1,l and ˆal, respectively.” Qian shows including a bias in a layer that has been binarized.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the reference of Liu_X and incorporate into the teachings of Qian because both references teach a binarized model with attention network and a feedforward network.
One of ordinary skill in the art would be motivated to do so because this would bring a method whose “ frame work can be used for knowledge distillation from the full precision floating-point deep model to the binary model. Experiments show that with this knowledge distillation approach, the performance of a binary neural network can be significantly improved”, (see Qian in page 708, section 5).
Further, Vaswani teaches “scale each element of the second initial binarized output by applying a LayerNorm operation to the second initial binarized output to generate a second binarized output for the position.”
See Vaswani in pages 2-3, section 3.1 Encoder and Decoder Stacks describe "Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network. We employ a residual connection [10] around each of the two sub-layers, followed by layer normalization [1]. That is, the output of each sub-layer is LayerNorm(x + Sublayer(x)), where Sublayer(x) is the function implemented by the sub-layer itself. To facilitate these residual connections, all sub-layers in the model, as well as the embedding layers, produce outputs of dimension dmodel = 512. " Vaswani here also describes using LayerNorm function around each set of sub-layers.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Liu_X , Qian, Fraser, and Hou and incorporate with the teachings of Vaswani by using the teachings of Liu_X, Qian, Fraser, and Hou, with the teaching of Vaswani of using a LayerNorm operation.
One of ordinary skill in the art would be motivated to do so because by integrating Vaswani’s framework into the methods of Liu_X, Qian, Fraser, and Hou, one with ordinary skill in the art would provide “models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU,” (see Vaswani in page 1, abstract).
However, Liu_X in view of Qian, further in view of Fraser, further in view of Hou, and further in view of Vaswani, did not teach:
“ … adding a second binarized bias to a product of the second binarized matrix multiplication to generate a binarized sum;”
or “scale each element of the second initial binarized output by applying a LayerNorm operation to the second initial binarized output to generate a second binarized output for the position.”
or “… and, for each position: generate a second initial binarized output for the position,”
In an analogous art, Umuroglu teaches “… and, for each position: generate a second initial binarized output for the position,” and “scale each element of the second initial binarized output by applying a LayerNorm operation to the second initial binarized output to generate a second binarized output for the position,” and “… adding a second binarized bias to a product of the second binarized matrix multiplication to generate a binarized sum;”
See Umuroglu in page 68, section 4.2 BNN-specific Operator Optimizations describe “ BNNs have several properties that enable a more efficient mapping to FPGAs without affecting the network accuracy, which we describe in the following subsections. We assume that the methodology described in [5] is used for training all BNNs in this paper, where all BNN layers have the following properties (unless otherwise stated): • Using 1-bit values for all input activations, weights and output activations (full binarization), where an unset bit represents-1 and a set bit represents +1. • Batch normalization prior to the activation function. • Using the following activation function: Sign(x) = {+1 if x ≥ 0,−1 if x < 0}”. Umuroglu shows binarizing all layers, which also generates output activations and other outputs. Here, Umuroglu means more than one layer and includes a second layer that can produce second outputs.
See Umuroglu in page 67, section 3.2 Accuracy–Computation Tradeoffs “ We also binarize the input images for the BNN as our experiments show that input binarization works well for MNIST. Since the space of possible network topologies that can be trained is infinite, we adopted the approach in [27] to simplify the problem. We fix the network topology to a 3 hidden layer, fully connected network while scaling the number of neurons in each layer, and plot the resulting accuracy in Table 1 along with the number of parameters and operations per frame.” Here, Umuroglu shows binarizing input for each position, where each position includes each layer. Scaling the number of neurons in each layer shows normalizing the values of the layers in the network.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Liu_X , Qian, Fraser, Hou, and Vaswani, and incorporate with the teachings of Umuroglu by using the teachings of Liu_X, Qian, Fraser, Hou, and Vaswani, with the teaching of Umuroglu of using a second binarized output.
One of ordinary skill in the art would be motivated to do so because by integrating Umuroglu’s framework into the methods of Liu_X, Qian, Fraser, Hou, and Vaswani, one with ordinary skill in the art would find a method “by utilizing a novel set of optimizations that enable efficient mapping of binarized neural networks to hardware, we implement fully connected, convolutional and pooling layers, with per-layer compute resources being tailored to user-provided throughput requirements… To the best of our knowledge, ours are the fastest classification rates reported to date on these benchmarks,” (see Umuroglu in page 65, abstract).
Claim 19:
Regarding claim 19, the claim recites similar limitations as corresponding claim 5, and is rejected for similar reasons as claim 5 using similar teachings and rationale.
Claim 6 is rejected under 35 U.S.C. 103 over Liu_X in view of Qian, further in view of Fraser, further in view of Hou, further in view of Vaswani, and further in view of Kim, S., et al., in “Incremental binarization on recurrent neural networks for single-channel source separation,” published on May 2019, available at https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8682595 , (hereafter, Kim_S).
Claim 6:
Regarding claim 6, Liu_X in view of Qian, further in view of Fraser, further in view of Hou, and further in view of Vaswani, teach the limitations in claim 4.
However, Liu_X in view of Qian, further in view of Fraser, further in view of Hou, and further in view of Vaswani, did not teach “6. The system of claim 4, wherein the second feedforward layer is binarized using bipolar binarization.”
In an analogous system, Kim_S discloses “6. The system of claim 4, wherein the second feedforward layer is binarized using bipolar binarization.”
See Kim_S describe in page 376, abstract that “Recurrent Neural Networks (RNN) require several sets of weights within its cells, which significantly increases the computational cost compared to the fully-connected networks. To mitigate this increased computation, we focus on the GRU cells and quantize the feedforward procedure with binarized values and bitwise operations”. Also, see Kim_S in page 376 in abstract mention “It eventually achieves the full binarization by incrementally increasing the amount of binarization over the iterations. Our experiments show that the proposed BGRU method produces source separation results greater than that of a real-valued fully connected network, with 11-12 dB mean Signal-to-Distortion Ratio (SDR). A fully binarized BGRU still outperforms a Bitwise Neural Network (BNN) by 1-2 dB even with less number of layers.” Kim_S mentions binarized BGRU model has more than one layer, since “increasing the amount of binarization over the iterations” show that there is more than one layer to be binarized. The examiner construes second layer to be any layer of a neural network model that a binarization method is implemented upon.
Further, see Kim_S describe in page 376, third and fourth paragraphs in Introduction that “By limiting the network to bipolar binary values, the space complexity of the network can be significantly reduced. In addition, all real-valued operations during the feedforward procedure can be replaced with bitwise logic, which further reduces both spatial and time complexity [19, 20, 21, 22]. Transforming real-valued weights into bipolar binaries results in heavy quantization loss [23, 24]. To alleviate this effect, the weights are converted into binary values through a gentle training procedure.” Kim_S mentions that “limiting the network to bipolar binary values” (i.e. binarized using bipolar binarization) for the “feedforward procedure” helps reduce time and spatial complexity, improving efficiency for the system.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Liu_X , Qian, Fraser, Hou, and Vaswani, and incorporate with the teachings of Kim_S by using the teachings of Liu_X, Qian, Fraser, Hou, Vaswani, with the teaching of Kim_S of using bipolar binarization.
One of ordinary skill in the art would be motivated to do so because by integrating the framework of Kim_S into the methods of Liu_X, Qian, Fraser, Hou, Vaswani, one with ordinary skill in the art would find “an efficient method to reduce the computational and spatial complexity of the GRU network for the source separation problem while maintaining high performance results,” (see Kim_S in 1. Introduction section on page 376).
Claims 7, 8, 9, 10, 11, 12, 13, and 14 are rejected under 35 U.S.C. 103 over Liu_X in view of Qian, further in view of Fraser, and further in view of Vaswani.
Claim 7:
Regarding claim 7, Liu_X in view of Qian, further in view of Fraser, teach the limitations in claim 1. However, Liu_X in view of Qian, further in view of Fraser, did not teach “7. The system of claim 1, wherein generating an attended input sequence at least in part by applying an attention mechanism to the input sequence for the layer comprises: for each one or more attention heads: applying query-key-value attention to a set of queries for the attention head, a set of keys for the attention head, and a set of values for the attention head to generate an attention output; and generating the attended input sequence from the attention outputs for the one or more attention heads.”
In an analogous field, Vaswani teaches “7. The system of claim 1, wherein generating an attended input sequence at least in part by applying an attention mechanism to the input sequence for the layer comprises: for each one or more attention heads: applying query-key-value attention to a set of queries for the attention head, a set of keys for the attention head, and a set of values for the attention head to generate an attention output; and generating the attended input sequence from the attention outputs for the one or more attention heads.”
See Vaswani in pages 3-4, in section 3.2.1 Scaled Dot-Product Attention describe “We call our particular attention "Scaled Dot-Product Attention" (Figure 2). The input consists of queries and keys of dimension dk, and values of dimension dv. We compute the dot products of the query with all keys, divide each by √dk, and apply a softmax function to obtain the weights on the values. In practice, we compute the attention function on a set of queries simultaneously, packed together into a matrix Q. The keys and values are also packed together into matrices K and V . We compute the matrix of outputs as:
PNG
media_image5.png
108
799
media_image5.png
Greyscale
Vaswani mentions “we compute the attention function on a set of queries simultaneously… The keys and values are also packed together into matrices K and V .” to show applying the attention mechanism to the input that “consists of queries and keys of dimension dk, and values of dimension dv.”
Further, see Vaswani in page 4, section 3.2.2 Multi-Head Attention describe “Instead of performing a single attention function with d model-dimensional keys, values and queries, we found it beneficial to linearly project the queries, keys and values h times with different, learned linear projections to dk, dk and dv dimensions, respectively. On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions.” When Vaswani mentions “On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding dv-dimensional output values”, this shows using multi-headed attention layer to the scaled dot-product attention operation, which calculates for multiple heads in parallel to generate independent attention outputs (i.e. applying query-key-value attention to a set of queries, keys or values, for the attention head to generate an attention output).
When Vaswani mentions individual attention outputs (or multiple attention heads) "are concatenated and once again projected, resulting in the final values," these final values refer to the generating attended input sequence. This represent the original input sequence after it has been processed and updated by the attention mechanism. When Vaswani describes the model projects the attention function "h times" in parallel, yielding outputs that are then "concatenated and once again projected," Vaswani is showing that the final output is constructed by combining the individual outputs from all the parallel attention heads (i.e. from the attention outputs for the one or more attention heads).
Also, see Vaswani in page 4, figure 2 illustrate a multi-head attention system having different attention layers that work in parallel on each query, key, value the system receives.
PNG
media_image9.png
586
1056
media_image9.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Liu_X , Qian, and Fraser, and incorporate with the teachings of Vaswani by using the teachings of Liu_X, Qian, Fraser, with the teaching of Vaswani of using creating attended input sequence.
One of ordinary skill in the art would be motivated to do so because by integrating Vaswani’s framework into the methods of Liu_X, Qian, Fraser, one with ordinary skill in the art would provide “models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU,” (see Vaswani in page 1, abstract).
Claim 8:
Regarding claim 8, Liu_X in view of Qian, further in view of Fraser, and further in view of Vaswani, teach the limitations in claim 7.
Further, Vaswani teaches “8. The system of claim 7, wherein for at least one of the blocks, the set of queries, the set of keys, and the set of values for each attention head are derived from the input sequence to the block.”
See Vaswani describe in page 5, section 3.2.3, Applications of Attention in our Model that “the Transformer uses multi-head attention in three different ways: • In "encoder-decoder attention" layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder. This allows every position in the decoder to attend over all positions in the input sequence. This mimics the typical encoder-decoder attention mechanisms in sequence-to-sequence models such as [31, 2, 8]. • The encoder contains self-attention layers. In a self-attention layer all of the keys, values and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder.” Here, Vaswani shows that “In a self-attention layer all of the keys, values and queries come from the same place” means that the queries, keys, values for each part of the attention layer arise from the same place, which is viewed as an input sequence here. This describes a Self-Attention layer. In this mechanism, all three components—the queries, keys, and values—are derived directly from the exact same input sequence, which acts as a dynamic lookup system where the sequence "queries" itself. Vaswani shows the query, key, and value components come from the exact same previous layer in that specific stack. This helps the model understand the internal context and relationships within that specific sequence.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Liu_X , Qian, and Fraser, and incorporate with the teachings of Vaswani by using the teachings of Liu_X, Qian, Fraser, with the teaching of Vaswani of derived from the input sequence to the attention block.
One of ordinary skill in the art would be motivated to do so because by integrating Vaswani’s framework into the methods of Liu_X, Qian, Fraser, one with ordinary skill in the art would provide “models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU,” (see Vaswani in page 1, abstract).
Claim 9:
Regarding claim 9, Liu_X in view of Qian, further in view of Fraser, and further in view of Vaswani, teach the limitations in claim 8.
Further, Vaswani teaches “9. The system of claim 8, wherein for at least one of the blocks for which the set of queries, the set of keys, and the set of values for each attention head are derived from the input sequence to the block: the query-key-value attention is causal query-key-value attention.”
See Vaswani in page 3, section 3.1, in Decoder subsection describe "We also modify the self-attention sub-layer in the decoder stack to prevent positions from attending to subsequent positions. This masking, combined with fact that the output embeddings are offset by one position, ensures that the predictions for position i can depend only on the known outputs at positions less than i." Here, Vaswani shows the decoder has predictions for position that depend on known outputs that the model has seen, or causal relations. The examiner construes causal to mean any information that is already known, and precedes the information of the predictions.
See Vaswani in page 5, section 3.2.3. Applications of Attention in our Model, mentions " In "encoder-decoder attention" layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder. This allows every position in the decoder to attend over all positions in the input sequence. This mimics the typical encoder-decoder attention mechanisms in sequence-to-sequence models such as [31, 2, 8]. • The encoder contains self-attention layers. In a self-attention layer all of the keys, values and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder." When Vaswani mentions “self-attention layer all of the keys, values and queries come from the same place,” Vaswani illustrates a causal relation, of a known origin, where information from query, key, value, is information that comes from a known attention mechanism.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Liu_X , Qian, and Fraser, and incorporate with the teachings of Vaswani by using the teachings of Liu_X, Qian, Fraser, with the teaching of Vaswani of a causal attention mechanism.
One of ordinary skill in the art would be motivated to do so because by integrating Vaswani’s framework into the methods of Liu_X, Qian, Fraser, one with ordinary skill in the art would provide “models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU,” (see Vaswani in page 1, abstract).
Claim 10:
Regarding claim 10, Liu_X in view of Qian, further in view of Fraser, and further in view of Vaswani, teach the limitations in claim 7.
Further, Vaswani teaches “10. The system of claim 7, wherein for at least one of the blocks, the set of queries are derived from the input sequence to the block, and the set of keys and the set of values are derived from an encoded representation of the network input generated by an encoder neural network.”
See Vaswani in page 5, section 3.2.3 mention "3.2.3 Applications of Attention in our Model The Transformer uses multi-head attention in three different ways: • In "encoder-decoder attention" layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder. This allows every position in the decoder to attend over all positions in the input sequence. This mimics the typical encoder-decoder attention mechanisms in sequence-to-sequence models". Here, Vaswani shows that the encoder decoder attention mechanisms create queries from input sequence from the previous decoder layer, and the keys and values are created from the encoder's output
See Vaswani in page 3 in 3.2 Attention describe “An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.” See also in Vaswani in page 3, section 3.2.1 Scaled Dot-Product Attention, mention “We call our particular attention "Scaled Dot-Product Attention" (Figure 2). The input consists of queries and keys of dimension dk, and values of dimension dv. .." Here, Vaswani shows “mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors” illustrating the encoder neural network generates vectors (i.e. encoded representations) of the queries, keys and values.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Liu_X , Qian, and Fraser, and incorporate with the teachings of Vaswani by using the teachings of Liu_X, Qian, Fraser, with the teaching of Vaswani of the set of keys and the set of values are derived from an encoded representation of the network input generated by an encoder neural network.
One of ordinary skill in the art would be motivated to do so because by integrating Vaswani’s framework into the methods of Liu_X, Qian, Fraser, one with ordinary skill in the art would provide “models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU,” (see Vaswani in page 1, abstract).
Claim 11:
Regarding claim 11, Liu_X in view of Qian, further in view of Fraser, and further in view of Vaswani, teach the limitations in claim 10.
Further, Vaswani teaches “11. The system of claim 10, wherein the attention neural network comprises the encoder neural network and a subset of the plurality of blocks are in the encoder neural network.”
See Vaswani in page 5, section 3.2.3 Applications of Attention in our Model describe “The Transformer uses multi-head attention in three different ways … The encoder contains self-attention layers. In a self-attention layer all of the keys, values and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder.” Here, Vaswani shows that the attention network contains an encoder, which has its own self-attention layers which are part of the subset of the layers (i.e. plurality of blocks in the encoder neural network).
See Vaswani in page 5, section 3.3 Position-wise Feed-Forward Networks describe “In addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully connected feed-forward network, which is applied to each position separately and identically.” Here, Vaswani shows each encoder has a fully connected feed-forward network.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Liu_X , Qian, and Fraser, and incorporate with the teachings of Vaswani by using the teachings of Liu_X, Qian, Fraser, with the teaching of Vaswani of an encoder and blocks in encoder neural network.
One of ordinary skill in the art would be motivated to do so because by integrating Vaswani’s framework into the methods of Liu_X, Qian, Fraser, one with ordinary skill in the art would provide “models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU,” (see Vaswani in page 1, abstract).
Claim 12:
Regarding claim 12, Liu_X in view of Qian, further in view of Fraser, and further in view of Vaswani, teach the limitations in claim 7.
Further, Liu_X teaches “ The system of claim 7, wherein the attention block is a binarized attention block”
See Liu_X in [n0043 -n0047] describe “Each converter module consists of two parts: a multi-head attention (MHA) module and a feed-forward network (FFN). The computation of the MHA module depends on the query Q, the key K, and the value V, all of which are derived from the hidden state. Here, N represents the length of the sequence, and D represents the dimension of the feature. 9 For a specific transformation layer, the computation in the attention head can be represented as: [n0044] Q＝bi-linearQ(H), K＝bi-linearK(H), V＝bi linearV (H) (2) [n0045] Among them, bi-linearQ, bi-linearK, and bi linearV represent three different binary linear layers. The inventors then calculated the attention score A as follows: [n0047] Among them, BQ and BK are the binarized query and key, respectively.” Each block is construed to mean different layer of the attention mechanism model, and each layer here shows a query, a key, or a value. Here, Liu_X shows binarizing each of these layers to get binarized attention blocks.
Further, Liu_X teaches “and applying a binarized query-key-value attention to the scaled sets of queries, keys, and values for the attention head to generate a binarized attention output.”
See Liu_X in [n0078] describe “Where, ‖·‖ represents l2 normalization. Previous research has shown that matrices constructed in this way are considered to reflect specific patterns in network semantic understanding. 13 The inventors further discovered that because matrices focus more on endogenous relative relationships, they are also scale-normalized and stable in terms of numerical values, making them suitable for extraction between binarized networks and full-precision networks.”
Further, see Liu_X in [n0043 -n0047] describe “Each converter module consists of two parts: a multi-head attention (MHA) module and a feed-forward network (FFN). The computation of the MHA module depends on the query Q, the key K, and the value V, all of which are derived from the hidden state. Here, N represents the length of the sequence, and D represents the dimension of the feature. 9 For a specific transformation layer, the computation in the attention head can be represented as: [n0044] Q＝bi-linearQ(H), K＝bi-linearK(H), V＝bi linearV (H) (2) [n0045] Among them, bi-linearQ, bi-linearK, and bi linearV represent three different binary linear layers. The inventors then calculated the attention score A as follows: [n0047] Among them, BQ and BK are the binarized query and key, respectively.” Liu_X in [n0043] describes a multi-head attention “MHA module that depends on the query Q, the key K, and the value V”, and in [n0044], Liu_X shows applying a binarized attention operation on bi-linear () to the set of queries, keys, and values. In [n0045], Liu_X describes a calculated attention score, which includes a binarized attention output.
Further, see Liu_X in [n0050] mention “the extraction step can alleviate the performance degradation of quantized BERT models under ultra-low bit width settings, and it can be applied to any architecture without hindrance. One feasible approach is to extract the attention score ATl, the output MTl of the MHA module, and the hidden state HTl from the full-precision teacher network in a hierarchical manner, and then transfer them to the binarized student counterparts network.” This further emphasizes Liu’s description of generating the binarized attention output from the MHA module.
Further, Vaswani teaches “...and applying a binarized query-key-value attention to the scaled sets of queries, keys, and values for the attention head to generate a binarized attention output.”
See Vaswani in page 4, second paragraph, section 3.2.1. Scaled Dot-Product Attention where Vaswani describes "in practice, we compute the attention function on a set of queries simultaneously, packed together into a matrix Q. The keys and values are also packed together into matrices K and V ." Here, Vaswani shows that the attention function includes a set of queries, keys and values.
Later, see Vaswani in page 4, fourth paragraph, section 3.2.1. describe “we suspect that for large values of dk, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients. To counteract this effect, we scale the dot products by 1 / √ dk”. Here, Vaswani describes scaling the set of queries, keys, and values using the factor 1/ √ dk. The scaled sets are construed to mean using a scaling operation on the set of queries, set of keys, and set of values to get scaled sets. See Vaswani in figure 2 illustrates generating an attention output of a scaled dot-product attention operation on a set of queries, keys and values for the multi-head attention model.
PNG
media_image10.png
780
1378
media_image10.png
Greyscale
Further, Vaswani teaches “The system of claim 7, wherein the attention block is a binarized attention block and wherein applying query-key-value attention comprises: scaling the set of queries for the attention head, the set of keys for the attention head, and the set of values for the attention head by applying LayerNorm to the set of queries for the attention head, a set of keys for the attention head, and a set of values;”
See Vaswani in page 3, section 3.2 Attention describe “An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.”
Further, see Vaswani in page 4, second paragraph, section 3.2.1. Scaled Dot-Product Attention where Vaswani describes "in practice, we compute the attention function on a set of queries simultaneously, packed together into a matrix Q. The keys and values are also packed together into matrices K and V ." Here, Vaswani shows that the attention function includes a set of queries, keys and values. Later, see Vaswani in page 4, fourth paragraph, section 3.2.1. describe for more info.
Further, see Vaswani in pages 2-3, section 3.1 Encoder and Decoder Stacks describe "Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network. We employ a residual connection [10] around each of the two sub-layers, followed by layer normalization [1]. That is, the output of each sub-layer is LayerNorm(x + Sublayer(x)), where Sublayer(x) is the function implemented by the sub-layer itself. To facilitate these residual connections, all sub-layers in the model, as well as the embedding layers, produce outputs of dimension dmodel = 512. " Vaswani here also describes using LayerNorm function around each set of sub-layers, which includes the attention mechanism function having a set of queries, keys and values. LayerNorm here acts as a scaling mechanism by normalizing the sequence individually.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Liu_X , Qian, and Fraser, and incorporate with the teachings of Vaswani by using the teachings of Liu_X, Qian, Fraser, with the teaching of Vaswani of applying a binarized attention and scaling.
One of ordinary skill in the art would be motivated to do so because by integrating Vaswani’s framework into the methods of Liu_X, Qian, Fraser, one with ordinary skill in the art would provide “models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU,” (see Vaswani in page 1, abstract).
Claim 13:
Regarding claim 13, Liu_X in view of Qian, further in view of Fraser, and further in view of Vaswani, teach the limitations in claim 12.
Further, Liu_X teaches “13. The system of claim 12, wherein generating an attended input sequence at least in part by applying an attention mechanism to the input sequence for the layer comprises: for each of the one or more attention heads: applying a binarized linear projection to the binarized attention output to generate a binarized projected output;
See Liu_X in [n0040-n0042] describe “Regarding the weights of the binarized linear layer, in this embodiment of the invention, the weights are redistributed to zero mean to preserve representational information, and a scaling factor is applied to minimize quantization error. To improve computational efficiency, activations are symbolically binarized without rescaling. Therefore, the calculation result can be expressed as: [n0042] Where W and X represent full-precision weights and activations, μ(·) represents the mean, αw is the scaling factor for the weights, and represents matrix multiplication.”
See also Liu_X in [n0043-n0047] describe “The computation of the MHA module depends on the query Q, the key K, and the value V, all of which are derived from the hidden state. Here, N represents the length of the sequence, and D represents the dimension of the feature. For a specific transformation layer, the computation in the attention head can be represented as: [n0044] Q＝bi-linearQ(H), K＝bi-linearK(H), V＝bi linearV(H) (2) [n0045] Among them, bi-linearQ, bi-linearK, and bi linearV represent three different binary linear layers. The inventors then calculated the attention score A as follows: [n0047] Among them, BQ and BK are the binarized query and key, respectively. It is important to note that the resulting attention weights are truncated by the attention mask, and each row in A can be viewed as a k-dim vector, where k is the number of unmasked elements. Then, the inventors binarized the attention weights”. Liu_X describes weights have been binarized, and in [n0043-n0047], Liu_X also mentions the attention model’s outputs of query, key, and value have been projected and binarized. The examiner construes projection to refer to transforming a continuous, high-dimensional set of features into specialized spaces for Queries (Q), Keys (K), and Values (V). Here Liu_X shows that instead of the attention network directly learning from the raw hidden variables, the model learns specific linear transformations (or, in this case, "bi-linear" interactions that model pairwise feature relationships) to generate Q, K, and V and their binarized outputs.
Further, Liu_X teaches “scaling the binarized projected output by applying LayerNorm to the binarized projected output to generate a scaled binarized projected output;”
See Liu_X in [n0047-n0049] describe “Among them, BQ and BK are the binarized query and key, respectively. It is important to note that the resulting attention weights are truncated by the attention mask, and each row in A can be viewed as a k-dim vector, where k is the number of unmasked elements. Then, the inventors binarized the attention weights. [n0049] The inventors followed the original BERT model architecture, incorporating the MHA module and the remainder of the feedforward network into a binarized network.” Liu_X here mentions using the binarized projected outputs such as BQ and BK ,
See Liu_X in [n0040] describe “Regarding the weights of the binarized linear layer, in this embodiment of the invention, the weights are redistributed to zero mean to preserve representational information, and a scaling factor is applied to minimize quantization error. To improve computational efficiency, activations are symbolically binarized without rescaling. Therefore, the calculation result can be expressed as:
PNG
media_image11.png
65
1000
media_image11.png
Greyscale
[n0042] Where W and X represent full-precision weights and activations, μ(·) represents the mean, αw is the scaling factor for the weights, and represents matrix multiplication.” Here, Liu_X shows a scaled binarized projected output.
However, Liu_X did not teach “scaling the binarized projected output by applying LayerNorm to the binarized projected output to generate a scaled binarized projected output” or “and summing the binarized attention output and the scaled binarized projected output to generate a binarized output of the attention head”
In an analogous art, Vaswani teaches “scaling the binarized projected output by applying LayerNorm to the binarized projected output to generate a scaled binarized projected output”,
See Vaswani in pages 2-3, section 3.1 Encoder and Decoder Stacks describe "Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network. We employ a residual connection [10] around each of the two sub-layers, followed by layer normalization [1]. That is, the output of each sub-layer is LayerNorm(x + Sublayer(x)), where Sublayer(x) is the function implemented by the sub-layer itself. To facilitate these residual connections, all sub-layers in the model, as well as the embedding layers, produce outputs of dimension dmodel = 512. " Vaswani talks about applying LayerNorm to scale the output.
Further, see Vaswani in page 4, section 3.2.2 Multi-Head Attention describe “Instead of performing a single attention function with d model-dimensional keys, values and queries, we found it beneficial to linearly project the queries, keys and values h times with different, learned linear projections to dk, dk and dv dimensions, respectively. On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions.” Here, Vaswani describes generating learned linear projections from queries, keys and values after applying LayerNorm.
Further, Vaswani teaches “and summing the binarized attention output and the scaled binarized projected output to generate a binarized output of the attention head,”
See Vaswani in pages 2-3, section 3.1 Encoder and Decoder Stacks describe "Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network. We employ a residual connection [10] around each of the two sub-layers, followed by layer normalization [1]. That is, the output of each sub-layer is LayerNorm(x + Sublayer(x)), where Sublayer(x) is the function implemented by the sub-layer itself. To facilitate these residual connections, all sub-layers in the model, as well as the embedding layers, produce outputs of dimension dmodel = 512. " Vaswani shows that using a residual connection or the operation LayerNorm(x + Sublayer(x)), means summing the sublayers (output from the attention module) and a scaled output (from LayerNorm) to generate an output.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Liu_X , Qian, and Fraser, and incorporate with the teachings of Vaswani by using the teachings of Liu_X, Qian, Fraser, with the teaching of Vaswani of using LayerNorm and then summing an attention module output and a scaled output.
One of ordinary skill in the art would be motivated to do so because by integrating Vaswani’s framework into the methods of Liu_X, Qian, Fraser, one with ordinary skill in the art would provide “models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU,” (see Vaswani in page 1, abstract).
Claim 14:
Regarding claim 14, Liu_X in view of Qian, further in view of Fraser, and further in view of Vaswani, teach the limitations in claim 12.
Further, Liu_X teaches “14. The system of claim 12, wherein each attention head is configured to generate the set of queries for the attention head by applying a first binarized linear projection to a corresponding input, the set of keys for the attention head by applying a second binarized linear projection to a corresponding input, and the set of values for the attention head by applying a third binarized linear projection to a corresponding input.”
See Liu_X in [n0013-n0018] describe “Preferably, the bidirectional attention mechanism is implemented using the following formula: [n0015] Where BA is the binarized attention weight, BV is the binarized value sign(V), and is a bitwise-Affine matrix multiplier consisting of and shifts. [n0016] Preferably, the extracted parameters are query Q, key K, and value V; wherein... [n0017] Q=bi-linearQ(H), K=bi-linearK(H), V=bi linearV(H); [n0018] bi-linearQ, bi-linearK, and bi-linearV represent three different binary linear layers.” Here, Liu_X shows an attention mechanism (i.e. attention head) generate the first, second, and third binarized linear projections, each representing set of query parameters, set of key parameters, and set of value parameters, applied to each of the linear projections, respectively.
Further, Vaswani teaches “14. The system of claim 12, wherein each attention head is configured to generate the set of queries for the attention head by applying a first binarized linear projection to a corresponding input, the set of keys for the attention head by applying a second binarized linear projection to a corresponding input, and the set of values for the attention head by applying a third binarized linear projection to a corresponding input.”
See Vaswani in pages 4-5, section 3.2.2. Multi-Head Attention, describe “we found it beneficial to linearly project the queries, keys and values h times with different, learned linear projections to dk, dk and dv dimensions, respectively. On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding dv-dimensional output values…
PNG
media_image12.png
105
694
media_image12.png
Greyscale
” Vaswani shows using a different linear projection for each of queries, keys, and values respectively into a first QWi Q, a second KWiK, and a third VWiV .
Also, see Vaswani in page 3, section 3.2.1 Scaled Dot-Product Attention mention “we call our particular attention "Scaled Dot-Product Attention" (Figure 2). The input consists of queries and keys of dimension dk, and values of dimension dv.” Here, Vaswani notes the input is a corresponding input to a set of query, keys, and values.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Liu_X , Qian, and Fraser, and incorporate with the teachings of Vaswani by using the teachings of Liu_X, Qian, Fraser, with the teaching of Vaswani of applying a binarized linear projection on queries, keys, and values.
One of ordinary skill in the art would be motivated to do so because by integrating Vaswani’s framework into the methods of Liu_X, Qian, Fraser, one with ordinary skill in the art would provide “models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU,” (see Vaswani in page 1, abstract).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WENWEI ZENG whose telephone number is (571)272-7111. The examiner can normally be reached Monday-Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached at (571) 272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WenWei Zeng/Examiner, Art Unit 2146
/USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146
Prosecution Insights