Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on April 14, 2026 has been entered.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-5, 7-13, and 15-22 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1,
Claim 1 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 1 is directed to a system for performing a machine learning task on a network input to generate a network output, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“applying hk query linear transformations to the input vectors to generate hk sets of query vectors, wherein hk is an integer greater than one”
“applying hk key linear transformations to the memory vectors to generate hk sets of key vectors, wherein each of the hk sets of query vectors corresponds to one of the hk sets of key vectors”
“applying hv value linear transformations to the memory vectors to generate hv sets of value vectors, wherein each set of value vectors corresponds to a respective one of the hv attention heads”
“for each input vector, generating hv weighted value vectors”
“for each of the hv sets of value vectors: for each query vector in each of the hk sets of query vectors, generating a corresponding set of attention-logits for the query vector that includes a respective attention-logit for each key vector in the corresponding set of key vectors for the set of query vectors, comprising applying an attention function between the query vector and the corresponding set of key vectors for the set of query vectors”
“generating a corresponding set of transformed attention weights that includes a respective transformed attention weight for each value vector in the set of value vectors, wherein the corresponding set of transformed attention weights for the set of value vectors depends on all of the hk sets of query vectors and all of the hk sets of key vectors, and wherein generating the corresponding set of transformed attention weights comprises, for each query vector-key vector combination, applying an attention-logit linear transformation to the hk attention logits corresponding to the query vector-key vector combination to generate h transformed attention-logits for the query vector-key vector combination, wherein h is an integer greater than one”
“computing a weighted sum of the value vectors in the set of value vectors weighted by the corresponding set of transformed attention weights that depends on all of the hk sets of query vectors and all of the hk sets of key vectors to generate a corresponding weighted value vector associated with the respective attention head”
“generating a respective attended vector for each input vector from the respective weighted value vectors for the hv attention heads for the input vector”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical relationships and calculations.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are mere instructions to apply (See MPEP 2106.05(f)) and insignificant extra-solution activity (See MPEP 2106.05(g)).
The limitations:
“A system for performing a machine learning task on a network input to generate a network output, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement: an attention neural network configured to perform the machine learning task, the attention neural network comprising a plurality of layers, each layer comprising an attention sub- layer, the attention sub-layer comprising hv attention heads, wherein hv is an integer greater than one, and configured to perform operations”
“wherein the query linear transformations, the key linear transformation, the value linear transformations, and the attention-logit linear transformation of each attention sub-layer have been learned during training of the attention neural network to perform the machine learning task through adjustments to parameters of the attention neural network to determine trained values for the parameters, and wherein the attention neural network has been trained on an unsupervised data set through unsupervised learning”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
The limitations:
“obtaining an input sequence for the layer comprising a respective input vector at each of n positions, wherein n is an integer greater than or equal to one”
“obtaining m memory vectors, wherein m is an integer greater than or equal to one”
As drafted, are additional elements that amount to no more than insignificant extra-solution activity. See MPEP 2106.05(g).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, all of the additional elements are “mere instructions to apply an exception” and “insignificant extra-solution activity”. Specifically, the obtaining limitations recite the well-understood, routine, and conventional activity of receiving and transmitting data over a network. MPEP 2106.05(d)(II); OIP Techs., Inc v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network). Mere instructions to apply an exception and insignificant extra-solution activity cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 2,
Claim 2 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 2 is directed to a system for performing a machine learning task on a network input to generate a network output, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“applying an output linear transformation to a concatenation of the weighted value vectors for the input vector to generate the respective attended vector for the input vector”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical relationships and calculations.
Step 2A Prong Two Analysis: See corresponding analysis of claim 1.
Step 2B Analysis: See corresponding analysis of claim 1.
Regarding Claim 3,
Claim 3 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 3 is directed to a system for performing a machine learning task on a network input to generate a network output, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are mere instructions to apply (See MPEP 2106.05(f)).
The limitations:
“wherein the attention sub-layer is a self-attention sub-layer and wherein the one or more input vectors are the same as the one or more memory vectors”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, all of the additional elements are “mere instructions to apply an exception”. Mere instructions to apply an exception cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 4,
Claim 4 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 4 is directed to a system for performing a machine learning task on a network input to generate a network output, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 3.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are mere instructions to apply (See MPEP 2106.05(f)).
The limitations:
“wherein the attention sub-layer is a masked self- attention sub-layer and wherein the attention function is masked”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, all of the additional elements are “mere instructions to apply an exception”. Mere instructions to apply an exception cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 5,
Claim 5 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 5 is directed to a system for performing a machine learning task on a network input to generate a network output, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“generating, from the sets of attention-logits for the query vectors in the sets of query vectors, a plurality of sets of transformed attention-logits that each include a respective transformed attention-logit for each memory vector, comprising applying the attention-logit linear transformation to the sets of attention-logits”
“for each of the sets of transformed attention-logits, generating a corresponding set of attention weights that includes a respective attention weight for each memory vector”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical relationships and calculations.
Step 2A Prong Two Analysis: See corresponding analysis of claim 1.
Step 2B Analysis: See corresponding analysis of claim 1.
Regarding Claim 7,
Claim 7 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 7 is directed to a system for performing a machine learning task on a network input to generate a network output, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“applying a plurality of learned input logit linear transformations to the input vectors to generate a plurality of dynamic input attention-logit matrices, and applying a plurality of learned memory logit linear transformations to the memory vectors to generate a plurality of dynamic memory attention-logit matrices, and wherein generating the plurality of sets of transformed attention-logits that each include a respective transformed attention-logit for each memory vector further comprises: applying the plurality of dynamic input attention-logit matrices to the sets of attention-logits, and applying the plurality of dynamic memory attention-logit matrices to the sets of attention-logits”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical relationships and calculations.
Step 2A Prong Two Analysis: See corresponding analysis of claim 5.
Step 2B Analysis: See corresponding analysis of claim 5.
Regarding Claim 8,
Claim 8 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 8 is directed to a system for performing a machine learning task on a network input to generate a network output, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“generating the sets of transformed attention weights from the sets of attention weights, comprising applying an attention weight linear transformation to the sets of attention weights to generate the sets of transformed attention weights”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical relationships and calculations.
Step 2A Prong Two Analysis: See corresponding analysis of claim 5.
Step 2B Analysis: See corresponding analysis of claim 5.
Regarding Claim 9,
Claim 9 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 9 is directed to a system for performing a machine learning task on a network input to generate a network output, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 8.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are mere instructions to apply (See MPEP 2106.05(f)).
The limitations:
“wherein the attention weight linear transformation is learned during the training of the attention neural network”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, all of the additional elements are “mere instructions to apply an exception”. Mere instructions to apply an exception cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 10,
Claim 10 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 10 is directed to a system for performing a machine learning task on a network input to generate a network output, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“applying a plurality of learned input attention weight linear transformations to the input vectors to generate a plurality of dynamic input attention weight matrices, and applying a plurality of learned memory attention weight linear transformations to the memory vectors to generate a plurality of dynamic memory attention weight matrices, and wherein generating the plurality of sets of transformed attention weights comprises: applying the plurality of dynamic input attention weight matrices to the sets of attention weights, and applying the plurality of dynamic memory attention weight matrices to the sets of attention weights”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical relationships and calculations.
Step 2A Prong Two Analysis: See corresponding analysis of claim 9.
Step 2B Analysis: See corresponding analysis of claim 9.
Regarding Claim 11,
Claim 11 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 11 is directed to a system for performing a machine learning task on a network input to generate a network output, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein for each of the sets of transformed attention-logits, generating a corresponding set of attention weights that includes a respective attention weight for each memory vector comprises: for each of the sets of transformed attention-logits, applying a softmax function to the transformed attention-logits in the set to generate the corresponding set of attention weights”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical relationships and calculations.
Step 2A Prong Two Analysis: See corresponding analysis of claim 5.
Step 2B Analysis: See corresponding analysis of claim 5.
Regarding Claim 12,
Claim 12 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 12 is directed to a system for performing a machine learning task on a network input to generate a network output, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein the attention function is a dot-product attention function or a scaled dot-product attention function”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical relationships/formulas and calculations.
Step 2A Prong Two Analysis: See corresponding analysis of claim 1.
Step 2B Analysis: See corresponding analysis of claim 1.
Regarding Claim 13,
Claim 13 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 13 is directed to a system for performing a machine learning task on a network input to generate a network output, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“generate an output sequence for the layer from the attended input sequence, the output sequence comprising a respective layer output vector at each of the one or more positions”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical relationships/formulas and calculations.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are mere instructions to apply (See MPEP 2106.05(f)).
The limitations:
“wherein the layer also comprises a feed-forward sub layer that is configured to: receive an attended input sequence that includes the respective attended input vectors for each of the input vectors”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, all of the additional elements are “mere instructions to apply an exception”. Mere instructions to apply an exception cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 15,
Claim 15 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 15 is directed to one or more non-transitory computer-readable storage media storing instructions, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“applying hk query linear transformations to the input vectors to generate hk sets of query vectors, wherein hk is an integer greater than one”
“applying hk key linear transformations to the memory vectors to generate hk sets of key vectors, wherein each of the hk sets of query vectors corresponds to one of the hk sets of key vectors”
“applying hv value linear transformations to the memory vectors to generate hv sets of value vectors, wherein each set of value vectors corresponds to a respective one of the hv attention heads”
“for each input vector, generating hv weighted value vectors”
“for each of the hv sets of value vectors: for each query vector in each of the hk sets of query vectors, generating a corresponding set of attention-logits for the query vector that includes a respective attention-logit for each key vector in the corresponding set of key vectors for the set of query vectors, comprising applying an attention function between the query vector and the corresponding set of key vectors for the set of query vectors”
“generating a corresponding set of transformed attention weights that includes a respective transformed attention weight for each value vector in the set of value vectors, wherein the corresponding set of transformed attention weights for the set of value vectors depends on all of the hk sets of query vectors and all of the hk sets of key vectors, and wherein generating the corresponding set of transformed attention weights comprises, for each query vector-key vector combination, applying an attention-logit linear transformation to the hk attention logits corresponding to the query vector-key vector combination to generate h transformed attention-logits for the query vector-key vector combination, wherein h is an integer greater than one”
“computing a weighted sum of the value vectors in the set of value vectors weighted by the corresponding set of transformed attention weights that depends on all of the hk sets of query vectors and all of the hk sets of key vectors to generate a corresponding weighted value vector associated with the respective attention head”
“generating a respective attended vector for each input vector from the respective weighted value vectors for the hv attention heads for the input vector”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical relationships and calculations.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are mere instructions to apply (See MPEP 2106.05(f)) and insignificant extra-solution activity (See MPEP 2106.05(g)).
The limitations:
“One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to implement: an attention neural network configured to perform a machine learning task, the attention neural network comprising a plurality of layers, each layer comprising an attention sub- layer, the attention sub-layer comprising hv attention heads, wherein hv is an integer greater than one, and configured to perform operations”
“wherein the query linear transformations, the key linear transformation, the value linear transformations, and the attention-logit linear transformation of each attention sub-layer have been learned during training of the attention neural network to perform the machine learning task through adjustments to parameters of the attention neural network to determine trained values for the parameters, and wherein the attention neural network has been trained on an unsupervised data set through unsupervised learning”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
The limitations:
“obtaining an input sequence for the layer comprising a respective input vector at each of n positions, wherein n is an integer greater than or equal to one”
“obtaining m memory vectors, wherein m is an integer greater than or equal to one”
As drafted, are additional elements that amount to no more than insignificant extra-solution activity. See MPEP 2106.05(g).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, all of the additional elements are “mere instructions to apply an exception” and “insignificant extra-solution activity”. Specifically, the obtaining limitations recite the well-understood, routine, and conventional activity of receiving and transmitting data over a network. MPEP 2106.05(d)(II); OIP Techs., Inc v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network). Mere instructions to apply an exception and insignificant extra-solution activity cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 16,
Claim 16 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 16 is directed to a method for receiving and processing a network input, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“applying hk query linear transformations to the input vectors to generate hk sets of query vectors, wherein hk is an integer greater than one”
“applying hk key linear transformations to the memory vectors to generate hk sets of key vectors, wherein each of the hk sets of query vectors corresponds to one of the hk sets of key vectors”
“applying hv value linear transformations to the memory vectors to generate hv sets of value vectors, wherein each set of value vectors corresponds to a respective one of the hv attention heads”
“for each input vector, generating hv weighted value vectors”
“for each of the hv sets of value vectors: for each query vector in each of the hk sets of query vectors, generating a corresponding set of attention-logits for the query vector that includes a respective attention-logit for each key vector in the corresponding set of key vectors for the set of query vectors, comprising applying an attention function between the query vector and the corresponding set of key vectors for the set of query vectors”
“generating a corresponding set of transformed attention weights that includes a respective transformed attention weight for each value vector in the set of value vectors, wherein the corresponding set of transformed attention weights for the set of value vectors depends on all of the hk sets of query vectors and all of the hk sets of key vectors, and wherein generating the corresponding set of transformed attention weights comprises, for each query vector-key vector combination, applying an attention-logit linear transformation to the hk attention logits corresponding to the query vector-key vector combination to generate h transformed attention-logits for the query vector-key vector combination, wherein h is an integer greater than one”
“computing a weighted sum of the value vectors in the set of value vectors weighted by the corresponding set of transformed attention weights that depends on all of the hk sets of query vectors and all of the hk sets of key vectors to generate a corresponding weighted value vector associated with the respective attention head”
“generating a respective attended vector for each input vector from the respective weighted value vectors for the hv attention heads for the input vector”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical relationships and calculations.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are mere instructions to apply (See MPEP 2106.05(f)) and insignificant extra-solution activity (See MPEP 2106.05(g)).
The limitations:
“processing the network input using an attention neural network that is configured to perform a machine learning task on the network input to generate a network output, the attention neural network comprising a plurality of layers, each layer comprising an attention sub-layer, the attention sub-layer configured to perform operations”
“wherein the query linear transformations, the key linear transformation, the value linear transformations, and the attention-logit linear transformation of each attention sub-layer have been learned during training of the attention neural network to perform the machine learning task through adjustments to parameters of the attention neural network to determine trained values for the parameters, and wherein the attention neural network has been trained on an unsupervised data set through unsupervised learning”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
The limitations:
“receiving a network input”
“obtaining an input sequence for the layer comprising a respective input vector at each of n positions, wherein n is an integer greater than or equal to one”
“obtaining m memory vectors, wherein m is an integer greater than or equal to one”
As drafted, are additional elements that amount to no more than insignificant extra-solution activity. See MPEP 2106.05(g).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, all of the additional elements are “mere instructions to apply an exception” and “insignificant extra-solution activity”. Specifically, the receiving and obtaining limitations recite the well-understood, routine, and conventional activity of receiving and transmitting data over a network. MPEP 2106.05(d)(II); OIP Techs., Inc v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network). Mere instructions to apply an exception and insignificant extra-solution activity cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 17,
Claim 17 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 17 is directed to a method for receiving and processing a network input, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“applying an output linear transformation to a concatenation of the weighted value vectors for the input vector to generate the respective attended vector for the input vector”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical relationships and calculations.
Step 2A Prong Two Analysis: See corresponding analysis of claim 16.
Step 2B Analysis: See corresponding analysis of claim 16.
Regarding Claim 18,
Claim 18 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 18 is directed to a method for receiving and processing a network input, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 16.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are mere instructions to apply (See MPEP 2106.05(f)).
The limitation:
“wherein the attention sub-layer is a self-attention sub-layer and wherein the one or more input vectors are the same as the one or more memory vectors”
As drafted, is an additional element that amounts to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, all of the additional elements are “mere instructions to apply an exception”. Mere instructions to apply an exception cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 19,
Claim 19 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 19 is directed to a method for receiving and processing a network input, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 18.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are mere instructions to apply (See MPEP 2106.05(f)).
The limitation:
“wherein the attention sub-layer is a masked self-attention sub-layer and wherein the attention function is masked”
As drafted, is an additional element that amounts to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, all of the additional elements are “mere instructions to apply an exception”. Mere instructions to apply an exception cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 20,
Claim 20 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 20 is directed to a method for receiving and processing a network input, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“generating, from the sets of attention-logits for the query vectors in the sets of query vectors, a plurality of sets of transformed attention-logits that each include a respective transformed attention-logit for each memory vector, comprising applying the attention-logit linear transformation to the sets of attention-logits”
“for each of the sets of transformed attention-logits, generating a corresponding set of attention weights that includes a respective attention weight for each memory vector”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical relationships and calculations.
Step 2A Prong Two Analysis: See corresponding analysis of claim 16.
Step 2B Analysis: See corresponding analysis of claim 16.
Regarding Claim 21,
Claim 21 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 21 is directed to a method for receiving and processing a network input, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 16.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that merely indicate a field of use or technological environment in which to apply a judicial exception (See MPEP 2106.05(h)).
The limitations:
“wherein the machine learning task is a text generation task, the network input comprises a sequence of text, an image, or both, and the network output comprises another sequence of text”
As drafted, are additional elements that merely indicate a field of use or technological environment in which to apply a judicial exception. See MPEP 2106.05(h).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements merely indicate a field of use or technological environment in which to apply a judicial exception. The claim is not patent eligible.
Regarding Claim 22,
Claim 22 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 22 is directed to a method for receiving and processing a network input, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 16.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that merely indicate a field of use or technological environment in which to apply a judicial exception (See MPEP 2106.05(h)).
The limitations:
“wherein the machine learning task is an image generation task, the network input comprises a sequence of text, and the network output comprises an image”
As drafted, are additional elements that merely indicate a field of use or technological environment in which to apply a judicial exception. See MPEP 2106.05(h).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements merely indicate a field of use or technological environment in which to apply a judicial exception. The claim is not patent eligible.
Response to Arguments
Regarding the rejection applied under 35 U.S.C. 101, Applicant firstly asserts that the claims, as amended, recite a specific technological solution to a technological problem, and specifically, the claims are limited to a particular technical implementation that achieves an improvement in computer functionality (“Remarks”, Page 1).
Applicant further asserts that MPEP 2106.04(d)(1) now states that improvements to system performance based upon "adjustments to parameters of a machine learning model associated with tasks" are "tantamount to how the machine learning model itself would function in operation" and are not subsumed in a mathematical calculation (“Remarks”, Page 2).
Applicant further asserts that the claimed invention provides an innovative approach for configuring and training a machine learning model, which enables an improvement in machine learning technology, specifically by preventing model quality degradation when increasing the number of attention heads in a neural network (“Remarks”, Page 2). Applicant further asserts that the claimed invention provides an improvement in machine learning performance, yielding higher accuracy (e.g., better perplexities) on a variety of machine learning tasks relative to existing state-of-the-art multi-head attention schemes (“Remarks”, Page 2).
Applicant further asserts that the claimed invention achieves these improvements by configuring the attention sub-layers of the neural network to perform "talking heads" attention rather than independent multi-head attention, which inserts a learned linear projection across the attention-heads dimension of the attention-logits tensor, altering the neural network by allowing communication between the attention heads, meaning each attention function depends on all of the keys and queries rather than operating in complete independence (“Remarks”, Page 2).
Applicant further asserts that the amended claims now explicitly recite steps that enable the asserted improvements to the functioning of the neural network, such as:
“generating a corresponding set of transformed attention weights that
includes a respective transformed attention weight for each value vector in the set
of value vectors, wherein the corresponding set of transformed attention weights for the set of value vectors depends on all of the hk sets of query vectors and all of the hk sets of key vectors, and wherein generating the corresponding set of transformed attention weights comprises, for each query vector-key vector combination, applying an attention-logit linear transformation to the hk attention logits corresponding to the query vector- key vector combination to generate h transformed attention-logits for the query vector-key vector combination, wherein h is an integer greater than one; …wherein the query linear transformations, the key linear transformation, the value linear transformations, and the attention-logit linear transformation of each attention sub-layer have been learned during training of the attention neural network to perform the machine learning task through adjustments to parameters of the attention neural network to determine trained values for the parameters…” (“Remarks”, Pages 3-4).
Applicant further asserts that the above limitations correspond directly to the technical improvements described in the specification, and the claimed steps are specific adjustments to the architecture and parameters of a neural network that govern how the neural network routes data between attention heads (“Remarks”, Page 4). Applicant further asserts that under MPEP 2106.04(d)(1), adjusting parameters and operations of a machine learning model to solve a specific technological problem is tantamount to an improvement in how a machine learning model operates, and that the claims should be found eligible because this optimization of model parameters directly aligns with the updated MPEP 2106.4(d)(1) guidance (“Remarks”, Page 4).
However, the updated MPEP 2106.4(d)(1) guidance from December 5, 2025 does not explicitly state "adjustments to parameters of a machine learning model associated with tasks are tantamount to how the machine learning model itself would function in operation and are not subsumed in a mathematical calculation”. Rather, regarding Desjardins, the updated MPEP 2106.4(d)(1) guidance states:
“Indeed, enumerated improvements identified in the Desjardins specification included disclosures of the effective learning of new tasks in succession in connection with specifically protecting knowledge concerning previously accomplished tasks; allowing the system to reduce use of storage capacity; and the enablement of reduced complexity in the system. Such improvements were tantamount to how the machine learning model itself would function in operation and therefore not subsumed in the identified mathematical calculation.”
Therefore, the updated MPEP 2106.4(d)(1) guidance includes that the identified improvements in Desjardins, such as the effective learning of new tasks in succession in connection with specifically protecting knowledge concerning previously accomplished tasks; allowing the system to reduce use of storage capacity; and the enablement of reduced complexity in the system, were “tantamount to how the machine learning model itself would function in operation and therefore not subsumed in the identified mathematical calculation”, and not merely “adjustments to parameters of a machine learning model associated with tasks”.
Further, the updated MPEP 2106.4(d)(1) guidance states “Importantly, the ARP evaluated the claims as a whole in discerning at least the limitation ‘adjust the first values of the plurality of parameters to optimize performance of the machine learning model on the second machine learning task while protecting performance of the machine learning model on the first machine learning task’ reflected the improvement disclosed in the specification.”
Further, regarding Desjardins, the updated MPEP 2106.4(d)(1) guidance states “claims to a method of training a machine learning model were directed to improvements in the machine learning technology itself and additionally included data structure elements reciting adjustments in values to plurality of performance parameters while preserving prior values”.
Therefore, Desjardins provided a specific training strategy that allows the model to preserve performance on earlier tasks even as it learns new ones, which was reflected in the limitation reciting “adjust the first values of the plurality of parameters to optimize performance of the machine learning model on the second machine learning task while protecting performance of the machine learning model on the first machine learning task”. Merely reciting that parameters are adjusted during training of the neural network is not sufficient to demonstrate an improvement, and instead recites an additional element (i.e., mere instructions to apply an exception), as discussed above in the 35 U.S.C. 101 rejection of claim 1 above.
Further, the independent claims recite a plurality of mathematical calculations, which, while complex, are still a series of calculations, and fall within the mathematical concepts grouping of abstract ideas. Further, the above limitations that Applicant asserts provides the improvement also fall within the mathematical concepts grouping of abstract ideas. Therefore, even if the claims did recite an improvement, it would be an improvement in the mathematical calculation, and therefore an improvement in the abstract idea itself. The MPEP notes that it is important to keep in mind that an improvement in the abstract idea itself is not an improvement in technology. MPEP 2106.05(a)(II). Therefore, even if the claims did recite an improvement, as written, it would be an improvement in the abstract idea. Therefore, the 35 U.S.C. 101 rejection of claims 1-5, 7-13 and 15-22 is maintained.
Allowable Subject Matter
Upon proper overcome of the rejection applied under 35 U.S.C. 101 discussed above, claims 1-5, 7-13, and 15-22 would be allowable.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KURT NICHOLAS PRESSLY whose telephone number is (703)756-4639. The examiner can normally be reached M-F 8-4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KURT NICHOLAS PRESSLY/Examiner, Art Unit 2125
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125
Prosecution Insights