DETAILED ACTION
This action is responsive to the amendment filed on 07/30/2026. Claims 1, 3-10, and 12-20 are pending in the case. Claims 1, 3, 10, 12, and 19 are currently amended. Claims 1, 10, and 19 are independent claims.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 07/30/2026 has been entered.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 3-10, and 12-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1:
Step 1 Statutory Category: Claim 1 is directed to a system, which falls under one of the four statutory categories.
Step 2A Prong 1 Judicial Exception: Claim 1 recites, in part, “wherein each of the plurality of normalization sub-layers is configured to apply layer normalization to a sum of: a first scaling parameter multiplied by an input vector of the sub-layer, …; and an output vector of the sub-layer, such that the first scaling parameter weights respective contributions of the input vector and the output vector to the sum on which layer normalization is performed”. This limitation, under the broadest reasonable interpretation, covers the recitation of a mathematical calculation, as directed to “a claim that recites a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification, will be considered as falling within the "mathematical concepts" grouping. A mathematical calculation is a mathematical operation (such as multiplication) or an act of calculating using mathematical methods to determine a variable or number”. See MPEP §2106.04(a)(2)(I)(C). Further, the claim recites “wherein, at each of the plurality of layers, the processor is further configured to scale a plurality of value projection weights and a plurality of output projection weights of the attention sub-layer and a plurality of feed-forward weights of the feed-forward sub-layer by a second scaling parameter when training the transformer network”. This limitation, under the broadest reasonable interpretation, covers the recitation of a mathematical calculation, as directed to “a claim that recites a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification, will be considered as falling within the "mathematical concepts" grouping. A mathematical calculation is a mathematical operation (such as multiplication) or an act of calculating using mathematical methods to determine a variable or number”. See MPEP §2106.04(a)(2)(I)(C).
Step 2A Prong 2 Integration into a Practical Application: This judicial exception is not integrated into a practical application. In particular, the claim recites: “a computing system” and “a processor”. These limitations are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f). Further, the claim recites: “receive a training set”. This limitation amounts to mere data gathering. It is necessary to acquire the data in order to use the recited judicial exception. Therefore, this limitation is insignificant extra-solution activity to the judicial exception, see MPEP §2106.05(g). Further, the claim recites: “based at least in part on the training data set, train a transformer network that includes a plurality of layers”. This limitation is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f). Further, the claim recites: “wherein the plurality of layers each respectively include a plurality of sub-layers including: an attention sub-layer; a feed-forward sub-layer; and a plurality of normalization sub-layers downstream from corresponding sub-layers of the plurality of sub-layers”. This limitation is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §2106.05(h). Further, the claim recites: “wherein the input vector is passed to the normalization sub-layer in a residual stream, of the transformer network”. This limitation is an additional element that amounts to insignificant extra-solution activity to the judicial exception, see MPEP §2106.05(g).
Step 2B Significantly More: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements: “a computing system”, “a processor”, and “based at least in part on the training data set, train a transformer network that includes a plurality of layers” amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept. Further, the additional element “receive a training set” is insignificant extra-solution activity to the judicial exception and is directed to receiving or transmitting data over a network which courts have recognized as well-understood, routine, and conventional when they are claimed in a generic manner, see MPEP §2106.05(d)(II). Further, the additional element “wherein the plurality of layers each respectively include a plurality of sub-layers including: an attention sub-layer; a feed-forward sub-layer; and a plurality of normalization sub-layers downstream from corresponding sub-layers of the plurality of sub-layers” generally links the use of the judicial exception to a particular technological environment or field of use. Elements that merely generally link the use of the judicial exception to a particular technological environment or field of use cannot provide an inventive concept. Finally, the additional element “wherein the input vector is passed to the normalization sub-layer in a residual stream, of the transformer network” amounts to insignificant extra-solution activity to the judicial exception and further, the additional element is well‐understood, routine, and conventional as taught by activity is supported under Berkheimer Option 2, Dey et al., U.S. Patent Application Publication No. 20220292266, Paragraph 0002, Lines 21-28, “The original Transformer design uses L-layers in the encoder 110, where L=6 to perform sequential operations (e.g., 110_1, 110_2, . . . , 110_L, where L=6) and L layers for the decoder 102. Layers of encoder 110 are processed by Add and Norm components 112, 114 via skip feeds 115, 116 to perform residual connection followed by layer normalization, which are well known functions applied in deep architectures”; See also, Figure 1. The claim is not patent eligible.
Regarding claim 3, the rejection of claim 1 is incorporated, and further, the claim recites: “wherein the processor is further configured to determine the first scaling parameter and the second scaling parameter based at least in part on a number of the plurality of layers”. This limitation recites mathematical concepts in addition to those identified in the rejection of the parent claim. Thus, the claim recites a judicial exception.
The claim does not include any additional elements that amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception. The claim is not patent eligible.
Regarding claim 4, the rejection of claim 3 is incorporated, and further, the claim recites: “the processor is further configured to determine the first scaling parameter and the second scaling parameter based at least in part on whether or not the transformer network includes both an encoder and a decoder”. This limitation is a continuation of the “wherein the processor is further configured to determine the first scaling parameter and the second scaling parameter based at least in part on a number of the plurality of layers” limitation of the parent claim. Thus, the claim recites a judicial exception.
Further, the claim recites: “the transformer network includes an encoder and/or a decoder”. This limitation is an additional element that amounts to generally linking the use of the judicial exception to a particular technological environment or field of use. See MPEP §2106.05(h). Elements that merely generally link the use of the judicial exception to a particular technological environment or field of use cannot provide an inventive concept. The claim is not patent eligible.
Regarding claim 5, the rejection of claim 4 is incorporated, and further, the claim recites: “wherein: the transformer network includes the encoder without including the decoder or includes the decoder without including the encoder; the first scaling parameter is equal to
(
2
N
)
1
4
, where N is the number of the plurality of layers; and the second scaling parameter is equal to
(
8
N
)
-
1
4
”. This limitation recites mathematical concepts in addition to those identified in the rejection of the parent claim. Thus, the claim recites a judicial exception.
The claim does not include any additional elements that amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception. The claim is not patent eligible.
Regarding claim 6, the rejection of claim 4 is incorporated, and further, the claim recites: “the first scaling parameter and the second scaling parameter differ between the encoder and the decoder”. This limitation is a continuation of the “the processor is further configured to determine the first scaling parameter and the second scaling parameter based at least in part on whether or not the transformer network includes both an encoder and a decoder” limitation of the parent claim. Thus, the claim recites a judicial exception.
Further, the claim recites: “the transformer network includes both the encoder and the decoder”. This limitation is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §2106.05(h). Elements that merely generally link the use of the judicial exception to a particular technological environment or field of use cannot provide an inventive concept. The claim is not patent eligible.
Regarding claim 7, the rejection of claim 6 is incorporated, and further, the claim recites: “at the encoder: the first scaling parameter is equal to
0.81
(
N
4
M
)
1
16
, where N is a number of encoder layers included in the encoder and M is a number of decoder layers included in the decoder; and the second scaling parameter is equal to
0.87
(
N
4
M
)
-
1
16
; and at the decoder: the first scaling parameter is equal to
(
3
M
)
1
4
; and the second scaling parameter is equal to
(
12
M
)
-
1
4
”. This limitation recites mathematical concepts in addition to those identified in the rejection of the parent claim. Thus, the claim recites a judicial exception.
The claim does not include any additional elements that amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception. The claim is not patent eligible.
Regarding claim 8, the rejection of claim 1 is incorporated, and further, the claim recites: “wherein the transformer network includes 100 or more layers”. This limitation is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §2106.05(h). Elements that merely generally link the use of the judicial exception to a particular technological environment or field of use cannot provide an inventive concept. The claim is not patent eligible.
Regarding claim 9, the rejection of claim 1 is incorporated, and further, the claim recites: “wherein the transformer network is a machine translation model”. This limitation is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §2106.05(h). Elements that merely generally link the use of the judicial exception to a particular technological environment or field of use cannot provide an inventive concept. The claim is not patent eligible.
Regarding claim 10:
Step 1 Statutory Category: Claim 10 is directed to a method, which falls under one of the four statutory categories.
Step 2A Prong 1 Judicial Exception: Claim 10 recites, in part, “wherein each of the plurality of normalization sub-layers is configured to apply layer normalization to a sum of: a first scaling parameter multiplied by an input vector of the sub-layer, …; and an output vector of the sub-layer, such that the first scaling parameter weights respective contributions of the input vector and the output vector to the sum on which layer normalization is performed”. This limitation, under the broadest reasonable interpretation, covers the recitation of a mathematical calculation, as directed to “a claim that recites a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification, will be considered as falling within the "mathematical concepts" grouping. A mathematical calculation is a mathematical operation (such as multiplication) or an act of calculating using mathematical methods to determine a variable or number”. See MPEP §2106.04(a)(2)(I)(C). Further, the claim recites: “wherein, at each of the plurality of layers, training the transformer network further includes scaling a plurality of value projection weights and a plurality of output projection weights of the attention sub-layer and a plurality of feed-forward weights of the feed-forward sub-layer by a second scaling parameter”. This limitation, under the broadest reasonable interpretation, covers the recitation of a mathematical calculation, as directed to “a claim that recites a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification, will be considered as falling within the "mathematical concepts" grouping. A mathematical calculation is a mathematical operation (such as multiplication) or an act of calculating using mathematical methods to determine a variable or number”. See MPEP §2106.04(a)(2)(I)(C).
Step 2A Prong 2 Integration into a Practical Application: This judicial exception is not integrated into a practical application. In particular, the claim recites: “a computing system”. This limitation is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f). Further, the claim recites: “receiving a training set”. This limitation amounts to mere data gathering. It is necessary to acquire the data in order to use the recited judicial exception. Therefore, this limitation is insignificant extra-solution activity to the judicial exception, see MPEP §2106.05(g). Further, the claim recites: “based at least in part on the training data set, training a transformer network that includes a plurality of layers”. This limitation is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f). Further, the claim recites: “wherein the plurality of layers each respectively include a plurality of sub-layers including: an attention sub-layer; a feed-forward sub-layer; and a plurality of normalization sub-layers downstream from corresponding sub-layers of the plurality of sub-layers”. This limitation is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §2106.05(h). Finally, the claim recites: “wherein the input vector is passed to the normalization sub-layer in a residual stream of the transformer network”. This limitation is an additional element that amounts to insignificant extra-solution activity to the judicial exception, see MPEP §2106.05(g).
Step 2B Significantly More: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements: “a computing system” and “based at least in part on the training data set, training a transformer network that includes a plurality of layers” amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept. Further, the additional element “receiving a training set” is insignificant extra-solution activity to the judicial exception and is directed to receiving or transmitting data over a network which courts have recognized as well-understood, routine, and conventional when they are claimed in a generic manner, see MPEP §2106.05(d)(II). Further, the additional element “wherein the plurality of layers each respectively include a plurality of sub-layers including: an attention sub-layer; a feed-forward sub-layer; and a plurality of normalization sub-layers downstream from corresponding sub-layers of the plurality of sub-layers” generally links the use of the judicial exception to a particular technological environment or field of use. Elements that merely generally link the use of the judicial exception to a particular technological environment or field of use cannot provide an inventive concept. Finally, the additional element: “wherein the input vector is passed to the normalization sub-layer in a residual stream of the transformer network” amounts to insignificant extra-solution activity to the judicial exception, and further, the additional element is well‐understood, routine, and conventional as taught by activity is supported under Berkheimer Option 2, Dey et al., U.S. Patent Application Publication No. 20220292266, Paragraph 0002, Lines 21-28, “The original Transformer design uses L-layers in the encoder 110, where L=6 to perform sequential operations (e.g., 110_1, 110_2, . . . , 110_L, where L=6) and L layers for the decoder 102. Layers of encoder 110 are processed by Add and Norm components 112, 114 via skip feeds 115, 116 to perform residual connection followed by layer normalization, which are well known functions applied in deep architectures”; See also, Figure 1. The claim is not patent eligible.
Regarding claim 12, the rejection of claim 10 is incorporated, and further, claim 12 is substantially similar to claim 3 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 13, the rejection of claim 12 is incorporated, and further, claim 13 is substantially similar to claim 4 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 14, the rejection of claim 13 is incorporated, and further, claim 14 is substantially similar to claim 5 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 15, the rejection of claim 13 is incorporated, and further, claim 15 is substantially similar to claim 6 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 16, the rejection of claim 15 is incorporated, and further, claim 16 is substantially similar to claim 7 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 17, the rejection of claim 10 is incorporated, and further, claim 17 is substantially similar to claim 8 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 18, the rejection of claim 10 is incorporated, and further, claim 18 is substantially similar to claim 9 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 19:
Step 1 Statutory Category: Claim 19 is directed to a system, which falls under one of the four statutory categories.
Step 2A Prong 1 Judicial Exception: Claim 19 recites, in part, “process the inferencing input data … to generate inferencing output data”. This limitation, under the broadest reasonable interpretation, covers the recitation of a mathematical calculation, as directed to “a claim that recites a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification, will be considered as falling within the "mathematical concepts" grouping. A mathematical calculation is a mathematical operation (such as multiplication) or an act of calculating using mathematical methods to determine a variable or number”. See MPEP §2106.04(a)(2)(I)(C). Further, the claim recites: “wherein each of the plurality of normalization sub-layers is configured to apply layer normalization to a sum of: a first scaling parameter multiplied by an input vector of the sub-layer, …; and an output vector of the sub-layer, such that the first scaling parameter weights respective contributions of the input vector and the output vector to the sum on which layer normalization is performed”. This limitation, under the broadest reasonable interpretation, covers the recitation of a mathematical calculation, as directed to “a claim that recites a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification, will be considered as falling within the "mathematical concepts" grouping. A mathematical calculation is a mathematical operation (such as multiplication) or an act of calculating using mathematical methods to determine a variable or number”. See MPEP §2106.04(a)(2)(I)(C). Further, the claim recites: “wherein, at each of the plurality of layers, the processor is further configured to scale a plurality of value projection weights and a plurality of output projection weights of the attention sub-layer and a plurality of feed-forward weights of the feed-forward sub-layer by a second scaling parameter when training the transformer network”. This limitation, under the broadest reasonable interpretation, covers the recitation of a mathematical calculation, as directed to “a claim that recites a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification, will be considered as falling within the "mathematical concepts" grouping. A mathematical calculation is a mathematical operation (such as multiplication) or an act of calculating using mathematical methods to determine a variable or number”. See MPEP §2106.04(a)(2)(I)(C).
Step 2A Prong 2 Integration into a Practical Application: This judicial exception is not integrated into a practical application. In particular, the claim recites: “a computing system” and “a processor”. These limitations are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f). Further, the claim recites: “receive inferencing input data”. This limitation amounts to mere data gathering. It is necessary to acquire the data in order to use the recited judicial exception. Therefore, this limitation is insignificant extra-solution activity to the judicial exception, see MPEP §2106.05(g). Further, the claim recites: “at a transformer network”. This limitation is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §2106.05(h). Further, the claim recites: “wherein the transformer network includes a plurality of layers that each respectively include a plurality of sub-layers including: an attention sub-layer; a feed-forward sub-layer; and a plurality of normalization sub-layers downstream from corresponding sub-layers of the plurality of sub-layers”. This limitation is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §2106.05(h). Further, the claim recites: “wherein the input vector is passed to the normalization sub-layer in a residual stream of the transformer network”. This limitation is an additional element that amounts to insignificant extra-solution activity to the judicial exception, see MPEP §2106.05(g). Finally, the claim recites: “output the inferencing output data”. This limitation is insignificant extra-solution activity to the judicial exception, see MPEP §2106.05(g).
Step 2B Significantly More: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements: “a computing system” and “a processor”, amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept. Further, the additional elements “receive inferencing input data” and “output the inferencing output data” are insignificant extra-solution activity to the judicial exception and are directed to receiving or transmitting data over a network which courts have recognized as well-understood, routine, and conventional when they are claimed in a generic manner, see MPEP §2106.05(d)(II). Further, the additional elements “at a transformer network” and “wherein the transformer network includes a plurality of layers that each respectively include a plurality of sub-layers including: an attention sub-layer; a feed-forward sub-layer; and a plurality of normalization sub-layers downstream from corresponding sub-layers of the plurality of sub-layers” generally link the use of the judicial exception to a particular technological environment or field of use. Elements that merely generally link the use of the judicial exception to a particular technological environment or field of use cannot provide an inventive concept. Finally, the additional element: “wherein the input vector is passed to the normalization sub-layer in a residual stream of the transformer network” amounts to insignificant extra-solution activity to the judicial exception, and further, the additional element is well‐understood, routine, and conventional as taught by activity is supported under Berkheimer Option 2, Dey et al., U.S. Patent Application Publication No. 20220292266, Paragraph 0002, Lines 21-28, “The original Transformer design uses L-layers in the encoder 110, where L=6 to perform sequential operations (e.g., 110_1, 110_2, . . . , 110_L, where L=6) and L layers for the decoder 102. Layers of encoder 110 are processed by Add and Norm components 112, 114 via skip feeds 115, 116 to perform residual connection followed by layer normalization, which are well known functions applied in deep architectures”; See also, Figure 1. The claim is not patent eligible.
Regarding claim 20, the rejection of claim 19 is incorporated, and further, the claim recites: “wherein the transformer network is a machine translation model configured to: receive, as the inferencing input data, a text input in a first language; and output, as the inferencing output data, the text input translated into a second language”. This limitation is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §2106.05(h). Elements that merely generally link the use of the judicial exception to a particular technological environment or field of use cannot provide an inventive concept. The claim is not patent eligible.
Response to Arguments
Applicant’s arguments regarding the 35 U.S.C. 101 rejections of the claims have been fully considered but are unpersuasive.
Argument 1:
Applicant first argues, in page 11, paragraph 1 of the response, that the claims reflect an improvement to the structure of LayerNorm which is inherently an improvement to a machine learning model architecture, and the model architecture that uses the first scaling parameter and the second scaling parameter reflects an improvement because it has higher training stability at high layer counts.
Examiner’s Response:
Examiner respectfully disagrees. While the claim does recite limitations that could be considered “the structure” of the model architecture (e.g., claim 1, lines 4-10), the claim does not reflect an improvement in that “structure” but rather what “each of the plurality of normalization sub-layers is configured” (claim 1, lines 10-11) to do and the operation the normalization sub-layers are configured to do is a mathematical concept. Further, an inventive concept cannot be furnished by the unpatentable abstract idea itself, see MPEP 2106.05(I). An improvement to the mathematical operation performed by the normalization sub-layer may be an improvement in an abstract idea, but not an improvement in the functioning of a computer, as a computer nor an improvement in the technology.
Argument 2:
Applicant next argues, on page 12, paragraph 2 of the response, that claim 1 recites an improvement to machine learning model architecture and as such is directed to eligible subject matter according to Recentive Analytics, Inc. v. Fox Corp. and Ex Parte Desjardins.
Examiner’s Response:
Examiner Respectfully disagrees. Claim 1 does not recite an improvement to machine learning model architecture, but rather, may recite an improvement to the mathematical operation performed by the normalization sub-layers. An improvement to the mathematical operation performed by the normalization sub-layer may be an improvement in an abstract idea, but not an improvement in the functioning of a computer, as a computer nor an improvement in the technology. Further, an inventive concept cannot be furnished by the unpatentable abstract idea itself, see MPEP 2106.05(I).
Applicant's arguments regarding the remainder of the claims rely upon the arguments asserted with respect to the independent claims, and are thus unpersuasive.
Conclusion
Claims 1, 3-10, and 12-20 have been rejected under 35 U.S.C. 101 only. A complete prior art search was performed for these claims; however no prior art was uncovered that disclose or fairly suggest the following claimed features:
In claims 1 and 19 (and substantially similarly in claim 10):
…wherein, at each of the plurality of layers, the processor is further configured to scale a plurality of value projection weights and a plurality of output projection weights of the attention sub-layer and a plurality of feed-forward weights of the feed-forward sub-layer by a second scaling parameter when training the transformer network
The closest prior art of record includes:
Yin et al., Patent Application Publication No. 20240127000 discloses a transformer network that performs layer normalization on a sum of an input vector of the sub-layer and an output vector of the sub-layer but does not disclose the first nor second scaling parameter required by claim 1. Yin does disclose computing a scaled dot-product of the queries and keys in an attention head, where
d
k
could be considered a scaling parameter but it does not scale the value projection weights, output projections weights and feed-forward weights as required by the claims.
Xu et al., Understanding and Improving Layer Normalization, 11/16/2019, https://arxiv.org/pdf/1911.07013 discloses AdaNorm which adopts a new transformation function which can adaptively control scaling weights towards different inputs. However, this scaling function is used to replace the bias and gain of typical LayerNorm, and not to scale value projection weights, output projections weights and feed-forward weights as required by the claims.
Liu et al., Rethinking Skip Connection with Layer Normalization in Transformers and ResNets, 05/15/2021, https://arxiv.org/pdf/2105.07205 discloses expanded skip connection with Layer Normalization which teaches layer normalization on a sum of an input vector of the sub-layer multiplied by a scaling parameter and an output vector of the sub-layer. However, this method does not disclose a second scaling parameter used to scale value projection weights, output projections weights and feed-forward weights as required by the claims.
Ahmed et al., WEIGHTED TRANSFORMER NETWORK FOR MACHINE TRANSLATION, 11/06/2017, https://arxiv.org/pdf/1711.02132 discloses a weighted transformer which uses multi-branch encoders and decoders and combines the output of the different branches using multiple scaling parameters. However, this method does not disclose a second scaling parameter used to scale value projection weights, output projections weights and feed-forward weights as required by the claims.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOLLY CLARKE SIPPEL whose telephone number is (571)272-3270. The examiner can normally be reached Monday - Friday, 7:30 a.m. - 4:30 p.m. ET..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571)272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.C.S./ Examiner, Art Unit 2122
/KAKALI CHAKI/ Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights