Prosecution Insights
Last updated: November 20, 2026
Application No. 18/787,752

ATTENTION NEURAL NETWORKS WITH PARALLEL ATTENTION AND FEED-FORWARD LAYERS

Final Rejection §103
Filed
Jul 29, 2024
Priority
Apr 01, 2022 — provisional 63/326,764 +1 more
Examiner
BEJCEK II, ROBERT H
Art Unit
2148
Tech Center
2100 — Computer Architecture & Software
Assignee
Google LLC
OA Round
2 (Final)
63%
Grant Probability
Moderate
3-4
OA Rounds
1y 6m
Est. Remaining
85%
With Interview

Examiner Intelligence

Grants 63% of resolved cases
63%
Career Allowance Rate
162 granted / 258 resolved
+7.8% vs TC avg
Strong +22% interview lift
Without
With
+22.5%
Interview Lift
resolved cases with interview
Typical timeline
3y 9m
Avg Prosecution
23 currently pending
Career history
292
Total Applications
across all art units

Statute-Specific Performance

§101
22.6%
-17.4% vs TC avg
§103
41.1%
+1.1% vs TC avg
§102
10.5%
-29.5% vs TC avg
§112
22.4%
-17.6% vs TC avg
Black line = Tech Center average estimate • Based on career data from 258 resolved cases

Office Action

§103
DETAILED ACTION Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Priority Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) is acknowledged. Applicant has not complied with one or more conditions for receiving the benefit of an earlier filing date under 35 U.S.C. 120 as follows: The later-filed application must be an application for a patent for an invention which is also disclosed in the prior application (the parent or original nonprovisional application or provisional application). The disclosure of the invention in the parent application and in the later-filed application must be sufficient to comply with the requirements of 35 U.S.C. 112(a) or the first paragraph of pre-AIA 35 U.S.C. 112, except for the best mode requirement. See Transco Products, Inc. v. Performance Contracting, Inc., 38 F.3d 551, 32 USPQ2d 1077 (Fed. Cir. 1994). The disclosure of the prior-filed application, Provisional Application No. 63/326,764 dated 4/1/2022, fails to provide adequate support or enablement in the manner provided by 35 U.S.C. 112(a) or pre-AIA 35 U.S.C. 112, first paragraph for one or more claims of this application. Specifically, exemplary claim 22 recites a Gaussian error linear unit (GeLU) activation function, which is not found in the disclosure of the provisional application or parent application. Accordingly, claims 22, 29, and 36 are not entitled to the benefit of the prior date of the provisional application or the parent application. Therefore, the priority date of these claims is 7/29/2024, which is the filing date of this application. Claim Rejections - 35 USC § 103 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. Claim(s) 21, 25-28, 32-35, 39-41 is/are rejected under 35 U.S.C. 103 as being unpatentable over Vaswani et al. (hereinafter Vaswani), Attention Is All You Need in view of Merrill et al. (hereinafter Merrill), Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent, further in view of Dey et al. (hereinafter Dey), U.S. Patent Application Publication 2022/0292266. Regarding Claim 21, Vaswani discloses a system for performing a machine learning task on a network input to generate a network output, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement: an attention neural network configured to perform the machine learning task, the attention neural network comprising a stack of a plurality of attention layers, each attention layer comprising an attention sub-layer and a feed-forward sub-layer [“The Transformer follows this overall architecture using stacked self-attention and point-wise, fully connected layers” §3 ¶2; Fig. 1], and each attention layer configured to: receive an input sequence for the attention layer [“an input sequence” §3 ¶1; “Inputs” Fig. 1]; provide the input sequence to the attention sub-layer and to the feed-forward sub- layer [Fig. 1], wherein the attention sub-layer is configured to generate an attended input sequence based at least in part on applying an attention mechanism to the input sequence by using one or more query linear transformation matrices, one or more key linear transformation matrices, and one or more value linear transformation matrices included in the attention sub-layer [“An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors.” §3.2 ¶1; Fig. 2; “we compute the attention function on a set of queries simultaneously, packed together into a matrix Q. The keys and values are also packed together into matrices K and V.” §3.2.1 ¶2], wherein the one or more query linear transformation matrices, the one or more key linear transformation matrices, and the one or more value linear transformation matrices each include respective weights [“parameter matrices WiQ є Rdmodelxdk ,WiK є Rdmodelxdk ,WiV є Rdmodelxdv and WO є Rhdvxdmodel” §3.2.1 ¶3] but no biases, and wherein the feed-forward sub-layer configured to generate a transformed input sequence based on applying a sequence of transformations on the input sequence by using two or more weight matrices included in the feed-forward sub-layer [“In addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully connected feed-forward network” §3.3 ¶1; Fig. 1], wherein the two or more weight matrices each include respective weights [§3.3 equation 2] but no biases; and generate the output sequence for the attention layer by determining a combination of the attended input sequence and the transformed input sequence [“We also use the usual learned linear transformation and softmax function to convert the decoder output to predicted next-token probabilities.” §3.4 ¶1; Fig. 1]. However, while a computer and memory are implicit in the execution of transformers, Vaswani fails to explicitly disclose the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers; but no biases; but no biases. Merrill discloses the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers [“on NVIDIA A100 GPUs” §A ¶3]; but no biases [“the self attention and feed-forward sublayers have no bias terms” §D ¶1]; but no biases [“the self attention and feed-forward sublayers have no bias terms” §D ¶1]. It would have been obvious to one having ordinary skill in the art, having the teachings of Vaswani and Merrill before him before the effective filing date of the claimed invention, to modify the system of Vaswani to incorporate using no biases of Merrill. Given the advantage of removing a bias calculation to simplify the process since the sublayers are normalized, one having ordinary skill in the art would have been motivated to make this obvious modification. However, Vaswani fails to explicitly disclose wherein the attention sub-layer and the feed-forward sub-layer are configured to receive a same input sequence in parallel. Dey discloses wherein the attention sub-layer and the feed-forward sub-layer are configured to receive a same input sequence in parallel [“In contrast with a conventional encoder with multiple layers of a 2 sub-layer stack in sequential processing, the enhanced encoder of this disclosure operates with a single layer of a multi-head attention component in parallel with a feed forward network” §10; “Similar to enhanced encoder, enhanced decoder is enhanced by parallel configuration of multi-head attention components and feed-forward network” §14; Fig. 2]. It would have been obvious to one having ordinary skill in the art, having the teachings of Vaswani, Merrill, and Dey before him before the effective filing date of the claimed invention, to modify the combination to incorporate the attention sub-layer and the feed-forward sub-layer configured to receive a same input sequence in parallel of Dey. Given the advantage of superior performance of the enhanced encoder with parallel integration demonstrating significantly reduced number of parameters related to neural network operations, which results in computation resources being greatly conserved and model learning accelerated, one having ordinary skill in the art would have been motivated to make this obvious modification. Regarding Claim 25, Vaswani, Merrill, and Dey disclose the system of claim 21. Vaswani further discloses wherein generating the transformed input sequence based on applying the sequence of transformations on the input sequence by using the two or more fully-connected weight matrices included in the feed-forward sub-layer comprises: multiplying the input sequence by a weight matrix that represents weights of a first fully-connected layer to generate a product [“we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation, similar to [24]. In the embedding layers, we multiply those weights by p dmodel” §3.4 ¶1]. However, Vaswani fails to explicitly disclose adding no bias to the product. Merrill discloses adding no bias to the product [“the self attention and feed-forward sublayers have no bias terms” §D ¶1]. It would have been obvious to one having ordinary skill in the art, having the teachings of Vaswani, Merrill, and Dey before him before the effective filing date of the claimed invention, to modify the combination to incorporate using no biases of Merrill. Given the advantage of removing a bias calculation to simplify the process since the sublayers are normalized, one having ordinary skill in the art would have been motivated to make this obvious modification. Regarding Claim 26, Vaswani, Merrill, and Dey disclose the system of claim 21. Vaswani further discloses wherein determining the combination of the attended input sequence and the transformed input sequence comprises: computing a sum of the attended input sequence and the transformed input sequence [“the sums of the embeddings” §5.4 ¶2]. Regarding Claim 27, Vaswani, Merrill, and Dey disclose the system of claim 21. Vaswani further discloses wherein the attention mechanism comprises a multi-head self-attention mechanism [“Multi-Head Attention” §3.2.2; Fig. 1, 2]. Claim 28 is rejected on the same grounds as claim 21. Claims 32-34 are rejected on the same grounds as claims 25-27 respectively. Claim 35 is rejected on the same grounds as claim 21. Claims 39-41 are rejected on the same grounds as claims 25-27 respectively. Claim(s) 22, 29, 36 is/are rejected under 35 U.S.C. 103 as being unpatentable over Vaswani, Merrill, and Dey in view of So et al. (hereinafter So), Primer: Searching for Efficient Transformers for Language Modeling. Regarding Claim 22, Vaswani, Merrill, and Dey disclose the system of claim 21. Vaswani further discloses wherein the sequence of transformations comprises a first linear transformation followed by a Gaussian error linear unit (GeLU) activation function followed by a second linear transformation [“This consists of two linear transformations with a ReLU activation in between.” §3.3 ¶1]. However, Vaswani fails to explicitly disclose wherein the sequence of transformations comprises a first linear transformation followed by a Gaussian error linear unit (GeLU) activation function followed by a second linear transformation. So discloses wherein the sequence of transformations comprises a first linear transformation followed by a Gaussian error linear unit (GeLU) activation function [“the most commonly used activation functions: ReLU, GELU and Swish” §3 ¶3] followed by a second linear transformation. It would have been obvious to one having ordinary skill in the art, having the teachings of Vaswani, Merrill, Dey, and So before him before the effective filing date of the claimed invention, to modify the combination to incorporate the GELU of So. Given the advantage of substituting one known activation function for another, one having ordinary skill in the art would have been motivated to make this obvious modification. Claims 29 and 36 are rejected on the same grounds as claim 22. Claim(s) 23-24, 30-31, 37-38 is/are rejected under 35 U.S.C. 103 as being unpatentable over Vaswani, Merrill, and Dey in view of Narang et al. (hereinafter Narang), Do Transformer Modifications Transfer Across Implementations and Applications? Regarding Claim 23, Vaswani, Merrill, and Dey disclose the system of claim 21. Vaswain further discloses wherein the sequence of transformations comprises a first linear transformation followed by a Swish activation function followed by a second linear transformation [“This consists of two linear transformations with a ReLU activation in between.” §3.3 ¶1]. However, Vaswani fails to explicitly disclose wherein the sequence of transformations comprises a first linear transformation followed by a Swish activation function followed by a second linear transformation. Narang discloses wherein the sequence of transformations comprises a first linear transformation followed by a Swish activation function [“The activation functions that we explored are: (1) GeLU (Hendrycks & Gimpel, 2016), (2) Swish (Ramachandran et al., 2017)” §3.1 ¶1] followed by a second linear transformation. It would have been obvious to one having ordinary skill in the art, having the teachings of Vaswani, Merrill, Dey, and Narang before him before the effective filing date of the claimed invention, to modify the combination to incorporate the Swish activation function of Narang. Given the advantage of a simple substitution of one activation function for another, one having ordinary skill in the art would have been motivated to make this obvious modification. Regarding Claim 24, Vaswani, Merrill, and Dey disclose the system of claim 21. Vaswani further discloses wherein the sequence of transformations comprises a first linear transformation followed by a SwiGLU activation function followed by a second linear transformation [“This consists of two linear transformations with a ReLU activation in between.” §3.3 ¶1]. However, Vaswani fails to explicitly disclose wherein the sequence of transformations comprises a first linear transformation followed by a SwiGLU activation function followed by a second linear transformation. Narang discloses wherein the sequence of transformations comprises a first linear transformation followed by a SwiGLU activation function [“We also explore Gated Linear Unit" (GLU) variants” §3.1 ¶1; “We find that several activation functions improve performance over the ReLU activation used in the standard Transformer architecture. Specifically, SwiGLU and GeGLU improve performance” §4.1 ¶2] followed by a second linear transformation. It would have been obvious to one having ordinary skill in the art, having the teachings of Vaswani, Merrill, Dey, and Narang before him before the effective filing date of the claimed invention, to modify the combination to incorporate the SwiGLU activation function of Narang. Given the advantage of a simple substitution of one activation function for another and improved performance, one having ordinary skill in the art would have been motivated to make this obvious modification. Claims 30-31 are rejected on the same grounds as claims 23-24 respectively. Claims 37-38 are rejected on the same grounds as claims 23-24 respectively. Examiner’s Note The Examiner respectfully requests of the Applicant in preparing responses, to fully consider the entirety of the reference(s) as potentially teaching all or part of the claimed invention. It is noted, REFERENCES ARE RELEVANT AS PRIOR ART FOR ALL THEY CONTAIN. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain.” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)). A reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art, including non-preferred embodiments (see MPEP 2123). The Examiner has cited particular locations in the reference(s) as applied to the claim(s) above for the convenience of the Applicant. Although the specified citations are representative of the teachings of the art and are applied to the specific limitations within the individual claim(s), typically other passages and figures will apply as well. Additionally, any claim amendments for any reason should include remarks indicating clear support in the originally filed specification. Response to Arguments Regarding the priority of claims 22, 29, and 36, Applicant asserts that amended claims are supported in the provisional application but does not provide any citation to confirm such support. Examiner was unable to find support in both the provision and parent application. Accordingly, these claims are not entitled to the priority date of either earlier filed application. Regarding the prior art rejections, Applicant's arguments with respect to the claims have been considered but are moot because the arguments do not apply to the references being used in the current rejection of the limitations. Conclusion Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT H BEJCEK II whose telephone number is (571)270-3610. The examiner can normally be reached Monday - Friday: 9:00am - 5:00pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T. Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /R.B./ Examiner, Art Unit 2148 /MICHELLE T BECHTOLD/ Supervisory Patent Examiner, Art Unit 2148
Read full office action

Prosecution Timeline

Jul 29, 2024
Application Filed
Sep 26, 2024
Response after Non-Final Action
Apr 29, 2026
Non-Final Rejection mailed — §103
Jul 17, 2026
Examiner Interview Summary
Jul 17, 2026
Applicant Interview (Telephonic)
Jul 23, 2026
Response Filed
Oct 02, 2026
Final Rejection mailed — §103 (current)

Precedent Cases

Applications granted by this same examiner with similar technology

Patent 12737689
Scale-Permuted Machine Learning Architecture
2y 1m to grant Granted Sep 15, 2026
Patent 12737592
PARTIAL ACTIVATION OF MULTIPLE PATHWAYS IN NEURAL NETWORKS
1y 10m to grant Granted Sep 15, 2026
Patent 12651203
SYSTEM AND METHOD FOR TRAINING TEACHER MODEL BASED ON STRUCTURE OF STUDENT MODEL
4y 9m to grant Granted Jun 09, 2026
Patent 12639600
DATABASE UTILIZING SPATIAL PROBABILITY MODELS FOR DATA COMPRESSION
3y 7m to grant Granted May 26, 2026
Patent 12619909
STUDENT PERFORMANCE EVALUATION METHOD AND SYSTEM BASED ON AI IDENTIFICATION DATA
4y 2m to grant Granted May 05, 2026
Study what changed to get past this examiner. Based on 5 most recent grants.

Strategy Recommendation AI-generated — please review before filing

Get a prosecution strategy drawn from examiner precedents, rejection analysis, and claim mapping.
Typically takes 5-10 seconds — AI-generated, attorney review required before filing

Prosecution Projections

3-4
Expected OA Rounds
63%
Grant Probability
85%
With Interview (+22.5%)
3y 9m (~1y 6m remaining)
Median Time to Grant
Moderate
PTA Risk
Based on 258 resolved cases by this examiner. Grant probability derived from career allowance rate.

Sign in with your work email

Enter your email to receive a magic link. No password needed.

Personal email addresses (Gmail, Yahoo, etc.) are not accepted.

Free tier: 3 strategy analyses per month