DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on XXXXXXXXXXXXXX has been entered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims X are canceled.
Claims X are new.
Claims 1-26 are pending and have been examined.
This action is in reply to the papers filed on 07/29/2024 (effective filing date 01/28/2022).
Information Disclosure Statement
The information disclosure statements submitted: 08/26/2025, 09/23/2025, 12/22/2025, 04/16/2026, have been considered by the Examiner and made of record in the application file.
Preliminary Amendment
The present Office Action is based upon the original patent application filed on 07/29/2024 as modified by the preliminary amendment filed on 07/29/2024.
Claim Objections – minor drafting errors
Claim 9 recites “The system claim 8” rather than “The system of claim 8.” As worded, claim 9 does not use the standard dependency phrasing required by 37 C.F.R. § 1.75(c) to clearly designate it as an improvement of claim 8, which raises a clarity/form issue.
Claims 12–14 and 23 alternate between “the respective quadratic attended input for the layer input” and “the respective quadratic attended input for layer input” (missing “the” before the second occurrence of “layer input”).
Claim 12 uses a semicolon where a colon appears to be intended (“…comprises;”).
Applicant should confirm and correct these on the record together with any response to this Office Action.
Reasons For Allowance
Prior-Art Rejection withdrawn
Claims xxx are potentially allowable over the prior-art, but, are still subject to the 35 USC §101 and/or 35 USC §112 rejections herein. The closest prior art (See PTO-892, Notice of References Cited) does not teach the claimed:
Claims xxx are allowed. Independent claims X, Y, and Z all contain the same inventive scope. The closest prior art (See PTO-892, Notice of References Cited) does not teach the claimed:
The closest prior-art (xxx) teach the features as disclosed in Non-final Rejection (xxxx), however, these cited references do not teach and the prior-art does not teach at least the following combination of features and/or elements:
Claim Rejections - 35 USC §101 - Withdrawn
Per Applicant’s amendments and arguments and considering new guidance in the MPEP, the rejections are withdrawn. Specifically, in Applicant’s Remarks (dated 03/14/2017, pgs. 8-11), Applicant traverses the 35 USC §101 rejections arguing that the amended claims recite new limitations that are not abstract, amount to significantly more, are directed to a practical application, etc… For example, Applicant argues….
In support of their arguments, Applicant cites to the following recent Fed. Cir. court cases (i.e., Alice Corp. v. CLS Bank Int’l, SRI Int’l, Inc. v. Cisco Systems, Inc., Ultramercial, Inc. v. Hulu, LLC, Berkheimer, Core Wireless, McRO, Enfish, Bascom, DDR, etc…).
Claim Rejections - 35 USC § 101
35 U.S.C. § 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-26 are rejected under 35 U.S.C. § 101 as being directed to non-statutory subject matter because the claimed invention is directed to an abstract idea without significantly more.
Legal Standard (plain-language summary)
Under Alice Corp. Pty. Ltd. v. CLS Bank Int’l, 573 U.S. 208 (2014), and Mayo Collaborative Services v. Prometheus Laboratories, Inc., 566 U.S. 66 (2012), a claim is evaluated for patent eligibility using a two-step framework (implemented by the Office as Step 1, Step 2A [two prongs], and Step 2B — see MPEP § 2106):
• Step 1: Is the claim to a process, machine, manufacture, or composition of matter?
• Step 2A, Prong One: Does the claim recite a judicial exception (an abstract idea, law of nature, or natural phenomenon)?
• Step 2A, Prong Two: If so, is the exception integrated into a practical application?
• Step 2B: If not, do the claim elements — alone or as an ordered combination — add “significantly more” than the exception itself (an “inventive concept”)?
Step 1 — Statutory Category
Claim 1 recites “a system… comprising one or more computers and one or more storage devices” — this is a machine, a statutory category.
Claim 25 recites “one or more non-transitory computer storage media” — because the claim expressly limits the storage media to “non-transitory,” it is directed to an article of manufacture and does not read on a transitory propagating signal. It therefore falls within a statutory category.
Claim 26 recites “a method performed by one or more computers” — this is a process, a statutory category.
All claims pass Step 1. The analysis proceeds to Step 2A.
Step 2A, Prong One — The Claims Recite an Abstract Idea (Mathematical Concepts)
Independent claim 1 recites, as its central steps:
“generating a respective projected input for each layer input by processing the layer inputs using one or more first feed-forward neural network layers; applying an attention mechanism over the input sequence to generate a respective attended layer input for each layer input; generating a respective initial output for each layer input by computing an element-wise product between the respective projected input for the layer input and the respective attended layer input for the layer input; and generating the output sequence for the layer from the respective initial outputs for the layer inputs…”
Stripped of the “feed-forward neural network layer” and “attention mechanism” labels, these steps describe: (1) transforming numeric values by a mathematical function, (2) computing a second numeric transformation (weighted combinations of values, i.e., a matrix multiplication/attention weighting), and (3) multiplying the two results together element-by-element to produce an output value. This is a set of mathematical relationships, mathematical formulas/equations, and mathematical calculations, which is one of the enumerated groupings of abstract ideas in MPEP § 2106.04(a)(2). See SAP Am., Inc. v. InvestPic, LLC, 898 F.3d 1161 (Fed. Cir. 2018) (claims to selecting, arranging, and mathematically manipulating data through statistical formulas were directed to an abstract idea, even though performed by a computer); Digitech Image Techs., LLC v. Elecs. for Imaging, Inc., 758 F.3d 1344 (Fed. Cir. 2014) (a process of organizing information through mathematical correlations is abstract); Parker v. Flook, 437 U.S. 584 (1978); Gottschalk v. Benson, 409 U.S. 63 (1972) (a mathematical algorithm/formula for converting data is not patent-eligible standing alone).
This characterization is reinforced by several dependent claims that recite the mathematical relationship explicitly, in equation form, which the August 4, 2025 USPTO memorandum on evaluating subject matter eligibility (reaffirming MPEP § 2106.04(a)(2) and the 2024 AI subject-matter-eligibility examples) identifies as claim language that plainly “recites” (rather than merely “involves”) a mathematical concept:
Claim 8: “a matrix A that includes the respective sets of attention weights for the layer inputs satisfies: A=relu²(Qg diag(γ)
K
g
T
+ b), where relu2 is a squared ReLU element-wise activation function… diag(γ) is a diagonal matrix that has a vector γ along the diagonal and zeroes at all other entries… and b is a bias.”
Claim 14: the same relu²-based attention-weight formula, applied “for the layer inputs in the chunk g.”
Claim 15: “a matrix
V
^
g
l
i
n
of the linear attended inputs for the layer inputs in chunk g… satisfies: V̂g^lin=Q_g diag(λ)(Σ(h=1)^(T/C) K_g^T V_h).”
Claim 16 and claim 23: the causal variant of the same linear-attention summation formula.
Claims 8, 14, 15, 16, and 23 recite nothing but a mathematical formula relating inputs (Q, K, V, γ, λ, b) to an output (A or V̂ⁱⁱⁿ) — this is a mathematical relationship/formula within MPEP § 2106.04(a)(2)(I), directly comparable to the ineligible formula claims in Parker v. Flook and SAP v. InvestPic.
The remaining dependent claims narrow the same abstract mathematical scheme without changing its character: claims 2–4 add further generic mathematical processing steps (“processing the initial outputs using one or more second feed-forward neural network layers,” “applying a normalization operation, a residual connection, or both”); claims 5–7, 9–13, and 17–18 describe additional mathematical sub-steps of the attention calculation (generating queries/keys/values, splitting the calculation into “chunks,” combining “linear” and “quadratic” attention results, sharing representations, and reducing dimensionality); and claims 19–22 and 24 describe how the layers are arranged (initial layers, output layers, auto-regressive generation, causal vs. non-causal attention) without adding anything outside the mathematical/data-manipulation scheme. Each of these is itself a mathematical operation on data or a description of how/when the mathematical operations are sequenced, and each therefore falls within the same abstract-idea grouping as claim 1. See MPEP § 2106.04(a)(2)(I).
Mental-process note: Because these formulas operate on matrices across an entire input sequence (potentially thousands of positions and hundreds of dimensions) and require impractical volumes of arithmetic to perform “in the mind,” the mathematical-concepts grouping — not the mental-process grouping — is the applicable basis for this rejection, consistent with the August 2025 USPTO memorandum’s caution against over-extending the mental-process grouping.
Step 2A, Prong Two — No Integration Into a Practical Application
The only elements of claim 1 beyond the mathematical scheme itself are: “one or more computers,” “one or more storage devices,” and the label “attention neural network.” Claim 25 adds only “non-transitory computer storage media.” Claim 26 adds only “one or more computers” and the generic steps “receiving a network input” and “processing the network input… to generate a network output.”
These are recited at a high level of generality and do not integrate the abstract mathematical scheme into a practical application:
The claims do not recite what the “network input,” “network output,” or “machine learning task” actually are (e.g., no claim requires that the input be text, an image, audio, or any other specific data type, and no claim recites a specific downstream use — see, e.g., claim 1: “a machine learning task on a network input to generate a network output”). Without such a limitation, the claims cover the mathematical scheme applied to any data, for any purpose, which is a hallmark of a claim not integrated into a practical application. See MPEP § 2106.04(d); Electric Power Grp., LLC v. Alstom S.A., 830 F.3d 1350 (Fed. Cir. 2016) (claims to gathering, analyzing, and presenting data in the abstract, without more, are not integrated into a practical application, regardless of alleged usefulness); Two-Way Media Ltd. v. Comcast Cable Commc’ns, LLC, 874 F.3d 1329 (Fed. Cir. 2017) (functional, result-oriented claiming of a desired outcome, without reciting how a technical problem is solved, is not enough); BSG Tech LLC v. Buyseasons, Inc., 899 F.3d 1281 (Fed. Cir. 2018).
“one or more computers” and “one or more storage devices” are recited as generic, off-the-shelf computing components used simply to carry out the mathematical operations; the claims do not tie the mathematical scheme to a specific improvement in how a computer operates (contrast Enfish, LLC v. Microsoft Corp., 822 F.3d 1327 (Fed. Cir. 2016), where the claim recited a specific new data-structure architecture that changed how the computer’s memory itself was organized). No claim recites, for example, a specific hardware accelerator, a specific memory-access pattern, or any other structural computer improvement.
The claims do not recite the technical justification for the particular formula chosen (for example, no claim recites that the “element-wise product” gating step, or the split between “quadratic” and “linear” attention, reduces computation time or memory relative to a standard self-attention layer, or otherwise recites a stated technical improvement). The specification itself states this benefit only at the level of the written description (e.g., ¶¶ [0009]–[0012], describing the O(n²) cost of ordinary self-attention and stating that the disclosed design “reduc[es]… quadratic complexity… to a variant with linear complexity” and trains “in less time”), not in the claim language. An unclaimed benefit described only in the specification cannot supply eligibility that is missing from the claim language itself. See Two-Way Media, 874 F.3d at 1337; MPEP § 2106.04(d)(1).
For these reasons, the claims as currently drafted do not integrate the recited mathematical concept into a practical application, and the analysis proceeds to Step 2B.
Step 2B — No Inventive Concept (“Significantly More”)
Considered individually and as an ordered combination, the additional elements of claims 1, 25, and 26 — “one or more computers,” “one or more storage devices,” and “non-transitory computer storage media” — are well-understood, routine, conventional computer components performing their generic functions (storing instructions; executing instructions). See MPEP § 2106.05(d); Alice, 573 U.S. at 225–26 (generic computer implementation of an abstract idea does not supply an inventive concept); TLI Commc’ns LLC Patent Litig., 823 F.3d 607 (Fed. Cir. 2016) (same, for generic recitation of a server and telephone unit). The claims, taken as an ordered combination, do no more than instruct that the abstract mathematical scheme be implemented on generic computer hardware — which is not enough. See BSG Tech, 899 F.3d at 1290–91 (narrowing an abstract idea, without more, does not add an inventive concept).
Under Berkheimer v. HP Inc., 881 F.3d 1360 (Fed. Cir. 2018), whether an additional element is well-understood, routine, and conventional is a factual question; this rejection is based on the fact that the claims recite only generic computer components (computers, storage devices, storage media) at the highest level of generality, which is consistent with what the specification itself describes as background, general-purpose computing hardware (e.g., implementation “in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware… or in combinations of one or more of them”).
Claims 1–26 are therefore rejected under 35 U.S.C. § 101.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. §112(b):
(B) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 19-24 are rejected under 35 U.S.C. §112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention. Claim 1 introduces “an attention neural network,” but claim 19 (from which claims 20–24 depend, directly or indirectly) refers back to “the neural network” (“wherein the neural network includes one or more initial layers…”). Because claim 1 never introduces a plain “neural network” apart from “the attention neural network,” “the neural network” in claim 19 lacks a clear antecedent basis. It is unclear whether “the neural network” in claim 19 is intended to refer back to “the attention neural network” of claim 1, or to some other, unclaimed neural network. This ambiguity is carried into claims 20–24, each of which depends from claim 19. Suggested correction: amend claim 19 to recite “wherein the attention neural network includes…” to match the antecedent established in claim 1.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-7, 10, 11, 19-22, 24-26 are rejected under 35 U.S.C. 103 as being unpatentable over: Shazeer et al. 2018/0341860; in view of “GLU Variants Improve Transformer”, Naom Shazeer, Google, Feb. 14, 2020. Ar Xiv:2002.05202v1 (hereinafter, NPL Google Shazeer).
18/834,202 – Claim 1. Shazeer et al. 2018/0341860 teaches A system for performing a machine learning task on a network input to generate a network output, the system comprising one or more computers and one or more storage devices storing instructions (Shazeer et al. 2018/0341860 [0004 - system implemented as computer programs on one or more computers in one or more locations that generates an output sequence…][0008; 0097; 0098; claim 1][0003 - Neural networks are machine learning models that employ one or more layers…][0087 - process 300 can be performed repeatedly on inputs selected from a set of training data as part of a conventional machine learning training technique to train the initial neural network layers…]) that, when executed by the one or more computers, cause the one or more computers to implement: an attention neural network configured to perform the machine learning task, the attention neural network comprising a plurality of attentive layers (Shazeer et al. 2018/0341860 [0068-0072 – attention layers]), each attentive layer configured to receive an input sequence for the layer (Shazeer et al. 2018/0341860 [0020 - the neural network includes an encoder neural network and a decoder neural network… both the encoder and the decoder are attention-based, i.e., both apply an attention mechanism over their respective received inputs while transducing the input sequence…][0008][0024 - the neural network system 100 can perform any of a variety of tasks that require processing sequential inputs to generate sequential outputs…]) and to generate an output sequence for the layer by performing operations comprising: obtaining the input sequence for the layer (Shazeer et al. 2018/0341860 [0051; 0085]), the input sequence comprising a respective layer input at each of one or more positions (Shazeer et al. 2018/0341860 [0029 - embedding layer 120 mapping each network input to a numeric representation and providing it, position by position, to the first encoder subnetwork]); generating a respective projected input for each layer input by processing the layer inputs using one or more first feed-forward neural network layers (Shazeer et al. 2018/0341860 [0040 - position-wise feed-forward layer 134, “two or more learned linear transformations each separated by an activation function”]); applying an attention mechanism over the input sequence to generate a respective attended layer input for each layer input (Shazeer et al. 2018/0341860 [0061-0062 - scaled dot-product attention mechanism 230… computes the dot products of the query with all of the keys, divides each of the dot products by a scaling factor… and then applies a softmax function]); generating a respective initial output for each layer input by computing an element-wise product between the respective projected input for the layer input and the respective attended layer input (Shazeer et al. 2018/0341860 [0040-0041; 0056-0057]) for the layer input; and generating the output sequence for the layer from the respective initial outputs for the layer inputs, wherein the output sequence for the layer includes a respective layer output for each layer input (Shazeer et al. 2018/0341860 [0039 – “Add & Norm” residual-connection and layer-normalization output of each sub-layer, passed forward as the sub-layer’s output]).
Shazeer et al. 2018/0341860 may not expressly disclose the “generating a respective initial output for each layer input by computing an element-wise product between the respective projected input… and the respective attended layer input…” features, however, NPL Google Shazeer teaches (NPL Google Shazeer [pg 1, eqns. (2 – baseline Transformer FFN) and (4 - GLU); pg. 3, § 3.2 (“The GEGLU and SwiGLU variants produce the best perplexities”) and p. 3, § 3.3 (“the new GLU-variants perform best on most of the tasks”) for the express quality-improvement motivation] discloses computing an element-wise product between two parallel projections of the input as a replacement for a Transformer’s feed-forward sub-layer, expressly for improved quality). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Shazeer et al. 2018/0341860 to include the features as taught by NPL Google Shazeer. One of ordinary skill in the art would have been motivated to do so to utilize well known tools and features useful for implementing ‘attention neural networks with gated attention units’. Combining NPL Google Shazeer’s element-wise-product gating structure with Shazeer et al. 2018/0341860 attention output as the gating branch (in place of NPL Google Shazeer’s own simpler sigmoid/GELU gate) is the combination of known elements — an attention computation and a GLU-style element-wise gate — according to each element’s own known function, to obtain the predictable result of a gated, attention-informed output.
18/834,202 – Claim 25. Shazeer et al. 2018/0341860 further teaches One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to implement (Shazeer et al. 2018/0341860 [0088; 0094; 0095; claims 1 and 29]): … an attention neural network configured to perform a machine learning task, the attention neural network comprising a plurality of attentive layers, each attentive layer configured to receive an input sequence for the layer and to generate an output sequence for the layer by performing operations comprising: obtaining the input sequence for the layer, the input sequence comprising a respective layer input at each of one or more positions; generating a respective projected input for each layer input by processing the layer inputs using one or more first feed-forward neural network layers; applying an attention mechanism over the input sequence to generate a respective attended layer input for each layer input; generating a respective initial output for each layer input by computing an element-wise product between the respective projected input for the layer input and the respective attended layer input for the layer input; and generating the output sequence for the layer from the respective initial outputs for the layer inputs, wherein the output sequence for the layer includes a respective layer output for each layer input.
18/834,202 – Claim 26. Shazeer et al. 2018/0341860 further teaches A method performed by one or more computers, the method comprising (Shazeer et al. 2018/0341860 [0088; 0094; 0095; claims 1 and 29]): … receiving a network input; and processing the network input using an attention neural network to generate a network output for the network input, the attention neural network comprising a plurality of attentive layers, each attentive layer configured to receive an input sequence for the layer and to generate an output sequence for the layer by performing operations comprising: obtaining the input sequence for the laver, the input sequence comprising a respective layer input at each of one or more positions; generating a respective projected input for each layer input by processing the layer inputs using one or more first feed-forward neural network layers; applying an attention mechanism over the input sequence to generate a respective attended layer input for each layer input; generating a respective initial output for each layer input by computing an element-wise product between the respective projected input for the layer input and the respective attended layer input for the layer input; and generating the output sequence for the layer from the respective initial outputs for the layer inputs, wherein the output sequence for the layer includes a respective layer output for each layer input.
Claims 25 and 26 recite the same operations on, respectively, “one or more non-transitory computer storage media” and as steps of “a method performed by one or more computers.” The same combination of Shazeer et al. 2018/0341860 and ‘NPL Google Shazeer’ discloses or renders obvious every corresponding limitation, for the same reasons.
Claims 2-7, 10, 11, 19-22, 24 are rejected under 35 U.S.C. 103 as being unpatentable over: Shazeer et al. 2018/0341860; in view of ‘NPL Google Shazeer’.
18/834,202 – Claim 2. Shazeer et al. 2018/0341860 further teaches The system of claim 1, the operations further comprising: generating a respective updated output for each layer input by processing the initial outputs using one or more second feed-forward neural network layers (Shazeer et al. 2018/0341860 [0040 - position-wise feed-forward layer 134] Obvious — routine retention of Shazeer et al.’s existing downstream FFN).
18/834,202 – Claim 3. Shazeer et al. 2018/0341860 further teaches The system of claim 2, wherein the respective layer output for each layer input is the respective updated output for the layer input (Shazeer et al. 2018/0341860 [0040 - position-wise feed-forward layer 134] Obvious — routine retention of Shazeer et al.’s existing downstream FFN).
18/834,202 – Claim 4. Shazeer et al. 2018/0341860 further teaches The system of claim 2, the operations further comprising: generating the respective layer output for each layer input by applying a normalization operation, a residual connection, or both to the respective updated output for the layer input (Shazeer et al. 2018/0341860 [0039 - expressly discloses residual connections and layer normalization (“Add & Norm”) after each sub-layer][Fig. 1]).
18/834,202 – Claim 5. Shazeer et al. 2018/0341860 further teaches The system of claim 1, wherein applying an attention mechanism over the input sequence to generate a respective attended layer input for each respective layer input comprises: generating a respective query and a respective key for each layer input by processing the layer inputs using one or more third feed-forward neural network layers; generating a respective value for each layer input by processing the layer inputs using one or more fourth feed-forward neural network layers; generating a respective set of attention weights for each layer input from the respective query for the layer input and the respective keys for the layer inputs; and for each layer input, applying the respective set of attention weights for the layer input to the respective values for the layer inputs (Shazeer et al. 2018/0341860 [0069-0070 - learned linear transformations forming Q, K, V][0061-0062 - dot-product attention] Shazeer et al. — discloses standard scaled dot-product/multi-head attention (Q, K, V linear projections; softmax(QKᵀ)V)).
18/834,202 – Claim 6. Shazeer et al. 2018/0341860 further teaches The system of claim 5, wherein the respective key and the respective value for each layer input are derived from a same shared representation generated by processing the layer inputs using the one or more third feed-forward neural network layers (Shazeer et al. 2018/0341860 [0074 - “all of the keys, values and queries come from the same place… the output of the previous subnetwork”] Shazeer et al. — discloses self-attention already derives Q, K, and V from the same input representation; using one shared projection for K and V specifically is a routine simplification).
18/834,202 – Claim 7. Shazeer et al. 2018/0341860 further teaches The system of claim 5, wherein the dimensionality of each query and key is smaller than the dimensionality of the layer inputs and the layer outputs (Shazeer et al. 2018/0341860 [0072 - “the sub-layer may reduce the dimensionality of the original keys, values, and queries to d/h”] Shazeer et al. — discloses multi-head attention already uses a smaller per-head dimension (d/h) than the full model dimension).
18/834,202 – Claim 10. Shazeer et al. 2018/0341860 further teaches The system of claim 1, wherein the attention mechanism is a linear attention mechanism (Shazeer et al. 2018/0341860 [0038 – attention mechanism][0069 - Each attention layer is configured to transform the original queries, and keys, and values using learned linear transformations and then apply the attention mechanism 230 to the transformed queries, keys, and values…][0078; 0080; 0082]).
18/834,202 – Claim 11. Shazeer et al. 2018/0341860 further teaches The system of claim 1, wherein the attention mechanism is a partial attention mechanism (Shazeer et al. 2018/0341860 [0038; 0069; 0078; 0080]).
18/834,202 – Claim 19. Shazeer et al. 2018/0341860 further teaches The system of claim 1, wherein the neural network includes one or more initial layers, a sequence of the attentive layers, and one or more output layers (Shazeer et al. 2018/0341860 [0029 - embedding layer 120][0020 - encoder/decoder stack] Shazeer et al. — discloses an embedding/input stage, an encoder/decoder stack, and an output stage.).
18/834,202 – Claim 20. Shazeer et al. 2018/0341860 further teaches The system of claim 19, wherein the initial layers include an embedding layer (Shazeer et al. 2018/0341860 [0029 - embedding layer 120][0020 - encoder/decoder stack] Shazeer et al. — discloses an embedding/input stage, an encoder/decoder stack, and an output stage.).
18/834,202 – Claim 21. Shazeer et al. 2018/0341860 further teaches The system of claim 19, wherein the network output is a sequence, wherein the neural network auto-regressively generates the network output over multiple time steps, and wherein at each time step the neural network processes the network input and any already generated elements of the network output (Shazeer et al. 2018/0341860 [0043 - “generating a network output for a corresponding output position conditioned on (i) the encoded representations and (ii) network outputs at output positions preceding the output position”] Shazeer et al. — discloses autoregressive decoding using previously generated tokens).
18/834,202 – Claim 22. Shazeer et al. 2018/0341860 further teaches The system of claim 21, wherein the attention mechanism in each attentive layer is a causal attention mechanism (Shazeer et al. 2018/0341860 [0066 - “masks out (sets to negative infinity) all values in the scaled output matrix that correspond to positions after the current output position”] Shazeer et al. — discloses masked (causal) self-attention in the decoder).
18/834,202 – Claim 24. Shazeer et al. 2018/0341860 further teaches The system of claim 19, wherein the network input is a sequence and wherein the attention mechanism in each attentive layer is non-causal (Shazeer et al. 2018/0341860 [0074 - contrasting encoder self-attention (“each position in the encoder can attend to all positions in the input order”) with masked decoder attention] Shazeer et al. — discloses non-causal (bidirectional) encoder self-attention).
Claims 8-9 are rejected under 35 U.S.C. 103 as being unpatentable over: Shazeer et al. 2018/0341860; in view of ‘NPL Google Shazeer’; in further view of ‘Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention’ Katharopoulos et al., arXiv:2006.16236v2. (hereinafter NPL Katharopoulos).
18/834,202 – Claim 8. Shazeer et al. 2018/0341860 further teaches The system of claim 5, wherein a matrix A that includes the respective sets of attention weights for the layer inputs satisfies: A=relu2(Qgdiag(γ)KgT+b), where relu.sup.2 is a squared ReLU element-wise activation function, Q is a matrix of the respective queries for the layer inputs, diag(γ) is a diagonal matrix that has a vector γ along the diagonal and zeroes at all other entries, K is a matrix of the respective keys for the layer inputs, and b is a bias (Shazeer et al. 2018/0341860 [0040 – ReLU activation function][0060 – weighted sum of values…][0062-0066 - matrix]).
Shazeer et al. 2018/0341860 may not expressly disclose the “squared ReLU attention-weight formula in place of softmax” features, however, NPL Katharopoulos teaches (NPL Katharopoulos [pg. 4, Eq. 7 (φ(x) = elu(x) + 1, replacing softmax); pg. 3, Eq. 3 (generalized similarity-function attention that softmax is one instance of]). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Shazeer et al. 2018/0341860 to include the features as taught by NPL Katharopoulos. One of ordinary skill in the art would have been motivated to do so to utilize well known tools and features useful for implementing ‘attention neural networks with gated attention units’. Obvious substitution of known equivalents, KSR, 550 U.S. at 416 - Routine substitution of one known, art-recognized attention-normalizing activation for another; NPL Katharopoulos teaches replacing softmax with an alternative (elu(x)+1) feature map for the same attention-weight role, evidencing that a POSITA had reason to substitute non-softmax activations here.
18/834,202 – Claim 9. Shazeer et al. 2018/0341860 further teaches The system claim 8, wherein b is a relative position bias (Shazeer et al. 2018/0341860 [0030 - combining an embedded representation with a positional embedding of the input’s position] offered as evidence the art already treated position-dependent terms as a known, routine addition to tention computations - Obvious over common knowledge in the art; MPEP § 2144.03).
No Prior-art Rejection / Potentially Allowable
Claims 12-18 and 23 cannot be rejected with prior-art. Individual claimed features are taught in the prior-art, however, the unique combination of features and elements are not taught by the prior-art without hindsight reasoning. These claims are further rejected as being dependent upon a rejected base claim but might possibly be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
18/834,202 – Claim 12. Shazeer et al. 2018/0341860 further teaches The system of claim 1, wherein the layer inputs are partitioned into a plurality of chunks, and wherein applying an attention mechanism over the input sequence to generate a respective attended layer input for each respective layer input comprises; for each chunk: generating a respective query and a respective key for each layer input in the chunk by processing the layer inputs using one or more third feed-forward neural network layers for the chunk; generating a respective value for each layer input in the chunk by processing the layer inputs using one or more fourth feed-forward neural network layers for the chunk; and applying a quadratic attention mechanism over the layer inputs in the chunk using the respective queries, keys, and values for the layer inputs in the chunk to generate a respective quadratic attended input for each layer input in the chunk; and applying a linear attention mechanism across the plurality of chunks to generate a respective linear attended input for each layer input; and for each layer input, combining the respective linear attended input for the layer input and the respective quadratic attended input for layer input to generate the attended input for the layer input ([]).
18/834,202 – Claim 13. Shazeer et al. 2018/0341860 further teaches The system of claim 12, wherein combining the respective linear attended input for the layer input and the respective quadratic attended input for layer input to generate the attended input for the layer input comprises adding the respective linear attended input for the layer input and the respective quadratic attended input for layer input to generate the attended input for the layer input ([]).
18/834,202 – Claim 14. Shazeer et al. 2018/0341860 further teaches The system of claim 12, wherein applying a quadratic attention mechanism over the layer inputs in the chunk using the respective queries, keys, and values for the layer inputs in the chunk to generate a respective quadratic attended input for each layer input in the chunk comprises: generating a respective set of attention weights for each layer input in the chunk from the respective query for the layer input and the respective keys for the layer inputs in the chunk; and for each layer input, applying the respective set of attention weights for the layer input to the respective values for the layer inputs to generate the respective quadratic attended input for the layer input, wherein a matrix A.sub.g that includes the respective sets of attention weights for the layer inputs in the chunk g satisfies: Ag=relu2(Qgdiag(γ)KgT+b), where relu.sup.2 is a squared ReLU element-wise activation function, Q.sub.g is a matrix of the respective queries for the layer inputs in the chunk g, diag(γ) is a diagonal matrix that has a vector γ along the diagonal and zeroes at all other entries, K.sub.g is a matrix of the respective keys for the layer inputs in the chunk g, and b is a bias ([]).
18/834,202 – Claim 15. Shazeer et al. 2018/0341860 further teaches The system of claim 12, wherein a matrix {circumflex over (V)}.sub.g.sup.lin of the linear attended inputs for the layer inputs in chunk g generated by applying the linear attention mechanism satisfies: V^glin=Qgdiag(λ)(Σh=1T/CKgTVh), where Q.sub.g is a matrix of the respective queries for the layer inputs in the chunk g, diag(λ) is a diagonal matrix that has a vector λ along the diagonal and zeroes at all other entries, T is a total number of layer inputs in the input sequence, C is the number of layer inputs in each chunk, K.sub.h is a matrix of the respective keys for the layer inputs in the chunk h, and V.sub.h is a matrix of the respective values for the layer inputs in the chunk h ([]).
18/834,202 – Claim 16. Shazeer et al. 2018/0341860 further teaches The system of claim 12, wherein the linear attention mechanism is a causal linear attention mechanism, and wherein a matrix {circumflex over (V)}.sub.g.sup.lin of the linear attended inputs for the layer inputs in chunk g generated by applying the causal linear attention mechanism satisfies: V^glin=Qgdiag(λ)(Σh=1g-1KgTVh), where Q.sub.g is a matrix of the respective queries for the layer inputs in the chunk g, diag(λ) is a diagonal matrix that has a vector λ along the diagonal and zeroes at all other entries, the sum is over the chunks that are before the chunk g in an ordering of the chunks for the causal linear attention mechanism, K.sub.h is a matrix of the respective keys for the layer inputs in the chunk h, and V.sub.h is a matrix of the respective values for the layer inputs in the chunk h ([]).
18/834,202 – Claim 17. Shazeer et al. 2018/0341860 further teaches The system of claim 12, wherein the one or more first feed-forward neural network layers include a respective set of one or more first feed-forward neural network layers for each chunk, and wherein generating a respective projected input for each layer input by processing the layer inputs using one or more first feed-forward neural network layers comprises: for each chunk, generating a respective projected input for each layer input in the chunk by processing the layer inputs in the chunk using the respective set of one or more first feed-forward neural network layers for the chunk ([]).
18/834,202 – Claim 18. Shazeer et al. 2018/0341860 further teaches The system of claim 17, the operations further comprising: generating a respective updated output for each layer input by processing the initial outputs using one or more second feed-forward neural network layers, wherein the one or more second feed-forward neural network layers include a respective set of one or more second feed-forward neural network layers for each chunk, and wherein generating a respective updated output for each layer input by processing the initial outputs using one or more second feed-forward neural network layers comprises: for each chunk, generating a respective updated output for each layer input in the chunk by processing the initial outputs for the layer inputs in the chunk using the respective set of one or more second feed-forward neural network layers for the chunk ([]).
18/834,202 – Claim 23. Shazeer et al. 2018/0341860 further teaches The system of claim 22, wherein the linear attention mechanism is a causal linear attention mechanism, and wherein a matrix {circumflex over (V)}.sub.g.sup.lin of the linear attended inputs for the layer inputs in chunk g generated by applying the causal linear attention mechanism satisfies: V^glin=Qgdiag(λ)(Σh=1g-1KgTVh), where Q.sub.g is a matrix of the respective queries for the layer inputs in the chunk g, diag(λ) is a diagonal matrix that has a vector λ along the diagonal and zeroes at all other entries, the sum is over the chunks that are before the chunk g in an ordering of the chunks for the causal linear attention mechanism, K.sub.h is a matrix of the respective keys for the layer inputs in the chunk h, and V.sub.h is a matrix of the respective values for the layer inputs in the chunk, wherein the quadratic attention mechanism within each chunk is causal (Shazeer et al. 2018/0341860 []).
Examiner’s Response to Arguments
Per Applicants’ amendments/arguments, the rejections are withdrawn.
Applicant's arguments have been considered but are moot in view of the new ground(s) of rejection.
Applicants’ amendments have necessitated the new grounds of rejection noted above.
Examiner’s Response: Claim Rejections – 35 USC §112
Per Applicants’ amendments/arguments, the rejections are withdrawn.
Applicant's arguments have been considered but are moot in view of the new ground(s) of rejection.
Applicants’ amendments have necessitated the new grounds of rejection noted above.
Examiner’s Response: Claim Rejections – 35 USC §101
Per Applicants’ amendments/arguments, the rejections are withdrawn. See notes above for additional reasoning and rationale for dropping 35 USC 101 rejection including Applicant’s amendments, arguments, lack of abstract idea, and practical integration.
Applicant's arguments have been considered but are moot in view of the new ground(s) of rejection.
Applicants’ amendments have necessitated the new grounds of rejection noted above.
Regarding Claims 1-15, on page(s) 6-12 of Applicant’s Remarks (dated 12/27/2016), Applicants traverse the 35 USC §101 rejections arguing the following:
Examiner’s Response: Claim Rejections – 35 USC § 103
Per Applicants’ amendments/arguments, the rejections are withdrawn. See notes above for additional reasoning and rationale for dropping prior-art rejection including Applicant’s amendments and arguments and unique combination of features and elements not taught by the prior-art without hindsight reasoning.
Applicant's arguments have been considered but are moot in view of the new ground(s) of rejection.
Applicants’ amendments have necessitated the new grounds of rejection noted above.
Regarding Claim X, on page(s) 8-9 of Applicant’s Remarks / After Final Amendments (dated 07/15/2011), Applicant(s) argues that the cited reference(s) (Ellis and Vandermolen) fails to teach, describe, or suggest the amended features. Specifically, Applicant(s) argues that cited reference(s) do not teach, describe, or suggest the following: . With respect, Applicant’s arguments are deemed unpersuasive and the amended feature(s) remain rejected as follows.
With respect, Applicant’s arguments are deemed unpersuasive and the amended feature(s) remain rejected as follows.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
PERTINENT PRIOR ART – Patent Literature
The prior-art made of record and considered pertinent to applicant's disclosure.
Socher, et al. US 2016/0350653 A1 (“Dynamic Memory Network,” Socher, Kumar, Irsoy, Iyyer, Xiong, Merity & Paulus, assigned to salesforce.com, inc.; filed June 1, 2016; published December 1, 2016). Socher, et al. US 2016/0350653 A1 discloses a “Dynamic Memory Network” (DMN) — an input module that converts sentences/words into vectors, and an episodic memory module that, in each of one or more “passes,” computes a scalar attention/gate value for each input fact as a function of that fact, the current memory state, and the question ([0067]: “a series of attention gate values are computed, one for each sentence in the input. An attention gate value represents how relevant that sentence is for the question”). Critically, and unlike claim 1, that gate value in Socher is always a single scalar number per fact (or, in the “Attention Based GRU” variant, a scalar per dimension via a sigmoid/softmax vector but still computed from a small set of similarity features of the fact — not a second, independently-computed full vector “attended” representation). Socher uses that scalar gate in one of two ways: (1) as a softmax weight in a weighted sum of the fact vectors ([0071], Equation 16: eⁱ = Σₜ softmax(gⁱₜ)·sₜ) — summation, not an element-wise product; or (2) to interpolate, element-by-element, between two GRU hidden-state vectors in an “Attention Based GRU” cell ([0099]–[0101], Equation 31: hᵢ = gᵢᵗ∘h̃ᵢ + (1−gᵢᵗ)∘hᵢ₋₁) — a standard GRU-style convex-combination gate, not a product of a feed-forward-projected vector and an attention-derived vector. Socher’s disclosed element-wise (“∘”) product operator itself appears in exactly two contexts, neither of which matches claim 1: (a) as raw similarity features concatenated into the gate’s input vector — e.g., [0069]/Equation 14: z(s,m,q) = [s∘q, s∘m, |s−q|, |s−m|, s, m, sᵀW⁽ᵇ⁾q, sᵀW⁽ᵇ⁾m] — where “s∘q” and “s∘m” are just two of eight concatenated features feeding a small feed-forward network that outputs the single scalar gate value g = G(s,m,q) ([0070], Equation 15); and (b) as the standard internal gating math of a GRU/LSTM cell ([0048]–[0052], Equations 1–12, e.g., hₜ = zₜ∘hₜ₋₁ + (1−zₜ)∘h̃ₜ), which is generic recurrent-network mechanics, not an attentive-layer output-gating step.
Shazeer et al. US 2020/0089755 A1 [0056] In some implementations the multi task multi modal machine learning model 100 may further include an input output mixer neural network. The input output mixer neural network may be configured to process encoded inputs, e.g., received from the encoder neural network 104, and decoder outputs, e.g., received from the decoder neural network 106. The input output mixer neural network may further be configured to generate encoded outputs which may be received and processed by the decoder neural network 106. The input output mixer neural network may include one or more attention neural network layers configured to perform respective attention mechanisms, and one or more convolutional neural network layers. An example input output mixer neural network is illustrated and described in more detail below with reference to FIG. 4.
Shazeer et al. US 2019/0130213 A1 [0006] Attention-based neural networks have been shown to perform well on sequence processing tasks, e.g., tasks that involve processing a sequential input, autoregressively generating a sequential output, or both. However, for each position in a given output or input sequence, self-attention layers in an attention-based neural network attend over (at least) all of the preceding positions in the sequence. This makes attention-based neural networks difficult to apply to tasks that require generating an output image. While an image can be represented as a sequence of color values, images are composed of a large number of color values. For example, a 32×32×3 (RGB) image has over 3,000 color values, resulting in a very long sequence representation. Additionally, images have a two-dimensional (and, for color images, three-dimensional) structure that is difficult to represent as a one-dimensional sequence.
PERTINENT PRIOR ART – Non-Patent Literature (NPL)
The NPL prior-art made of record and considered pertinent to applicant's disclosure.
Zaheer (Big Bird) = Zaheer, Guruganesh, Dubey, Ainslie, et al., “Big Bird: Transformers for Longer Sequences,” NeurIPS 2020 / arXiv:2007.14062 (July 28, 2020) — discloses a sparse attention mechanism in which each position attends only to a restricted subset of positions (a local window, a fixed set of global positions, and/or random positions), constraining the full attention matrix to a sparse pattern, and expressly reports this as reducing attention’s computational and memory cost while preserving quality on long sequences.
NPL Google Shazeer = Noam Shazeer, “GLU Variants Improve Transformer,” arXiv:2002.05202 (Feb. 12, 2020) — a separate, non-patent-literature reference by the same first author as the patent reference above; labeled “NPL Google Shazeer” throughout this Office Action to distinguish it from the Shazeer et al. patent publication. Discloses replacing a Transformer feed-forward sub-layer with a Gated Linear Unit that computes an element-wise product between two different learned projections of the same input (one passed through a gating nonlinearity), and expressly reports that this gating improves Transformer quality.
Katharopoulos = Katharopoulos, Vyas, Pappas & Fleuret, “Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention,” ICML 2020 / arXiv:2006.16236 (June 29, 2020) — discloses reformulating the attention computation as a linear-complexity operation via kernel feature maps and running (cumulative) sums, including a causal/autoregressive variant using a causal cumulative sum, and expressly reports the resulting linear-time complexity as a benefit over ordinary (quadratic) attention.
THIS ACTION IS MADE FINAL
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
THIS ACTION IS MADE FINAL
Applicant’s amendment necessitated new grounds of rejection and FINAL Rejection.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW T. SITTNER whose telephone number is (571) 270-7137 and email: matthew.sittner@uspto.gov. The examiner can normally be reached on Monday-Friday, 8:00am - 5:00pm (Mountain Time Zone). Please schedule interview requests via email: matthew.sittner@uspto.gov
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sarah M. Monfeldt can be reached on (571) 270-1833.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW T SITTNER/
Primary Examiner, Art Unit 3629b
Prosecution Insights