DETAILED ACTION
This action is responsive to the claims filed on 12/09/2024. Claims 1-18 and 25-26 are pending for examination.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 04/28/2025 and 02/25/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 10-11 and 25-26 are rejected under 35 U.S.C. 103 as being unpatentable over Shazeer et al. (US 2018/0341860 A1), hereafter referred to as Shazeer, in view of Press et al., (Press, O., Smith, N. A., & Lewis, M. (2021). Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409.), hereafter referred to as Press.
Claim 1: Shazeer teaches the following
• A method performed by one or more computers, the method comprising: receiving a network input; processing the network input using a neural network that comprises a plurality of neural network layers arranged as a directed graph to generate a network output for the network input, (Shazeer, ¶¶ 21-25, 28, and 37, “The neural network system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations”; “The neural network system 100 receives an input sequence 102 and processes the input sequence 102 to transduce the input sequence 102 into an output sequence 152”; “The encoder neural network 110 includes an embedding layer 120 and a sequence of one or more encoder subnetworks 130”; and “for each encoder subnetwork other than the first encoder subnetwork in the sequence, the encoder subnetwork input is the encoder subnetwork output of the preceding encoder subnetwork in the sequence,” the claimed neural-network layers arranged as a directed graph are interpreted as the embedding layer 120 and ordered encoder subnetworks 130 forming the graph nodes, with the output of each preceding layer being supplied in one direction as the input to the next layer, thereby forming directed graph edges from the network input to the generated network output.)
• processing the set of input embeddings, in accordance with values of a set of regularized attention layer parameters, to generate: (i) a set of value embeddings, comprising a respective value embedding for each input embedding, and (ii) a set of intermediate attention scores; (Shazeer, ¶¶ 62-65, 69-70, and 74, teaches that “the attention sub-layer packs the queries into a matrix Q, packs the keys into a matrix K, and packs the values into a matrix V”; “performs a matrix multiply (MatMul) between the matrix Q and the transpose of the matrix K to generate a matrix of compatibility function outputs”; “Each attention layer is configured to transform the original queries, and keys, and values using learned linear transformations”; and “there is a respective key, value, and query for each position in the input order.” Shazeer’s learned query, key, and value linear transformations are the underlying attention-layer parameters; the transformed values forming matrix V are the set of value embeddings, including a respective value for each input position; and the individual entries of the QK^T compatibility matrix, prior to softmax, are the intermediate attention scores. Shazeer is relied upon for these underlying parameterized attention operations, rather than for teaching that its unmodified self-attention sub-layer is itself regularized.)
• and providing a layer output for the attention layer based on the set of output embeddings. (Shazeer, ¶ 71, "The attention sub-layer then combines the initial outputs of the attention layers to generate the final output of the attention sub-layer... [and] applies a learned linear transformation to the concatenated output to generate the output of the attention sub-layer", Shazeer expressly provides the attention layer's output based on the generated attention outputs)
Press, in the same field of transformer attention, teaches the following which Shazeer fails to expressly teach:
wherein the plurality of neural network layers comprise one or more regularized attention layers, and wherein processing the network input comprises, for each regularized attention layer: receiving a layer input to the regularized attention layer, wherein the layer input to the regularized attention layer comprises a set of input embeddings; (Press, p. 3, § 2.1, “The model has 16 transformer layers of dimension 1024, with 8 heads”; p. 4, § 2.2, “The output of a self-attention sublayer is a linearly transformed, weighted sum of the input value vectors”; and p. 5, Fig. 3 and § 3, “our linearly biased attention method, ALiBi, adds a constant bias… to each attention score,” “m is a head-specific scalar that is set and not learned throughout training,” and “we add position information at every layer to the keys and queries but not to the values,” Press applies its fixed, non-learned ALiBi bias to the attention operation at the transformer layers. The claimed regularized attention layer is therefore interpreted as Press’s ALiBi-modified attention layer, rather than as an unmodified self-attention layer merely bearing a different name. Each such layer receives the input representation vectors used to produce its queries, keys, and input value vectors, which collectively constitute the claimed set of input embeddings under the broadest reasonable interpretation. Because Press applies the ALiBi modification at every layer, each layer receiving those input vectors performs the claimed regularized attention processing.)
and applying a regularized attention operation over the set of input embeddings to generate a set of output embeddings, comprising: (Press, pp. 4-5, §§ 2.2-3, “The output of a self-attention sublayer is a linearly transformed, weighted sum of the input value vectors”; “the attention sublayer computes the attention scores for the ith query qi… given the first i keys K”; “These attention scores are then multiplied by the values to return the output of the attention sublayer”; and “The only modification we apply is after the query-key dot product, where we add a static, non-learned bias,” Press’s regularized attention operation is specifically the attention operation after it has been modified by the static, non-learned ALiBi bias. The input representation vectors from which the queries, keys, and input value vectors are obtained are the claimed input embeddings, and the respective weighted-sum vectors returned by the attention sublayer are the claimed output embeddings. Thus, it is Press’s fixed-bias modification of the attention operation—not an ordinary self-attention operation standing alone—that teaches the claimed regularized attention operation.)
transforming the intermediate attention scores using a set of shaping constants to generate a set of transformed attention scores, wherein: values of the shaping constants are initialized prior to training of the neural network and are not adjusted during the training of the neural network; and the values of the shaping constants are selected to regularize the set of output embeddings; (Press, p. 5, Fig. 3 and § 3, “The only modification we apply is after the query-key dot product, where we add a static, non-learned bias: softmax(qiK^T + m · [−(i − 1), …, −2, −1, 0]), where scalar m is a head-specific slope fixed before training”; and “ALiBi has an inductive bias towards recency; it penalizes attention scores between distant query-key pairs, with the penalty increasing as the distance between a key and a query grows,” the entries of qiK^T are the intermediate attention scores; the corresponding entries of the fixed vector m · [−(i − 1), …, −2, −1, 0] are the shaping constants; and the sums inside the softmax are the transformed attention scores. Because m is fixed before training and remains non-learned, the shaping constants are not adjusted during training. The constants regularize the output embeddings by imposing a predetermined recency bias on the attention weights that are multiplied by the values to form the output embeddings.)
and generating the set of output embeddings using: (i) the set of value embeddings, and (ii) the set of transformed attention scores; (Press, p. 5, Fig. 3 and § 3, “softmax(qiK^T + m · [−(i − 1), …, −2, −1, 0])”; “These attention scores are then multiplied by the values to return the output of the attention sublayer”; and “the rest of the computation is unmodified,” Press applies softmax to the transformed, bias-containing attention scores and multiplies the resulting attention weights by the value embeddings to generate the output embeddings of the ALiBi-modified attention layer.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated Press's fixed linear attention biases into the attention layers of Shazeer. Press applies its modification at the same query-key score stage used by Shazeer, expressly states that the remainder of the attention computation is unmodified, and implements the biases through the existing mask before softmax. A person of ordinary skill would have been motivated to make this predictable substitution to improve length extrapolation and efficiency while retaining Shazeer's established query, key, value, softmax, and weighted-sum architecture.
Claim 10: Shazeer and Press teach the limitations of claim 1, and Press further teaches the following
• The method of claim 1, wherein transforming the intermediate attention scores using the set of shaping constants to generate the set of transformed attention scores comprises, for each intermediate attention score: generating a corresponding transformed attention score by combining the intermediate attention score with a corresponding shaping constant. (Press, p. 5, Fig. 3 and § 3, "ALiBi, adds a constant bias... to each attention score"; p. 5, § 3, "The only modification we apply is after the query-key dot product, where we add a static, non-learned bias", each query-key score is combined with its corresponding fixed distance-and-slope bias to produce the score supplied to softmax)
Claim 11: Shazeer and Press teach the limitations of claim 10, and Press further teaches the following
• The method of claim 10, wherein for each intermediate attention score, generating the corresponding transformed attention score comprises: summing the intermediate attention score with the corresponding shaping constant. (Press, p. 5, Fig. 3 and § 3, “The only modification we apply is after the query-key dot product, where we add a static, non-learned bias: softmax(qiK^T + m · [−(i − 1), …, −2, −1, 0]),” in Press’s expression, each element of qiK^T is the claimed intermediate attention score, each corresponding element of m · [−(i − 1), …, −2, −1, 0] is the claimed corresponding shaping constant, the “+” operator expressly sums those corresponding elements, and the resulting element inside the softmax is the corresponding transformed attention score.)
Claim 25: Claim 25 recites limitations substantially similar to claim 1, as such a similar analysis applies, Shazeer further teaches the following additional limitation for consideration:
• A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: (Shazeer, ¶ 21, "The neural network system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations"; ¶¶ 88 and 94, "Embodiments... can be implemented as one or more computer programs... encoded on a tangible non-transitory storage medium for execution by... data processing apparatus... The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data", Shazeer teaches a system of one or more computers and storage devices containing executable instructions that cause the computers to perform the neural-network operations)
Claim 26: Claim 26 recites limitations substantially similar to claim 1, as such a similar analysis applies, Shazeer further teaches the following additional limitation for consideration:
• One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: (Shazeer, ¶¶ 88 and 95, "Embodiments... can be implemented as one or more computer programs... encoded on a tangible non-transitory storage medium for execution by... data processing apparatus... Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices", Shazeer expressly teaches non-transitory computer storage media encoded with instructions for execution by one or more computers)
Claims 2-3 are rejected under 35 U.S.C. 103 as being unpatentable over Shazeer in view of Press, as applied to claim 1 and 10-11 above, and in further view of Noci et al., (Noci, L., Anagnostidis, S., Biggio, L., Orvieto, A., Singh, S. P., & Lucchi, A. (2022). Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 35, 27198-27211.), hereafter referred to as Noci.
Claim 2: Shazeer and Press teach the limitations of claim 1, and Noci, in the same field of representation learning, further teaches the following which Shazeer and Press fails to teach:
• The method of claim 1, wherein the values of the shaping constants are selected to maintain or increase a rank of the set of output embeddings. (Noci, pp. 1-2, "stacking self-attention layers... can result in rank collapse of the tokens' representations at initialization... [and] an appropriate depth-dependent scaling of the residual branches... approximately preserves the cosine of the angle between tokens, hence stabilizing the rank of the propagating sequence"; p. 7, "the 1/sqrt(L) scaling... [provides] new insights on its role in the context of rank preservation", Noci teaches selecting fixed scaling values at initialization specifically to preserve the rank of propagating token/output representations and avoid rank collapse)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to select Press's fixed shaping slopes in view of Noci's rank-preservation teaching. Press already reports manually selecting robust fixed slope sets to impose a desired inductive bias, while Noci explains that rank collapse in transformer representations causes vanishing query/key gradients and that appropriately selected fixed scaling at initialization preserves token diversity and rank. Applying that known design objective when selecting the constants in the combination of Shazeer and Press would predictably maintain or increase the rank of the resulting output embeddings and improve trainability.
Claim 3: Shazeer, Press, and Noci teach the limitations of claim 2, and Noci, in the same field of representation learning, further teaches the following which Shazeer and Press fails to teach:
• The method of claim 2, wherein the values of the shaping constants are selected to increase a likelihood that the rank of the set of output embeddings exceeds a threshold. (Noci, pp. 6-7, Theorem 3.3, “without the depth-dependent residual scaling… the correlation between different tokens… converges to 1, implying rank collapse”; “ρℓ = 1 if and only if every pair of tokens is perfectly aligned”; and, under the disclosed 1/√L scaling, “the correlation term is one if and only if… all the input tokens are perfectly aligned,” Noci identifies the rank-collapse threshold as rank one: when ρℓ equals one, every token representation is perfectly aligned and the representation matrix has rank one. For non-degenerate inputs, Noci’s selected 1/√L scaling instead maintains ρℓ below one, such that the token representations are not all perfectly aligned and the representation rank exceeds the threshold rank of one. Because Noci analyzes this behavior for a randomly initialized transformer, selecting the disclosed scaling values increases the likelihood that the output-representation rank remains greater than one rather than collapsing to the threshold rank.)
Claims 4-7 and 12-16 are rejected under 35 U.S.C. 103 as being unpatentable over Shazeer in view of Press, as applied above, and in further view of Dao et al., (FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022)), hereafter referred to as Dao.
Claim 4: Shazeer and Press teach the limitations of claim 1, and Dao, in the same field of representation learning, further teaches the following which Shazeer and Press fails to teach:
• The method of claim 1, wherein the values of the set of shaping constants are derived from a shaping matrix by operations comprising: determining a decomposition of the shaping matrix into a product of: (i) a diagonal matrix, and (ii) a partition matrix, wherein the partition matrix has row sums equal to one; (Dao, pp. 3-4, §§ 2.2-3.1, "P = softmax(S)... where softmax is applied row-wise... f(x) := [e^(x_1-m(x)) ... e^(x_B-m(x))], l(x) := sum_i f(x)_i, softmax(x) := f(x)/l(x)"; p. 5, Algorithm 1, lines 10-12, "l-tilde_ij = rowsum(P-tilde_ij)... O_i <- diag(l_i^new)^-1 (... + ... P-tilde_ij V_j)", Dao decomposes the positive exponentiated score matrix into a diagonal row-sum matrix and a row-normalized softmax matrix P. Equivalently, the unnormalized shaping matrix equals diag(l)P, and every row of P sums to one)
• and applying a logarithm to the partition matrix, the shaping matrix being based on the result of applying the logarithm to the partition matrix. (Dao, p. 4, § 3.1, “f(x) := [e^(x1−m(x)) … e^(xB−m(x))], l(x) := Σi f(x)i, softmax(x) := f(x)/l(x),” Dao’s equations define each entry of the row-normalized partition matrix P as Pj = exp(xj − m(x))/l(x). Applying the natural logarithm entrywise gives ln(Pj) = xj − m(x) − ln(l(x)), or equivalently xj = ln(Pj) + m(x) + ln(l(x)). Thus, the pre-softmax score or shaping matrix x is based on the result of applying the logarithm to the row-stochastic partition matrix P, with only a row-specific additive constant. Because adding the same constant to every entry of a row does not alter its softmax distribution, a person of ordinary skill starting from a desired partition matrix P would have found it obvious to apply the entrywise logarithm to P to obtain an algebraically equivalent shaping-score matrix that reproduces P after softmax.)
It would have been obvious to incorporate Dao's explicit row-normalization and diagonal-scaling formulation into the combined attention system of Shazeer and Press. Dao teaches an exact, algebraically equivalent implementation of softmax attention that reduces memory access while preserving O = softmax(QK^T)V. A person of ordinary skill would have used Dao's diagonal row-sum matrix and row-stochastic partition representation to construct and apply Press's fixed shaping matrix because this produces the same intended biased attention probabilities while enabling an efficient exact implementation.
Claim 5: Shazeer, Press, and Dao teach the limitations of claim 4, and Press further teaches the following
• The method of claim 4, wherein the shaping matrix is derived from at least one base matrix, wherein values of off-diagonal entries of the base matrix decay exponentially based on a distance from a diagonal of the base matrix. (Press, p. 5, Fig. 3 and § 3, "softmax(q_i K^T + m · [-(i - 1), ..., -2, -1, 0])"; p. 5, § 3, "it penalizes attention scores between distant query-key pairs, with the penalty increasing as the distance between a key and a query grows", Press's additive entries are -m times distance from the diagonal. Softmax exponentiation therefore forms underlying positive base entries exp(-m·distance), which decay exponentially as distance from the diagonal increases)
Claim 6: Shazeer, Press, and Dao teach the limitations of claim 5, and Press further teaches the following
• The method of claim 5, wherein each on-diagonal entry of the base matrix has a same value. (Press, p. 5, Fig. 3, "[the bias matrix has rows ending in] ... 3 2 1 0; ... 4 3 2 1 0", each diagonal position has distance zero and thus the same additive value zero; in the exponentiated base matrix each corresponding diagonal entry consequently has the same value exp(0))
Claim 7: Shazeer, Press, and Dao teach the limitations of claim 6, and Press further teaches the following
• The method of claim 6, wherein each of the on-diagonal entries of the base matrix have value one. (Press, p. 5, Fig. 3 and § 3, “softmax(qiK^T + m · [−(i − 1), …, −2, −1, 0]),” and Fig. 3 showing a value of zero at every diagonal position of the ALiBi bias matrix, Press’s fixed additive bias corresponds after softmax exponentiation to the positive base matrix Bij = exp(−m · distance(i,j)). At each on-diagonal position, distance(i,i) equals zero and the corresponding bias is m · 0 = 0. Therefore, every on-diagonal entry of the resulting base matrix is Bii = exp(0) = 1.)
Claim 12: Shazeer, Press, and Dao teach the limitations of claim 4, and Shazeer further teaches the following
• The method of claim 4, wherein processing the set of input embeddings to generate the set of intermediate attention scores comprises: processing the set of input embeddings to generate: (i) a respective query embedding, and (ii) a respective key embedding, for each input embedding; and generating each intermediate attention score based on a measure of similarity between a corresponding query embedding and a corresponding key embedding. (Shazeer, ¶¶ 60-63 and 69-70, "each attention layer is configured to apply a learned query linear transformation to each original query to generate a layer-specific query... [and] a learned key linear transformation to each original key to generate a layer-specific key... [and] performs a matrix multiply (MatMul) between the matrix Q and the transpose of the matrix K to generate a matrix of compatibility function outputs", Shazeer generates a respective query and key representation for each position and uses their dot product, a recognized similarity measure, as the corresponding intermediate attention score)
Claim 13: Shazeer, Press, and Dao teach the limitations of claim 4, and Shazeer further teaches the following
• The method of claim 4, wherein generating the set of output embeddings using: (i) the set of value embeddings, and (ii) the set of transformed attention scores, comprises: generating a set of final attention scores by applying a causal masking operation followed by a non-linear transformation to the set of transformed attention scores; (Shazeer, ¶¶ 65-66, "The attention sub-layer then applies a softmax over the scaled output matrix to generate a matrix of weights... For sub-layers that use masking... the attention sub-layer masks the scaled output matrix before applying the softmax... [and] masks out (sets to negative infinity) all values... that correspond to positions after the current output position", Shazeer's decoder attention performs causal masking and then applies the nonlinear softmax transformation to generate final attention scores)
• and generating the set of output embeddings using: (i) the set of value embeddings, and (ii) the set of transformed attention scores. (Shazeer, ¶ 65, "[the attention sub-layer] performs a matrix multiply (MatMul) between the weight matrix and the matrix V to generate an output matrix that includes the output of the attention mechanism for each of the values", Shazeer combines the score-derived weight matrix with the value embeddings to generate the output embeddings)
Claim 14: Shazeer, Press, and Dao teach the limitations of claim 13, and Shazeer further teaches the following
• The method of claim 13, wherein the non-linear transformation is a soft-max transformation. (Shazeer, ¶¶ 61 and 65-66, "[the attention sub-layer] applies a softmax function over the scaled dot products to obtain the weights on the values... [and] masks the scaled output matrix before applying the softmax", the nonlinear transformation following the mask is expressly a softmax transformation)
Claim 15: Shazeer, Press, and Dao teach the limitations of claim 13, and Shazeer further teaches the following
• The method of claim 13, wherein generating the set of output embeddings using: (i) the set of value embeddings, and (ii) the set of transformed attention scores, comprises: generating each output embedding based on a linear combination of the set of value embeddings, wherein coefficients of the linear combination are defined by respective transformed attention scores from the set of transformed attention scores. (Shazeer, ¶¶ 60-65, "The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key... [the sub-layer] performs a matrix multiply (MatMul) between the weight matrix and the matrix V", a weighted sum is a linear combination of value embeddings, and the attention-score-derived weights are the coefficients of that linear combination)
Claim 16: Shazeer, Press, and Dao teach the limitations of claim 15, and Dao further teaches the following
• The method of claim 15, wherein generating the set of output embeddings comprises: applying an embedding-specific rescaling to the set output embeddings based on the diagonal matrix. (Dao, p. 5, Algorithm 1, lines 10-12, "l-tilde_ij = rowsum(P-tilde_ij)... Write O_i <- diag(l_i^new)^-1 (diag(l_i)... O_i + ... P-tilde_ij V_j)"; p. 4, § 3.1, "By scaling the output of each block by the right normalization factor before adding them up, we get the correct result at the end", Dao applies a separate row/embedding normalization factor through the inverse diagonal matrix diag(l)^-1 when generating each block of output embeddings)
Claims 8-9 are rejected under 35 U.S.C. 103 as being unpatentable over Shazeer in view of Press and Dao, as applied to claim 4 above, and in further view of Zhao et al., (Zero Initialization: Initializing Residual Networks with Only Zeros and Ones (2021)), hereafter referred to as Zhao.
Claim 8: Shazeer, Press, and Dao teach the limitations of claim 4, and Zhao further teaches the following which Shazeer, Press, and Dao fails to teach:
• The method of claim 4, wherein the shaping matrix is derived from at least one base matrix, wherein diagonal entries of the base matrix each have a same first value and off-diagonal entries of the base matrix each have a same second value. (Zhao, p. 6, Eq. 1, “K-hat(i,i) = 1 for 0 ≤ i ≤ cmin − 1” and “K-hat(i,j) = 0 for 0 ≤ i ≤ cin − 1, 0 ≤ j ≤ cout − 1, and j ≠ i”; p. 6, § 3, “we initialize the largest square matrix in K-hat as identity matrix and initialize the rest components to zero,” the claimed base matrix is interpreted as Zhao’s disclosed identity or partial-identity matrix K-hat. Equation 1 assigns the same first value of one to every on-diagonal entry K-hat(i,i) and assigns the same second value of zero to every off-diagonal entry K-hat(i,j), where j ≠ i. Zhao therefore expressly discloses a base matrix whose diagonal entries share a common first value and whose off-diagonal entries share a common second value.)
It would have been obvious to use Zhao's fixed identity-type matrix as a base matrix in the shaping-matrix construction of Shazeer, Press, and Dao. Zhao teaches that deterministic zero-and-one initialization improves reproducibility and that identity paths preserve signal propagation. An identity base is also full rank and supplies a predictable self-retaining attention prior. A person of ordinary skill therefore would have used that known matrix as a simple, stable starting point for the fixed shaping matrix before row normalization and logarithmic conversion.
Claim 9: Shazeer, Press, Dao, and Zhao teach the limitations of claim 8, and Zhao further teaches the following
• The method of claim 8, wherein the first value is one and the second value is strictly less than one. (Zhao, p. 6, Eq. 1, "K-hat_i,i = 1... [and] K-hat_i,j = 0... and j != i", the first, diagonal value is one and the second, off-diagonal value is zero, which is strictly less than one)
Claims 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Shazeer in view of Press, as applied to claim 1 above, and in further view of Zhao.
Claim 17: Shazeer and Press teach the limitations of claim 1, and Zhao further teaches the following which Shazeer and Press fail to teach:
• The method of claim 1, wherein prior to training of the neural network, the values of the regularized attention layer parameters are initialized to cause a value of each of the intermediate attention scores to be zero. (Zhao, pp. 1-2, "ZerO... initializes residual networks with only zeros and ones... [and] allows us to start the training from zeros and ones entirely"; p. 3, § 2.1, "if we initialize the weights to zero, every layer will output zero constantly no matter what the inputs are", applying Zhao's zero initialization to at least one of Shazeer's query or key projection matrices before training makes each query-key dot product, and therefore each intermediate attention score, zero)
It would have been obvious to initialize the query/key parameters of the Shazeer-Press attention layer using Zhao's deterministic zero initialization. Zhao teaches improved reproducibility, reduced run-to-run variance, and successful signal propagation when zero-initialized residual branches are paired with identity/skip paths. Shazeer's transformer already includes residual connections. A person of ordinary skill therefore would have expected zero-initializing a query or key projection to provide a controlled initial attention-score state while residual paths and Press's fixed bias continue to propagate differentiated signals.
Claim 18: Shazeer and Press teach the limitations of claim 1, and Zhao further teaches the following
• The method of claim 1, wherein prior to training of the neural network, the values of the regularized attention layer parameters are initialized to encourage a value of each of the intermediate attention scores to be near zero. (Zhao, p. 1, § 1, "initializing the residual branches... at (or close to) zero benefits signal propagation and optimization"; p. 5, § 2.4, "most of our weights are close to zero", initializing the query/key parameters at or close to zero correspondingly encourages their dot-product intermediate attention scores to be near zero before training)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Dong, Y., Cordonnier, J. B., & Loukas, A. (2021, July). Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International conference on machine learning (pp. 2793-2803). PMLR.
Raganato, A., Scherrer, Y., & Tiedemann, J. (2020, November). Fixed encoder self-attention patterns in transformer-based machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 556-568).
Sander, M. E., Ablin, P., Blondel, M., & Peyré, G. (2022, May). Sinkformers: Transformers with doubly stochastic attention. In International Conference on Artificial Intelligence and Statistics (pp. 3515-3530). PMLR.
Bachlechner, T., Majumder, B. P., Mao, H., Cottrell, G., & McAuley, J. (2021, December). Rezero is all you need: Fast convergence at large depth. In Uncertainty in artificial intelligence (pp. 1352-1361). PMLR.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HYUNGJUN B YI whose telephone number is (703)756-4799. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached at (571) 272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/H.B.Y./Examiner, Art Unit 2146
/USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146
Prosecution Insights