DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is in response to the claims filed on 10/23/2023. Claims 1-20 are currently pending and have been examined.
Specification
Applicant is reminded of the proper content of an abstract of the disclosure.
A patent abstract is a concise statement of the technical disclosure of the patent and should include that which is new in the art to which the invention pertains. The abstract should not refer to purported merits or speculative applications of the invention and should not compare the invention with the prior art.
If the patent is of a basic nature, the entire technical disclosure may be new in the art, and the abstract should be directed to the entire disclosure. If the patent is in the nature of an improvement in an old apparatus, process, product, or composition, the abstract should include the technical disclosure of the improvement. The abstract should also mention by way of example any preferred modifications or alternatives.
Where applicable, the abstract should include the following: (1) if a machine or apparatus, its organization and operation; (2) if an article, its method of making; (3) if a chemical compound, its identity and use; (4) if a mixture, its ingredients; (5) if a process, the steps.
Extensive mechanical and design details of an apparatus should not be included in the abstract. The abstract should be in narrative form and generally limited to a single paragraph within the range of 50 to 150 words in length.
As per MPEP § 608.01(b),
The content of a patent abstract should be such as to enable the reader thereof, regardless of his or her degree of familiarity with patent documents, to determine quickly from a cursory inspection of the abstract the nature and gist of the technical disclosure and that which is new in the art to which the invention pertains.
The abstract fails to adequately summarize the disclosure. The abstract merely mentions a set of learnable multi-head binder vectors used to determine attention weights over the input sequence. It fails to recite some features that are present in the specification, including the use of encoder and decoder, the key/value vector processing, SoftMax operation, concatenation of the first and second attention output. Accordingly, the abstract should be amended to more accurately reflect the invention.
See MPEP § 608.01(b) for guidelines for the preparation of patent abstracts.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1, 11 and 20 recite the limitation “generating, a decoder of the neural network based NLP model, a task output based on the one or more text representations in response to the text input”. It does not distinctly define the meets and bounds of what is being generated here because it is unclear if the generating step is generating a decoder of the neural network or generating a task output. For purpose of the examination this limitation is interpreted as generating a task output based on one or more text representations in response to the text input using a decoder.
Claim 2-10 and 12-19 are rejected as being dependent on rejected base claim 1 and 11 without curing any of the deficiencies.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1,2,6, 11,12, 16 and 20 are rejected under 35 U.S.C. 102 (a)(1) as being anticipated by Wang et al., “Improved Transformer With Multi-Head Dense Collaboration”
Regarding Claim 1, Wang discloses,
A method for performing a natural language processing (NLP) task at a neural network-based NLP model, comprising; (Wang, pg.2755, section I, “We conduct extensive experiments on various NLP tasks, such as machine translation, sentence classification, etc. CollFormer, i.e., Transformer equipped with our proposed SLC and CLCs, achieves consistent improvement over vanilla Transformer and even many strong baselines”)
receiving, via a communication interface (Wang, pg.2765, section V.C.6, “We perform this experiment on Transformer (base) model for the En-De translation task and run it on a V100 GPU” implying the use of computer which has at least one form of communication interface to receive data)
a text input of a sequence of tokens; (Wang, pg. 2755, section II. A, “For example, it is demonstrated that the self-attention mechanism can be treated as a graph-structure inductive bias, and all the tokens in an input sequence are connected via a relevance-based pooling operation [27].”)
encoding, by an encoder of the neural network-based NLP model, the text input into one or more text representations, wherein the encoding comprises: (Wang, pg. 2759, section IV.C, “For this architecture, both the encoder and the decoder contain multiple MHA layers. We name the attention layer between the encoder and decoder as the cross-attention” stating the presence of encoder and figure 2, showing the input into representation R.
[AltContent: textbox (Input)][AltContent: ][AltContent: ]
PNG
media_image1.png
631
718
media_image1.png
Greyscale
[AltContent: textbox (Representation)]
computing, by a first attention head, a first attention output based at least in part on a first tunable vector, and an encoder layer input (Wang, pg. 2756, section III. A “Here, Wq, Wk, and Wv are trainable parameters of the linear projection” which are the tunable vectors and pg.2757, section III.B “Vanilla MHA projects the input sequence into several different subspaces with separate projection matrices” showing the input and “In order to make different attention heads interact with each other in Transformer, a nonlinear Feed-Forward Network (FFN) is applied to the output of MHA.” showing the output. )
computing, by a second attention head, a second attention output based on second tunable vector and the encoder layer input; (Wang, pg. 2757, section III. B “Vanilla MHA projects the input sequence into several different subspaces with separate projection matrices, so that multiple scaled dot-product attentions are computed in parallel.” multiple attention being computed in parallel supports at least first and second heads and “Formally, MHA is computed as: MultiHead(Xq,Xkv)=[Z1;...;Zh]Wo, where h is the total number of heads, [·;...;·] indicates the concatenation operation, Zi is the attention corresponding to the i-th head, and Wo is a matrix of trainable parameters.” showing the tunable vector and “In order to make different attention heads interact with each other in Transformer, a nonlinear Feed-Forward Network (FFN) is applied to the output of MHA.” showing the output.)
and concatenating the first attention output and the second attention output as a context vector for the encoding;(Wang, pg.2757, section III. B, “and “Formally, MHA is computed as: MultiHead(Xq,Xkv)=[Z1;...;Zh]Wo, where h is the total number of heads, [·;...;·] indicates the concatenation operation, Zi is the attention corresponding to the i-th head, and Wo is a matrix of trainable parameters” concatenates outputs of multiple attention heads. )
and generating, a decoder of the neural network based NLP model, a task output based on the one or more text representations in response to the text input (Wang, pg.2759, section IV, figure 2,
[AltContent: textbox (Output based on text representation and the text input )][AltContent: rect]
PNG
media_image1.png
631
718
media_image1.png
Greyscale
And pg.2759, section IV. C, “For this architecture, both the encoder and the decoder contain multiple MHA layers. We name the attention layer between the encoder and decoder as the cross-attention. It helps the decoder focus on relevant parts of the input sentence.”)
Regarding Claim 2, Wang teaches,
The method of claim 1 (and thus the rejection of claim 1 is incorporated) , wherein the first attention output is computed based on further computing a first key vector and a first value vector corresponding to a first feature vector corresponding to the encoder layer input, (Wang, pg. 2756, section III. A, “We denote the feature matrix of the n-token query sequence as Xq ∈ Rn× d, and the m-token key-value sequence as Xkv ∈ Rm× d. Here, d is the dimension of features. Then, the dot-product attention is computed as:
Z =Att(Xq,Xkv)= Softmax(A)XkvWv,
where A =( XqWq(XkvWk)T )/√ d
Here, Wq, Wk, and Wv are trainable parameters of the linear projection, A is the attention score matrix.” showing that self-attention takes a query sequence and a key-value sequence as the input and computes the equations. Thus, the encoder-layer input Xkv (corresponding to the first feature vector) is project ted using Wk and Wv to get key/value vectors that can be used in computing the attention output.)
Regarding Claim 6, Wang teaches,
The method of claim 1 (and thus the rejection of claim 1 is incorporated), wherein generating, by the decoder of the neural network based NLP model the task output further comprising: (Wang, pg.2759, section IV. C, “For this architecture, both the encoder and the decoder contain multiple MHA layers”)
by a third attention head, a third attention output based at least in part on a third tunable vector, and a decoder layer input,(Wang, page 2757, section III. B “Vanilla MHA projects the input sequence into several different subspaces with separate projection matrices, so that multiple scaled dot-product attentions are computed in parallel.” multiple attention being computed in parallel supports multiple head including third and fourth and “Formally, MHA is computed as: MultiHead(Xq,Xkv)=[Z1;...;Zh]Wo, where h is the total number of heads, [·;...;·] indicates the concatenation operation, Zi is the attention corresponding to the i-th head, and Wo is a matrix of trainable parameters.” showing the tunable vector and “In order to make different attention heads interact with each other in Transformer, a nonlinear Feed-Forward Network (FFN) is applied to the output of MHA.” showing the output.)
computing, by a fourth attention head, a fourth attention output based at least I part on a fourth tunable vector, and a decoder layer input, (Wang, page 2757, section III. B “Vanilla MHA projects the input sequence into several different subspaces with separate projection matrices, so that multiple scaled dot-product attentions are computed in parallel.” multiple attention being computed in parallel supports multiple head including third and fourth and “Formally, MHA is computed as: MultiHead(Xq,Xkv)=[Z1;...;Zh]Wo, where h is the total number of heads, [·;...;·] indicates the concatenation operation, Zi is the attention corresponding to the i-th head, and Wo is a matrix of trainable parameters.” showing the tunable vector and “In order to make different attention heads interact with each other in Transformer, a nonlinear Feed-Forward Network (FFN) is applied to the output of MHA.” showing the output.)
and concatenating the third attention output and the fourth attention output as a decoding context vector for the decoding;(Wang, pg.2757, section III.B, “and “Formally, MHA is computed as: MultiHead(Xq,Xkv)=[Z1;...;Zh]Wo, where h is the total number of heads, [·;...;·] indicates the concatenation operation, Zi is the attention corresponding to the i-th head, and Wo is a matrix of trainable parameters” concatenates outputs of multiple attention heads )
Regarding Claim 11, Wang discloses,
A system for performing a natural language processing (NLP) task at a neural network-based NLP model, the system comprising; (Wang, pg.2755, section I, “We conduct extensive experiments on various NLP tasks, such as machine translation, sentence classification, etc. CollFormer, i.e., Transformer equipped with our proposed SLC and CLCs, achieves consistent improvement over vanilla Transformer and even many strong baselines”)
a memory that stores the neural network based NLP model and a plurality of processor executable instructions; (Wang, pg. 2759, section IV. D, “At the training stage, since the activation outputs of each layer need to be saved in the memory for computing the gradients, the memory cost will increase when equipped with CLC.”)
a communication interface (Wang, pg.2765, section V.C.6, “We perform this experiment on Transformer (base) model for the En-De translation task and run it on a V100 GPU” implying the use of computer which has at least one form of communication interface to receive data)
that receives a text input of a sequence of tokens; (Wang, pg.2755, section II.A, “For example, it is demonstrated that the self-attention mechanism can be treated as a graph-structure inductive bias, and all the tokens in an input sequence are connected via a relevance-based pooling operation [27].”)
and one or more hardware processors that read and execute the plurality of processor- executable instructions from the memory to perform operations comprising: (Wang, pg.2760, section V.A, “We train all the models with four K80 or P40 GPUs. Limited by the GPU memory size, more epochs are required than the results in [5].”
encoding, by an encoder of the neural network-based NLP model, the text input into one or more text representations, wherein the encoding comprises: (Wang, pg. 2759, section IV.C, “For this architecture, both the encoder and the decoder contain multiple MHA layers. We name the attention layer between the encoder and decoder as the cross-attention” stating the presence of encoder and figure 2, showing the input into representation R.
[AltContent: ][AltContent: ]
PNG
media_image1.png
631
718
media_image1.png
Greyscale
[AltContent: textbox (Representation)][AltContent: textbox (Input )]
computing, by a first attention head, a first attention output based at least in part on a first tunable vector, and an encoder layer input (Wang, pg. 2756, section III. A “Here, Wq, Wk, and Wv are trainable parameters of the linear projection” which are the tunable vectors and pg.2757, section III.B “Vanilla MHA projects the input sequence into several different subspaces with separate projection matrices” showing the input and “In order to make different attention heads interact with each other in Transformer, a nonlinear Feed-Forward Network (FFN) is applied to the output of MHA.” showing the output. )
computing, by a second attention head, a second attention output based on second tunable vector and the encoder layer input; (Wang, pg.2757, section III. B “Vanilla MHA projects the input sequence into several different subspaces with separate projection matrices, so that multiple scaled dot-product attentions are computed in parallel.” multiple attention being computed in parallel supports at least first and second heads and “Formally, MHA is computed as: MultiHead(Xq,Xkv)=[Z1;...;Zh]Wo, where h is the total number of heads, [·;...;·] indicates the concatenation operation, Zi is the attention corresponding to the i-th head, and Wo is a matrix of trainable parameters.” showing the tunable vector and “In order to make different attention heads interact with each other in Transformer, a nonlinear Feed-Forward Network (FFN) is applied to the output of MHA.” showing the output.)
and concatenating the first attention output and the second attention output as a context vector for the encoding;(Wang, pg.2757, section III.B, “and “Formally, MHA is computed as: MultiHead(Xq,Xkv)=[Z1;...;Zh]Wo, where h is the total number of heads, [·;...;·] indicates the concatenation operation, Zi is the attention corresponding to the i-th head, and Wo is a matrix of trainable parameters” concatenates outputs of multiple attention heads. )
and generating, a decoder of the neural network based NLP model, a task output based on the one or more text representations in response to the text input (Wang, pg.2759, section IV, figure 2,
[AltContent: textbox (Output based on text representation and the text input )][AltContent: rect]
PNG
media_image1.png
631
718
media_image1.png
Greyscale
and pg.2759, section IV.C, “For this architecture, both the encoder and the decoder contain multiple MHA layers. We name the attention layer between the encoder and decoder as the cross-attention. It helps the decoder focus on relevant parts of the input sentence.”)
Regarding Claim 12, Wang teaches,
The system of claim 11 (and thus the rejection of claim 11 is incorporated), wherein the first attention output is computed based on further computing a first key vector and a first value vector corresponding to a first feature vector corresponding to the encoder layer input, (Wang, pg. 2756, section III. A, “We denote the feature matrix of the n-token query sequence as Xq ∈ Rn× d, and the m-token key-value sequence as Xkv ∈ Rm× d. Here, d is the dimension of features. Then, the dot-product attention is computed as:
Z =Att(Xq,Xkv)= Softmax(A)XkvWv,
where A =( XqWq(XkvWk)T )/√ d
Here, Wq, Wk, and Wv are trainable parameters of the linear projection, A is the attention score matrix.” showing that self-attention takes a query sequence and a key-value sequence as the input and computes the equations. Thus, the encoder-layer input Xkv (corresponding to the first feature vector) is project ted using Wk and Wv to get key/value vectors that can be used in computing the attention output.)
Regarding Claim 16, Wang teaches,
The system of claim 11(and thus the rejection of claim 11 is incorporated), wherein the operation of generating, by the decoder of the neural network based NLP model the task output further comprising: (Wang, pg. 2759, section IV. C, “For this architecture, both the encoder and the decoder contain multiple MHA layers”)
by a third attention head, a third attention output based at least in part on a third tunable vector, and a decoder layer input,(Wang, pg.2757, section III. B “Vanilla MHA projects the input sequence into several different subspaces with separate projection matrices, so that multiple scaled dot-product attentions are computed in parallel.” multiple attention being computed in parallel supports multiple head including third and fourth and “Formally, MHA is computed as: MultiHead(Xq,Xkv)=[Z1;...;Zh]Wo, where h is the total number of heads, [·;...;·] indicates the concatenation operation, Zi is the attention corresponding to the i-th head, and Wo is a matrix of trainable parameters.” showing the tunable vector and “In order to make different attention heads interact with each other in Transformer, a nonlinear Feed-Forward Network (FFN) is applied to the output of MHA.” showing the output.)
computing, by a fourth attention head, a fourth attention output based at least I part on a fourth tunable vector, and a decoder layer input, (Wang, pg.2757, section III. B “Vanilla MHA projects the input sequence into several different subspaces with separate projection matrices, so that multiple scaled dot-product attentions are computed in parallel.” multiple attention being computed in parallel supports multiple head including third and fourth and “Formally, MHA is computed as: MultiHead(Xq,Xkv)=[Z1;...;Zh]Wo, where h is the total number of heads, [·;...;·] indicates the concatenation operation, Zi is the attention corresponding to the i-th head, and Wo is a matrix of trainable parameters.” showing the tunable vector and “In order to make different attention heads interact with each other in Transformer, a nonlinear Feed-Forward Network (FFN) is applied to the output of MHA.” showing the output.)
and concatenating the third attention output and the fourth attention output as a decoding context vector for the decoding;(Wang, pg.2757, section III. B, “and “Formally, MHA is computed as: MultiHead(Xq,Xkv)=[Z1;...;Zh]Wo, where h is the total number of heads, [·;...;·] indicates the concatenation operation, Zi is the attention corresponding to the i-th head, and Wo is a matrix of trainable parameters” concatenates outputs of multiple attention heads )
Regarding Claim 20, Wang discloses,
A non-transitory machine-readable medium comprising a plurality of machine-executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising: (Wang, pg.2760, section V.A, “We train all the models with four K80 or P40 GPUs. Limited by the GPU memory size, more epochs are required than the results in [5].”
receiving, via a communication interface (Wang, pg.2765, section V.C.6, “We perform this experiment on Transformer (base) model for the En-De translation task and run it on a V100 GPU” implying the use of computer which has at least one form of communication interface to receive data)
a text input of a sequence of tokens; (Wang, pg.2755, section II.A, “For example, it is demonstrated that the self-attention mechanism can be treated as a graph-structure inductive bias, and all the tokens in an input sequence are connected via a relevance-based pooling operation [27].”)
encoding, by an encoder of the neural network-based NLP model, the text input into one or more text representations, wherein the encoding comprises: (Wang, pg. 2759, section IV.C, “For this architecture, both the encoder and the decoder contain multiple MHA layers. We name the attention layer between the encoder and decoder as the cross-attention” stating the presence of encoder and figure 2, showing the input into representation R.
[AltContent: ][AltContent: ]
PNG
media_image1.png
631
718
media_image1.png
Greyscale
[AltContent: textbox (Representation)][AltContent: textbox (Input )]
computing, by a first attention head, a first attention output based at least in part on a first tunable vector, and an encoder layer input (Wang, pg. 2756, section III. A “Here, Wq, Wk, and Wv are trainable parameters of the linear projection” which are the tunable vectors and pg.2757, section III.B “Vanilla MHA projects the input sequence into several different subspaces with separate projection matrices” showing the input and “In order to make different attention heads interact with each other in Transformer, a nonlinear Feed-Forward Network (FFN) is applied to the output of MHA.” showing the output. )
computing, by a second attention head, a second attention output based on second tunable vector and the encoder layer input; (Wang, pg.2757, section III. B “Vanilla MHA projects the input sequence into several different subspaces with separate projection matrices, so that multiple scaled dot-product attentions are computed in parallel.” multiple attention being computed in parallel supports at least first and second heads and “Formally, MHA is computed as: MultiHead(Xq,Xkv)=[Z1;...;Zh]Wo, where h is the total number of heads, [·;...;·] indicates the concatenation operation, Zi is the attention corresponding to the i-th head, and Wo is a matrix of trainable parameters.” showing the tunable vector and “In order to make different attention heads interact with each other in Transformer, a nonlinear Feed-Forward Network (FFN) is applied to the output of MHA.” showing the output.)
and concatenating the first attention output and the second attention output as a context vector for the encoding;(Wang, pg.2757, section III.B, “and “Formally, MHA is computed as: MultiHead(Xq,Xkv)=[Z1;...;Zh]Wo, where h is the total number of heads, [·;...;·] indicates the concatenation operation, Zi is the attention corresponding to the i-th head, and Wo is a matrix of trainable parameters” concatenates outputs of multiple attention heads. )
and generating, a decoder of the neural network based NLP model, a task output based on the one or more text representations in response to the text input (Wang, pg.2759, section IV, figure 2,
[AltContent: textbox (Output based on text representation and the text input )][AltContent: rect]
PNG
media_image1.png
631
718
media_image1.png
Greyscale
and pg.2759, section IV.C, “For this architecture, both the encoder and the decoder contain multiple MHA layers. We name the attention layer between the encoder and decoder as the cross-attention. It helps the decoder focus on relevant parts of the input sentence.”)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3,7,13,17 are rejected under 35 U.S.C. 103 as being unpatentable over Wang, in view of Henry et al, “Query-Key Normalization for Transformers”
Regarding Claim 3, Wang discloses,
The method of claim 2 (and thus the rejection of claim 2 is incorporated),
However, Wang fails to mention the normalization of the vectors.
Henry, in the same field of endeavor as Wang (modifying transformer architecture), discloses,
wherein the first attention output is computed based on a softmax operation between a normalized first tunable vector, the first key vector and the first value vector (Henry, pg.1, section 1 “The operation softmax(QKT) defines a distribution for each token over all the others in its sequence that sums to 1. Multiplying by V then yields a new matrix where the embedding of each token is a weighted average of the vectors in V . and abstract, “Specifically, we apply l2 normalization along the head dimension of each query and key matrix prior to multiplying them and then scale up by a learnable parameter instead of dividing by the square root of the embedding dimension.” where the learnable parameter is the tunable vector.)
wherein the first key vector and the first value vector corresponds to a same first position in the sequence of tokens (Henry, pg.1, section 1, “Given a matrix X embed ding a sequence of tokens,….”and “the vectors of X are projected linearly into Query, Key and Value matrices.” showing that the K and V rows originate from the same token vector in X.)
Wang and Henry are both considered to be analogous to the claimed invention because they are in the same field of modifying transformers. Therefore, it would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Wang and Henry to use softmax operation between the vectors where the first key and value vectors are in the same first position in the sequence of tokens. The motivation to do so is to “modify the attention mechanism to make the softmax function less prone to arbitrary saturation without sacrificing expressivity” (Henry, abstract).
Regarding Claim 7, Wang teaches,
The method of claim 6 (and thus the rejection of claim 6 is incorporated),
Wang further discloses,
and a third key vector derived from the decoder layer input (Wang, pg.2757, section III.B, “Vanilla MHA projects the input sequence into several different subspaces with separate projection matrices...” implying that each head has its own key projection.)
However, Wang fails to mention the third attention output and the multiplication between the result and the key value.
Henry discloses,
wherein the third attention output is computed based attending a normalized third tunable vector (Henry, pg.2, section 2, “QKNORM applies l2 normalization to Q and K only, and it does so along the head dimension, (which is the same dimension as the embedding dimension, but after multihead attention has split its input into separate heads)”)
and then multiplying an attended result with a third key value derived from the decoder layer input.(Henry, pg.3, section 3, “
PNG
media_image2.png
63
257
media_image2.png
Greyscale
” which is the attended result “where ˆQ and ˆK are Q and K with 2-normalization applied along their head dimensions and g is a learnable scalar parameter initialized with g0 as computed” where the learnable parameter is the tunable vector)
It would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Wang and Henry to have a third key vector and the third attention output and the multiplication between the result and the key value. The motivation to do so would be to “let the model effectively ignore connections between words it should not attend to” (Henry, section 3)
Regarding Claim 13, Wang discloses,
The system of claim 12 (and thus the rejection of claim 12 is incorporated),
However, Wang fails to mention the normalization of the vectors.
Henry, in the same field of endeavor as Wang (modifying transformer architecture), discloses,
wherein the first attention output is computed based on a softmax operation between a normalized first tunable vector, the first key vector and the first value vector (Henry, pg.1, section 1 “The operation softmax(QKT) defines a distribution for each token over all the others in its sequence that sums to 1. Multiplying by V then yields a new matrix where the embedding of each token is a weighted average of the vectors in V . and abstract, “Specifically, we apply l2 normalization along the head dimension of each query and key matrix prior to multiplying them and then scale up by a learnable parameter instead of dividing by the square root of the embedding dimension.” where the learnable parameter is the tunable vector.)
wherein the first key vector and the first value vector corresponds to a same first position in the sequence of tokens (Henry, pg.1, section 1, “Given a matrix X embed ding a sequence of tokens,….”and “the vectors of X are projected linearly into Query, Key and Value matrices.” showing that the K and V rows originate from the same token vector in X.)
Wang and Henry are both considered to be analogous to the claimed invention because they are in the same field of modifying transformers. Therefore, it would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Wang and Henry to use softmax operation between the vectors where the first key and value vectors are in the same first position in the sequence of tokens. The motivation to do so is to “modify the attention mechanism to make the softmax function less prone to arbitrary saturation without sacrificing expressivity” (Henry, abstract).
Regarding Claim 17, Wang teaches,
The system of claim 16 (and thus the rejection of claim 16 is incorporated),
Wang further discloses,
and a third key vector derived from the decoder layer input (Wang, pg. 2757, section III. B, “Vanilla MHA projects the input sequence into several different subspaces with separate projection matrices...” implying that each head has its own key projection.)
However, Wang fails to mention the third attention output and the multiplication between the result and the key value.
Henry discloses,
wherein the third attention output is computed based attending a normalized third tunable vector (Henry, pg.2, section 2, “QKNORM applies l2 normalization to Q and K only, and it does so along the head dimension, (which is the same dimension as the embedding dimension, but after multihead attention has split its input into separate heads)”)
and then multiplying an attended result with a third key value derived from the decoder layer input.(Henry,pg.3, section 3, “
PNG
media_image2.png
63
257
media_image2.png
Greyscale
” which is the attended result “where ˆQ and ˆK are Q and K with 2-normalization applied along their head dimensions and g is a learnable scalar parameter initialized with g0 as computed” where the learnable parameter is the tunable vector)
It would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Wang and Henry to have a third key vector and the third attention output and the multiplication between the result and the key value. The motivation to do so would be to “let the model effectively ignore connections between words it should not attend to” (Henry, section 3)
Claims 4,5, 14 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Wang, in view of Luong et al, “Effective Approaches to Attention-based Neural Machine Translation”
Regarding Claim 4, Wang teaches,
The method of claim 1 (and thus the rejection of claim 1 is incorporated).
Wang further discloses,
and feeding a concatenated vector to a feed forward layer to obtain an encoding layer output (Wang, pg. 2757, section III.B, “In order to make different attention heads interact with each other in Transformer, a nonlinear Feed-Forward Network (FFN) is applied to the output of MHA” and pg. 2759, section IV. C “For this architecture, both the encoder and the decoder contain multiple MHA layers”)
However, Wang fails to disclose the concatenation of the context vector with the encoder layer input.
Luong, in the same field of endeavor as Wang (improving attention based neural network model), discloses,
wherein the encoding further comprises: concatenating the context vector with the encoder layer input (Luong, pg.3, section 3, “Common to these two types of models is the fact that at each time step t in the decoding phase, both approaches first take as input the hidden state ht at the top layer of a stacking LSTM” and “Specifically, given the target hidden state ht and the source-side context vector ct, we employ a simple concatenation layer to combine the information from both vectors to produce an attentional hidden state”)
Wang and Luong are both considered to be analogous to the claimed invention because they are in the same field of improving attention based neural network model. Therefore, it would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Wang and Henry to concatenate the context vector with encoder layer input and feed it to a feed forward layer to obtain output. The motivation to do so is to “ make different attention heads interact with each other in Transformer” (Wang, section III, part B)
Regarding Claim 5, the combination of Wang/Luong, discloses,
The method of claim 4 (and thus the rejection of claim 4 is incorporated)
Wang further teaches,
wherein the encoding further comprises: computing, at a next encoding layer, a next encoding layer output using feature vectors in the encoding layer output as an input (Wang, pg. 2759, section IV, figure 2,
[AltContent: rect][AltContent: textbox (Shows the consecutive stacked encoder layers where the output of the encoding layer is input to the another)]
PNG
media_image1.png
631
718
media_image1.png
Greyscale
)
Regarding Claim 15, the combination of Wang/Luong, discloses,
The system of claim 14 (and thus the rejection of claim 14 is incorporated)
Wang further teaches,
wherein the encoding further comprises: computing, at a next encoding layer, a next encoding layer output using feature vectors in the encoding layer output as an input (Wang, pg.2759, section IV, figure 2,
[AltContent: rect][AltContent: textbox (Shows the consecutive stacked encoder layers where the output of the encoding layer is input to the another)]
PNG
media_image1.png
631
718
media_image1.png
Greyscale
)
Regarding Claim 14, Wang teaches,
The system of claim 11 (and thus the rejection of claim 11 is incorporated).
Wang further discloses,
and feeding a concatenated vector to a feed forward layer to obtain an encoding layer output (Wang, pg.2757, section III. B, “In order to make different attention heads interact with each other in Transformer, a nonlinear Feed-Forward Network (FFN) is applied to the output of MHA” and pg.2759, section IV. C “For this architecture, both the encoder and the decoder contain multiple MHA layers”)
However, Wang fails to disclose the concatenation of the context vector with the encoder layer input.
Luong, in the same field of endeavor as Wang (improving attention based neural network model), discloses,
wherein the encoding further comprises: concatenating the context vector with the encoder layer input (Luong, pg. 3, section 3, “Common to these two types of models is the fact that at each time step t in the decoding phase, both approaches first take as input the hidden state ht at the top layer of a stacking LSTM” and “Specifically, given the target hidden state ht and the source-side context vector ct, we employ a simple concatenation layer to combine the information from both vectors to produce an attentional hidden state”)
Wang and Luong are both considered to be analogous to the claimed invention because they are in the same field of improving attention based neural network model. Therefore, it would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Wang and Henry to concatenate the context vector with encoder layer input and feed it to a feed forward layer to obtain output. The motivation to do so is to “make different attention heads interact with each other in Transformer” (Wang, section III, part B)
Claim 8 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Wang, in view of Li et al, “Transformer-based online speech recognition with decoder-end adaptive computations”
Regarding Claim 8, Wang teaches,
The method of claim 7 (and thus the rejection of claim 7 is incorporated),
However, Wang fails to mention the cumulative sum of the result.
Li, in the same field of endeavor as Wang (improving transformer architecture in NLP) discloses,
wherein the third attention output is computed using cumulative sum of the attended result (Li, pg.3, section 3, “A possible solution is to convert the Bernoulli process based on Pi, j into a more stable accumulation of the attention weights, and the process is halted once the accumulation exceeds a certain threshold……From j = 1, we keep inspecting the accumulation of Pi, j, and halt the computation as soon as the cumulative sum is greater than 1…… As the output of headh, a context vector is calculated using the halting probabilities {Pi,..} as weighs”)
Wang and Li are both considered to be analogous to the claimed invention because they are in the same field of improving transformer architecture in NLP. Therefore, it would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Wang and Li to compute the attention output using sum of results. The motivation to do so is “to control latency during inference” (Li, section 3)
Regarding Claim 18, Wang teaches,
The system of claim 17 (and thus the rejection of claim 17 is incorporated),
However, Wang fails to mention the cumulative sum of the result.
Li, in the same field of endeavor as Wang (improving transformer architecture in NLP) discloses,
wherein the third attention output is computed using cumulative sum of the attended result (Li, pg.3, section 3, “A possible solution is to convert the Bernoulli process based on Pi, j into a more stable accumulation of the attention weights, and the process is halted once the accumulation exceeds a certain threshold……From j = 1, we keep inspecting the accumulation of Pi, j, and halt the computation as soon as the cumulative sum is greater than 1…… As the output of headh, a context vector is calculated using the halting probabilities {Pi,..} as weighs”)
Wang and Li are both considered to be analogous to the claimed invention because they are in the same field of improving transformer architecture in NLP. Therefore, it would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Wang and Li to compute the attention output using sum of results. The motivation to do so is “to control latency during inference” (Li, section 3)
Claim 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al, in view of Li et al, further in view of Henry et al.
Regarding Claim 9, Wang/Li combination teaches,
The method of claim 8 (and thus the rejection of claim 8 is incorporated).
Wang further teaches,
wherein the third attention output corresponds to the third attention head (Wang, pg. 2757,section III.B, “Formally, MHA is computed as: MultiHead(Xq,Xkv)=[Z1;...;Zh]Wo, where h is the total number of heads, [·;...;·] indicates the concatenation operation, Zi is the attention corresponding to the i-th head” each Zi is an output mapped to attention head, so Z3 would have its corresponding head)
the fourth attention output corresponds to the fourth attention head(Wang, pg. 2757, section III. B, “Formally, MHA is computed as: MultiHead(Xq,Xkv)=[Z1;...;Zh]Wo, where h is the total number of heads, [·;...;·] indicates the concatenation operation, Zi is the attention corresponding to the i-th head” each Zi is an output mapped to attention head, so Z4 would have its corresponding head )
and the decoding context vector is position-wise (Wang, pg. 2757, section III. B,” In order to make different attention heads interact with each other in Transformer, a nonlinear Feed-Forward Network (FFN) is applied to the output of MHA. And the same FFN is applied to each position of the sequence on each layer of Transformer”).
However, Wang fails to mention the specific position in the token where the outputs and the context vector would correspond to.
Henry discloses,
the third and fourth attention output corresponds to the specific position in the sequence of tokens, (Henry, pg.2, section 2, “Q and K thus become ˆ Q and ˆK, where the ith row vector of ˆQ (the ith embedding in the sequence) is given by..”the third and fourth attention output can be associated with the same ith token position)
and the decoding context vector corresponds to the specific position in the sequence of tokens (Henry, pg.1, section 1, “Multiplying by V then yields a new matrix where the embedding of each token is a weighted average of the vectors in V”)
Wang, Li and Henry are all considered to be analogous to the claimed invention because they are in the same field of improving transformer architecture in NLP. Therefore, it would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Wang, Li and Henry so that the output of individual attention head and the context vector correspond to the position in the token sequence. The motivation to do so would be to make sure each attention head is attending for their respective token.
Regarding Claim 19, Wang/Li combination teaches,
The system of claim 18 (and thus the rejection of claim 18 is incorporated).
Wang further teaches,
wherein the third attention output corresponds to the third attention head (Wang, pg. 2757, section III. B, “Formally, MHA is computed as: MultiHead(Xq,Xkv)=[Z1;...;Zh]Wo, where h is the total number of heads, [·;...;·] indicates the concatenation operation, Zi is the attention corresponding to the i-th head” each Zi is an output mapped to attention head, so Z3 would have its corresponding head)
the fourth attention output corresponds to the fourth attention head (Wang, pg.2757, section III. B, “Formally, MHA is computed as: MultiHead(Xq,Xkv)=[Z1;...;Zh]Wo, where h is the total number of heads, [·;...;·] indicates the concatenation operation, Zi is the attention corresponding to the i-th head” each Zi is an output mapped to attention head, so Z4 would have its corresponding head )
and the decoding context vector is position-wise (Wang, pg.2757, section III.B,” In order to make different attention heads interact with each other in Transformer, a nonlinear Feed-Forward Network (FFN) is applied to the output of MHA. And the same FFN is applied to each position of the sequence on each layer of Transformer”).
However, Wang fails to mention the specific position in the token where the outputs and the context vector would correspond to.
Henry discloses,
the third and fourth attention output corresponds to the specific position in the sequence of tokens, (Henry, pg.2, section 2, “Q and K thus become ˆ Q and ˆK, where the ith row vector of ˆQ (the ith embedding in the sequence) is given by..”the third and fourth attention output can be associated with the same ith token position)
and the decoding context vector corresponds to the specific position in the sequence of tokens (Henry, pg.1, section 1, “Multiplying by V then yields a new matrix where the embedding of each token is a weighted average of the vectors in V”)
Wang, Li and Henry are all considered to be analogous to the claimed invention because they are in the same field of improving transformer architecture in NLP. Therefore, it would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Wang, Li and Henry so that the output of individual attention head and the context vector correspond to the particular position in the token sequence. The motivation to do so would be to make sure each attention head is attending for their respective token.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al, in view of Ding et al “Convolutional Transformer: An Enhanced Attention Mechanism Architecture for Remaining Useful Life Estimation of Bearings”
Regarding Claim 10, Wang discloses,
The method of claim 1 (and thus the rejection of claim 1 is incorporated),
Wang fails to teach the backpropagation of the NLP model. Ding, in the same field of endeavor and Wang (enhancing the attention architecture in NLP), discloses,
further comprising: updating the first tunable vector and the second tunable vector at a backpropagation of the neural network based NLP model during training (Ding, pg. 4, section III, “….denote the ith version of linear projection on embeddings X to obtain different queries, keys, and values, respectively” and pg.5, section III, algorithm 1, “
[AltContent: rect][AltContent: textbox (First and second tunable vector/trainable parameters)][AltContent: textbox (Updating the trainable parameters/vectors)][AltContent: rect][AltContent: rect][AltContent: textbox (Calculating the gradient of the loss function using back propagation)]
PNG
media_image3.png
382
359
media_image3.png
Greyscale
Wang and Ding are both considered to be analogous to the claimed invention because they are in the same field of enhancing transformer architecture in NLP. Therefore, it would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Wang and Ding to update the first tunable vector and the second tunable vector at a backpropagation of the neural network. The motivation to do so would be to minimize errors.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PRAYUKTA GHIMIRE whose telephone number is (571)270-5484. The examiner can normally be reached M-F, 9 am to 5 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/P.N.G./Examiner, Art Unit 2122
/MICHAEL H HOANG/PRIMARY EXAMINER, Art Unit 2122
Prosecution Insights