DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on November 8, 2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Specification
The disclosure is objected to because of the following informalities:
In paragraph 19, line 15, "fewer compute resources" should read "fewer computing resources."
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-28 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claims 1-9 and 19-28 are system claims. Claims 10-18 are method type claims. Therefore, the claims are directed to either a process, machine, manufacture or composition of matter.
Claim 1:
Regarding claim 1, in step 1 of the 101-analysis set forth in MPEP 2106, the claim recites “a processing system.” This is a device or machine, which is one of the four statutory categories of invention.
In step 2A prong 1 of the 101-analysis set forth in MPEP 2106, the examiner has determined that the following limitations recite a process that, under the broadest reasonable interpretation, covers as mathematical concept but for recitation of generic computer components:
“generate an attention output in the transformer neural network, the attention output being generated such that outlier values for the attention output are attenuated in the transformer neural network” (this is a mathematical concept; the attention output, per the specification (see paragraph 0032) is generated using the equation
A
t
t
e
n
t
i
o
n
Q
,
K
,
V
≔
s
o
f
t
m
a
x
(
Q
K
T
d
)
V
), see MPEP § 2106.05(a)(2)(I))
In step 2A prong 2 of the 101-analysis set forth in MPEP 2106, the examiner has determined that the following additional elements do not integrate this judicial exception into a practical application:
i. “A processing system comprising” (a processing system is considered a generic computer component being used as a tool to perform the judicial exception – see MPEP § 2106.05(f))
ii. “at least one memory having executable instructions stored thereon” (a memory is considered a generic computer component being used as a tool to perform the judicial exception – see MPEP § 2106.05(f))
iii. “one or more processors configured to execute the executable instructions” (processors are considered a generic computer component being used as a tool to perform the judicial exception – see MPEP § 2106.05(f))
iv. “receive an input for processing using a transformer neural network” (receiving an input is considered insignificant extra-solution activity of mere data gathering – see MPEP § 2106.05(g))
v. “generate an output of the transformer neural network based on the generated attention output” (generating an output is considered mere instructions to apply the exception using a generic computer – see MPEP § 2106.05(g))
In step 2B of the 101-analysis set forth in the 2019 PEG, the examiner has determined that the claim does not include additional elements that are sufficient to amount significantly more than the judicial exception.
As discussed above, the additional element iv recites insignificant extra-solution activity of
mere data gathering, which is a well-understood routine and conventional activity, receiving or
transmitting data over a network, e.g., using the Internet to gather data, see Symantec, 838
F.3d at 1321, 120 USPQ2d at 1362. Additional element v recites mere instructions to apply the exception using a generic computer. In addition, additional elements i, ii, and iii recite a generic computer component being used to perform the judicial exception, which are not indicative of significantly more.
Considering that the additional elements individually and in combination, and the claim
as a whole, the additional elements do not provide significantly more than the abstract idea.
Therefore, the claim is not patent eligible.
Claim 2
Regarding claim 2, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis to claim 1. Further, claim 2 recites the following additional elements:
“generate the attention output based on a clipped softmax function having a dynamic range controlled by a first hyperparameter and a second hyperparameter” (this is a mathematical concept; the softmax function, as described in paragraph 0034 of the specification, is
c
l
i
p
p
e
d
s
o
f
t
m
a
x
=
x
;
ξ
,
γ
≔
c
l
i
p
(
ξ
-
γ
∙
s
o
f
t
m
a
x
x
+
γ
,
0,1
)
, see MPEP § 2106.05(a)(2)(I)))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“The processing system of Claim 1, wherein to generate the attention output in the transformer neural network, the one or more processors are configured to cause the processing system to” (in step 2A, prong 2, this is considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 3
Regarding claim 3, it is dependent upon claim 2, and thereby incorporates the limitations of, and corresponding analysis to claim 2. Further, claim 3 recites the following additional elements:
“the first hyperparameter comprises a hyperparameter greater than or equal to 1 and wherein the second hyperparameter comprises a hyperparameter less than or equal to 0” (this is a mathematical concept, as per paragraph 0034, in the equation
c
l
i
p
p
e
d
s
o
f
t
m
a
x
=
x
;
ξ
,
γ
≔
c
l
i
p
(
ξ
-
γ
∙
s
o
f
t
m
a
x
x
+
γ
,
0,1
)
, two parameters
ξ
and
γ
are defined, wherein
ξ
≥
1
,
and
γ
≤
0
, see MPEP § 2106.05(a)(2)(I)))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“The processing system of Claim 2” (in step 2A, prong 2, this is considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 4
Regarding claim 4, it is dependent upon claim 3, and thereby incorporates the limitations of, and corresponding analysis to claim 3. Further, claim 4 recites the following additional elements:
“wherein the clipped softmax function is configured to output values up to and including a value of 1 when a value of the first hyperparameter is greater than 1” (this is a mathematical concept, as per paragraph 0034, in the equation
c
l
i
p
p
e
d
s
o
f
t
m
a
x
=
x
;
ξ
,
γ
≔
c
l
i
p
(
ξ
-
γ
∙
s
o
f
t
m
a
x
x
+
γ
,
0,1
)
, the function
c
l
i
p
(
)
outputs a value of 1 for inputs larger than 1, see MPEP § 2106.05(a)(2)(I))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“The processing system of claim 3,” (in step 2A, prong 2, this is considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 5
Regarding claim 5, it is dependent upon claim 3, and thereby incorporates the limitations of, and corresponding analysis to claim 3. Further, claim 5 recites the following additional elements:
“wherein the clipped softmax function is configured to output values down to and including a value of 0 when a value of the second hyperparameter is less than 0” (this is a mathematical concept, as per paragraph 0034, in the equation
c
l
i
p
p
e
d
s
o
f
t
m
a
x
=
x
;
ξ
,
γ
≔
c
l
i
p
(
ξ
-
γ
∙
s
o
f
t
m
a
x
x
+
γ
,
0,1
)
, the function
c
l
i
p
(
)
outputs a value of 0 for inputs smaller than 0, see MPEP § 2106.05(a)(2)(I))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“The processing system of claim 3” (in step 2A, prong 2, this is considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 6
Regarding claim 6, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis to claim 1. Further, claim 6 recites the following additional elements:
“generate the attention output based on a gated attention block configured to output a minimum value of 0” (this is a mathematical concept; the equation given in paragraph 0039 is
P
^
x
≔
s
i
g
m
o
i
d
G
x
∙
s
o
f
t
m
a
x
(
Q
x
K
x
T
d
)
. In paragraph 0040, it is stated that the output of the softmax function may be between 0 and 1, but not exactly 0 or 1. It is multiplied by a sigmoid function, which may output a value between 0 and 1. Because neither function can output a negative number, the product of the two functions’ outputs must be at least 0, see MPEP § 2106.05(a)(2)(I))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“The processing system of Claim 1, wherein to generate the attention output in the transformer neural network, wherein to generate the attention output in the transformer neural network, the one or more processors are configured to cause the processing system to” (in step 2A, prong 2, this is considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 7
Regarding claim 7, it is dependent upon claim 6, and thereby incorporates the limitations of, and corresponding analysis to claim 6. Further, claim 7 recites the following additional elements:
“the gated attention block applies a bounded nonlinear function to one or more gating parameters defined for the transformer neural network” (this is a mathematical concept; per paragraph 0046 of the specification this may be, for example, a sigmoid function or other nonlinear function having defined maximum and minimum values which may be output by the function, see MPEP § 2106.05(a)(2)(I))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“The processing system of Claim 6” (in step 2A, prong 2, this is considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 8
Regarding claim 8, it is dependent upon claim 7, and thereby incorporates the limitations of, and corresponding analysis to claim 7. Further, claim 8 recites the following additional elements:
“the bounded nonlinear function comprises a sigmoid function” (this is a mathematical concept, see MPEP § 2106.05(a)(2)(I)).
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“The processing system of Claim 7” (in step 2A, prong 2, this is considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 9
Regarding claim 9, it is dependent upon claim 6, and thereby incorporates the limitations of, and corresponding analysis to claim 6. Further, claim 9 recites the following additional elements:
“wherein the gated attention block is applied for each token generated by the transformer neural network for the received input” (this is a mathematical concept; as per the equation in paragraph 0038
A
X
≔
P
^
(
x
)
V
(
x
)
,
P
^
(
x
)
represents the gated output of a function and
V
(
x
)
represents a value matrix associated with input
x
, wherein input
x
may be a sequence of tokens in a transformer neural network, see MPEP § 2106.05(a)(2)(I))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“The processing system of Claim 6” (in step 2A, prong 2, this is considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 10
Regarding claim 10, in step 1 of the 101-analysis set forth in MPEP 2106, the claim recites “a processor-implemented method.” This is a process, which is one of the four statutory categories of invention.
In step 2A prong 1 of the 101-analysis set forth in MPEP 2106, the examiner has determined that the following limitations recite a process that, under the broadest reasonable interpretation, covers as mathematical concept but for recitation of generic computer components:
“generating an attention output in the transformer neural network, the attention output being generated such that outlier values for the attention output are attenuated in the transformer neural network” (this is a mathematical concept; the attention output, per the specification (see paragraph 0032) is generated using the equation
A
t
t
e
n
t
i
o
n
Q
,
K
,
V
≔
s
o
f
t
m
a
x
(
Q
K
T
d
)
V
), see MPEP § 2106.05(a)(2)(I))
In step 2A prong 2 of the 101-analysis set forth in MPEP 2106, the examiner has determined that the following additional elements do not integrate this judicial exception into a practical application:
i. “a processor-implemented method” (this is considered mere instructions to apply the exception using a generic computer
ii. “receiving an input for processing using a transformer neural network” (receiving an input is considered insignificant extra-solution activity of mere data gathering – see MPEP § 2106.05(g))
iii. “generating an output of the transformer neural network based on the generated attention output” (generating an output is considered mere instructions to apply the exception using a generic computer – see MPEP § 2106.05(g))
In step 2B of the 101-analysis set forth in the 2019 PEG, the examiner has determined that the claim does not include additional elements that are sufficient to amount significantly more than the judicial exception.
As discussed above, the additional element ii recites insignificant extra-solution activity of
mere data gathering, which is a well-understood routine and conventional activity, receiving or
transmitting data over a network, e.g., using the Internet to gather data, see Symantec, 838
F.3d at 1321, 120 USPQ2d at 1362. Additional elements i and iii recite mere instructions to apply the exception using a generic computer.
Considering that the additional elements individually and in combination, and the claim
as a whole, the additional elements do not provide significantly more than the abstract idea.
Therefore, the claim is not patent eligible.
Claim 11
Regarding claim 11, it is dependent upon claim 10, and thereby incorporates the limitations of, and corresponding analysis to claim 10. Further, claim 11 recites the following additional elements:
“generating the attention output based on a clipped softmax function having a dynamic range controlled by a first hyperparameter and a second hyperparameter” (this is a mathematical concept; the softmax function, as described in paragraph 0034 of the specification, is
c
l
i
p
p
e
d
s
o
f
t
m
a
x
=
x
;
ξ
,
γ
≔
c
l
i
p
(
ξ
-
γ
∙
s
o
f
t
m
a
x
x
+
γ
,
0,1
)
, see MPEP § 2106.05(a)(2)(I)))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“The method of Claim 10, wherein generating the attention output in the transformer neural network” (in step 2A, prong 2, this is considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 12
Regarding claim 12, it is dependent upon claim 11, and thereby incorporates the limitations of, and corresponding analysis to claim 11. Further, claim 12 recites the following additional elements:
“the first hyperparameter comprises a hyperparameter greater than or equal to 1 and wherein the second hyperparameter comprises a hyperparameter less than or equal to 0” (this is a mathematical concept, as per paragraph 0034, in the equation
c
l
i
p
p
e
d
s
o
f
t
m
a
x
=
x
;
ξ
,
γ
≔
c
l
i
p
(
ξ
-
γ
∙
s
o
f
t
m
a
x
x
+
γ
,
0,1
)
, two parameters
ξ
and
γ
are defined, wherein
ξ
≥
1
,
and
γ
≤
0
, see MPEP § 2106.05(a)(2)(I)))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 13
Regarding claim 13, it is dependent upon claim 12, and thereby incorporates the limitations of, and corresponding analysis to claim 12. Further, claim 13 recites the following additional elements:
“the clipped softmax function is configured to output values up to and including a value of 1 when a value of the first hyperparameter is greater than 1” (this is a mathematical concept, as per paragraph 0034, in the equation
c
l
i
p
p
e
d
s
o
f
t
m
a
x
=
x
;
ξ
,
γ
≔
c
l
i
p
(
ξ
-
γ
∙
s
o
f
t
m
a
x
x
+
γ
,
0,1
)
, the function
c
l
i
p
(
)
outputs a value of 1 for inputs larger than 1, see MPEP § 2106.05(a)(2)(I))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 14
Regarding claim 14, it is dependent upon claim 12, and thereby incorporates the limitations of, and corresponding analysis to claim 12. Further, claim 14 recites the following additional elements:
“wherein the clipped softmax function is configured to output values down to and including a value of 0 when a value of the second hyperparameter is less than 0” (this is a mathematical concept, as per paragraph 0034, in the equation
c
l
i
p
p
e
d
s
o
f
t
m
a
x
=
x
;
ξ
,
γ
≔
c
l
i
p
(
ξ
-
γ
∙
s
o
f
t
m
a
x
x
+
γ
,
0,1
)
, the function
c
l
i
p
(
)
outputs a value of 0 for inputs smaller than 0, see MPEP § 2106.05(a)(2)(I))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 15
Regarding claim 15, it is dependent upon claim 10, and thereby incorporates the limitations of, and corresponding analysis to claim 10. Further, claim 15 recites the following additional elements:
“generating the attention output in the transformer neural network comprises generating the attention output based on a gated attention block configured to output a minimum value of 0” (this is a mathematical concept; the equation given in paragraph 0039 is
P
^
x
≔
s
i
g
m
o
i
d
G
x
∙
s
o
f
t
m
a
x
(
Q
x
K
x
T
d
)
. In paragraph 0040, it is stated that the output of the softmax function may be between 0 and 1, but not exactly 0 or 1. It is multiplied by a sigmoid function, which may output a value between 0 and 1. Because neither function can output a negative number, the product of the two functions’ outputs must be at least 0, see MPEP § 2106.05(a)(2)(I))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 16
Regarding claim 16, it is dependent upon claim 15, and thereby incorporates the limitations of, and corresponding analysis to claim 15. Further, claim 16 recites the following additional elements:
“the gated attention block applies a bounded nonlinear function to one or more gating parameters defined for the transformer neural network” (this is a mathematical concept; per paragraph 0046 of the specification this may be, for example, a sigmoid function or other nonlinear function having defined maximum and minimum values which may be output by the function, see MPEP § 2106.05(a)(2)(I))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 17
Regarding claim 17, it is dependent upon claim 16, and thereby incorporates the limitations of, and corresponding analysis to claim 16. Further, claim 17 recites the following additional elements:
“the bounded nonlinear function comprises a sigmoid function” (this is a mathematical concept, see MPEP § 2106.05(a)(2)(I))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 18
Regarding claim 18, it is dependent upon claim 15, and thereby incorporates the limitations of, and corresponding analysis to claim 15. Further, claim 18 recites the following additional elements:
“wherein the gated attention block is applied for each token generated by the transformer neural network for the received input” (this is a mathematical concept; as per the equation in paragraph 0038
A
X
≔
P
^
(
x
)
V
(
x
)
,
P
^
(
x
)
represents the gated output of a function and
V
(
x
)
represents a value matrix associated with input
x
, wherein input
x
may be a sequence of tokens in a transformer neural network, see MPEP § 2106.05(a)(2)(I))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 19
Regarding claim 19, in step 1 of the 101-analysis set forth in MPEP 2106, the claim recites “a processing system.” This is a device or machine, which is one of the four statutory categories of invention.
In step 2A prong 1 of the 101-analysis set forth in MPEP 2106, the examiner has determined that the following limitations recite a process that, under the broadest reasonable interpretation, covers as mathematical concept but for recitation of generic computer components:
“generating an attention output in the transformer neural network, the attention output being generated such that outlier values for the attention output are attenuated in the transformer neural network” (this is a mathematical concept; the attention output, per the specification (see paragraph 0032) is generated using the equation
A
t
t
e
n
t
i
o
n
Q
,
K
,
V
≔
s
o
f
t
m
a
x
(
Q
K
T
d
)
V
), see MPEP § 2106.05(a)(2)(I))
In step 2A prong 2 of the 101-analysis set forth in MPEP 2106, the examiner has determined that the following additional elements do not integrate this judicial exception into a practical application:
i. "A processing system comprising" (a processing system is considered a generic computer component being used as a tool to perform the judicial exception - see MPEP § 2106.05(f))
ii. "means for receiving an input" (this is considered mere instructions to apply the exception using a generic computer component - see MPEP § 2106.05(f))
iii. "receiving an input for processing using a transformer neural network" (receiving an input is considered insignificant extra-solution activity of mere data gathering - see MPEP § 2106.05(g))
iv. "means for generating an attention output" (this is considered mere instructions to apply the exception using a generic computer component - see MPEP § 2106.05(f))
v. "means for generating an output" (this is considered mere instructions to apply the exception using a generic computer component - see MPEP § 2106.05(f))
vi. "generate an output of the transformer neural network based on the generated attention output" (generating an output is considered mere instructions to apply the exception using a generic computer - see MPEP § 2106.05(g))
In step 2B of the 101-analysis set forth in the 2019 PEG, the examiner has determined that the claim does not include additional elements that are sufficient to amount significantly more than the judicial exception.
As discussed above, additional element i recites a generic computer component being used to perform the judicial exception, which are not indicative of significantly more. In addition, additional elements ii, iv, v, and vi recite mere instructions to apply the exception using a generic computer. In addition, additional element iii recites insignificant extra-solution activity of mere data gathering, which is a well-understood routine and conventional activity, receiving or transmitting data over a network, e.g., using the Internet to gather data, see Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362.
Considering that the additional elements individually and in combination, and the claim
as a whole, the additional elements do not provide significantly more than the abstract idea.
Therefore, the claim is not patent eligible.
Claim 20
Regarding claim 20, it is dependent upon claim 19, and thereby incorporates the limitations of, and corresponding analysis to claim 19. Further, claim 20 recites the following additional elements:
“generating the attention output based on a clipped softmax function having a dynamic range controlled by a first hyperparameter and a second hyperparameter” (this is a mathematical concept; the softmax function, as described in paragraph 0034 of the specification, is
c
l
i
p
p
e
d
s
o
f
t
m
a
x
=
x
;
ξ
,
γ
≔
c
l
i
p
(
ξ
-
γ
∙
s
o
f
t
m
a
x
x
+
γ
,
0,1
)
, see MPEP § 2106.05(a)(2)(I)))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“The processing system of Claim 19, wherein the means for generating the attention output in the transformer neural network comprises means for” (in step 2A, prong 2, this is considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 21
Regarding claim 21, it is dependent upon claim 20, and thereby incorporates the limitations of, and corresponding analysis to claim 20. Further, claim 21 recites the following additional elements:
“the first hyperparameter comprises a hyperparameter greater than or equal to 1 and wherein the second hyperparameter comprises a hyperparameter less than or equal to 0” (this is a mathematical concept, as per paragraph 0034, in the equation
c
l
i
p
p
e
d
s
o
f
t
m
a
x
=
x
;
ξ
,
γ
≔
c
l
i
p
(
ξ
-
γ
∙
s
o
f
t
m
a
x
x
+
γ
,
0,1
)
, two parameters
ξ
and
γ
are defined, wherein
ξ
≥
1
,
and
γ
≤
0
, see MPEP § 2106.05(a)(2)(I)))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“The processing system of Claim 20” (in step 2A, prong 2, this is considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 22
Regarding claim 22, it is dependent upon claim 21, and thereby incorporates the limitations of, and corresponding analysis to claim 21. Further, claim 22 recites the following additional elements:
“the clipped softmax function is configured to output values up to and including a value of 1 when a value of the first hyperparameter is greater than 1” (this is a mathematical concept, as per paragraph 0034, in the equation
c
l
i
p
p
e
d
s
o
f
t
m
a
x
=
x
;
ξ
,
γ
≔
c
l
i
p
(
ξ
-
γ
∙
s
o
f
t
m
a
x
x
+
γ
,
0,1
)
, the function
c
l
i
p
(
)
outputs a value of 1 for inputs larger than 1, see MPEP § 2106.05(a)(2)(I))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“The processing system of claim 21” (in step 2A, prong 2, this is considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 23
Regarding claim 23, it is dependent upon claim 21, and thereby incorporates the limitations of, and corresponding analysis to claim 21. Further, claim 23 recites the following additional elements:
“wherein the clipped softmax function is configured to output values down to and including a value of 0 when a value of the second hyperparameter is less than 0” (this is a mathematical concept, as per paragraph 0034, in the equation
c
l
i
p
p
e
d
s
o
f
t
m
a
x
=
x
;
ξ
,
γ
≔
c
l
i
p
(
ξ
-
γ
∙
s
o
f
t
m
a
x
x
+
γ
,
0,1
)
, the function
c
l
i
p
(
)
outputs a value of 0 for inputs smaller than 0, see MPEP § 2106.05(a)(2)(I))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“The processing system of Claim 21” (in step 2A, prong 2, this is considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 24
Regarding claim 24, it is dependent upon claim 19, and thereby incorporates the limitations of, and corresponding analysis to claim 19. Further, claim 24 recites the following additional elements:
“generating the attention output based on a gated attention block configured to output a minimum value of 0” (this is a mathematical concept; the equation given in paragraph 0039 is
P
^
x
≔
s
i
g
m
o
i
d
G
x
∙
s
o
f
t
m
a
x
(
Q
x
K
x
T
d
)
. In paragraph 0040, it is stated that the output of the softmax function may be between 0 and 1, but not exactly 0 or 1. It is multiplied by a sigmoid function, which may output a value between 0 and 1. Because neither function can output a negative number, the product of the two functions’ outputs must be at least 0, see MPEP § 2106.05(a)(2)(I))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“The processing system of Claim 19, wherein the means for generating the attention output in the transformer neural network comprises means for” (in step 2A, prong 2, this is considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 25
Regarding claim 25, it is dependent upon claim 24, and thereby incorporates the limitations of, and corresponding analysis to claim 24. Further, claim 25 recites the following additional elements:
“the gated attention block applies a bounded nonlinear function to one or more gating parameters defined for the transformer neural network” (this is a mathematical concept; per paragraph 0046 of the specification this may be, for example, a sigmoid function or other nonlinear function having defined maximum and minimum values which may be output by the function, see MPEP § 2106.05(a)(2)(I))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“The processing system of Claim 6” (in step 2A, prong 2, this is considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 26
Regarding claim 26, it is dependent upon claim 25, and thereby incorporates the limitations of, and corresponding analysis to claim 25. Further, claim 26 recites the following additional elements:
“the bounded nonlinear function comprises a sigmoid function” (this is a mathematical concept, see MPEP § 2106.05(a)(2)(I))
“The processing system of Claim 25” (in step 2A, prong 2, this is considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 27
Regarding claim 27, it is dependent upon claim 24, and thereby incorporates the limitations of, and corresponding analysis to claim 24. Further, claim 27 recites the following additional elements:
“wherein the gated attention block is applied for each token generated by the transformer neural network for the received input” (this is a mathematical concept; as per the equation in paragraph 0038
A
X
≔
P
^
(
x
)
V
(
x
)
,
P
^
(
x
)
represents the gated output of a function and
V
(
x
)
represents a value matrix associated with input
x
, wherein input
x
may be a sequence of tokens in a transformer neural network, see MPEP § 2106.05(a)(2)(I))
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mathematical concept but for the recitation of generic computer components, then it falls within the mathematical concepts grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“The processing system of Claim 24” (in step 2A, prong 2, this is considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using a generic computer, see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 28
Regarding claim 28, in step 1 of the 101-analysis set forth in MPEP 2106, the claim recites “a non-transitory computer readable medium.” This is a device or machine, which is one of the four statutory categories of invention.
In step 2A prong 1 of the 101-analysis set forth in MPEP 2106, the examiner has determined that the following limitations recite a process that, under the broadest reasonable interpretation, covers as mathematical concept but for recitation of generic computer components:
“generate an attention output in the transformer neural network, the attention output being generated such that outlier values for the attention output are attenuated in the transformer neural network” (this is a mathematical concept; the attention output, per the specification (see paragraph 0032) is generated using the equation
A
t
t
e
n
t
i
o
n
Q
,
K
,
V
≔
s
o
f
t
m
a
x
(
Q
K
T
d
)
V
), see MPEP § 2106.05(a)(2)(I))
In step 2A prong 2 of the 101-analysis set forth in MPEP 2106, the examiner has determined that the following additional elements do not integrate this judicial exception into a practical application:
i. “A non-transitory computer-readable medium having executable instructions stored thereon which, when executed by one or more processors, cause the one or more processors to perform an operation comprising:” (this is considered mere instructions to apply the exception using a generic computer – see MPEP § 2106.05(g))
ii. “receive an input for processing using a transformer neural network” (receiving an input is considered insignificant extra-solution activity of mere data gathering – see MPEP § 2106.05(g))
iii. “generate an output of the transformer neural network based on the generated attention output” (generating an output is considered mere instructions to apply the exception using a generic computer – see MPEP § 2106.05(g))
In step 2B of the 101-analysis set forth in the 2019 PEG, the examiner has determined that the claim does not include additional elements that are sufficient to amount significantly more than the judicial exception.
As discussed above, the additional element ii recites insignificant extra-solution activity of
mere data gathering, which is a well-understood routine and conventional activity, receiving or
transmitting data over a network, e.g., using the Internet to gather data, see Symantec, 838
F.3d at 1321, 120 USPQ2d at 1362. Additional elements i and iii recites mere instructions to apply the exception using a generic computer.
Considering that the additional elements individually and in combination, and the claim
as a whole, the additional elements do not provide significantly more than the abstract idea.
Therefore, the claim is not patent eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 10, 19, and 28 are rejected under 35 U.S.C. 103 as being unpatentable over Zhen et al. (https://ojs.aaai.org/index.php/AAAI/article/view/20397) (hereinafter Zhen) in view of Spagnoli (US 20220300807 A1).
Claim 1
Regarding claim 1, Zhen teaches “receive an input for processing using a transformer neural network;”
See Zhen in page 4719, col. 1 where it describes “We normalize the vanilla multi-head attention module by multiplying the single head representation
H
i
by multiplying the single head representation
H
i
with a learnable gate parameter
g
i
. Consequently, the proposed GMHA can be written as:
G
-
M
u
l
t
i
H
e
a
d
Q
,
K
,
V
=
C
o
n
c
a
t
g
i
∙
H
i
W
O
, (7)” Here, Zhen teaches an equation that receives input (in this case, a weight matrix
W
O
) that is processed by the DETR (detection transformer). Further see Zhen in page 4716, col. 2 and page 4717, col. 1 where it describes “In this work, we propose to compress the DETR models through low-rank tensor decomposition. The neural network weights can be represented as a combination of multiple very small tensor arrays.” Here, Zhen teaches that the detection transformer is a transformer neural network.
Further Zhen teaches “generate an attention output in the transformer neural network, the attention output being generated such that outlier values for the attention output are attenuated in the transformer neural network;”
See Zhen in page 4719, col 1 where it describes “We normalize the vanilla multi-head attention module by multiplying the single head representation
H
i
by multiplying the single head representation
H
i
with a learnable gate parameter
g
i
. Consequently, the proposed GMHA can be written as:
G
-
M
u
l
t
i
H
e
a
d
Q
,
K
,
V
=
C
o
n
c
a
t
g
i
∙
H
i
W
O
, (7)” Here, Zhen teaches generating output using the gated multi-head attention module with a detection transformer (DETR) model. Because the output is being generated with an attention module, it is therefore an attention output. Further see Zhen in page 4719, col 2 where it describes “Given the learnable parameter set
q
for all heads in a MHA module, our gate parameter set
g
can be derived as follows:
s
=
S
i
g
m
o
i
d
(
(
q
+
l
o
g
u
-
log
1
-
u
)
/
T
)
, (8)
s
-
=
s
λ
-
μ
+
μ
(9),
g
=
c
l
a
m
p
(
s
-
,
0,1
)
(10)… Since we want heads that can be totally opened or closed in GMHA modules, we introduce two parameters
μ
,
λ
that can stretch the valve of s to
(
μ
,
λ
)
with
μ
<
0
and
λ
>
1
as described in Eq. (9). Then we can apply a hard-sigmoid function to clip the value of gate g into range [0, 1] following Eq. (10).” Here, Zhen teaches a series of functions that achieve similar functionality to outlier attenuation, namely by setting the two parameters
λ
and
μ
in equation 9 and by clipping the values of
g
into the range [0, 1] in equation 10. Because the function clips the values of g into the range [0, 1], it can be said that values outside of this range (that is, outlier values) are reduced (attenuated).
However, Zhen did not explicitly teach “a processing system,” “at least one memory having executable instructions stored thereon,” “one or more processors configured to execute the executable instructions in order to cause the processing system to,” or “generate an output of the transformer neural network based on the generated attention output.”
Further, Spagnoli in the same field of art teaches “a processing system;”
See Spagnoli in paragraph 0085 where it describes “The example versions of the embodiments described herein may be implemented as logical operations in a distributed processing system such as the system (1800) including the servers (1815).” Here, Spagnoli teaches a processing system for carrying out the operations.
Further Spagnoli teaches “at least one memory having executable instructions stored thereon;”
See Spagnoli in paragraph 0038 where it describes “Although not required, aspects of the technology may be described herein in the general context of computer-executable instructions, such as routines executed by a general- or special-purpose data processing device (e.g., a server or client computer). Aspects of the technology described herein may be stored or distributed on tangible computer-readable media, including magnetically or optically readable computer discs, hard-wired or preprogrammed chips (e.g., EEPROM semiconductor chips or chip sets), nanotechnology memory, biological memory, or other data storage media.” Here, Spagnoli describes storing instructions on a type of memory.
Further Spagnoli teaches “one or more processors configured to execute the executable instructions in order to cause the processing system to;”
See Spagnoli in paragraph 0076 where it describes “One or more processors in a multi-processing arrangement may also be employed to execute the sequences of instructions contained in the main memory (1804).” Here, Spagnoli describes processors that execute instructions.
Further Spagnoli teaches “generate an output of the transformer neural network based on the generated attention output.”
See Spagnoli in paragraph 0055 where it describes “The Decoder Multi-Head Attention Layer (1000c) also receives the output from the final Encoder Block (803), and then passes the processed output therefrom to the next, or fourth, Add & Normalize Layer (903d), which adds it with the original input received by the Decoder Multi-Head Attention Layer (1000). The fourth Add & Normalize Layer (903d) normalizes the input and output and passes it to both a second Linear Layer (904b) and a fifth Add & Normalize Layer (903e). The final or fifth Add & Normalize Layer (903) sends the output thereof to the next Decoder Block (804), or if it is the final Decoder BlockN.sub.x (804), the output is provided to one or more Output Linear Layers (907) and then to the final Output Layer (908) to create the Final Output (801) of the Transformer Neural Network (800).” Here, Spagnoli describes generating a final output of a transformer neural network using an output that has been processed through an attention layer.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen with the teachings of Spagnoli by using Zhen’s teachings of receiving an input using a transformer neural network, generating an attention output in the TNN such that outliers are attenuated, and combine with Spagnoli’s teachings of incorporating a processing system, a memory, and processors, and generating an output based on the attention output.
One of ordinary skill in the art would be motivated to do so because by integrating Spagnoli’s frameworks into the methods of Zhen, one with ordinary skill in the art would “allow faster and cheaper computation… capable of Multi-task learning” (Spagnoli, paragraph 0042).
Claim 10
Regarding claim 10, Zhen teaches “receiving an input for processing using a transformer neural network;”
See Zhen in page 4719, col. 1 where it describes “We normalize the vanilla multi-head attention module by multiplying the single head representation
H
i
by multiplying the single head representation
H
i
with a learnable gate parameter
g
i
. Consequently, the proposed GMHA can be written as:
G
-
M
u
l
t
i
H
e
a
d
Q
,
K
,
V
=
C
o
n
c
a
t
g
i
∙
H
i
W
O
, (7)” Here, Zhen teaches an equation that receives input (in this case, a weight matrix
W
O
) that is processed by the DETR (detection transformer). Further see Zhen in page 4716, col. 2 and page 4717, col. 1 where it describes “In this work, we propose to compress the DETR models through low-rank tensor decomposition. The neural network weights can be represented as a combination of multiple very small tensor arrays.” Here, Zhen teaches that the detection transformer is a transformer neural network.
Further Zhen teaches “generating an attention output in the transformer neural network, the attention output being generated such that outlier values for the attention output are attenuated in the transformer neural network;”
See Zhen in page 4719, col 1 where it describes “We normalize the vanilla multi-head attention module by multiplying the single head representation
H
i
by multiplying the single head representation
H
i
with a learnable gate parameter
g
i
. Consequently, the proposed GMHA can be written as:
G
-
M
u
l
t
i
H
e
a
d
Q
,
K
,
V
=
C
o
n
c
a
t
g
i
∙
H
i
W
O
, (7)” Here, Zhen teaches generating output using the gated multi-head attention module with a detection transformer (DETR) model. Because the output is being generated with an attention module, it is therefore an attention output. Further see Zhen in page 4719, col 2 where it describes “Given the learnable parameter set
q
for all heads in a MHA module, our gate parameter set
g
can be derived as follows:
s
=
S
i
g
m
o
i
d
(
(
q
+
l
o
g
u
-
log
1
-
u
)
/
T
)
, (8)
s
-
=
s
λ
-
μ
+
μ
(9),
g
=
c
l
a
m
p
(
s
-
,
0,1
)
(10)… Since we want heads that can be totally opened or closed in GMHA modules, we introduce two parameters
μ
,
λ
that can stretch the valve of s to
(
μ
,
λ
)
with
μ
<
0
and
λ
>
1
as described in Eq. (9). Then we can apply a hard-sigmoid function to clip the value of gate g into range [0, 1] following Eq. (10).” Here, Zhen teaches a series of functions that achieve similar functionality to outlier attenuation, namely by setting the two parameters
λ
and
μ
in equation 9 and by clipping the values of
g
into the range [0, 1] in equation 10. Because the function clips the values of g into the range [0, 1], it can be said that values outside of this range (that is, outlier values) are reduced (attenuated).
However, Zhen did not explicitly teach “a processor-implemented method” or “generating an output of the transformer neural network based on the generated attention output.”
Further, Spagnoli in the same field of art teaches “a processor-implemented method;”
See Spagnoli in paragraph 0026 where it describes “In One aspect, the described technology concerns one or more methods, systems, apparatuses, and mediums storing processor-executable process.” Here, Spagnoli teaches methods carried out by processors.
Further Spagnoli teaches “generating an output of the transformer neural network based on the generated attention output.”
See Spagnoli in paragraph 0055 where it describes “The Decoder Multi-Head Attention Layer (1000c) also receives the output from the final Encoder Block (803), and then passes the processed output therefrom to the next, or fourth, Add & Normalize Layer (903d), which adds it with the original input received by the Decoder Multi-Head Attention Layer (1000). The fourth Add & Normalize Layer (903d) normalizes the input and output and passes it to both a second Linear Layer (904b) and a fifth Add & Normalize Layer (903e). The final or fifth Add & Normalize Layer (903) sends the output thereof to the next Decoder Block (804), or if it is the final Decoder BlockN.sub.x (804), the output is provided to one or more Output Linear Layers (907) and then to the final Output Layer (908) to create the Final Output (801) of the Transformer Neural Network (800).” Here, Spagnoli describes generating a final output of a transformer neural network using an output that has been processed through an attention layer.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen with the teachings of Spagnoli by using Zhen’s teachings of receiving an input using a transformer neural network, generating an attention output in the TNN such that outliers are attenuated, and combine with Spagnoli’s teachings of a processor-implemented method and generating an output based on the attention output.
One of ordinary skill in the art would be motivated to do so because by integrating Spagnoli’s frameworks into the methods of Zhen, one with ordinary skill in the art would “allow faster and cheaper computation… capable of Multi-task learning” (Spagnoli, paragraph 0042).
Claim 19
Regarding claim 19, Zhen teaches “means for receiving an input for processing using a transformer neural network;”
See Zhen in page 4719, col. 1 where it describes “We normalize the vanilla multi-head attention module by multiplying the single head representation
H
i
by multiplying the single head representation
H
i
with a learnable gate parameter
g
i
. Consequently, the proposed GMHA can be written as:
G
-
M
u
l
t
i
H
e
a
d
Q
,
K
,
V
=
C
o
n
c
a
t
g
i
∙
H
i
W
O
, (7)” Here, Zhen teaches an equation that receives input (in this case, a weight matrix
W
O
) that is processed by the DETR (detection transformer). Further see Zhen in page 4716, col. 2 and page 4717, col. 1 where it describes “In this work, we propose to compress the DETR models through low-rank tensor decomposition. The neural network weights can be represented as a combination of multiple very small tensor arrays.” Here, Zhen teaches that the detection transformer is a transformer neural network.
Further Zhen teaches “means for generating an attention output in the transformer neural network, the attention output being generated such that outlier values for the attention output are attenuated in the transformer neural network;”
See Zhen in page 4719, col 1 where it describes “We normalize the vanilla multi-head attention module by multiplying the single head representation
H
i
by multiplying the single head representation
H
i
with a learnable gate parameter
g
i
. Consequently, the proposed GMHA can be written as:
G
-
M
u
l
t
i
H
e
a
d
Q
,
K
,
V
=
C
o
n
c
a
t
g
i
∙
H
i
W
O
, (7)” Here, Zhen teaches generating output using the gated multi-head attention module with a detection transformer (DETR) model. Because the output is being generated with an attention module, it is therefore an attention output. Further see Zhen in page 4719, col 2 where it describes “Given the learnable parameter set
q
for all heads in a MHA module, our gate parameter set
g
can be derived as follows:
s
=
S
i
g
m
o
i
d
(
(
q
+
l
o
g
u
-
log
1
-
u
)
/
T
)
, (8)
s
-
=
s
λ
-
μ
+
μ
(9),
g
=
c
l
a
m
p
(
s
-
,
0,1
)
(10)… Since we want heads that can be totally opened or closed in GMHA modules, we introduce two parameters
μ
,
λ
that can stretch the valve of s to
(
μ
,
λ
)
with
μ
<
0
and
λ
>
1
as described in Eq. (9). Then we can apply a hard-sigmoid function to clip the value of gate g into range [0, 1] following Eq. (10).” Here, Zhen teaches a series of functions that achieve similar functionality to outlier attenuation, namely by setting the two parameters
λ
and
μ
in equation 9 and by clipping the values of
g
into the range [0, 1] in equation 10. Because the function clips the values of g into the range [0, 1], it can be said that values outside of this range (that is, outlier values) are reduced (attenuated).
However, Zhen did not explicitly teach “a processing system” or “means for generating an output of the transformer neural network based on the generated attention output.”
Further, Spagnoli in the same field of art teaches “a processing system;”
See Spagnoli in paragraph 0085 where it describes “The example versions of the embodiments described herein may be implemented as logical operations in a distributed processing system such as the system (1800) including the servers (1815).” Here, Spagnoli teaches a processing system for carrying out the operations.
Further Spagnoli teaches “means for generating an output of the transformer neural network based on the generated attention output.”
See Spagnoli in paragraph 0055 where it describes “The Decoder Multi-Head Attention Layer (1000c) also receives the output from the final Encoder Block (803), and then passes the processed output therefrom to the next, or fourth, Add & Normalize Layer (903d), which adds it with the original input received by the Decoder Multi-Head Attention Layer (1000). The fourth Add & Normalize Layer (903d) normalizes the input and output and passes it to both a second Linear Layer (904b) and a fifth Add & Normalize Layer (903e). The final or fifth Add & Normalize Layer (903) sends the output thereof to the next Decoder Block (804), or if it is the final Decoder BlockN.sub.x (804), the output is provided to one or more Output Linear Layers (907) and then to the final Output Layer (908) to create the Final Output (801) of the Transformer Neural Network (800).” Here, Spagnoli describes generating a final output of a transformer neural network using an output that has been processed through an attention layer.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen with the teachings of Spagnoli by using Zhen’s teachings of receiving an input using a transformer neural network, generating an attention output in the TNN such that outliers are attenuated, and combine with Spagnoli’s teachings of a processing system and generating an output based on the attention output.
One of ordinary skill in the art would be motivated to do so because by integrating Spagnoli’s frameworks into the methods of Zhen, one with ordinary skill in the art would “allow faster and cheaper computation… capable of Multi-task learning” (Spagnoli, paragraph 0042).
Claim 28
Regarding claim 28, Zhen teaches “receiving an input for processing using a transformer neural network;”
See Zhen in page 4719, col. 1 where it describes “We normalize the vanilla multi-head attention module by multiplying the single head representation
H
i
by multiplying the single head representation
H
i
with a learnable gate parameter
g
i
. Consequently, the proposed GMHA can be written as:
G
-
M
u
l
t
i
H
e
a
d
Q
,
K
,
V
=
C
o
n
c
a
t
g
i
∙
H
i
W
O
, (7)” Here, Zhen teaches an equation that receives input (in this case, a weight matrix
W
O
) that is processed by the DETR (detection transformer). Further see Zhen in page 4716, col. 2 and page 4717, col. 1 where it describes “In this work, we propose to compress the DETR models through low-rank tensor decomposition. The neural network weights can be represented as a combination of multiple very small tensor arrays.” Here, Zhen teaches that the detection transformer is a transformer neural network.
Further Zhen teaches “generating an attention output in the transformer neural network, the attention output being generated such that outlier values for the attention output are attenuated in the transformer neural network;”
See Zhen in page 4719, col 1 where it describes “We normalize the vanilla multi-head attention module by multiplying the single head representation
H
i
by multiplying the single head representation
H
i
with a learnable gate parameter
g
i
. Consequently, the proposed GMHA can be written as:
G
-
M
u
l
t
i
H
e
a
d
Q
,
K
,
V
=
C
o
n
c
a
t
g
i
∙
H
i
W
O
, (7)” Here, Zhen teaches generating output using the gated multi-head attention module with a detection transformer (DETR) model. Because the output is being generated with an attention module, it is therefore an attention output. Further see Zhen in page 4719, col 2 where it describes “Given the learnable parameter set
q
for all heads in a MHA module, our gate parameter set
g
can be derived as follows:
s
=
S
i
g
m
o
i
d
(
(
q
+
l
o
g
u
-
log
1
-
u
)
/
T
)
, (8)
s
-
=
s
λ
-
μ
+
μ
(9),
g
=
c
l
a
m
p
(
s
-
,
0,1
)
(10)… Since we want heads that can be totally opened or closed in GMHA modules, we introduce two parameters
μ
,
λ
that can stretch the valve of s to
(
μ
,
λ
)
with
μ
<
0
and
λ
>
1
as described in Eq. (9). Then we can apply a hard-sigmoid function to clip the value of gate g into range [0, 1] following Eq. (10).” Here, Zhen teaches a series of functions that achieve similar functionality to outlier attenuation, namely by setting the two parameters
λ
and
μ
in equation 9 and by clipping the values of
g
into the range [0, 1] in equation 10. Because the function clips the values of g into the range [0, 1], it can be said that values outside of this range (that is, outlier values) are reduced (attenuated).
However, Zhen did not explicitly teach “a non-transitory computer-readable medium having executable instructions stored thereon which, when executed by one or more processors, cause the one or more processors to perform an operation comprising” or “generating an output of the transformer neural network based on the generated attention output.”
Further, Spagnoli in the same field of art teaches “a non-transitory computer-readable medium having executable instructions stored thereon which, when executed by one or more processors, cause the one or more processors to perform an operation comprising;”
See Spagnoli in paragraph 0078 where it describes “Generally, the term “computer-readable medium” as used herein refers to any medium that participated in providing instructions to the processor (1808) for execution.” Here, Spagnoli teaches computer-readable mediums that store instructions.
Further Spagnoli teaches “generating an output of the transformer neural network based on the generated attention output.”
See Spagnoli in paragraph 0055 where it describes “The Decoder Multi-Head Attention Layer (1000c) also receives the output from the final Encoder Block (803), and then passes the processed output therefrom to the next, or fourth, Add & Normalize Layer (903d), which adds it with the original input received by the Decoder Multi-Head Attention Layer (1000). The fourth Add & Normalize Layer (903d) normalizes the input and output and passes it to both a second Linear Layer (904b) and a fifth Add & Normalize Layer (903e). The final or fifth Add & Normalize Layer (903) sends the output thereof to the next Decoder Block (804), or if it is the final Decoder BlockN.sub.x (804), the output is provided to one or more Output Linear Layers (907) and then to the final Output Layer (908) to create the Final Output (801) of the Transformer Neural Network (800).” Here, Spagnoli describes generating a final output of a transformer neural network using an output that has been processed through an attention layer.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen with the teachings of Spagnoli by using Zhen’s teachings of receiving an input using a transformer neural network, generating an attention output in the TNN such that outliers are attenuated, and combine with Spagnoli’s teachings of a computer-readable medium storing instructions and generating an output based on the attention output.
One of ordinary skill in the art would be motivated to do so because by integrating Spagnoli’s frameworks into the methods of Zhen, one with ordinary skill in the art would “allow faster and cheaper computation… capable of Multi-task learning” (Spagnoli, paragraph 0042).
Claims 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, and 27 are rejected under 35 U.S.C. 103 as being unpatentable over Zhen in view of Spagnoli, and further in view of Long et al. (https://link.springer.com/chapter/10.1007/978-3-030-91560-5_1) (hereinafter Long).
Claim 2 Regarding claim 2, Zhen in view of Spagnoli teaches the limitations in claim 1.
Further Zhen teaches “to generate the attention output in the transformer neural network, the one or more processors are configured to cause the processing system to generate the attention output based on a clipped … function having a dynamic range controlled by a first hyperparameter and a second hyperparameter.”
See Zhen in page 4719, cols. 1-2 where it describes “Multi-head attention (MHA) is a powerful and ubiquitous model in the DETR model, which allows the network to obtain information from different mapping spaces. In vanilla DETR models, we usually have 8 heads in each attention module. However, not all the heads contribute equally for making predictions. The redundant information causes the DETR model to overfit. We normalize the vanilla multi-head attention module by multiplying the single head representation
H
i
with a learnable gate parameter
g
i
.” Here, Zhen teaches generating an attention output with a function. Further see Zhen in page 4719, col. 2 where it describes “Given the learnable parameter set
q
for all heads in a MHA module, our gate parameter set
g
can be derived as follows:
s
=
S
i
g
m
o
i
d
(
(
q
+
log
u
-
l
o
g
(
1
-
u
)
)
/
T
)
,
s
-
=
s
(
λ
-
μ
)
+
μ
,
g
=
c
l
a
m
p
(
s
-
,
0,1
)
.” Here, Zhen teaches generating the gate parameters (or in other words, output) using a function. Further see Zhen in page 4719, col. 2 where it describes “Since we want the heads can be totally opened or closed in GMHA modules, we introduce two parameters
μ
,
λ
that can stretch the value of
s
to
(
μ
,
λ
)
with
μ
<
0
and
λ
>
1
as described in Eq. (9). Then we can apply a hard-sigmoid function to clip the value of gate
g
into range
[
0
,
1
]
following Eq. (10).” Here, Zhen describes clipping the function using two hyperparameters.
Neither Zhen or Spagnoli appear to teach “a softmax function.”
However, Long in the same field of art teaches “a softmax function.”
See Long in page 9 where it describes “Then we distinguish the importance of feature interactions by using both the attention mechanism and the gating mechanism:
α
i
=
s
o
f
t
m
a
x
(
Q
i
K
i
T
d
K
)
(7)” Here, Long teaches a softmax function used as part of an attention mechanism.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen and Spagnoli to incorporate the teachings of Long and incorporate a softmax function when generating the attention output.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, and incorporate with Long’s teachings of using a softmax equation in the attention output. One of ordinary skill in the art would be motivated to do so because by integrating Long’s frameworks into the methods of Zhen and Spagnoli, one of ordinary skill in the art would have a mechanism that “proved effective in different domains, such as machine translation and recommender systems” (Long, page 9).
Claim 3
Regarding claim 3, Zhen in view of Spagnoli in view of Long teaches the limitations in claim 2.
Further Zhen teaches “the first hyperparameter comprises a hyperparameter greater than or equal to 1 and wherein the second hyperparameter comprises a hyperparameter less than or equal to 0.”
See Zhen in page 4719, col. 2 where it describes “Since we want the heads can be totally opened or closed in GMHA modules, we introduce two parameters
μ
,
λ
that can stretch the value of
s
to
(
μ
,
λ
)
with
μ
<
0
and
λ
>
1
as described in Eq. (9).” Here, Zhen describes setting the parameter
μ
to less than 0 and the parameter
λ
to greater than 1.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen, Spagnoli, and Long to incorporate the teachings of Zhen and have two hyperparameters, one less than 0 and one greater than 1.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, and using a softmax equation in the attention output, and further incorporate two hyperparameters, one less than 0 and one greater than 1. One of ordinary skill in the art would be motivated to do so because by integrating Zhen’s frameworks into the methods of Zhen, Spagnoli, and Long, one of ordinary skill in the art would “(help) the vanilla DETR model achieve 1.6 points AP improvement” (Zhen, page 4721, col. 1).
Claim 4
Regarding claim 4, Zhen in view of Spagnoli in view of Long teaches the limitations in claim 3.
Further Zhen teaches “the clipped… function is configured to output values up to and including a value of 1 when a value of the first hyperparameter is greater than 1.”
See Zhen in page 4719, col. 2 where it describes “Then we can apply a hard-sigmoid function to clip the value of gate g into range [0, 1] following Eq. (10).” Here, Zhen describes a function that can output values in the range from 0 to 1, or in other words up to and including 1.
Further Long teaches “softmax function;”
See Long in page 9 where it describes “Then we distinguish the importance of feature interactions by using both the attention mechanism and the gating mechanism:
α
i
=
s
o
f
t
m
a
x
(
Q
i
K
i
T
d
K
)
(7)” Here, Long teaches a softmax function used as part of an attention mechanism.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen, Spagnoli, and Long to incorporate the teachings of Zhen and incorporate a function that output values up to and including 1 when the first hyperparameter is greater than 1, and further incorporate Long’s teachings of using a softmax function.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, incorporates two hyperparameters, one less than 0 and one greater than 1, and a function that outputs a value up to and including 1 if the first hyperparameter is greater than 1, and further incorporate Long’s teachings of a softmax function. One of ordinary skill in the art would be motivated to do so because by integrating Zhen and Long’s frameworks into the methods of Zhen, Spagnoli, and Long, one of ordinary skill in the art would “(help) the vanilla DETR model achieve 1.6 points AP improvement” (Zhen, page 4721, col. 1).
Claim 5
Regarding claim 5, Zhen in view of Spagnoli in view of Long teaches the limitations in claim 3.
Further Zhen teaches “the clipped… function is configured to output values down to and including a value of 0 when a value of the second hyperparameter is less than 0.”
See Zhen in page 4719, col. 2 where it describes “Then we can apply a hard-sigmoid function to clip the value of gate g into range [0, 1] following Eq. (10).” Here, Zhen describes a function that can output values in the range from 0 to 1, or in other words down to and including 0.
Further Long teaches “softmax function;”
See Long in page 9 where it describes “Then we distinguish the importance of feature interactions by using both the attention mechanism and the gating mechanism:
α
i
=
s
o
f
t
m
a
x
(
Q
i
K
i
T
d
K
)
(7)” Here, Long teaches a softmax function used as part of an attention mechanism.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen, Spagnoli, and Long to incorporate the teachings of Zhen and incorporate a function that output values down to and including 0 when the second hyperparameter is less than 0, and further incorporate Long’s teachings of using a softmax function.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, incorporates two hyperparameters, one less than 0 and one greater than 1, and a function that outputs a value down to and including 0 if the second hyperparameter is less than 0, and further incorporate Long’s teachings of a softmax function. One of ordinary skill in the art would be motivated to do so because by integrating Zhen and Long’s frameworks into the methods of Zhen, Spagnoli, and Long, one of ordinary skill in the art would “(help) the vanilla DETR model achieve 1.6 points AP improvement” (Zhen, page 4721, col. 1).
Claim 6
Regarding claim 6, Zhen in view of Spagnoli teaches the limitations in claim 1.
Neither Zhen or Spagnoli appear to teach “to generate the attention output in the transformer neural network, the one or more processors are configured to cause the processing system to generate the attention output based on a gated attention block configured to output a minimum value of 0.”
However, Long teaches “to generate the attention output in the transformer neural network, the one or more processors are configured to cause the processing system to generate the attention output based on a gated attention block configured to output a minimum value of 0.”
See Long in page 9 where it describes “Specifically, for the gated multi-head attention layer of the k-th encoder, the inputs are the representations of k-order cross features and the embedding vectors of raw features, which are denoted as
H
(
k
)
∈
R
m
×
d
and
H
(
1
)
∈
R
m
×
d
respectively… Finally, we obtain the representations by concatenating the outputs of all heads:
H
i
k
*
=
g
i
⊙
α
i
V
i
(9),
H
k
*
=
N
o
r
m
(
[
H
i
k
*
H
2
k
*
…
H
h
k
*
+
H
1
)
, (10),
V
i
=
W
i
V
H
k
, (11), where
W
i
v
∈
R
d
K
×
d
is the parameter of head
i
,
h
is the number of heads, and
N
o
r
m
represents the normalization.” Here, Long teaches generating attention outputs based on a gated attention layer’s outputs that are at least 0. Specifically in equation 9, the softmax function
α
i
is being multiplied by the sigmoid
g
i
, and since both of these functions will return values between 0 and 1 (as described in pages 6-7: “each record of the dataset is consisted of m categorical features and an associated label
y
∈
{
0,1
}
"
)
, will return a value of at least 0 when multiplied by each other.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen and Spagnoli to incorporate the teachings of Long and incorporate generating an attention output based on a gated attention block configured to output a minimum value of 0.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, and incorporate with Long’s teachings of generating an attention output based on a gated attention block configured to output a minimum value of 0. One of ordinary skill in the art would be motivated to do so because by integrating Long’s frameworks into the methods of Zhen and Spagnoli, one of ordinary skill in the art would have a mechanism that “proved effective in different domains, such as machine translation and recommender systems” (Long, page 9).
Claim 7
Regarding claim 7, Zhen in view of Spagnoli in view of Long teaches the limitations in claim 6.
Neither Zhen or Spagnoli appear to teach “the gated attention block applies a bounded nonlinear function to one or more gating parameters defined for the transformer neural network.”
However, Long teaches “the gated attention block applies a bounded nonlinear function to one or more gating parameters defined for the transformer neural network.”
See Long in page 9 where it describes “Specifically, for the gated multi-head attention layer of the k-th encoder, the inputs are the representations of k-order cross features and the embedding vectors of raw features, which are denoted as
H
(
k
)
∈
R
m
×
d
and
H
(
1
)
∈
R
m
×
d
respectively. In contrast to the vanilla multi-head attention network, we integrate a neural gating structure:
g
i
=
s
i
g
m
o
i
d
(
G
a
t
e
N
e
t
1
(
H
k
H
1
)
(6), where || denotes the concatenation operation,
G
a
t
e
N
e
t
1
is a feed-forward neural network, and
g
i
is the gaet learned for head
i
.” Here, Long describes applying a sigmoid function, which is a bounded nonlinear function, to the gating parameters.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen and Spagnoli to incorporate the teachings of Long and incorporate a bounded nonlinear function applied by a gated attention block.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, and incorporate with Long’s teachings of a bounded nonlinear function applied by a gated attention block. One of ordinary skill in the art would be motivated to do so because by integrating Long’s frameworks into the methods of Zhen and Spagnoli, one of ordinary skill in the art would have a mechanism that “proved effective in different domains, such as machine translation and recommender systems” (Long, page 9).
Claim 8
Regarding claim 8, Zhen in view of Spagnoli in view of Long teaches the limitations of claim 7.
Neither Zhen or Spagnoli appear to teach “the bounded nonlinear function comprises a sigmoid function.”
However, Long teaches “the bounded nonlinear function comprises a sigmoid function.”
See Long in page 9, equation 6 where it describes “In contrast to the vanilla multi-head attention network, we integrate a neural gating structure:
g
i
=
s
i
g
m
o
i
d
(
G
a
t
e
N
e
t
1
H
k
H
1
)
Here, Long describes using sigmoid function.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen and Spagnoli to incorporate the teachings of Long and incorporate a bounded nonlinear function that is a sigmoid function.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, and incorporate with Long’s teachings of a sigmoid bounded nonlinear function. One of ordinary skill in the art would be motivated to do so because by integrating Long’s frameworks into the methods of Zhen and Spagnoli, one of ordinary skill in the art would have a mechanism that “proved effective in different domains, such as machine translation and recommender systems” (Long, page 9).
Claim 9
Regarding claim 9, Zhen in view of Spagnoli in view of Long teaches the limitations in claim 6.
Neither Zhen or Spagnoli appear to teach “the gated attention block is applied for each token generated by the transformer neural network for the received input.”
However, Long teaches “the gated attention block is applied for each token generated by the transformer neural network for the received input.”
See Long in page 9 where it teaches “Then, we distinguish the importance of feature interactions by using both the attention mechanism and the gating mechanism…
Q
i
=
W
i
Q
H
k
,
K
i
=
W
i
K
H
1
(8) where
W
i
Q
∈
R
d
K
×
d
are parameters to learn for head
i
,
d
K
denotes the dimension of each head, and
α
i
is the final importance of cross features.” Here, Long teaches applying the attention layer for each of the received input parameters.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen and Spagnoli to incorporate the teachings of Long and incorporate applying a gated attention block for each input token received in the neural network.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, and incorporate with Long’s teachings of applying a gated attention block for each input token received in the neural network. One of ordinary skill in the art would be motivated to do so because by integrating Long’s frameworks into the methods of Zhen and Spagnoli, one of ordinary skill in the art would have a mechanism that “proved effective in different domains, such as machine translation and recommender systems” (Long, page 9).
Claim 11 Regarding claim 11, Zhen in view of Spagnoli teaches the limitations in claim 10.
Further Zhen teaches “generating the attention output in the transformer neural network comprises generating the attention output based on a clipped… function having a dynamic range controlled by a first hyperparameter and a second hyperparameter.”
See Zhen in page 4719, cols. 1-2 where it describes “Multi-head attention (MHA) is a powerful and ubiquitous model in the DETR model, which allows the network to obtain information from different mapping spaces. In vanilla DETR models, we usually have 8 heads in each attention module. However, not all the heads contribute equally for making predictions. The redundant information causes the DETR model to overfit. We normalize the vanilla multi-head attention module by multiplying the single head representation
H
i
with a learnable gate parameter
g
i
.” Here, Zhen teaches generating an attention output with a function. Further see Zhen in page 4719, col. 2 where it describes “Given the learnable parameter set
q
for all heads in a MHA module, our gate parameter set
g
can be derived as follows:
s
=
S
i
g
m
o
i
d
(
(
q
+
log
u
-
l
o
g
(
1
-
u
)
)
/
T
)
,
s
-
=
s
(
λ
-
μ
)
+
μ
,
g
=
c
l
a
m
p
(
s
-
,
0,1
)
.” Here, Zhen teaches generating the gate parameters (or in other words, output) using a function. Further see Zhen in page 4719, col. 2 where it describes “Since we want the heads can be totally opened or closed in GMHA modules, we introduce two parameters
μ
,
λ
that can stretch the value of
s
to
(
μ
,
λ
)
with
μ
<
0
and
λ
>
1
as described in Eq. (9). Then we can apply a hard-sigmoid function to clip the value of gate
g
into range
[
0
,
1
]
following Eq. (10).” Here, Zhen describes clipping the function using two hyperparameters.
Neither Zhen or Spagnoli appear to teach “a softmax function.”
However, Long in the same field of art teaches “a softmax function.”
See Long in page 9 where it describes “Then we distinguish the importance of feature interactions by using both the attention mechanism and the gating mechanism:
α
i
=
s
o
f
t
m
a
x
(
Q
i
K
i
T
d
K
)
(7)” Here, Long teaches a softmax function used as part of an attention mechanism.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen and Spagnoli to incorporate the teachings of Long and incorporate a softmax function when generating the attention output.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, and incorporate with Long’s teachings of using a softmax equation in the attention output. One of ordinary skill in the art would be motivated to do so because by integrating Long’s frameworks into the methods of Zhen and Spagnoli, one of ordinary skill in the art would have a mechanism that “proved effective in different domains, such as machine translation and recommender systems” (Long, page 9).
Claim 12
Regarding claim 12, Zhen in view of Spagnoli in view of Long teaches the limitations in claim 11.
Further Zhen teaches “the first hyperparameter comprises a hyperparameter greater than or equal to 1 and wherein the second hyperparameter comprises a hyperparameter less than or equal to 0.”
See Zhen in page 4719, col. 2 where it describes “Since we want the heads can be totally opened or closed in GMHA modules, we introduce two parameters
μ
,
λ
that can stretch the value of
s
to
(
μ
,
λ
)
with
μ
<
0
and
λ
>
1
as described in Eq. (9).” Here, Zhen describes setting the parameter
μ
to less than 0 and the parameter
λ
to greater than 1.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen, Spagnoli, and Long to incorporate the teachings of Zhen and have two hyperparameters, one less than 0 and one greater than 1.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, and using a softmax equation in the attention output, and further incorporate two hyperparameters, one less than 0 and one greater than 1. One of ordinary skill in the art would be motivated to do so because by integrating Zhen’s frameworks into the methods of Zhen, Spagnoli, and Long, one of ordinary skill in the art would “(help) the vanilla DETR model achieve 1.6 points AP improvement” (Zhen, page 4721, col. 1).
Claim 13
Regarding claim 13, Zhen in view of Spagnoli in view of Long teaches the limitations in claim 12.
Further Zhen teaches “the clipped… function is configured to output values up to and including a value of 1 when a value of the first hyperparameter is greater than 1.”
See Zhen in page 4719, col. 2 where it describes “Then we can apply a hard-sigmoid function to clip the value of gate g into range [0, 1] following Eq. (10).” Here, Zhen describes a function that can output values in the range from 0 to 1, or in other words up to and including 1.
Further Long teaches “softmax function;”
See Long in page 9 where it describes “Then we distinguish the importance of feature interactions by using both the attention mechanism and the gating mechanism:
α
i
=
s
o
f
t
m
a
x
(
Q
i
K
i
T
d
K
)
(7)” Here, Long teaches a softmax function used as part of an attention mechanism.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen, Spagnoli, and Long to incorporate the teachings of Zhen and incorporate a function that output values up to and including 1 when the first hyperparameter is greater than 1, and further incorporate Long’s teachings of using a softmax function.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, incorporates two hyperparameters, one less than 0 and one greater than 1, and a function that outputs a value up to and including 1 if the first hyperparameter is greater than 1, and further incorporate Long’s teachings of a softmax function. One of ordinary skill in the art would be motivated to do so because by integrating Zhen and Long’s frameworks into the methods of Zhen, Spagnoli, and Long, one of ordinary skill in the art would “(help) the vanilla DETR model achieve 1.6 points AP improvement” (Zhen, page 4721, col. 1).
Claim 14
Regarding claim 14, Zhen in view of Spagnoli in view of Long teaches the limitations in claim 12.
Further Zhen teaches “the clipped… function is configured to output values down to and including a value of 0 when a value of the second hyperparameter is less than 0.”
See Zhen in page 4719, col. 2 where it describes “Then we can apply a hard-sigmoid function to clip the value of gate g into range [0, 1] following Eq. (10).” Here, Zhen describes a function that can output values in the range from 0 to 1, or in other words down to and including 0.
Further Long teaches “softmax function;”
See Long in page 9 where it describes “Then we distinguish the importance of feature interactions by using both the attention mechanism and the gating mechanism:
α
i
=
s
o
f
t
m
a
x
(
Q
i
K
i
T
d
K
)
(7)” Here, Long teaches a softmax function used as part of an attention mechanism.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen, Spagnoli, and Long to incorporate the teachings of Zhen and incorporate a function that output values down to and including 0 when the second hyperparameter is less than 0, and further incorporate Long’s teachings of using a softmax function.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, incorporates two hyperparameters, one less than 0 and one greater than 1, and a function that outputs a value down to and including 0 if the second hyperparameter is less than 0, and further incorporate Long’s teachings of a softmax function. One of ordinary skill in the art would be motivated to do so because by integrating Zhen and Long’s frameworks into the methods of Zhen, Spagnoli, and Long, one of ordinary skill in the art would “(help) the vanilla DETR model achieve 1.6 points AP improvement” (Zhen, page 4721, col. 1).
Claim 15
Regarding claim 15, Zhen in view of Spagnoli teaches the limitations in claim 10.
Neither Zhen or Spagnoli appear to teach “generating the attention output in the transformer neural network comprises generating the attention output based on a gated attention block configured to output a minimum value of 0.”
However, Long teaches “generating the attention output in the transformer neural network comprises generating the attention output based on a gated attention block configured to output a minimum value of 0.”
See Long in page 9 where it describes “Specifically, for the gated multi-head attention layer of the k-th encoder, the inputs are the representations of k-order cross features and the embedding vectors of raw features, which are denoted as
H
(
k
)
∈
R
m
×
d
and
H
(
1
)
∈
R
m
×
d
respectively… Finally, we obtain the representations by concatenating the outputs of all heads:
H
i
k
*
=
g
i
⊙
α
i
V
i
(9),
H
k
*
=
N
o
r
m
(
[
H
i
k
*
H
2
k
*
…
H
h
k
*
+
H
1
)
, (10),
V
i
=
W
i
V
H
k
, (11), where
W
i
v
∈
R
d
K
×
d
is the parameter of head
i
,
h
is the number of heads, and
N
o
r
m
represents the normalization.” Here, Long teaches generating attention outputs based on a gated attention layer’s outputs that are at least 0. Specifically in equation 9, the softmax function
α
i
is being multiplied by the sigmoid
g
i
, and since both of these functions will return values between 0 and 1 (as described in pages 6-7: “each record of the dataset is consisted of m categorical features and an associated label
y
∈
{
0,1
}
"
)
, will return a value of at least 0 when multiplied by each other.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen and Spagnoli to incorporate the teachings of Long and incorporate generating an attention output based on a gated attention block configured to output a minimum value of 0.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, and incorporate with Long’s teachings of generating an attention output based on a gated attention block configured to output a minimum value of 0. One of ordinary skill in the art would be motivated to do so because by integrating Long’s frameworks into the methods of Zhen and Spagnoli, one of ordinary skill in the art would have a mechanism that “proved effective in different domains, such as machine translation and recommender systems” (Long, page 9).
Claim 16
Regarding claim 16, Zhen in view of Spagnoli in view of Long teaches the limitations in claim 15.
Neither Zhen or Spagnoli appear to teach “the gated attention block applies a bounded nonlinear function to one or more gating parameters defined for the transformer neural network.”
However, Long teaches “the gated attention block applies a bounded nonlinear function to one or more gating parameters defined for the transformer neural network.”
See Long in page 9 where it describes “Specifically, for the gated multi-head attention layer of the k-th encoder, the inputs are the representations of k-order cross features and the embedding vectors of raw features, which are denoted as
H
(
k
)
∈
R
m
×
d
and
H
(
1
)
∈
R
m
×
d
respectively. In contrast to the vanilla multi-head attention network, we integrate a neural gating structure:
g
i
=
s
i
g
m
o
i
d
(
G
a
t
e
N
e
t
1
(
H
k
H
1
)
(6), where || denotes the concatenation operation,
G
a
t
e
N
e
t
1
is a feed-forward neural network, and
g
i
is the gaet learned for head
i
.” Here, Long describes applying a sigmoid function, which is a bounded nonlinear function, to the gating parameters.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen and Spagnoli to incorporate the teachings of Long and incorporate a bounded nonlinear function applied by a gated attention block.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen ang Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, and incorporate with Long’s teachings of a bounded nonlinear function applied by a gated attention block. One of ordinary skill in the art would be motivated to do so because by integrating Long’s frameworks into the methods of Zhen and Spagnoli, one of ordinary skill in the art would have a mechanism that “proved effective in different domains, such as machine translation and recommender systems” (Long, page 9).
Claim 17
Regarding claim 17, Zhen in view of Spagnoli in view of Long teaches the limitations of claim 16.
Neither Zhen or Spagnoli appear to teach “the bounded nonlinear function comprises a sigmoid function.”
However, Long teaches “the bounded nonlinear function comprises a sigmoid function.”
See Long in page 9, equation 6 where it describes “In contrast to the vanilla multi-head attention network, we integrate a neural gating structure:
g
i
=
s
i
g
m
o
i
d
(
G
a
t
e
N
e
t
1
H
k
H
1
)
Here, Long describes using sigmoid function.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen and Spagnoli to incorporate the teachings of Long and incorporate a bounded nonlinear function that is a sigmoid function.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, and incorporate with Long’s teachings of a sigmoid bounded nonlinear function. One of ordinary skill in the art would be motivated to do so because by integrating Long’s frameworks into the methods of Zhen and Spagnoli, one of ordinary skill in the art would have a mechanism that “proved effective in different domains, such as machine translation and recommender systems” (Long, page 9).
Claim 18
Regarding claim 18, Zhen in view of Spagnoli in view of Long teaches the limitations in claim 15.
Neither Zhen or Spagnoli appear to teach “the gated attention block is applied for each token generated by the transformer neural network for the received input.”
However, Long teaches “the gated attention block is applied for each token generated by the transformer neural network for the received input.”
See Long in page 9 where it teaches “Then, we distinguish the importance of feature interactions by using both the attention mechanism and the gating mechanism…
Q
i
=
W
i
Q
H
k
,
K
i
=
W
i
K
H
1
(8) where
W
i
Q
∈
R
d
K
×
d
are parameters to learn for head
i
,
d
K
denotes the dimension of each head, and
α
i
is the final importance of cross features.” Here, Long teaches applying the attention layer for each of the received input parameters.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen and Spagnoli to incorporate the teachings of Long and incorporate applying a gated attention block for each input token received in the neural network.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, and incorporate with Long’s teachings of applying a gated attention block for each input token received in the neural network. One of ordinary skill in the art would be motivated to do so because by integrating Long’s frameworks into the methods of Zhen and Spagnoli, one of ordinary skill in the art would have a mechanism that “proved effective in different domains, such as machine translation and recommender systems” (Long, page 9).
Claim 20 Regarding claim 20, Zhen in view of Spagnoli teaches the limitations in claim 19.
Further Zhen teaches “the means for generating the attention output in the transformer neural network comprises means for generating the attention output based on a clipped… function having a dynamic range controlled by a first hyperparameter and a second hyperparameter.”
See Zhen in page 4719, cols. 1-2 where it describes “Multi-head attention (MHA) is a powerful and ubiquitous model in the DETR model, which allows the network to obtain information from different mapping spaces. In vanilla DETR models, we usually have 8 heads in each attention module. However, not all the heads contribute equally for making predictions. The redundant information causes the DETR model to overfit. We normalize the vanilla multi-head attention module by multiplying the single head representation
H
i
with a learnable gate parameter
g
i
.” Here, Zhen teaches generating an attention output with a function. Further see Zhen in page 4719, col. 2 where it describes “Given the learnable parameter set
q
for all heads in a MHA module, our gate parameter set
g
can be derived as follows:
s
=
S
i
g
m
o
i
d
(
(
q
+
log
u
-
l
o
g
(
1
-
u
)
)
/
T
)
,
s
-
=
s
(
λ
-
μ
)
+
μ
,
g
=
c
l
a
m
p
(
s
-
,
0,1
)
.” Here, Zhen teaches generating the gate parameters (or in other words, output) using a function. Further see Zhen in page 4719, col. 2 where it describes “Since we want the heads can be totally opened or closed in GMHA modules, we introduce two parameters
μ
,
λ
that can stretch the value of
s
to
(
μ
,
λ
)
with
μ
<
0
and
λ
>
1
as described in Eq. (9). Then we can apply a hard-sigmoid function to clip the value of gate
g
into range
[
0
,
1
]
following Eq. (10).” Here, Zhen describes clipping the function using two hyperparameters.
Neither Zhen or Spagnoli appear to teach “a softmax function.”
However, Long in the same field of art teaches “a softmax function.”
See Long in page 9 where it describes “Then we distinguish the importance of feature interactions by using both the attention mechanism and the gating mechanism:
α
i
=
s
o
f
t
m
a
x
(
Q
i
K
i
T
d
K
)
(7)” Here, Long teaches a softmax function used as part of an attention mechanism.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen and Spagnoli to incorporate the teachings of Long and incorporate a softmax function when generating the attention output.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, and incorporate with Long’s teachings of using a softmax equation in the attention output. One of ordinary skill in the art would be motivated to do so because by integrating Long’s frameworks into the methods of Zhen and Spagnoli, one of ordinary skill in the art would have a mechanism that “proved effective in different domains, such as machine translation and recommender systems” (Long, page 9).
Claim 21
Regarding claim 21, Zhen in view of Spagnoli in view of Long teaches the limitations in claim 20.
Further Zhen teaches “the first hyperparameter comprises a hyperparameter greater than or equal to 1 and wherein the second hyperparameter comprises a hyperparameter less than or equal to 0.”
See Zhen in page 4719, col. 2 where it describes “Since we want the heads can be totally opened or closed in GMHA modules, we introduce two parameters
μ
,
λ
that can stretch the value of
s
to
(
μ
,
λ
)
with
μ
<
0
and
λ
>
1
as described in Eq. (9).” Here, Zhen describes setting the parameter
μ
to less than 0 and the parameter
λ
to greater than 1.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen, Spagnoli, and Long to incorporate the teachings of Zhen and have two hyperparameters, one less than 0 and one greater than 1.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, and using a softmax equation in the attention output, and further incorporate two hyperparameters, one less than 0 and one greater than 1. One of ordinary skill in the art would be motivated to do so because by integrating Zhen’s frameworks into the methods of Zhen, Spagnoli, and Long, one of ordinary skill in the art would “(help) the vanilla DETR model achieve 1.6 points AP improvement” (Zhen, page 4721, col. 1).
Claim 22
Regarding claim 22, Zhen in view of Spagnoli in view of Long teaches the limitations in claim 21.
Further Zhen teaches “the clipped… function is configured to output values up to and including a value of 1 when a value of the first hyperparameter is greater than 1.”
See Zhen in page 4719, col. 2 where it describes “Then we can apply a hard-sigmoid function to clip the value of gate g into range [0, 1] following Eq. (10).” Here, Zhen describes a function that can output values in the range from 0 to 1, or in other words up to and including 1.
Further Long teaches “softmax function;”
See Long in page 9 where it describes “Then we distinguish the importance of feature interactions by using both the attention mechanism and the gating mechanism:
α
i
=
s
o
f
t
m
a
x
(
Q
i
K
i
T
d
K
)
(7)” Here, Long teaches a softmax function used as part of an attention mechanism.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen, Spagnoli, and Long to incorporate the teachings of Zhen and incorporate a function that output values up to and including 1 when the first hyperparameter is greater than 1, and further incorporate Long’s teachings of using a softmax function.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, incorporates two hyperparameters, one less than 0 and one greater than 1, and a function that outputs a value up to and including 1 if the first hyperparameter is greater than 1, and further incorporate Long’s teachings of a softmax function. One of ordinary skill in the art would be motivated to do so because by integrating Zhen and Long’s frameworks into the methods of Zhen, Spagnoli, and Long, one of ordinary skill in the art would “(help) the vanilla DETR model achieve 1.6 points AP improvement” (Zhen, page 4721, col. 1).
Claim 23
Regarding claim 23, Zhen in view of Spagnoli in view of Long teaches the limitations in claim 21.
Further Zhen teaches “the clipped… function is configured to output values down to and including a value of 0 when a value of the second hyperparameter is less than 0.”
See Zhen in page 4719, col. 2 where it describes “Then we can apply a hard-sigmoid function to clip the value of gate g into range [0, 1] following Eq. (10).” Here, Zhen describes a function that can output values in the range from 0 to 1, or in other words down to and including 0.
Further Long teaches “softmax function;”
See Long in page 9 where it describes “Then we distinguish the importance of feature interactions by using both the attention mechanism and the gating mechanism:
α
i
=
s
o
f
t
m
a
x
(
Q
i
K
i
T
d
K
)
(7)” Here, Long teaches a softmax function used as part of an attention mechanism.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen, Spagnoli, and Long to incorporate the teachings of Zhen and incorporate a function that output values down to and including 0 when the second hyperparameter is less than 0, and further incorporate Long’s teachings of using a softmax function.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, incorporates two hyperparameters, one less than 0 and one greater than 1, and a function that outputs a value down to and including 0 if the second hyperparameter is less than 0, and further incorporate Long’s teachings of a softmax function. One of ordinary skill in the art would be motivated to do so because by integrating Zhen and Long’s frameworks into the methods of Zhen, Spagnoli, and Long, one of ordinary skill in the art would “(help) the vanilla DETR model achieve 1.6 points AP improvement” (Zhen, page 4721, col. 1).
Claim 24
Regarding claim 24, Zhen in view of Spagnoli teaches the limitations in claim 19.
Neither Zhen or Spagnoli appear to teach “the means for generating the attention output in the transformer neural network comprises means for generating the attention output based on a gated attention block configured to output a minimum value of 0.”
However, Long teaches “the means for generating the attention output in the transformer neural network comprises means for generating the attention output based on a gated attention block configured to output a minimum value of 0.”
See Long in page 9 where it describes “Specifically, for the gated multi-head attention layer of the k-th encoder, the inputs are the representations of k-order cross features and the embedding vectors of raw features, which are denoted as
H
(
k
)
∈
R
m
×
d
and
H
(
1
)
∈
R
m
×
d
respectively… Finally, we obtain the representations by concatenating the outputs of all heads:
H
i
k
*
=
g
i
⊙
α
i
V
i
(9),
H
k
*
=
N
o
r
m
(
[
H
i
k
*
H
2
k
*
…
H
h
k
*
+
H
1
)
, (10),
V
i
=
W
i
V
H
k
, (11), where
W
i
v
∈
R
d
K
×
d
is the parameter of head
i
,
h
is the number of heads, and
N
o
r
m
represents the normalization.” Here, Long teaches generating attention outputs based on a gated attention layer’s outputs that are at least 0. Specifically in equation 9, the softmax function
α
i
is being multiplied by the sigmoid
g
i
, and since both of these functions will return values between 0 and 1 (as described in pages 6-7: “each record of the dataset is consisted of m categorical features and an associated label
y
∈
{
0,1
}
"
)
, will return a value of at least 0 when multiplied by each other.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen and Spagnoli to incorporate the teachings of Long and incorporate generating an attention output based on a gated attention block configured to output a minimum value of 0.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, and incorporate with Long’s teachings of generating an attention output based on a gated attention block configured to output a minimum value of 0. One of ordinary skill in the art would be motivated to do so because by integrating Long’s frameworks into the methods of Zhen and Spagnoli, one of ordinary skill in the art would have a mechanism that “proved effective in different domains, such as machine translation and recommender systems” (Long, page 9).
Claim 25
Regarding claim 25, Zhen in view of Spagnoli in view of Long teaches the limitations in claim 24.
Neither Zhen or Spagnoli appear to teach “the gated attention block applies a bounded nonlinear function to one or more gating parameters defined for the transformer neural network.”
However, Long teaches “the gated attention block applies a bounded nonlinear function to one or more gating parameters defined for the transformer neural network.”
See Long in page 9 where it describes “Specifically, for the gated multi-head attention layer of the k-th encoder, the inputs are the representations of k-order cross features and the embedding vectors of raw features, which are denoted as
H
(
k
)
∈
R
m
×
d
and
H
(
1
)
∈
R
m
×
d
respectively. In contrast to the vanilla multi-head attention network, we integrate a neural gating structure:
g
i
=
s
i
g
m
o
i
d
(
G
a
t
e
N
e
t
1
(
H
k
H
1
)
(6), where || denotes the concatenation operation,
G
a
t
e
N
e
t
1
is a feed-forward neural network, and
g
i
is the gaet learned for head
i
.” Here, Long describes applying a sigmoid function, which is a bounded nonlinear function, to the gating parameters.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen and Spagnoli to incorporate the teachings of Long and incorporate a bounded nonlinear function applied by a gated attention block.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen ang Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, and incorporate with Long’s teachings of a bounded nonlinear function applied by a gated attention block. One of ordinary skill in the art would be motivated to do so because by integrating Long’s frameworks into the methods of Zhen and Spagnoli, one of ordinary skill in the art would have a mechanism that “proved effective in different domains, such as machine translation and recommender systems” (Long, page 9).
Claim 26
Regarding claim 26, Zhen in view of Spagnoli in view of Long teaches the limitations of claim 25.
Neither Zhen or Spagnoli appear to teach “the bounded nonlinear function comprises a sigmoid function.”
However, Long teaches “the bounded nonlinear function comprises a sigmoid function.”
See Long in page 9, equation 6 where it describes “In contrast to the vanilla multi-head attention network, we integrate a neural gating structure:
g
i
=
s
i
g
m
o
i
d
(
G
a
t
e
N
e
t
1
H
k
H
1
)
Here, Long describes using sigmoid function.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen and Spagnoli to incorporate the teachings of Long and incorporate a bounded nonlinear function that is a sigmoid function.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, and incorporate with Long’s teachings of a sigmoid bounded nonlinear function. One of ordinary skill in the art would be motivated to do so because by integrating Long’s frameworks into the methods of Zhen and Spagnoli, one of ordinary skill in the art would have a mechanism that “proved effective in different domains, such as machine translation and recommender systems” (Long, page 9).
Claim 27
Regarding claim 27, Zhen in view of Spagnoli in view of Long teaches the limitations in claim 24.
Neither Zhen or Spagnoli appear to teach “the gated attention block is applied for each token generated by the transformer neural network for the received input.”
However, Long teaches “the gated attention block is applied for each token generated by the transformer neural network for the received input.”
See Long in page 9 where it teaches “Then, we distinguish the importance of feature interactions by using both the attention mechanism and the gating mechanism…
Q
i
=
W
i
Q
H
k
,
K
i
=
W
i
K
H
1
(8) where
W
i
Q
∈
R
d
K
×
d
are parameters to learn for head
i
,
d
K
denotes the dimension of each head, and
α
i
is the final importance of cross features.” Here, Long teaches applying the attention layer for each of the received input parameters.
Zhen, Spagnoli, and Long are considered to be analogous to the claimed invention because they are in the same field of transformer neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhen and Spagnoli to incorporate the teachings of Long and incorporate applying a gated attention block for each input token received in the neural network.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhen and Spagnoli with the teachings of Long by using Zhen and Spagnoli’s teachings of a processing system with a memory and processors that receives an input for a transformer neural network, generates an attention output, generates an output, and attenuates outlier values, and incorporate with Long’s teachings of applying a gated attention block for each input token received in the neural network. One of ordinary skill in the art would be motivated to do so because by integrating Long’s frameworks into the methods of Zhen and Spagnoli, one of ordinary skill in the art would have a mechanism that “proved effective in different domains, such as machine translation and recommender systems” (Long, page 9).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW CHARLES YORKS whose telephone number is (571)270-1803. The examiner can normally be reached M-F, 9am to 5pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached at (571) 272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW CHARLES YORKS/Examiner, Art Unit 2146
/USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146
