DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitations use a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are: a processing device configured to execute instructions stored in memory in claim 8 and a processing device configured to iteratively train the neural network in claim 15.
Because these claim limitations are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have these limitations interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitations recite sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 6, 13, and 20 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 6, 13, and 20 recite the limitation determining diagonals of the products of the respective pairs of the plurality of matrices. There is insufficient antecedent basis for the products of the respective pairs because claims 6, 13, and 20 are dependent upon claims 2, 9, and 16 and “the products of the respective pairs” are disclosed in claims 3, 10, and 17.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the invention is directed to an abstract idea without significantly more.
Regarding claim 1:
Subject Matter Eligibility Analysis Step 1:
Claim 1 recites a method and is thus a process, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 1 recites
determining structural relationships between respective parameters of the plurality of matrices; (This limitation is a mental process as it could be performed mentally or with pen and paper by a human determining the relationships between matrix parameters and is thus a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
initializing the plurality of matrices based on the determined structural relationships; (This limitation is a mental process as it could be performed mentally or with pen and paper by a human determining an initialization and writing it down, which is a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
Therefore, claim 1 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 1 further recites additional elements of
A method of initializing and training a transformer neural network, wherein the transformer neural network includes a plurality of self-attention layers configured to operate in accordance with a plurality of matrices, the method comprising: (This element does not integrate the abstract idea into a practical application because it amounts to mere "apply it on a computer" (see MPEP 2106.05(f))).
providing, as input, the plurality of matrices to the transformer neural network to initialize the transformer neural network for training; (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g))).
generating, using the transformer neural network, an output based on the plurality of matrices. (This element does not integrate the abstract idea into a practical application because it amounts to mere "apply it on a computer" (see MPEP 2106.05(f)).)
Therefore, claim 1 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 1 do not provide significantly more than the abstract idea itself, taken alone and in combination because
A method of initializing and training a transformer neural network, wherein the transformer neural network includes a plurality of self-attention layers configured to operate in accordance with a plurality of matrices, the method comprising: merely recites steps that train a generic transformer neural network, which represents merely adding the words “apply it”, or an equivalent, which are not indicative of an inventive concept (MPEP 2106.05(f)).
providing, as input, the plurality of matrices to the transformer neural network to initialize the transformer neural network for training; is the well understood, routine, and conventional activity of “transmitting or receiving data over a network" (see MPEP 2106.05(d)(II); OIP Techs., Inc., V. Amazon.com, Inc., 788F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
generating, using the transformer neural network, an output based on the plurality of matrices merely recites steps that use a generic transformer neural network to generate output, which represents merely adding the words “apply it”, or an equivalent, which are not indicative of an inventive concept (MPEP 2106.05(f)).
Therefore, claim 1 is subject-matter ineligible.
Regarding claim 2:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 2 recites
wherein the plurality of matrices includes a value matrix, a projection matrix, a query matrix, and a key matrix. (This limitation is a mental process as it further defines the mental process of determining relationships between matrices recited in claim 1 and is thus a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
Therefore, claim 2 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 2 does not further recite any additional elements. Therefore, claim 2 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 2 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 2 is subject-matter ineligible.
Regarding claim 3:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 3 recites
wherein determining the structural relationships includes determining products of respective pairs of the plurality of matrices. (This limitation is a mental process as it further defines the mental process of determining relationships between matrices recited in claim 1 and is thus a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
Therefore, claim 3 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 3 does not further recite any additional elements. Therefore, claim 3 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 3 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 3 is subject-matter ineligible.
Regarding claim 4:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 4 recites
wherein determining the structural relationships includes determining (i) a first product of the value matrix and the projection matrix and (ii) a second product of the query matrix and the key matrix. (This limitation is a mental process as it further defines the mental process of determining relationships between matrices recited in claim 3 and is thus a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
Therefore, claim 4 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 4 does not further recite any additional elements. Therefore, claim 4 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 4 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 4 is subject-matter ineligible.
Regarding claim 5:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 5 recites
further comprising performing singular value decompositions based on the products of the respective pairs of the plurality of matrices, wherein performing the singular value decompositions includes using at least one of full-rank singular value decomposition factors and low-rank singular value decomposition factors. (This limitation is a mental process as it could be performed mentally or with pen and paper by a human computing matrix decompositions using a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
Therefore, claim 5 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 5 does not further recite any additional elements. Therefore, claim 5 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 5 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 5 is subject-matter ineligible.
Regarding claim 6:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 6 recites
wherein determining the structural relationships includes determining diagonals of the products of the respective pairs of the plurality of matrices. (This limitation is a mental process as it further defines the mental process of determining relationships between matrices recited in claim 1 and is thus a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
Therefore, claim 6 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 6 does not further recite any additional elements. Therefore, claim 6 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 6 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 6 is subject-matter ineligible.
Regarding claim 7:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 7 recites
wherein determining the structural relationships includes sampling a random normal matrix. (This limitation is a mental process as it further defines the mental process of determining relationships between matrices recited in claim 1 and is thus a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
Therefore, claim 7 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 7 does not further recite any additional elements. Therefore, claim 7 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 7 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 7 is subject-matter ineligible.
Regarding claim 8:
Subject Matter Eligibility Analysis Step 1:
Claim 8 recites a device and is thus a machine, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 8 recites
determine structural relationships between respective parameters of the plurality of matrices; (This limitation is a mental process as it could be performed mentally or with pen and paper by a human determining the relationships between matrix parameters and is thus a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
initialize the plurality of matrices based on the determined structural relationships; (This limitation is a mental process as it could be performed mentally or with pen and paper by a human determining an initialization and writing it down, which is a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
Therefore, claim 8 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 8 further recites additional elements of
A computing device configured to initialize and train a transformer neural network, wherein the transformer neural network includes a plurality of self-attention layers configured to operate in accordance with a plurality of matrices, the computing device including a processing device configured to execute instructions stored in memory to: (This element does not integrate the abstract idea into a practical application because it amounts to mere "apply it on a computer" (see MPEP 2106.05(f))).
provide, as input, the plurality of matrices to the transformer neural network to initialize the transformer neural network for training; (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g))).
generate, using the transformer neural network, an output based on the plurality of matrices. (This element does not integrate the abstract idea into a practical application because it amounts to mere "apply it on a computer" (see MPEP 2106.05(f)).)
Therefore, claim 8 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 8 do not provide significantly more than the abstract idea itself, taken alone and in combination because
A computing device configured to initialize and train a transformer neural network, wherein the transformer neural network includes a plurality of self-attention layers configured to operate in accordance with a plurality of matrices, the computing device including a processing device configured to execute instructions stored in memory to: merely recites steps that train a generic transformer neural network, which represents merely adding the words “apply it”, or an equivalent, which are not indicative of an inventive concept (MPEP 2106.05(f)).
provide, as input, the plurality of matrices to the transformer neural network to initialize the transformer neural network for training; is the well understood, routine, and conventional activity of “transmitting or receiving data over a network" (see MPEP 2106.05(d)(II); OIP Techs., Inc., V. Amazon.com, Inc., 788F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
generate, using the transformer neural network, an output based on the plurality of matrices. merely recites steps that use a generic transformer neural network to generate output, which represents merely adding the words “apply it”, or an equivalent, which are not indicative of an inventive concept (MPEP 2106.05(f)).
Therefore, claim 8 is subject-matter ineligible.
Regarding claim 9:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 9 recites
wherein the plurality of matrices includes a value matrix, a projection matrix, a query matrix, and a key matrix. (This limitation is a mental process as it further defines the mental process of determining relationships between matrices recited in claim 8 and is thus a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
Therefore, claim 9 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 9 does not further recite any additional elements. Therefore, claim 9 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 9 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 9 is subject-matter ineligible.
Regarding claim 10:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 10 recites
wherein, to determine the structural relationships…determine products of respective pairs of the plurality of matrices. (This limitation is a mental process as it further defines the mental process of determining relationships between matrices recited in claim 8 and is thus a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
Therefore, claim 10 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 10 further recites additional elements of
the processing device is configured to execute instructions to (This element does not integrate the abstract idea into a practical application because it amounts to mere "apply it on a computer" (see MPEP 2106.05(f))).
Therefore, claim 10 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 10 do not provide significantly more than the abstract idea itself, taken alone and in combination because
the processing device is configured to execute instructions to uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 10 is subject-matter ineligible.
Regarding claim 11:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 11 recites
wherein, to determine the structural relationships…determine (i) a first product of the value matrix and the projection matrix and (ii) a second product of the query matrix and the key matrix. (This limitation is a mental process as it further defines the mental process of determining relationships between matrices recited in claim 8 and is thus a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
Therefore, claim 11 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 11 further recites additional elements of
the processing device is configured to execute instructions to (This element does not integrate the abstract idea into a practical application because it amounts to mere "apply it on a computer" (see MPEP 2106.05(f))).
Therefore, claim 11 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 11 do not provide significantly more than the abstract idea itself, taken alone and in combination because
the processing device is configured to execute instructions to uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 11 is subject-matter ineligible.
Regarding claim 12:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 12 recites
perform singular value decompositions based on the products of the respective pairs of the plurality of matrices, and wherein performing the singular value decompositions includes using at least one of full-rank singular value decomposition factors and low-rank singular value decomposition factors. (This limitation is a mental process as it could be performed mentally or with pen and paper by a human computing matrix decompositions using a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
Therefore, claim 12 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 12 further recites additional elements of
wherein the processing device is configured to execute instructions to (This element does not integrate the abstract idea into a practical application because it amounts to mere "apply it on a computer" (see MPEP 2106.05(f))).
Therefore, claim 12 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 12 do not provide significantly more than the abstract idea itself, taken alone and in combination because
wherein the processing device is configured to execute instructions to uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 12 is subject-matter ineligible.
Regarding claim 13:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 13 recites
wherein, to determine the structural relationships…determine diagonals of the products of the respective pairs of the plurality of matrices. (This limitation is a mental process as it further defines the mental process of determining relationships between matrices recited in claim 8 and is thus a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
Therefore, claim 13 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 13 further recites additional elements of
the processing device is configured to execute instructions to (This element does not integrate the abstract idea into a practical application because it amounts to mere "apply it on a computer" (see MPEP 2106.05(f))).
Therefore, claim 13 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 13 do not provide significantly more than the abstract idea itself, taken alone and in combination because
the processing device is configured to execute instructions to uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 13 is subject-matter ineligible.
Regarding claim 14:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 14 recites
wherein, to determine the structural relationships…sample a random normal matrix. (This limitation is a mental process as it further defines the mental process of determining relationships between matrices recited in claim 8 and is thus a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
Therefore, claim 14 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 14 further recites additional elements of
the processing device is configured to execute instructions to (This element does not integrate the abstract idea into a practical application because it amounts to mere "apply it on a computer" (see MPEP 2106.05(f))).
Therefore, claim 14 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 14 do not provide significantly more than the abstract idea itself, taken alone and in combination because
the processing device is configured to execute instructions to uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 14 is subject-matter ineligible.
Regarding claim 15:
Subject Matter Eligibility Analysis Step 1:
Claim 15 recites a system with data storage/memory/processing device and is thus a machine, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 15 recites
determining structural relationships between respective parameters of the plurality of matrices, (This limitation is a mental process as it could be performed mentally or with pen and paper by a human determining the relationships between matrix parameters and is thus a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
initializing the plurality of matrices based on the determined structural relationships, (This limitation is a mental process as it could be performed mentally or with pen and paper by a human determining an initialization and writing it down, which is a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
Therefore, claim 15 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 15 further recites additional elements of
A system configured to train a transformer neural network, wherein the transformer neural network includes a plurality of self-attention layers configured to operate in accordance with a plurality of matrices, the system comprising…data storage that stores training data for training the transformer neural network; memory that stores a data representation of the transformer neural network; (This element does not integrate the abstract idea into a practical application because it amounts to mere "apply it on a computer" (see MPEP 2106.05(f))).
and a processing device configured to iteratively train the neural network using the training data to obtain a trained transformer neural network, wherein iteratively training the transformer neural network includes initializing the transformer neural network by (This element does not integrate the abstract idea into a practical application because it amounts to mere "apply it on a computer" (see MPEP 2106.05(f))).
providing, as input, the plurality of matrices to the transformer neural network to initialize the transformer neural network for training by the processing device. (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g))).
Therefore, claim 15 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 15 do not provide significantly more than the abstract idea itself, taken alone and in combination because
A system configured to train a transformer neural network, wherein the transformer neural network includes a plurality of self-attention layers configured to operate in accordance with a plurality of matrices, the system comprising…data storage that stores training data for training the transformer neural network; memory that stores a data representation of the transformer neural network; merely recites generic computer parts and steps that train a generic transformer neural network, which represents merely adding the words “apply it”, or an equivalent, which are not indicative of an inventive concept (MPEP 2106.05(f)).
a processing device configured to iteratively train the neural network using the training data to obtain a trained transformer neural network, wherein iteratively training the transformer neural network includes initializing the transformer neural network merely recites steps that train a generic transformer neural network, which represents merely adding the words “apply it”, or an equivalent, which are not indicative of an inventive concept (MPEP 2106.05(f)).
provide, as input, the plurality of matrices to the transformer neural network to initialize the transformer neural network for training; is the well understood, routine, and conventional activity of “transmitting or receiving data over a network" (see MPEP 2106.05(d)(II); OIP Techs., Inc., V. Amazon.com, Inc., 788F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
Therefore, claim 15 is subject-matter ineligible.
Regarding claim 16:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 16 recites
wherein the plurality of matrices includes a value matrix, a projection matrix, a query matrix, and a key matrix. (This limitation is a mental process as it further defines the mental process of determining relationships between matrices recited in claim 15 and is thus a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
Therefore, claim 16 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 16 does not further recite any additional elements. Therefore, claim 16 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 16 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 16 is subject-matter ineligible.
Regarding claim 17:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 17 recites
wherein determining the structural relationships includes determining products of respective pairs of the plurality of matrices. (This limitation is a mental process as it further defines the mental process of determining relationships between matrices recited in claim 15 and is thus a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
Therefore, claim 17 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 17 does not further recite any additional elements. Therefore, claim 17 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 17 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 17 is subject-matter ineligible.
Regarding claim 18:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 18 recites
wherein determining the structural relationships includes determining a first product of the value matrix and the projection matrix and a second product of the query matrix and the key matrix. (This limitation is a mental process as it further defines the mental process of determining relationships between matrices recited in claim 17 and is thus a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
Therefore, claim 18 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 18 does not further recite any additional elements. Therefore, claim 18 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 18 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 18 is subject-matter ineligible.
Regarding claim 19:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 19 recites
perform singular value decompositions based on the products of the respective pairs of the plurality of matrices, and wherein performing the singular value decompositions includes using at least one of full-rank singular value decomposition factors and low-rank singular value decomposition factors. (This limitation is a mental process as it could be performed mentally or with pen and paper by a human computing matrix decompositions using a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
Therefore, claim 19 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 19 further recites additional elements of
wherein the processing device is configured to (This element does not integrate the abstract idea into a practical application because it amounts to mere "apply it on a computer" (see MPEP 2106.05(f))).
Therefore, claim 19 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 19 do not provide significantly more than the abstract idea itself, taken alone and in combination because
wherein the processing device is configured to uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 19 is subject-matter ineligible.
Regarding claim 20:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 20 recites
wherein determining the structural relationships includes determining diagonals of the products of the respective pairs of the plurality of matrices. (This limitation is a mental process as it further defines the mental process of determining relationships between matrices recited in claim 15 and is thus a mental process of observation/evaluation/judgement (MPEP 2106.04(a)(2))).
Therefore, claim 20 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 20 does not further recite any additional elements. Therefore, claim 20 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 20 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 20 is subject-matter ineligible.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-4 and 6-7 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by He, et al., Non-Patent Literature “Deep Transformers Without Shortcuts: Modifying Self-Attention For Faithful Signal Propagation” (“He”).
Regarding claim 1, He discloses:
A method of initializing and training a transformer neural network, wherein the transformer neural network includes a plurality of self-attention layers configured to operate in accordance with a plurality of matrices. He, pg. 3, “The input to a transformer...is successively transformed via a series of L “transformer blocks”, with independently initialised parameters…Each transformer block consists of two component blocks...The attention block, which is the first of these component blocks...followed by multi-head attention [A method of initializing and training a transformer neural network, wherein the transformer neural network includes a plurality of self-attention layers]”
(Examiner’s Comment: the examiner interprets the “transformer blocks” comprising “The attention block” to be “self-attention layers”)
PNG
media_image1.png
100
1042
media_image1.png
Greyscale
PNG
media_image2.png
81
232
media_image2.png
Greyscale
[configured to operate in accordance with a plurality of matrices,]
(Examiner’s Comment: the examiner interprets “plurality of matrices” to include
W
Q
,
W
K
,
W
V
, and
W
O
)
the method comprising: determining structural relationships between respective parameters of the plurality of matrices; He, pg. 6:
PNG
media_image3.png
69
1044
media_image3.png
Greyscale
He, pg. 4:
[AltContent: rect]
PNG
media_image4.png
170
1041
media_image4.png
Greyscale
(Examiner’s Comment: the examiner interprets the highlighted terms to be the “structural relationships” between
W
Q
,
W
K
,
W
V
, and
W
O
. Note,
W
T
and
W
are products of
W
V
and
W
O
)
initializing the plurality of matrices based on the determined structural relationships; He, pg. 6:
[AltContent: rect]
PNG
media_image5.png
95
1044
media_image5.png
Greyscale
He, pg. 4:
[AltContent: rect]
PNG
media_image6.png
91
1041
media_image6.png
Greyscale
(Examiner’s Comment:
W
Q
is initialized to be 0 according to the structural relationship between
W
Q
and
W
K
.
W
V
and
W
O
are initialized “orthogonally” making their product the identity matrix according to their structural relationship.)
providing, as input, the plurality of matrices to the transformer neural network to initialize the transformer neural network for training; (He, pg. 3, “a series of L “transformer blocks” [the transformer neural network], with independently initialised parameters [providing, as input]...
W
Q
,
W
K
∈
R
d
×
d
k
a
n
d
W
V
∈
R
d
×
d
v
...
W
O
∈
R
h
d
v
×
d
[the plurality of matrices to the transformer neural network to initialize for training]” Examiner’s Comment: the “series of L transformer blocks” is interpreted to comprise a transformer neural network. The matrices
W
Q
,
W
K
,
W
V
, and
W
O
are interpreted as the “independently initialised parameters” of the transformer neural network. The step in which the matrices are “independently initialised” is interpreted as providing them as input to intialize the transformer neural network prior to training).
and generating, using the transformer neural network, an output based on the plurality of matrices. (He, pg. 26, “To assess whether our conclusions about training performance in pre-training transfer over to downstream tasks, we assess models trained on C4 on 5 common sense downstream tasks: BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), Winogrande (Sakaguchi et al., 2020), PIQA (Bisk et al., 2020), and SIQA (Sap et al., 2019). These datasets are commonly used to evaluate large pre-trained transformers” [and generating, using the transformer neural network, an output based on the plurality of matrices.] (Examiner’s Comment: the tasks posed by these datasets require the transformer neural network to process text and produce output, e.g. "The task is then to take a question and passage as input, and to return “yes” or “no” as output." from BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions, pg. 2 col 1)).
Regarding claim 2, He discloses the method of claim 1. He further discloses wherein the plurality of matrices includes a value matrix, a projection matrix, a query matrix, and a key matrix. He, pg. 3:
PNG
media_image1.png
100
1042
media_image1.png
Greyscale
PNG
media_image2.png
81
232
media_image2.png
Greyscale
(Examiner’s Comment: The examiner interprets
W
V
,
W
O
,
W
Q
, and
W
K
to be the value, projection, query and key matrices respectively)
Regarding claim 3, He discloses the method of claim 2. He further discloses wherein determining the structural relationships includes determining products of respective pairs of the plurality of matrices. He, pg. 6:
PNG
media_image3.png
69
1044
media_image3.png
Greyscale
He, pg. 4:
[AltContent: rect]
PNG
media_image4.png
170
1041
media_image4.png
Greyscale
(Examiner’s Comment: The examiner interprets the highlighted equations to denote the “products of respective pairs of the plurality of matrices”. Note,
W
T
and W are products of
W
V
and
W
O
)
Regarding claim 4, He discloses the method of claim 3. He further discloses wherein determining the structural relationships includes determining (i) a first product of the value matrix and the projection matrix and (ii) a second product of the query matrix and the key matrix. He, pg. 4:
[AltContent: rect]
PNG
media_image4.png
170
1041
media_image4.png
Greyscale
He, pg. 6:
PNG
media_image3.png
69
1044
media_image3.png
Greyscale
(Examiner’s Comment: the examiner interprets the highlighted equations to denote the “first product of the value matrix and the projection matrix” and the “second product of the query matrix and the key matrix”)
Regarding claim 6, He discloses the method of claim 2. He further discloses wherein determining the structural relationships includes determining diagonals of the products of the respective pairs of the plurality of matrices. He, pg. 6:
PNG
media_image3.png
69
1044
media_image3.png
Greyscale
He, pg. 4:
[AltContent: rect]
PNG
media_image4.png
170
1041
media_image4.png
Greyscale
(Examiner’s Comment: determining the products to be 0 and I also determines their diagonals to be 0 and 1 respectively.)
Regarding claim 7, He discloses the method of claim 1. He further discloses wherein determining the structural relationships includes sampling a random normal matrix. He, pg. 18, Algorithm 1:
[AltContent: rect][AltContent: rect]
PNG
media_image7.png
204
960
media_image7.png
Greyscale
(Examiner’s Comment: The highlighted notation denotes “sampling a random normal matrix”)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 5 is rejected under 35 U.S.C. as being unpatentable over He, et al., Non-Patent Literature “Deep Transformers Without Shortcuts: Modifying Self-Attention For Faithful Signal Propagation” (“He”) in view of Beren, et al., Non-Patent Literature “The Singular Value Decompositions of Transformer Weight Matrices are Highly Interpretable” (“Beren”).
Regarding claim 5, He discloses The method of claim 3. He does not teach:
further comprising performing singular value decompositions based on the products of the respective pairs of the plurality of matrices
wherein performing the singular value decompositions includes using at least one of full-rank singular value decomposition factors and low-rank singular value decomposition factors.
However, Beren teaches further comprising performing singular value decompositions based on the products of the respective pairs of the plurality of matrices (Beren, pg. 4, “we can interpret the query and key matrices and value and output matrices…as single bilinear matrices QK and OV… We call these the QK and OV circuits.”, Beren, pg. 1, “we take the SVD of the weight matrices of the OV circuit”, Beren, pg. 34, “we can use our SVD directions approach to generate interpretable directions…while our techniques have much less success when applied to the QK circuit of the attention layers. [performing singular value decompositions based on the products of the respective pairs of the plurality of matrices]”).
Beren also teaches wherein performing the singular value decompositions includes using at least one of full-rank singular value decomposition factors and low-rank singular value decomposition factors. (Beren, pg.17, “the OV circuits are low-rank matrices”).
He and Beren are both in the same field of endeavor (i.e. transformer neural networks). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine He and Beren to teach the above limitations. The motivation for doing so is that performing SVD on the attention matrices provides access to the internal semantic knowledge of the transformer neural network (cf. Beren, pg. 1, “If we take the SVD of the weight matrices of the OV circuit and of MLP layers of GPT models, and project them to token embedding space, we notice this results in highly interpretable semantic clusters…We can use this to both improve our understanding of transformer language models and edit their representations.”).
Claims 8-11, 13-18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over He, et al., Non-Patent Literature “Deep Transformers Without Shortcuts: Modifying Self-Attention For Faithful Signal Propagation” (“He”) in view of Fan, US Pre-Grant Publication US20250131753A1 (“Fan”).
Regarding claim 8, He discloses:
wherein the transformer neural network includes a plurality of self-attention layers configured to operate in accordance with a plurality of matrices He, pg. 3, “The input to a transformer...is successively transformed via a series of L “transformer blocks”, with independently initialised parameters…Each transformer block consists of two component blocks...The attention block, which is the first of these component blocks...followed by multi-head attention [wherein the transformer neural network includes a plurality of self-attention layers configured to operate in accordance with a plurality of matrices]”
(Examiner’s Comment: the examiner interprets the “transformer blocks” comprising “The attention block” to be “self-attention layers”)
PNG
media_image1.png
100
1042
media_image1.png
Greyscale
PNG
media_image2.png
81
232
media_image2.png
Greyscale
[configured to operate in accordance with a plurality of matrices,]
(Examiner’s Comment: the examiner interprets “plurality of matrices” to include
W
Q
,
W
K
,
W
V
, and
W
O
)
to: determine structural relationships between respective parameters of the plurality of matrices; He, pg. 6:
PNG
media_image3.png
69
1044
media_image3.png
Greyscale
He, pg. 4:
[AltContent: rect]
PNG
media_image4.png
170
1041
media_image4.png
Greyscale
(Examiner’s Comment: the examiner interprets the highlighted terms to be the “structural relationships” between
W
Q
,
W
K
,
W
V
, and
W
O
. Note,
W
T
and
W
are products of
W
V
and
W
O
)
initialize the plurality of matrices based on the determined structural relationships; He, pg. 6:
[AltContent: rect]
PNG
media_image5.png
95
1044
media_image5.png
Greyscale
He, pg. 4:
[AltContent: rect]
PNG
media_image6.png
91
1041
media_image6.png
Greyscale
(Examiner’s Comment:
W
Q
is initialized to be 0 according to the structural relationship between
W
Q
and
W
K
.
W
V
and
W
O
are intialized “orthogonally” making their product the identity matrix according to their structural relationship.)
provide, as input, the plurality of matrices to the transformer neural network to initialize the transformer neural network for training; (He, pg. 3, “a series of L “transformer blocks” [the transformer neural network], with independently initialised parameters [providing, as input]...
W
Q
,
W
K
∈
R
d
×
d
k
a
n
d
W
V
∈
R
d
×
d
v
...
W
O
∈
R
h
d
v
×
d
[the plurality of matrices to the transformer neural network to initialize for training]” Examiner’s Comment: the “series of L transformer blocks” is interpreted to comprise a transformer neural network. The matrices
W
Q
,
W
K
,
W
V
, and
W
O
are interpreted as the “independently initialised parameters” of the transformer neural network. The step in which the matrices are “independently initialised” is interpreted as providing them as input to intialize the transformer neural network prior to training”).
and generate, using the transformer neural network, an output based on the plurality of matrices. (He, pg. 26, “To assess whether our conclusions about training performance in pre-training transfer over to downstream tasks, we assess models trained on C4 on 5 common sense downstream tasks: BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), Winogrande (Sakaguchi et al., 2020), PIQA (Bisk et al., 2020), and SIQA (Sap et al., 2019). These datasets are commonly used to evaluate large pre-trained transformers” [and generate, using the transformer neural network, an output based on the plurality of matrices.] (Examiner’s Comment: the tasks posed by these datasets require the transformer neural network to process text and produce output, e.g. "The task is then to take a question and passage as input, and to return “yes” or “no” as output." from BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions, pg. 2 col 1)).
While He teaches a method for initializing and training a transformer neural network, He does not explicitly teach:
A computing device configured to initialize and train a transformer neural network,
the computing device including a processing device configured to execute instructions stored in memory
However, Fan teaches A computing device configured to initialize and train a transformer neural network, (Fan, paragraph [0078], “the computer-executable instructions of the image difference captioning system 102 cause the computing device(s) 800 to perform the operations described herein. [A computing device]”, Fan, paragraph [0058], “the image difference captioning system 102 utilizes a vision transformer neural network”, Fan, paragraph [0023], “the image difference captioning system uses the target captions as a basis for training and evaluating the cross-modal neural network. [configured to initialize and train a transformer neural network,]”).
Fan also teaches the computing device including a processing device configured to execute instructions stored in memory (Fan, paragraph [0078], ” When executed by the one or more processors, the computer-executable instructions [including a processing device configured to execute instructions stored in memory] of the image difference captioning system 102 cause the computing device(s) 800 [the computing device] to perform the operations described herein.”).
He and Fan are both in the same field of endeavor (i.e. transformer neural networks). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine He and Fan to teach the above limitations. The motivation for doing so is that the prior art contained a base method, the initialization and training of a transformer neural network of He, upon which the claimed invention is an improvement. The claimed invention is an improvement on the base method because the added computing and processing devices allow the initialization and training process to be run on a computer. The technique disclosed in Fan, the use of computing and processing devices, applies to the base method because it contains a transformer neural network. A person of ordinary skill in the art would have recognized that applying the known technique to the base method would have yielded the predictable result of running an initialization and training process on a computer and resulted in an improved system. Computing and processing devices would have been expected to make it possible to perform the method on a computer and therefore useable in a production environment. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply the known technique of Fan to the initialization and training method of He. This would yield the predictable result of an initialization and training system which can be run on a computer (See MPEP 2143).
Regarding claim 9, He in view of Fan discloses The computing device of claim 8. He further teaches wherein the plurality of matrices includes a value matrix, a projection matrix, a query matrix, and a key matrix. He, pg. 3:
PNG
media_image1.png
100
1042
media_image1.png
Greyscale
PNG
media_image2.png
81
232
media_image2.png
Greyscale
(Examiner’s Comment: The examiner interprets
W
V
,
W
O
,
W
Q
, and
W
K
to be the value, projection, query and key matrices respectively)
The motivation to combine is the same as claim 8 above because it applies equally to claim 9.
Regarding claim 10, He in view of Fan discloses The computing device of claim 9. He further discloses wherein, to determine the structural relationships…determine products of respective pairs of the plurality of matrices. He, pg. 6:
PNG
media_image3.png
69
1044
media_image3.png
Greyscale
He, pg. 4:
[AltContent: rect]
PNG
media_image4.png
170
1041
media_image4.png
Greyscale
(Examiner’s Comment: The examiner interprets the highlighted equations to denote the “products of respective pairs of the plurality of matrices”. Note,
W
T
and
W
are products of
W
V
and
W
O
)
He does not teach:
the processing device is configured to execute instructions to
However, Fan teaches the processing device is configured to execute instructions to (Fan, paragraph [0078], ” When executed by the one or more processors, the computer-executable instructions [the processing device is configured to execute instructions to] of the image difference captioning system 102 cause the computing device(s) 800 to perform the operations described herein.”).
The motivation to combine is the same as claim 8 above because it applies equally to claim 10.
Regarding claim 11, He in view of Fan discloses The computing device of claim 10. He further discloses wherein, to determine the structural relationships…determine (i) a first product of the value matrix and the projection matrix and (ii) a second product of the query matrix and the key matrix. He, pg. 4:
[AltContent: rect]
PNG
media_image4.png
170
1041
media_image4.png
Greyscale
He, pg. 6:
PNG
media_image3.png
69
1044
media_image3.png
Greyscale
(Examiner’s Comment: the examiner interprets the highlighted equations to denote the “first product of the value matrix and the projection matrix” and the “second product of the query matrix and the key matrix”)
He does not teach:
the processing device is configured to execute instructions to
However, Fan teaches the processing device is configured to execute instructions to. (Fan, paragraph [0078], ” When executed by the one or more processors, the computer-executable instructions [the processing device is configured to execute instructions to] of the image difference captioning system 102 cause the computing device(s) 800 to perform the operations described herein.”).
The motivation to combine is the same as claim 8 above because it applies equally to claim 11.
Regarding claim 13, He in view of Fan discloses The computing device of claim 9. He further discloses wherein, to determine the structural relationships…determine diagonals of the products of the respective pairs of the plurality of matrices. He, pg. 6:
PNG
media_image3.png
69
1044
media_image3.png
Greyscale
He, pg. 4:
[AltContent: rect]
PNG
media_image4.png
170
1041
media_image4.png
Greyscale
(Examiner’s Comment: determining the products to be 0 and I also determines their diagonals to be 0 and 1 respectively.)
He does not teach:
the processing device is configured to execute instructions to
However, Fan teaches the processing device is configured to execute instructions to. (Fan, paragraph [0078], ” When executed by the one or more processors, the computer-executable instructions [the processing device is configured to execute instructions to] of the image difference captioning system 102 cause the computing device(s) 800 to perform the operations described herein.”).
The motivation to combine is the same as claim 8 above because it applies equally to claim 13.
Regarding claim 14, He in view of Fan discloses The computing device of claim 8. He further discloses wherein, to determine the structural relationships…sample a random normal matrix. He, pg. 18, Algorithm 1:
[AltContent: rect][AltContent: rect]
PNG
media_image7.png
204
960
media_image7.png
Greyscale
(Examiner’s Comment: The highlighted notation denotes “sampling a random normal matrix”)
He does not teach:
the processing device is configured to execute instructions to
However, Fan teaches the processing device is configured to execute instructions to (Fan, paragraph [0078], ” When executed by the one or more processors, the computer-executable instructions [the processing device is configured to execute instructions to] of the image difference captioning system 102 cause the computing device(s) 800 to perform the operations described herein.”).
The motivation to combine is the same as claim 8 above because it applies equally to claim 14.
Regarding claim 15, He discloses:
A system configured to train a transformer neural network, wherein the transformer neural network includes a plurality of self-attention layers configured to operate in accordance with a plurality of matrices He, pg. 3, “The input to a transformer...is successively transformed via a series of L “transformer blocks”, with independently initialised parameters…Each transformer block consists of two component blocks...The attention block, which is the first of these component blocks...followed by multi-head attention [A system configured to train a transformer neural network, wherein the transformer neural network includes a plurality of self-attention layers]”
(Examiner’s Comment: the examiner interprets the “transformer blocks” comprising “The attention block” to be “self-attention layers”)
PNG
media_image1.png
100
1042
media_image1.png
Greyscale
PNG
media_image2.png
81
232
media_image2.png
Greyscale
[configured to operate in accordance with a plurality of matrices,]
(Examiner’s Comment: the examiner interprets “plurality of matrices” to include
W
Q
,
W
K
,
W
V
, and
W
O
)
wherein iteratively training the transformer neural network includes initializing the transformer neural network by determining structural relationships between respective parameters of the plurality of matrices He, pg. 6:
PNG
media_image3.png
69
1044
media_image3.png
Greyscale
He, pg. 4:
[AltContent: rect]
PNG
media_image4.png
170
1041
media_image4.png
Greyscale
(Examiner’s Comment: the examiner interprets the highlighted terms to be the “structural relationships” between
W
Q
,
W
K
,
W
V
, and
W
O
. Note,
W
T
and
W
are products of
W
V
and
W
O
)
initializing the plurality of matrices based on the determined structural relationships He, pg. 6:
[AltContent: rect]
PNG
media_image5.png
95
1044
media_image5.png
Greyscale
He, pg. 4:
[AltContent: rect]
PNG
media_image6.png
91
1041
media_image6.png
Greyscale
(Examiner’s Comment:
W
Q
is initialized to be 0 according to the structural relationship between
W
Q
and
W
K
.
W
V
and
W
O
are intialized “orthogonally” making their product the identity matrix according to their structural relationship.)
providing, as input, the plurality of matrices to the transformer neural network to initialize the transformer neural network for training (He, pg. 3, “a series of L “transformer blocks” [the transformer neural network], with independently initialised parameters [providing, as input]...
W
Q
,
W
K
∈
R
d
×
d
k
a
n
d
W
V
∈
R
d
×
d
v
...
W
O
∈
R
h
d
v
×
d
[the plurality of matrices to the transformer neural network to initialize for training]” Examiner’s Comment: the “series of L transformer blocks” is interpreted to comprise a transformer neural network. The matrices
W
Q
,
W
K
,
W
V
, and
W
O
are interpreted as the “independently initialised parameters” of the transformer neural network. The step in which the matrices are “independently initialised” is interpreted as providing them as input to intialize the transformer neural network prior to training).
While He teaches a system for initializing and training a transformer neural network, He does not explicitly teach:
the system comprising: data storage that stores training data for training the transformer neural network;
memory that stores a data representation of the transformer neural network;
and a processing device configured to iteratively train the neural network using the training data to obtain a trained transformer neural network
However, Fan teaches the system comprising: data storage that stores training data for training the transformer neural network; (Fan, paragraph [0084], “The data storage manager 810 also stores data for training a neural network [the system comprising: data storage that stores training data for training]”, Fan, paragraph [0037], “In some embodiments, a neural network includes one or more neural network layers including, but not limited to…a transformer-based neural network [the transformer neural network;]”).
Fan also teaches memory that stores a data representation of the transformer neural network; (Fan, paragraph [0084], “The data storage manager 810 also stores data…including one or more neural networks [memory that stores a data representation]”, Fan, paragraph [0037], “In some embodiments, a neural network includes one or more neural network layers including, but not limited to…a transformer-based neural network [of the transformer neural network;]”).
Fan also teaches a processing device configured to iteratively train the neural network using the training data to obtain a trained transformer neural network (Fan, paragraph [0078], “When executed by the one or more processors [a processing device], the computer-executable instructions of the image difference captioning system 102 cause the computing device(s) 800 to perform the operations described herein.”, Fan, paragraph [0058], “the image difference captioning system 102 utilizes a vision transformer neural network [to obtain a trained transformer neural network]”, Fan, paragraph [0023], “the image difference captioning system uses the target captions as a basis for training and evaluating the cross-modal neural network. [configured to iteratively train the neural network using the training data]”).
He and Fan are both in the same field of endeavor (i.e. transformer neural networks). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine He and Fan to teach the above limitations. The motivation for doing so is that the prior art contained a base method, the initialization and training of a transformer neural network of He, upon which the claimed invention is an improvement. The claimed invention is an improvement on the base method because the added data storage, memory and processing device allow the initialization and training process to be run on a computer. The technique disclosed in Fan, the use of data storage, memory and a processing device, applies to the base method because it contains a transformer neural network. A person of ordinary skill in the art would have recognized that applying the known technique to the base method would have yielded the predictable result of running an initialization and training process on a computer and resulted in an improved system. Data storage, memory and a processing device would have been expected to make it possible to perform the method on a computer and therefore useable in a production environment. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply the known technique of Fan to the initialization and training method of He. This would yield the predictable result of an initialization and training system which can be run on a computer (See MPEP 2143).
Regarding claim 16, He in view of Fan discloses The system of claim 15. He further teaches wherein the plurality of matrices includes a value matrix, a projection matrix, a query matrix, and a key matrix. He, pg. 3:
PNG
media_image1.png
100
1042
media_image1.png
Greyscale
PNG
media_image2.png
81
232
media_image2.png
Greyscale
(Examiner’s Comment: The examiner interprets
W
V
,
W
O
,
W
Q
, and
W
K
to be the value, projection, query and key matrices respectively)
The motivation to combine is the same as claim 15 above because it applies equally to claim 16.
Regarding claim 17, He in view of Fan discloses The system of claim 16. He further discloses wherein determining the structural relationships includes determining products of respective pairs of the plurality of matrices. He, pg. 6:
PNG
media_image3.png
69
1044
media_image3.png
Greyscale
He, pg. 4:
[AltContent: rect]
PNG
media_image4.png
170
1041
media_image4.png
Greyscale
(Examiner’s Comment: The examiner interprets the highlighted equations to denote the “products of respective pairs of the plurality of matrices”. Note,
W
T
and
W
are products of
W
V
and
W
O
)
The motivation to combine is the same as claim 15 above because it applies equally to claim 17.
Regarding claim 18, He in view of Fan discloses The system of claim 17. He further discloses wherein determining the structural relationships includes determining a first product of the value matrix and the projection matrix and a second product of the query matrix and the key matrix. He, pg. 4:
[AltContent: rect]
PNG
media_image4.png
170
1041
media_image4.png
Greyscale
He, pg. 6:
PNG
media_image3.png
69
1044
media_image3.png
Greyscale
(Examiner’s Comment: the examiner interprets the highlighted equations to denote the “first product of the value matrix and the projection matrix” and the “second product of the query matrix and the key matrix”)
The motivation to combine is the same as claim 15 above because it applies equally to claim 18.
Regarding claim 20, He in view of Fan discloses The system of claim 16. He further discloses wherein determining the structural relationships includes determining diagonals of the products of the respective pairs of the plurality of matrices. He, pg. 6:
PNG
media_image3.png
69
1044
media_image3.png
Greyscale
He, pg. 4:
[AltContent: rect]
PNG
media_image4.png
170
1041
media_image4.png
Greyscale
(Examiner’s Comment: determining the products to be 0 and I also determines their diagonals to be 0 and 1 respectively.)
The motivation to combine is the same as claim 15 above because it applies equally to claim 20.
Claims 12 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over He, et al., Non-Patent Literature “Deep Transformers Without Shortcuts: Modifying Self-Attention For Faithful Signal Propagation” (“He”) in view of Fan, US Pre-Grant Publication US20250131753A1 (“Fan”) and further in view of Beren, et al., Non-Patent Literature “The Singular Value Decompositions of Transformer Weight Matrices are Highly Interpretable” (“Beren”).
Regarding claim 12, He in view of Fan dicloses The computing device of claim 10.
He in view of Fan also discloses wherein the processing device is configured to execute instructions to (Fan, paragraph [0078], “When executed by the one or more processors, the computer-executable instructions [wherein the processing device is configured to execute instructions to] of the image difference captioning system 102 cause the computing device(s) 800 to perform the operations described herein.”).
He in view of Fan does not teach:
perform singular value decompositions based on the products of the respective pairs of the plurality of matrices
and wherein performing the singular value decompositions includes using at least one of full-rank singular value decomposition factors and low-rank singular value decomposition factors.
However, Beren teaches perform singular value decompositions based on the products of the respective pairs of the plurality of matrices (Beren, pg. 4, “we can interpret the query and key matrices and value and output matrices…as single bilinear matrices QK and OV…We call these the QK and OV circuits.”, Beren, pg. 1, “we take the SVD of the weight matrices of the OV circuit”, Beren, pg. 34, “we can use our SVD directions approach to generate interpretable directions…while our techniques have much less success when applied to the QK circuit of the attention layers.”).
Beren also teaches and wherein performing the singular value decompositions includes using at least one of full-rank singular value decomposition factors and low-rank singular value decomposition factors. (Beren, pg. 17, “the OV circuits are low-rank matrices”).
He in view of Fan and Beren are both in the same field of endeavor (i.e. transformer neural networks). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine He in view of Fan and Beren to teach the above limitations. The motivation for doing so is that performing SVD on the attention matrices provides access to the internal semantic knowledge of the transformer neural network (cf. Beren, pg. 1, “If we take the SVD of the weight matrices of the OV circuit and of MLP layers of GPT models, and project them to token embedding space, we notice this results in highly interpretable semantic clusters…We can use this to both improve our understanding of transformer language models and edit their representations.”).
Regarding claim 19, He in view of Fan dicloses The system of claim 17.
He in view of Fan also discloses wherein the processing device is configured (Fan, paragraph [0078], “When executed by the one or more processors, the computer-executable instructions [wherein the processing device is configured] of the image difference captioning system 102 cause the computing device(s) 800 to perform the operations described herein.”).
He in view of Fan does not teach:
to perform singular value decompositions based on the products of the respective pairs of the plurality of matrices
and wherein performing the singular value decompositions includes using at least one of full-rank singular value decomposition factors and low-rank singular value decomposition factors.
However, Beren teaches to perform singular value decompositions based on the products of the respective pairs of the plurality of matrices (Beren, pg. 4, “we can interpret the query and key matrices and value and output matrices…as single bilinear matrices QK and OV…We call these the QK and OV circuits.”, Beren, pg. 1, “we take the SVD of the weight matrices of the OV circuit”, Beren, pg. 34, “we can use our SVD directions approach to generate interpretable directions…while our techniques have much less success when applied to the QK circuit of the attention layers.”).
Beren also teaches and wherein performing the singular value decompositions includes using at least one of full-rank singular value decomposition factors and low-rank singular value decomposition factors. (Beren, pg. 17, “the OV circuits are low-rank matrices”).
He in view of Fan and Beren are both in the same field of endeavor (i.e. transformer neural networks). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine He in view of Fan and Beren to teach the above limitations. The motivation for doing so is that performing SVD on the attention matrices provides access to the internal semantic knowledge of the transformer neural network (cf. Beren, pg. 1, “If we take the SVD of the weight matrices of the OV circuit and of MLP layers of GPT models, and project them to token embedding space, we notice this results in highly interpretable semantic clusters…We can use this to both improve our understanding of transformer language models and edit their representations.”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Wang, et al., “Linformer: Self-Attention with Linear Complexity” discloses a method of approximating the self-attention mechanism by a low-rank matrix in order to reduce its time/space complexity
Wang, et al., US Pre-Grant Publication US20240320482A1 “Transformer network with normalization including scaling parameter” dicloses a method of initializing a transformer neural network by multiplying attention matrices by a scaling parameter to increase network stability.
Volkovs, et al., US Pre-Grant Publication US20210255862A1 “Initialization Of Parameters For Machine-Learned Transformer Neural Network Architectures” discloses a method of initializing a transformer neural network without normalization layers to improve computational efficiency.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL ELIJAH BROWN whose telephone number is (571)270-5401. The examiner can normally be reached Monday-Thursday (7:30 am - 5:00 pm) ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T Bechtold can be reached at 571-431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/D.E.B./Examiner, Art Unit 2148 /MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148
Prosecution Insights