DETAILED ACTION
Claims 1-20 are presented for examination.
This office action is in response to submission of application on 07/11/2024.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 04/18/2025 and 12/05/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claims 9 and 19 are objected to because of the following informalities: “C is a used-defined number previously determined” should be “C is a user-defined number” (as per disclosure ¶[0018]). Appropriate correction is required.
Specification
The disclosure is objected to because of the following informalities:
In paragraph 0008, “FIG.1 is a block diagram of illustrating…” should be “FIG.1 is a block diagram illustrating…”.
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“a layer input module, configured to receive hidden state“ in claim 11. See 35 U.S.C. 112 rejection below for further comments.
“a linear transformation module, configured to generate key data, first value data, and query data” in claim 11. See 35 U.S.C. 112 rejection below for further comments.
" a key positional encoder, configured to generate first positionally encoded key data by encoding the key data positionally" in claim 11. See 35 U.S.C. 112 rejection below for further comments.
"a query positional encoder, configured to generate positionally encoded query data by encoding the query data positionally" in claim 11. See 35 U.S.C. 112 rejection below for further comments.
“a first multiplication module, configured to perform first element-wise dot product operations” in claim 11. See 35 U.S.C. 112 rejection below for further comments.
“a second multiplication module, configured to perform second element-wise dot product operations” in claim 11. See 35 U.S.C. 112 rejection below for further comments.
“a first adder, configured to add the attention output and the hidden state” in claim 11. See 35 U.S.C. 112 rejection below for further comments.
“a first batch matrix multiplication module, configured to perform a first matrix dot product” in claim 12. See 35 U.S.C. 112 rejection below for further comments.
“a second batch matrix multiplication module, configured to perform a second matrix dot product” in claim 12. See 35 U.S.C. 112 rejection below for further comments.
“a concatenation module, configured to concatenate” in claim 12. See 35 U.S.C. 112 rejection below for further comments.
“a third batch matrix multiplication module, configured to perform a third matrix dot product” in claim 13. See 35 U.S.C. 112 rejection below for further comments.
“ a fourth batch matrix multiplication module, configured to perform a fourth matrix dot product” in claim 13. See 35 U.S.C. 112 rejection below for further comments.
“a second adder, configured to add” in claim 13. See 35 U.S.C. 112 rejection below for further comments.
“a softmax module, configured to perform a softmax function” in claim 14. See 35 U.S.C. 112 rejection below for further comments.
“an input layer normalization module, configured to normalize the hidden state” in claim 15. See 35 U.S.C. 112 rejection below for further comments.
“an output layer normalization module, configured to normalize the updated hidden output” in claim 15. See 35 U.S.C. 112 rejection below for further comments.
“a ring buffer, configured to save the first value data, the second value data, the first positionally encoded key data, and the second positionally encoded key data” in claim 16. This element is interpreted under 35 U.S.C. 112(f) as a memory buffer allocated in processor memory (¶[0029]) with the algorithm described in the specification (¶[0018]).
“first key cache is configured to save the first positionally encoded key data” in claim 16. This element is interpreted under 35 U.S.C. 112(f) as a cache memory buffer allocated in processor memory (¶[0018] , ¶[0029]) with the algorithm described in the specification (¶[0018]).
“the second key cache is configured to save the second positionally encoded key data” in claim 16. This element is interpreted under 35 U.S.C. 112(f) as a cache memory buffer allocated in processor memory (¶[0018] , ¶[0029]) with the algorithm described in the specification (¶[0018]).
“a third adder, configured to replace null values” in claim 20. See 35 U.S.C. 112 rejection below for further comments.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 11-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The following claim limitations of claims 11-15 and 20 invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
“a layer input module, configured to receive hidden state“ in claim 11,
“a linear transformation module, configured to generate key data, first value data, and query data” in claim 11,
" a key positional encoder, configured to generate first positionally encoded key data by encoding the key data positionally" in claim 11,
"a query positional encoder, configured to generate positionally encoded query data by encoding the query data positionally" in claim 11,
“a first multiplication module, configured to perform first element-wise dot product operations” in claim 11,
“a second multiplication module, configured to perform second element-wise dot product operations” in claim 11,
“a first adder, configured to add the attention output and the hidden state” in claim 11,
“a first batch matrix multiplication module, configured to perform a first matrix dot product” in claim 12,
“a second batch matrix multiplication module, configured to perform a second matrix dot product” in claim 12,
“a concatenation module, configured to concatenate” in claim 12,
“a third batch matrix multiplication module, configured to perform a third matrix dot product” in claim 13,
“ a fourth batch matrix multiplication module, configured to perform a fourth matrix dot product” in claim 13,
“a second adder, configured to add” in claim 13,
“a softmax module, configured to perform a softmax function” in claim 14,
“an input layer normalization module, configured to normalize the hidden state” in claim 15,
“an output layer normalization module, configured to normalize the updated hidden output” in claim 15,
“a third adder, configured to replace null values” in claim 20.
The limitations above invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed functions and to clearly link the structure, material, or acts to the functions. The specification is devoid of adequate structure to perform the claimed functions. Therefore, the claims are indefinite and are rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Dependent claims 16-19 inherit the deficiency from claim 11 and therefore are rejected on the same basis.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 11-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
As described above, the disclosure does not provide adequate structure to perform the claimed functions recited in claims 11-15 and 20. The specification does not demonstrate that applicant has made an invention that achieves the claimed function because the invention is not described with sufficient detail such that one of ordinary skill in the art can reasonably conclude that the inventor had possession of the claimed invention. (FP 7.31.01.)
Dependent claims 16-19 inherit the deficiency from claim 11 and therefore are rejected on the same basis.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
The claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 2A Prong 1: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Claim 1 includes the steps of:
receiving a hidden state associated with at least one token;
generating key data, first value data, and query data according to a received hidden state;
generating first positionally encoded key data by encoding the key data positionally;
generating positionally encoded query data by encoding the query data positionally;
performing first element-wise dot product operations according to the first positionally encoded key data, the positionally encoded query data, and second positionally encoded key data to generate an attention score;
performing second element-wise dot product operations according to the first value data, the attention score, and second value data to generate an attention output; and
adding the attention output and the hidden state to generate an updated hidden output;
wherein the second positionally encoded key data is obtained and cached before the first positionally encoded key data is generated, and
the second value data is obtained and cached before the first value data is generated.
The broadest reasonable interpretation of the bolded limitations above are directed to a mental process able to be performed in the human mind through the use of a physical aid, like a pen and paper. A human can:
generate key, value and query data according to a hidden state;
generate positionally encoded key data by encoding the key data positionally;
generate positionally encoded query data by encoding the query data positionally.
The broadest reasonable interpretation of the italicized limitations falls within the mathematical concepts groupings of abstract ideas because they cover mathematical relationships, mathematical formulas or equations, and mathematical calculations. See MPEP 2106.04(a)(2), subsection I.
performing first element-wise dot product operations according to the first positionally encoded key data, the positionally encoded query data, and second positionally encoded key data to generate an attention score;
performing second element-wise dot product operations according to the first value data, the attention score, and second value data to generate an attention output; and
adding the attention output and the hidden state to generate an updated hidden output;
As drafted and under their broadest reasonable interpretation, the above limitations recite abstract idea of mathematical concepts and calculations because performing dot product operations to generate scores and adding values comprises application of mathematical formulas, i.e., dot product and addition operations.
Step 2A Prong 2: Does the claim recite additional elements that integrate the judicial exception into a practical application?
As drafted and under their broadest reasonable interpretation, the following limitations recite additional elements which amount to generic computer components recited at a high level of generality, with merely the words “apply it” or an equivalent with the judicial exception, merely including instructions to implement an abstract idea on the additional elements, or merely using the additional elements as a tool to perform an abstract idea, as discussed in MPEP 2106.05(f).
a transformer-based large language model (LLM).
As drafted and under their broadest reasonable interpretation, the following limitations recite additional elements which amount to mere data gathering and output recited at a high level of generality, and thus are insignificant extra-solution activity. See MPEP 2106.05(g) (“whether the limitation is significant”). In addition, all uses of the recited judicial exceptions require such data gathering and output, and, as such, these limitations do not impose any meaningful limits on the claim. These limitations amount to necessary data gathering and outputting. See MPEP 2106.05.
receiving a hidden state associated with at least one token;
the second positionally encoded key data is obtained and cached before the first positionally encoded key data is generated, and
the second value data is obtained and cached before the first value data is generated.
The additional elements have been considered both individually and as an ordered combination in order to determine whether they integrates the exception into a practical application. Therefore, no meaningful claim limits are imposed practicing the abstract idea. Accordingly, at Step 2A, prong two, the additional elements do not integrate the judicial exception into a practical application.
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
The claim limitation(s) reciting generic computer elements amounts to no more than mere instructions to apply the exception using a generic computer.
The claim reciting the additional element(s) of “obtaining”, “caching”, “receiving” and/or “transmitting” amount to necessary data gathering and output.
The additional elements have been considered both individually and as an ordered combination in order to determine whether they warrant significantly more consideration. Thus, the claim does not provide an inventive concept.
The claim is ineligible.
Claims 2-10 further recite limitations that encompass mental evaluations that are practically performed in the human mind, but for the recitation of generic computer components, and mathematical concepts groupings of abstract ideas because they cover mathematical relationships, mathematical formulas or equations, and mathematical calculations. Claims 6-8 further recite storing and retrieving information in memory and are thus directed to well‐understood, routine, and conventional functions. The claims do not integrate the judicial exception into practical application. The claims do not recite additional elements that are sufficient to amount to significantly more than the judicial exception.
Claims 2-10 are ineligible.
Claims 11-20 are substantially similar to claims 1-10, and are rejected on the same basis as claims 1-10. These claims recite additional elements that amount to generic computer components recited at a high level of generality, with merely the words “apply it” or an equivalent with the judicial exception, merely including instructions to implement an abstract idea on the additional elements, or merely using the additional elements as a tool to perform an abstract idea.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 9-15, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Yu et al. (Pub. No.: US 2023/0177399 A1), as cited in the IDS dated 04/18/2025, hereafter Yu, in view of Wang et al. ("SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning"), hereafter Wang.
Regarding claim 1, Yu discloses:
An auto-regressive method for a transformer-based large language model (LLM), the auto-regressive method comprising (¶[0077]),
receiving a hidden state associated with at least one token (Fig. 2A and ¶[0041] teaches receiving a normalized tensor, i.e. a hidden state, associated with at least one input token),
generating key data, first value data, and query data according to a received hidden state (Fig. 2A and ¶[0041] teaches generating key data, first value data, and query data according to a received hidden state in the QKV operation block),
generating first positionally encoded key data by encoding the key data positionally; generating positionally encoded query data by encoding the query data positionally (Fig. 2A and ¶[0041] teaches positionally encoding the key and query data),
performing first … product operations according to the first positionally encoded key data, the positionally encoded query data, and second positionally encoded key data to generate an attention score (Fig. 2A and ¶[0061-0062] teaches performing product operations according to encoded key and query data, and second encoded key data, i.e. key cache tensors, to generate a multiplied tensor for self attention as an attention score),
performing second … product operations according to the first value data, the attention score, and second value data to generate an attention output (¶[0063] teaches performing a second product operation according to the value data, the multiplied tensor attention score, and value data cache as second value data to generate an attention output),
adding the attention output and the hidden state to generate an updated hidden output (Fig. 2A and ¶[0067] teaches adding the attention output to internal states to generate updated internal outputs),
wherein the second positionally encoded key data is obtained and cached before the first positionally encoded key data is generated (¶[0060] and [0062] teaches the second positionally encoded key data to be obtained and cached before the first positionally encoded key data is generated)
the second value data is obtained and cached before the first value data is generated (¶[0060] and [0063] teaches the second value data is obtained and cached before the first value data is generated).
While Yu discloses performing first … product operations according to the first positionally encoded key data, the positionally encoded query data, and second positionally encoded key data to generate an attention score and performing second … product operations according to the first value data, the attention score, and second value data to generate an attention output, they do not explicitly recite these product operations to be element-wise dot product operations.
Wang discloses:
performing first element-wise dot product operations… performing second element-wise dot product operations (Fig. 8 and Fig. 11 teaches multiplication modules that perform multiple elementwise dot product operations).
Yu and Wang are analogous art because they are from the same field of endeavor, attention and machine learning models.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Yu to include performing first element-wise dot product operations… performing second element-wise dot product operations, based on the teachings of Wang. One of ordinary skill in the art would have been motivated to make this modification for orders of magnitude speedup and energy savings, as suggested by Wang (page 13, conclusion paragraph).
Regarding claim 2, Yu, in view of Wang, discloses the computer-implemented method of claim 1 performing first element-wise dot product operations according to the first positionally encoded key data, the positionally encoded query data, and the second positionally encoded key data to generate an attention score (and thus the rejection of claim 1 is incorporated). Wang further discloses:
performing a first matrix dot product on the first positionally encoded key data and the positionally encoded query data to generate a first product output (Fig. 8, element “Q x KT”, Fig. 11 and page 8, left column, penultimate paragraph, lines 3-4 “a row of the key matrix Ki is loaded from the K SRAM, multiplied by Q with a multiplier array” teaches generating a first product output between a first key matrix and query data),
performing a second matrix dot product on the second positionally encoded key data and the positionally encoded query data to generate a second product output; and concatenating the first product output and the second product output to generate the attention score (Fig. 8, element “Q x KT”, Fig. 11 and page 8, left column, penultimate paragraph, lines 5-12 “Adder tree then computes attention scores by reducing all multiplied results… The corresponding multiple Kis are packed into the same line in the Key SRAM, and the query is broadcast for 512 times so that all Kis can have access to the same Q.” teaches performing subsequent product outputs on remainder of key data from storage before concatenation to output attention scores) .
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Yu to include performing a first matrix dot product on the first positionally encoded key data and the positionally encoded query data to generate a first product output, performing a second matrix dot product on the second positionally encoded key data and the positionally encoded query data to generate a second product output; and concatenating the first product output and the second product output to generate the attention score, based on the teachings of Wang. One of ordinary skill in the art would have been motivated to make this modification for orders of magnitude speedup and energy savings, as suggested by Wang (page 13, conclusion paragraph).
Regarding claim 3, Yu, in view of Wang, discloses the computer-implemented method of claim 2 (and thus the rejection of claim 2 is incorporated). Wang further discloses:
performing a third matrix dot product according to the attention score and the second value data to generate a third product output; performing a fourth matrix dot product according to the attention score and the first value data to generate a fourth product output; and adding the third product output and the fourth product output to generate the attention output (Fig. 8, top right element “attention prob x V” and page 8, right column, second paragraph, lines 1-3 “attention prob-value multiplication unit takes the attention probabilities as inputs, multiplies them with the Vs and then accumulates to get attention outputs” teaches generating third and fourth product output from different value data from storage and the attention score to generate the attention output).
Regarding claim 4, Yu, in view of Wang, discloses the computer-implemented method of claim 1 (and thus the rejection of claim 1 is incorporated). Yu further discloses:
performing a softmax function to the attention score for generating an attention weight (¶[0063] teaches performing a softmax function to the attention score for generating an attention weight P),
wherein the second … product operations are performed according to the first value data, the attention weight, and the second value data to generate the attention output (¶[0063] teaches the second product operations being performed according to the first value data, the attention weight P, and the second value data).
While Yu discloses wherein the second … product operations are performed according to the first value data, the attention weight, and the second value data to generate the attention output, they do not explicitly recite these product operations to be element-wise dot product operations.
Wang discloses:
second element-wise dot product operations are performed …(Fig. 8 teaches second element wise dot product operations being performed).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Yu to include second element-wise dot product operations are performed, based on the teachings of Wang. One of ordinary skill in the art would have been motivated to make this modification for orders of magnitude speedup and energy savings, as suggested by Wang (page 13, conclusion paragraph).
Regarding claim 5, Yu, in view of Wang, discloses the computer-implemented method of claim 1 (and thus the rejection of claim 1 is incorporated). Yu further discloses:
normalizing the hidden state as a normalized hidden state by adjusting a square sum of elements of the hidden state equal to a fixed number (Fig. 2A and ¶[0055] teaches normalizing the hidden state as a normalized hidden state by adjusting a square sum of elements of the hidden state equal to a fixed number),
wherein the key data, the first value data, and the query data are generated according to the normalized hidden state (Fig. 2A and ¶[0056]),
normalizing the updated hidden output as a normalized updated hidden output by adjusting a square sum of elements of the updated hidden output equal to the fixed number (Fig. 2A and ¶[0067] teaches normalizing the updated hidden output as a normalized updated hidden output by adjusting a square sum of elements of the updated hidden output equal to the fixed number).
Regarding claim 9, Yu, in view of Wang, discloses the computer-implemented method of claim 1 (and thus the rejection of claim 1 is incorporated). Yu further discloses:
a signal shape format of the first value data and the first positionally encoded key data is represented as (B,N,T,D), a signal shape format of the second value data and the second positionally encoded key data is represented as (B,N,C,D), wherein B is a batch size, T is a token amount, N is an attention head amount, D is a head dimension for each attention head, and C is a used-defined number previously determined (¶[0054-0056] and ¶[0058] teaches signal shape formats for values and keys represented by a batch size, a token amount, attention head amount, a head dimension for each attention head, and a used-defined number).
Regarding claim 10, Yu, in view of Wang, discloses the computer-implemented method of claim 4 (and thus the rejection of claim 4 is incorporated). Wang further discloses:
replacing null values of the attention score with mask values to generate an updated attention score (Fig. 8 and Fig. 9 teaches replacing null values of the attention score with mask values to generate an updated attention score through a zero eliminator).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Yu to include replacing null values of the attention score with mask values to generate an updated attention score, based on the teachings of Wang. One of ordinary skill in the art would have been motivated to make this modification for orders of magnitude speedup and energy savings, as suggested by Wang (page 13, conclusion paragraph).
Claims 11-15 and 19-20 are substantially similar to claims 1-5 and 9-10 respectively, and are rejected on the same basis as claims 1-5 and 9-10 respectively.
Claims 6, 7, 8, 16, 17, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Yu et al. (Pub. No.: US 2023/0177399 A1), as cited in the IDS dated 04/18/2025, hereafter Yu, in view of Wang et al. ("SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning"), hereafter Wang, in further view of Shi et al. ("Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition"), hereafter Shi.
Regarding claim 6, Yu, in view of Wang, discloses the computer-implemented method of claim 1 (and thus the rejection of claim 1 is incorporated). Yu further discloses:
wherein the first value cache is configured to save the first value data, the second value cache is configured to save the second value data, the first key cache is configured to save the first positionally encoded key data, and the second key cache is configured to save the second positionally encoded key data (¶[0088] teaches respective key and value caches).
While Yu discloses wherein the first value cache is configured to save the first value data, the second value cache is configured to save the second value data, the first key cache is configured to save the first positionally encoded key data, and the second key cache is configured to save the second positionally encoded key data, they do not disclose:
slicing a part of memory segments from a ring buffer to generate a first value cache and/or a first key cache;
slicing another part of memory segments from the ring buffer as sliding windows to generate a second value cache and/or a second key cache;
Shi discloses:
slicing a part of memory segments from a ring buffer to generate a first value cache and/or a first key cache (Fig. 1, page 1, right column, paragraph 2, lines 4-5 “AM-TRF introduces a memory bank. Each vector in the memory bank is an abstract embedding from the previous one segment”, page 2, right column, paragraph 2, lines 5-6 “Emformer only computes the key, value projections for the memory bank,” and page 2, right column, penultimate paragraph, lines 5-6 “the memory bank is implemented in ring buffer” teaches memory vectors segmented to generate a key or value cache),
slicing another part of memory segments from the ring buffer as sliding windows to generate a second value cache and/or a second key cache (Fig. 1, page 1, right column, paragraph 2, lines 4-5 “AM-TRF introduces a memory bank. Each vector in the memory bank is an abstract embedding from the previous one segment”, page 2, right column, paragraph 2, lines 5-6 “Emformer only computes the key, value projections for the memory bank,” and page 2, right column, penultimate paragraph, lines 5-6 “the memory bank is implemented in ring buffer” teaches memory vectors segmented in a sliding window fashion to generate a second key or value cache for the next layer).
Yu, Wang, and Shi are analogous art because they are from the same field of endeavor, attention and machine learning models.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Yu, in view of Wang, to include slicing a part of memory segments from a ring buffer to generate a first value cache and/or a first key cache and slicing another part of memory segments from the ring buffer as sliding windows to generate a second value cache and/or a second key cache, based on the teachings of Shi. One of ordinary skill in the art would have been motivated to make this modification to speed up training, as suggested by Shi (page 4, conclusion paragraph).
Regarding claim 7, Yu, in view of Wang, in further view of Shi, discloses the computer-implemented method of claim 6 (and thus the rejection of claim 1 is incorporated). Yu further discloses:
updating the part of memory segments…by incrementing an offset value of memory addresses after the first value cache and/or a first key cache is saved (¶[0099] teaches updating memory segments by incrementing memory addresses after first caches are saved),
wherein after the part of memory segments … are updated, the part of memory segments comprises the first value cache and/or a first key cache, and a part of second positionally encoded key data and/or a part of second value data (¶0088] and ¶[0099] teaches part of the memory comprises first caches and a part of the data).
the second value data and/or the second positionally encoded key data are updated after the part of memory segments are shifted ([0099] teaches updating value and key data for a new batch after memory segments are shifted).
While Yu discloses updating the part of memory segments … by incrementing an offset value of memory addresses after the first value cache and/or a first key cache is saved and wherein after the part of memory segments … are updated, the part of memory segments comprises the first value cache and/or a first key cache, and a part of second positionally encoded key data and/or a part of second value data, they do not explicitly recite updating the part of memory segments from the ring buffer.
Shi discloses:
updating the part of memory segments from the ring buffer … (Fig. 1, page 1, right column, paragraph 2, lines 4-5 “AM-TRF introduces a memory bank. Each vector in the memory bank is an abstract embedding from the previous one segment”, page 2, right column, paragraph 2, lines 5-6 “Emformer only computes the key, value projections for the memory bank,” and page 2, right column, penultimate paragraph, lines 5-6 “the memory bank is implemented in ring buffer” teaches updating the part of memory segments from the ring buffer).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Yu, in view of Wang, to include updating the part of memory segments from the ring buffer, based on the teachings of Shi. One of ordinary skill in the art would have been motivated to make this modification to speed up training, as suggested by Shi (page 4, conclusion paragraph).
Regarding claim 8, Yu, in view of Wang, in further view of Shi, discloses the computer-implemented method of claim 7 (and thus the rejection of claim 7 is incorporated). Shi further discloses:
copying the part of memory segments of the ring buffer according to their initialized memory addresses after the part of memory segments are shifted to hit an end memory segment of the ring buffer (Fig. 2, paragraph below Fig. 2, lines 7-9 “Emformer makes a hard copy of each segment’s look-ahead context and puts the look-ahead context copy at the input sequence’s beginning”, and page 2, right column, penultimate paragraph, lines 5-6 “the memory bank is implemented in ring buffer” teaches copying the part of memory segments of the ring buffer according to their initialized memory addresses after the part of memory segments are shifted to hit an end memory segment of the ring buffer).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Yu, in view of Wang, to include copying the part of memory segments of the ring buffer according to their initialized memory addresses after the part of memory segments are shifted to hit an end memory segment of the ring buffer, based on the teachings of Shi. One of ordinary skill in the art would have been motivated to make this modification to speed up training, as suggested by Shi (page 4, conclusion paragraph).
Claims 16-18 are substantially similar to claims 6-8, and are rejected on the same basis as claims 6-8.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's
disclosure.
U.S. Pub No. 11646017 B1: Shi et al. teaches self attention and buffer rings.
Li et al. (“Sequence Parallelism: Long Sequence Training from System Perspective”) teaches ring self attention.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HUMAIRA ZAHIN MAUNI whose telephone number is (703)756-5654. The examiner can normally be reached Monday - Friday, 9 am - 5 pm (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MATT ELL can be reached at (571) 270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/H.Z.M./Examiner, Art Unit 2141
/MATTHEW ELL/Supervisory Patent Examiner, Art Unit 2141
Prosecution Insights