DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
The present application is being examined under the claims filed 09/09/2025.
Claims 1-11 and 13-21 are pending.
Response to Amendment
This Office Action is in response to Applicant’s communication filed 09/09/2025 in response to office action mailed 07/11/2025. The Applicant’s remarks and any amendments to the claims or specification have been considered with the results that follow.
Response to Arguments
Regarding 35 U.S.C. 101
In Remarks page 9, Argument 1
Applicant respectfully submits the present amendments to claim 1 overcome the rejection because the amendments further demonstrate how the claimed subject matter 1) does not recite, set forth, describe, or otherwise encompass a recognized judicial exception; AND (2) amounts to significantly more than the alleged abstract idea to which the subject matter is directed.
Examiner’s response to Argument 1
Examiner disagrees. For the reasons given below, the rejections under 35 U.S.C. 101 are maintained.
In Remarks page 9, Argument 2
First, Applicant notes that claim 1 has been specifically been amended to specify limitations that cannot be performed in the human mind or using paper and pen. For example, claim 1 now specifies specific and non-general internal compute units as well as a plurality of functional abstraction layers of the cloud computing environment. Respectfully, as discussed in detail during an Examiner interview on September 5, 2025 between Attorney Derrick Breska (Att. Reg. No. 74,266) and Examiner Baker, because these elements are used in a detailed way ("internal compute units of the transformer are performed with resource provisioning in which the internal compute units are dynamically procured and controlled via a metering capability") that cannot be performed in the human mind or using paper and pen, the limitations of claim 1 cannot be alleged to be an ineligible abstract idea.
Examiner’s response to Argument 2
Examiner disagrees. Although the limitations are claimed as requiring a computer, MPEP 2106.04(a)(2) III. C. recites ““Claims can recite a mental process even if they are claimed as being performed on a computer. […] For instance, examiners should review the specification to determine if the claimed invention is described as a concept that is performed in the human mind and applicant is merely claiming that concept performed 1) on a generic computer, or 2) in a computer environment, or 3) is merely using a computer as a tool to perform the concept.” Abstract ideas that are merely performed using cloud computing as a tool, or in a cloud computing environment are still directed to abstract ideas. Thus adding language related to cloud computing to limitations that could otherwise be performed in the human mind would not change their interpretation as abstract ideas. Furthermore, the cloud computing limitations are directed to ordinary cloud computing management (e.g. charging users based on cloud computing usage).
In Remarks page 9, Argument 3
Second, Applicant notes that claim 1 has been specifically been amended to specify how the benefits and improvements to efficiencies enabled by claim 1 are achieved. These amendments specifically address the rejection's statement that claim 1 of the present application does not provide details for how the outcome is obtained. See NFOA at 5. More specifically, claim 1 of the present application has been amended to clarify affirmative steps that detail the efficient computations, e.g., "wherein the computations of the internal compute units of the transformer are performed with resource provisioning in which the internal compute units are dynamically procured and controlled via a metering capability to perform tasks within the cloud computing environment including the computations". For this reason, Applicant asserts that claim 1 amounts to significantly more by both providing other meaningful limitations beyond generally linking the use of the judicial exception to a particular technological environment and applying the judicial exception with, or by use of, a particular machine.
Examiner’s response to Argument 3
Examiner disagrees. MPEP 2106.05(f)(2) recites “Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more”. The added cloud computing limitations amount to merely using a computer in its ordinary capacity. Issuing instructions, using machine learning in a cloud computing environment, and dynamically controlling resources using metering are generic and ordinary computer functions and do not amount to an improvement to technology. Moreover, any alleged improvements appear to be caused by the mental process steps alone and the generic cloud computing is merely used as a tool. Therefore, the claims as amended do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception itself.
In Remarks page 10, Argument 4
(Examiner summarizes Applicant’s arguments) Applicant argues that claim 1 was amended solely to expedite prosecution, the rejection of claim 1 as amended and any claims depending from claim 1 is improper. Applicant further argues that claims 13-19 and 20 should be deemed eligible for similar reasons as claim 1.
Examiner’s response to Argument 4
Examiner disagrees for the reasons provided in responses to arguments above. The rejections of analogous and dependent claims under 35 U.S.C. 101 are maintained for similar reasons as claim 1. See rejections under 35 U.S.C. 101 for a complete analysis.
In Remarks pages 11-12, Argument 5
(Examiner summarizes Applicant’s arguments) Applicant argues that newly added limitations are not taught by any of the references currently of record and therefore claim 1 should be allowable.
Examiner’s response to Argument 5
Applicant’s amendments convincingly overcome the rejections under 35 U.S.C. 103. However, an updated search revealed new art that is used in new grounds of rejection necessitated by the amendment. See rejections under 35 U.S.C. 103 below.
In Remarks pages 12-16, Argument 6
(Examiner summarizes Applicant’s arguments) Applicant argues that the dependent and analogous claims should be allowable by virtue of claim 1.
Examiner’s response to Argument 6
New grounds of rejection is applied to claim 1, and thus the rejections of the analogous claims and dependent claims are maintained for similar reasons.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-11 and 13-21 are rejected under 35 U.S.C. 101 for containing an abstract idea without significantly more.
Regarding Claim 1:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes, the claim is to a process.
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
determining a threshold sequence-size for a transformer of the cloud computing environment — This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.). The limitation is directed to a mental process because it amounts to making a judgement of the maximum appropriate size for a sequence.
organizing a batch of sequences according to the threshold sequence-size — This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.). The limitation is directed to a mental process because it amounts to performing an evaluation of an ordering of a sequence based on given metrics.
causing, via use of a management layer of the functional abstraction layers of the cloud computing to issue instructions within the cloud computing environment, the transformer to process the input organized batch of sequences through multiple layers of the transformer — This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper. The limitation is directed to a mental process because it amounts to performing an evaluation of given data, for example, using a series of multiplications and activation functions. See MPEP 2106.04(a)(2) III. C. “Claims can recite a mental process even if they are claimed as being performed on a computer. […] For instance, examiners should review the specification to determine if the claimed invention is described as a concept that is performed in the human mind and applicant is merely claiming that concept performed 1) on a generic computer, or 2) in a computer environment, or 3) is merely using a computer as a tool to perform the concept.”
wherein the output includes translations of the sequences, wherein the translations are generated by an artificial intelligence (AI) model of the transformer during the processing of the input organized batch of sequences — This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.). The limitation is directed to a mental process because it amounts to performing an evaluation of text to determine a language translation by using a set of machine learning parameters.
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim does not recite additional elements that integrate the judicial exception into a practical application. The additional elements:
In a cloud computing environment having a plurality of functional abstraction layers for processing sequences, a computer-implemented method, comprising: — This limitation is directed to merely applying an abstract idea using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.04(d)).
inputting the organized batch of sequences into the transformer — This limitation is directed to mere data gathering and outputting which has been recognized by the courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) as insignificant extra-solution activity (see MPEP 2106.05(g)).
wherein an order of the sequences that results from the organizing, when computed by internal compute units of the transformer, results in relatively more efficient computations than would otherwise result from the internal compute units of the transformer computing the sequences without the organizing, wherein the efficiencies of the computations are based on metrics including a number of pipeline stalls that occur during the computations — This limitation is directed to mere instructions to apply a judicial exception. The limitation presents a mere idea of a solution to provide more efficient computations without the requisite details to achieve the solution (see MPEP 2106.05(f)(1)). Even if the limitation is implemented on a generic computer (see MPEP 2106.05(f)(2), 2106.04(d)), the limitation does not integrate the judicial exception into a practical application.
wherein the computations of the internal compute units of the transformer are performed with resource provisioning in which the internal compute units are dynamically procured and controlled via a metering capability to perform tasks within the cloud computing environment including the computations — This limitation is directed to mere instructions to apply a judicial exception. Using cloud computing metering to apply a judicial exception (see MPEP 2106.05(f)) is insufficient to integrate the judicial exception into a practical application. Even if the cloud computing metering is implemented on a generic computer (see MPEP 2106.05(f)(2), 2106.04(d)), the limitation does not integrate the judicial exception into a practical application.
and obtaining an output of the transformer that results from the processing of the input organized batch of sequences by the transformer — This limitation is directed to mere data gathering and outputting which has been recognized by the courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) as insignificant extra-solution activity (see MPEP 2106.05(g)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, the claim does not recite additional elements which amount to significantly more than the abstract idea itself. The additional elements as identified in step 2A prong 2:
In a cloud computing environment having a plurality of functional abstraction layers for processing sequences, a computer-implemented method, comprising: — Using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.05(d)) cannot amount to significantly more than the judicial exception itself.
inputting the organized batch of sequences into the transformer — This limitation is recited at a high level of generality and amounts to mere data gathering of transmitting and receiving data over a network, which is well-understood, routine, and conventional activity (see MPEP 2106.05(d) II.), which cannot amount to significantly more than the judicial exception.
wherein an order of the sequences that results from the organizing, when computed by internal compute units of the transformer, results in relatively more efficient computations than would otherwise result from the internal compute units of the transformer computing the sequences without the organizing, wherein the efficiencies of the computations are based on metrics including a number of pipeline stalls that occur during the computations — Mere instructions to apply a judicial exception (see MPEP 2106.05(f)) and using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.05(d)) cannot amount to significantly more than the judicial exception itself.
wherein the computations of the internal compute units of the transformer are performed with resource provisioning in which the internal compute units are dynamically procured and controlled via a metering capability to perform tasks within the cloud computing environment including the computations — Mere instructions to apply a judicial exception (see MPEP 2106.05(f)) and using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.05(d)) cannot amount to significantly more than the judicial exception itself.
and obtaining an output of the transformer that results from the processing of the input organized batch of sequences by the transformer — This limitation is recited at a high level of generality and amounts to mere data gathering of transmitting and receiving data over a network, which is well-understood, routine, and conventional activity (see MPEP 2106.05(d) II.), which cannot amount to significantly more than the judicial exception.
Regarding Claim 2
Claim 2 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which included an abstract idea (see rejection for claim 1). The claim recites the additional limitations:
Step 2A Prong 2:
wherein the transformer includes a Bidirectional Encoder Representations from Transformers (BERT) transformer — This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the field of the transformer.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
wherein the transformer includes a Bidirectional Encoder Representations from Transformers (BERT) transformer — Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 3
Claim 3 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which included an abstract idea (see rejection for claim 1). The claim recites the additional limitations:
Step 2A Prong 2:
wherein the transformer includes a Generative Pre-trained Transformer (GPT) — This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the field of the transformer.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
wherein the transformer includes a Generative Pre-trained Transformer (GPT) — Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 4
Claim 4 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which included an abstract idea (see rejection for claim 1). The claim recites the additional limitations:
Step 2A Prong 2:
wherein the transformer includes one or more vector matrix multipliers (VMMs)— This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the field of the transformer.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
wherein the transformer includes one or more vector matrix multipliers (VMMs) — Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 5
Claim 5 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which included an abstract idea (see rejection for claim 1). The claim recites the additional limitations:
Step 2A Prong 2:
wherein the transformer includes one or more attention-compute blocks — This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the field of the transformer.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
wherein the transformer includes one or more attention-compute blocks — Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 6
Claim 6 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which included an abstract idea (see rejection for claim 1). The claim recites the additional limitations:
Step 2A Prong 2:
where the layers include VMM blocks and attention-compute blocks — This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the field of the transformer.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
where the layers include VMM blocks and attention-compute blocks — Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 7
Claim 7 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which included an abstract idea (see rejection for claim 1). The claim recites the additional limitations:
Step 2A Prong 2:
wherein first sequences of the batch of sequences include a plurality of series of words, wherein second sequences of the batch of sequences include regions-of-interest from within an image — This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the field of the data of the batch of sequences.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
wherein first sequences of the batch of sequences include a plurality of series of words, wherein second sequences of the batch of sequences include regions-of-interest from within an image — Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 8
Claim 8 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which included an abstract idea (see rejection for claim 1). The claim recites the additional limitations:
Step 2A Prong 2:
wherein the threshold sequence- size indicates a representative length of a sequence that is input into the transformer, which can be compared to the length of each sequence that is input into the transformer — This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the field of the threshold sequence size.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
wherein the threshold sequence- size indicates a representative length of a sequence that is input into the transformer, wherein the representative length is compared to lengths of the sequences that are input into the transformer — Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 9
Claim 9 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which included an abstract idea (see rejection for claim 1). The claim recites the additional limitations:
Step 2A Prong 1:
where input excitation vectors are multiplied by trained-weight matrices — This limitation is directed to the abstract idea of a mathematical process, and mathematical calculations in particular (MPEP 2106.04(a)(2) I. C.). The claim describes the mathematical operations of multiplying a matrix and a vector in words.
Step 2A Prong 2:
wherein: the threshold sequence-size results in a first compute time for a sequence by vector matrix multipliers (VMMs) of the transformer — This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the field of the transformer.
the threshold sequence-size results in a second compute time for the sequence by attention-compute blocks of the transformer — This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the field of the transformer.
and the first compute time equals the second compute time for the threshold sequence-size — This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the fields of the first and second compute times.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
wherein: the threshold sequence-size results in a first compute time for a sequence by vector matrix multipliers (VMMs) of the transformer — Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
the threshold sequence-size results in a second compute time for the sequence by attention-compute blocks of the transformer — Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
and the first compute time equals the second compute time for the threshold sequence-size — Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 10
Claim 10 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which included an abstract idea (see rejection for claim 1). The claim merely recites the additional abstract ideas:
Step 2A Prong 1:
wherein organizing the batch of sequences includes: comparing lengths of the sequences to the threshold sequence- size, wherein the lengths of at least some of the sequences are different — This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
and arranging the sequences in an alternating order based on the comparisons — This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 11
Claim 11 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which included an abstract idea (see rejection for claim 1). The claim recites the additional limitations:
Step 2A Prong 2:
wherein the batch of sequences is input into the transformer in an alternating order determined utilizing the threshold sequence-size — This limitation is directed to mere data gathering and outputting which has been recognized by the courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) as insignificant extra-solution activity (see MPEP 2106.05(g)).
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
wherein the batch of sequences is input into the transformer in an alternating order determined utilizing the threshold sequence-size — This limitation is recited at a high level of generality and amounts to mere data gathering of transmitting and receiving data over a network, which is well-understood, routine, and conventional activity (see MPEP 2106.05(d) II.), which cannot amount to significantly more than the judicial exception.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 13
Independent claim 13 is a computer program product claim corresponding to method claim 1, which was directed to an abstract idea, therefore the same rejection and rationale applies. The only difference is that claim 13 recites the following additional elements treated under step 2A prong 2 and step 2B:
Step 2A Prong 2:
A computer program product comprising one or more non-transitory computer readable storage media, and program instructions collectively stored on the one or more non-transitory computer readable storage media, the program instructions comprising instructions configured to cause one or more processors to perform a method comprising — This limitation is directed to merely applying an abstract idea using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.04(d)).
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
A computer program product comprising one or more non-transitory computer readable storage media, and program instructions collectively stored on the one or more non-transitory computer readable storage media, the program instructions comprising instructions configured to cause one or more processors to perform a method comprising — Using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.05(d)) cannot amount to significantly more than the judicial exception itself.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 14
Dependent claim 14 is a computer program product claim corresponding to method claim 2, which was directed to an abstract idea, therefore the same rejection and rationale applies.
Regarding Claim 15
Dependent claim 15 is a computer program product claim corresponding to method claim 3, which was directed to an abstract idea, therefore the same rejection and rationale applies.
Regarding Claim 16
Dependent claim 16 is a computer program product claim corresponding to method claim 4, which was directed to an abstract idea, therefore the same rejection and rationale applies.
Regarding Claim 17
Dependent claim 17 is a computer program product claim corresponding to method claim 5, which was directed to an abstract idea, therefore the same rejection and rationale applies.
Regarding Claim 18
Dependent claim 18 is a computer program product claim corresponding to method claim 6, which was directed to an abstract idea, therefore the same rejection and rationale applies.
Regarding Claim 19
Dependent claim 19 is a computer program product claim corresponding to method claim 7, which was directed to an abstract idea, therefore the same rejection and rationale applies.
Regarding Claim 20
Independent claim 20 is a computer system claim corresponding to method claim 7, which was directed to an abstract idea, therefore the same rejection and rationale applies. The only difference is that claim 20 recites the following additional elements treated under step 2A prong 2 and step 2B:
Step 2A Prong 2:
A system of a transformer environment for processing sequences, comprising: a processor; and logic integrated with the processor, executable by the processor, or integrated with and executable by the processor, the logic being configured to: — This limitation is directed to merely applying an abstract idea using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.04(d)).
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
A system of a transformer environment for processing sequences, comprising: a processor; and logic integrated with the processor, executable by the processor, or integrated with and executable by the processor, the logic being configured to: — Using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.05(d)) cannot amount to significantly more than the judicial exception itself.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 21
Claim 21 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which included an abstract idea (see rejection for claim 1). The claim recites the additional limitations:
Step 2A Prong 2:
wherein the metrics further include an amount of time the internal compute units are kept occupied and a total latency — This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the field of the efficiency calculation measurements.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
wherein the metrics further include an amount of time the internal compute units are kept occupied and a total latency — Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4-5, 7, 13, 16-17, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Dai et al “Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context” herein referred to as Dai in view of Chen et al. “Efficient Tensor Core-Based GPU Kernels for Structured Sparsity under Reduced Precision” herein referred to as Chen, Yao et al. “Multimodal Transformer for Multimodal Machine Translation” herein referred to as Yao, and Jonnakuti “Scalable NLP in the Enterprise: Training Transformer Models on Distributed Cloud GPUs” herein referred to as Jonnakuti.
Regarding Claim 1
Dai teaches:
In a transformer environment for processing sequences, a computer-implemented method, comprising:
(page 1 abstract last sentences) “When trained only on WikiText-103, Transformer-XL[*Examiner notes: transformer environment] manages to generate reasonably coherent, novel text articles with thousands of tokens. Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch[*Examiner notes: computer-implemented method].”
determining a threshold sequence-size for a transformer of the transformer environment;
(page 3 column 2 paragraph 1 line 1) “To address the limitations of using a fixed-length context, we propose to introduce a recurrence mechanism to the Transformer architecture[*Examiner notes: mapped to transformer].”; (page 2 column 2 last paragraph) “One feasible but crude approximation is to split the entire corpus into shorter segments of manageable sizes, and only train the model within each segment, ignoring all contextual information from previous segments. […] in practice it has been standard practice to simply chunk long text into fixed-length segments[*Examiner notes: mapped to threshold sequence size] due to improved efficiency”; (page 3 column 2 paragraph 1 line 12) “Formally, let the two consecutive segments of length L[*Examiner notes: mapped to determining a threshold sequence size] be”
organizing a batch of sequences according to the threshold sequence-size
(page 3 column 2 paragraph 1 line 12) “Formally, let the two consecutive segments of length L be sτ = [xτ,1, · · · , xτ,L] and sτ+1 = [xτ+1,1, · · · , xτ+1,L] respectively[*Examiner notes: mapped to organizing a batch of sequence according to threshold sequence size].”; [*Examiner notes: The broadest reasonable interpretation of the term “organizing” includes arranging in a structured way. The consecutive sequences of length L sτ and sτ+1 above are arranged in a structured way and thus are organized.]
inputting the organized batch of sequences into the transformer
(page 4 column 2 line 9) “Then, the actual input to the Transformer is the element-wise addition of the word embeddings and the positional encodings. If we simply adapt this positional encoding to our recurrence mechanism, the hidden state sequence would be computed schematically by hτ+1 = f(hτ , Esτ+1 + U1:L) hτ = f(hτ−1, Esτ + U1:L), where Esτ ∈ RL×d is the word embedding sequence of sτ[*Examiner notes: inputting the organized batch of sequences into the transformer]”
causing, via issuing instructions, the transformer to process the input organized batch of sequences through multiple layers of the transformer;
(page 6 column 1 last paragraph) “The dataset enwik8 contains 100M bytes of unprocessed
Wikipedia text. We compare our architecture with the previous results in Table 2. Under the model size constraint, the 12-layer Transformer-XL[*Examiner notes: multiple layers of the transformer] achieves a new SoTA result”; (page 4 column 2 line 9) “Then, the actual input to the Transformer is the element-wise addition of the word embeddings and the positional encodings. If we simply adapt this positional encoding to our recurrence mechanism, the hidden state sequence would be computed schematically by hτ+1 = f(hτ , Esτ+1 + U1:L) hτ = f(hτ−1, Esτ + U1:L), where Esτ ∈ RL×d is the word embedding sequence of sτ[*Examiner notes: process the organized batch sequences through multiple layers]”
wherein an order of the sequences that results from the organizing, when computed by internal compute units of the transformer, results in relatively more efficient computations than would otherwise result from the internal compute units of the transformer computing the sequences without the organizing
(page 1 abstract) “As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation.”; (page 5 column 2 second to last paragraph) “In addition, it is worth mentioning that a naive way to compute A requires computing Wnk;RRi-j for all pairs (i; j), whose cost is quadratic w.r.t. the sequence length. However, noticing that the value of i - j only ranges from zero to the sequence length, we show a simple computation procedure in Appendix B, which reduces the cost to be linear w.r.t. the sequence length.”; [*Examiner notes: The computations are more efficient (i.e. linear vs. quadratic) in part due to the limited length of the sequences.]
and obtaining an output of the transformer that results from the processing of the input organized batch of sequences by the transformer
(page 8 column 2 section 4.4) “Trained only on WikiText-103 which is medium-sized, Transformer-XL is already able to generate relatively coherent articles[*Examiner notes: obtaining an output of the transformer that results] with thousands of tokens without manual cherry picking, despite minor flaws.”
Dai does not explicitly teach:
wherein the efficiencies of the computations are based on metrics including a number of pipeline stalls that occur during the computations
wherein the output includes translations of the sequences, wherein the translations are generated by an artificial intelligence (AI) model of the transformer during the processing of the input organized batch of sequences
However, Chen teaches
wherein the efficiencies of the computations are based on metrics including a number of pipeline stalls that occur during the computations;
(page 9 column 1 last paragraph) “We use the percentage of pipeline stall caused by "No Instruction", "Wait", and "Short Scoreboard"”; (page 9 column 2 paragraph 2) “Compared with our TCU-based 1-D Octet Tiling, the FPU baseline suffers more from the "No Instruction" and "Wait" stalls. We observe that the FPU baseline has 3776 and 6968 lines in their SASS code. Moreover, 3,402,752 and 3,407,872 HUML+FADD instructions are executed under 𝑉 = 4 and 𝑉 = 8, respectively”
Dai, Chen, and the instant application are analogous because they are all directed to machine learning.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transformer of Dai with the pipeline stall metrics as taught by Chen because (Chen page 1 abstract) “Based on these, we design SpMM and SDDMM kernels and achieve 1.71-7.19x speedup over cuSPARSE. Practical speedup is achieved over cuBLASHgemm under >70% and >90% sparsity with 4x1 grain size and half-precision.”
And Yao teaches:
wherein the output includes translations of the sequences, wherein the translations are generated by an artificial intelligence (AI) model of the transformer during the processing of the input organized batch of sequences
(page 4346 abstract) “The proposed method learns the representations of images based on the text, which avoids encoding irrelevant information in images.”; (page 4349 column 1) “Figure 3 depicts translations for two cases in the test set.”; Figure 3
Dai, Chen, Yao, and the instant application are analogous because they are all directed to machine learning.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transformer of Dai in view of Chen with the translation task as taught by Yao because (Yao page 4346 abstract) “The proposed method learns the representations of images based on the text, which avoids encoding irrelevant information in images. Experiments and visualization analysis demonstrate that our model benefits from visual information and substantially outperforms previous works and competitive baselines in terms of various metrics.”
Jonnakuti teaches:
In a cloud computing environment having a plurality of functional abstraction layers for processing sequences, a computer-implemented method, comprising:
(page 444 abstract) “Leveraging frameworks such as TensorFlow and PyTorch, along with orchestration via Kubernetes and Horovod, the paper examines techniques to achieve scalability, fault tolerance, and efficient resource utilization.”; (page 446 paragraph 1) “The transformer architecture, introduced by Vaswani et al. in 2017, represents a fundamental shift in how sequence data is processed in NLP.” [*Examiner notes: The figure shows different stages of cloud computing, which are the plurality of functional abstraction layers for processing sequences]
PNG
media_image1.png
411
1066
media_image1.png
Greyscale
causing, via use of a management layer of the functional abstraction layers of the cloud computing environment to issue instructions within the cloud computing environment
(page 448 last paragraph) “Resource orchestration and job scheduling are vital for managing heterogeneous GPU environments, where varying computational power and memory capacities must be efficiently leveraged.”
wherein the computations of the internal compute units of the transformer are performed with resource provisioning in which the internal compute units are dynamically procured and controlled via a metering capability to perform tasks within the cloud computing environment including the computations
(page 448 last paragraph) “Resource orchestration and job scheduling are vital for managing heterogeneous GPU environments, where varying computational power and memory capacities must be efficiently leveraged. Systems like Kubernetes can dynamically allocate resources based on the workload, optimizing GPU utilization. Job scheduling frameworks such as Apache Mesos or Kubernetes' native scheduler allow for the efficient distribution of training tasks across a large cluster, balancing the load and ensuring that GPUs are optimally utilized throughout the training process”
Dai, Chen, Yao, Jonnakuti, and the instant application are analogous because they are all directed to machine learning.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transformer of Dai in view of Chen and Yao with the cloud computing management of Jonnakuti because (Jonnakuti page 444 abstract) “Empirical results demonstrate the feasibility and performance trade-offs of scaling transformer architectures in production environments. The findings underscore the practical implications of marrying cutting-edge NLP with robust cloud-native infrastructure to drive operational efficiency in data-intensive domains.”
Regarding Claim 4
Dai in view of Chen, Yao, and Jonnakuti teaches:
The computer-implemented method of Claim 1
(see rejection of claim 1)
And Dai further teaches:
wherein the transformer includes one or more vector matrix multipliers (VMMs).
(page 12 section “B Efficient Computation of the Attention with Relative Positional Embedding”) “As we discussed in section 3.3, the naive way of computing the Wk,RRi−j for all pairs (i, j) is subject to a quadratic cost[*Examiner notes: part of the transformer]. Here, we present a simple method with only a linear cost.”; (page 13 line 3) “Hence, the main computation cost comes from the matrix-vector multiplication de = [Qv]T, which is not expensive any more.”
Regarding Claim 5
Dai in view of Chen, Yao, and Jonnakuti teaches:
The computer-implemented method of Claim 1
(see rejection of claim 1)
And Dai further teaches:
wherein the transformer includes one or more attention-compute blocks.
(page 5 column 2 paragraph 3 line 3) “For completeness, we summarize the computational procedure for a N-layer Transformer-XL with a single attention head here”
Regarding Claim 7
Dai in view of Chen, Yao, and Jonnakuti teaches:
The computer-implemented method of Claim 1
(see rejection of claim 1)
Dai further teaches:
wherein first sequences of the batch of sequences include a plurality of series of words,
(page 4 column 2 line 17) “where Esτ ∈ R L×d is the word embedding sequence of sτ”; (page 5 column 2 last paragraph) “We apply Transformer-XL to a variety of datasets on both word-level and character-level language modeling to have a comparison with state-of-the-art systems, including WikiText-103”; (page 6 column 1 paragraph 2) “WikiText-103 is the largest available word-level language modeling benchmark with long-term dependency”
And Yao further teaches:
wherein second sequences of the batch of sequences include regions-of-interest from within an image
(page 4347 column 1 above section 2.2) “Therefore, as the words are local semantic representations of the sentence, we extract the spatial features which are the semantic representations of local spatial regions of the image. We add the spatial features of the image as pseudo-words in the source sentence and feed it into the multimodal self-attention layer.”; Figure 3
Regarding Claim 13
Claim 13 is a computer program product claim corresponding to method claim 1. The only difference is that claim 13 recites a computer program product comprising a computer-readable storage medium:
Chen teaches:
A computer program product comprising one or more non-transitory computer readable storage media, and program instructions collectively stored on the one or more non-transitory computer readable storage media, the program instructions comprising instructions configured to cause one or more processors to perform a method comprising:
(page 2) “Memory Hierarchy[11]. NVIDIA GPU consists of an array of streaming multiprocessors (SMs). In Volta, all the SMs share a 6 MiB L2 Cache and a 16 GiB DRAM[*Examiner notes: mapped to computer-readable storage media]”; (page 3 column 1 paragraph 1) “Corresponding GPU kernels for SpMM and SDDMM[*Examiner notes: mapped to computer program product] are also proposed to deliver practical speedup”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transformer of Dai in view of Chen, Yao, and Jonnakuti with the non-transitory computer-readable storage media of Chen because (Chen page 1 abstract) “Column-vector-sparse-encoding can be applied to both SpMM & SDDMM, two major sparse DNN operations. We also introduce the Tensor-Core-based 1D Octet Tiling that has efficient memory access and computation patterns under small grain size. Based on these, we design SpMM and SDDMM kernels and achieve 1.71-7.19x speedup over cuSPARSE. Practical speedup is achieved over cuBLASHgemm under >70% and >90% sparsity with 4x1 grain size and half-precision.”
The remaining limitations of the claim are taught by the rejection of claim 1.
Regarding Claim 16
Claim 16 is a computer program product claim corresponding to method claim 4. The only difference is that claim 16 recites a computer program product comprising a computer-readable storage medium as taught in the rejection of claim 13 above. The remaining limitations of the claim are taught by the rejection of claim 4.
Regarding Claim 17
Claim 17 is a computer program product claim corresponding to method claim 5. The only difference is that claim 17 recites a computer program product comprising a computer-readable storage medium as taught in the rejection of claim 13 above. The remaining limitations of the claim are taught by the rejection of claim 5.
Regarding Claim 19
Claim 19 is a computer program product claim corresponding to method claim 7. The only difference is that claim 7 recites a computer program product comprising a computer-readable storage medium as taught in the rejection of claim 13 above. The remaining limitations of the claim are taught by the rejection of claim 7.
Regarding Claim 20
Claim 20 is a computer system claim corresponding to method claim 7. The only difference is that claim 20 recites a processor and logic
Chen teaches:
A system of a transformer environment for processing sequences, comprising: a processor; and logic integrated with the processor, executable by the processor, or integrated with and executable by the processor, the logic being configured to
(page 2) “Memory Hierarchy[11]. NVIDIA GPU[*Examiner notes: mapped to system in a transformer environment comprising a processor] consists of an array of streaming multiprocessors (SMs). In Volta, all the SMs share a 6 MiB L2 Cache and a 16 GiB DRAM” (page 3 column 1 paragraph 1) “Corresponding GPU kernels for SpMM and SDDMM[*Examiner notes: mapped to logic] are also proposed to deliver practical speedup”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transformer of Dai in view of Chen, Yao, and Jonnakuti with the system and processor of Chen because (Chen page 1 abstract) “Column-vector-sparse-encoding can be applied to both SpMM & SDDMM, two major sparse DNN operations. We also introduce the Tensor-Core-based 1D Octet Tiling that has efficient memory access and computation patterns under small grain size. Based on these, we design SpMM and SDDMM kernels and achieve 1.71-7.19x speedup over cuSPARSE. Practical speedup is achieved over cuBLASHgemm under >70% and >90% sparsity with 4x1 grain size and half-precision.”
The remaining limitations of the claim are taught by the rejection of claim 7 (and independent claim 1 from which claim 7 depends).
Claims 2 and 14 rejected under 35 U.S.C. 103 as being unpatentable over Dai in view of Chen, Yao, Jonnakuti, and further in view of NPL reference Devlin et al. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” herein referred to as Devlin.
Regarding Claim 2:
Dai in view of Chen, Yao, and Jonnakuti teaches:
The computer-implemented method of Claim 1
(see rejection of claim 1)
Dai in view of Chen, Yao, and Jonnakuti does not teach:
wherein the transformer includes a Bidirectional Encoder Representations from Transformers (BERT) transformer.
However, Devlin teaches:
wherein the transformer includes a Bidirectional Encoder Representations from Transformers (BERT) transformer.
(page 4171 abstract) “We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers.”
Dai, Chen, Yao, Jonnakuti, Devlin, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transformer of Dai in view of Chen, Yao, and Jonnakuti with the BERT of Devlin because (Devlin page 4171 abstract) “BERT is designed to pretrain deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be finetuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications.”
Regarding Claim 14
Claim 14 is a computer program product claim corresponding to method claim 2. The only difference is that claim 14 recites a computer program product comprising a computer-readable storage medium as taught in the rejection of claim 13 above. The remaining limitations of the claim are taught by the rejection of claim 2.
Claims 3 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Dai in view of Chen, Yao, Jonnakuti, and further in view of Radford et al. “Improving Language Understanding by Generative Pre-Training” herein referred to as Radford.
Regarding Claim 3:
Dai in view of Chen, Yao, and Jonnakuti teaches:
The computer-implemented method of Claim 1
(see rejection of claim 1)
Dai in view of Chen, Yao, and Jonnakuti does not teach:
wherein the transformer includes a Generative Pre-trained Transformer (GPT)
However, Radford teaches:
wherein the transformer includes a Generative Pre-trained Transformer (GPT)
(page 1 abstract) “We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text”; (page 2 paragraph 2 line 1) “For our model architecture, we use the Transformer”; [*Examiner notes: A transformer that uses generative pre-training is a generative pre-trained transformer (GPT)]
Dai, Chen, Yao, Jonnakuti, Radford, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transformer of Dai in view of Chen, Yao, and Jonnakuti with the GPT of Radford because (Radford page 1 abstract) “We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task” and (page 2 paragraph 2 line 1) “For our model architecture, we use the Transformer [62], which has been shown to perform strongly on various tasks such as machine translation [62], document generation [34], and syntactic parsing [29].”
Regarding Claim 15
Claim 15 is a computer program product claim corresponding to method claim 3. The only difference is that claim 15 recites a computer program product comprising a computer-readable storage medium as taught in the rejection of claim 13 above. The remaining limitations of the claim are taught by the rejection of claim 3.
Claims 8 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Dai in view of Chen, Yao, Jonnakuti, and further in view of Zhu et al. “Long-Short Transformer: Efficient Transformers for Language and Vision” herein referred to as Zhu.
Regarding Claim 8:
Dai in view of Chen, Yao, and Jonnakuti teaches:
The computer-implemented method of Claim 1
(see rejection of claim 1)
Dai in view of Chen, Yao, and Jonnakuti does not explicitly teach:
wherein the threshold sequence-size indicates a representative length of a sequence that is input into the transformer, wherein the representative length is compared to lengths of the sequences that are input into the transformer
However, Zhu teaches:
wherein the threshold sequence-size indicates a representative length of a sequence that is input into the transformer, wherein the representative length is compared to lengths of the sequences that are input into the transformer
(page 4 section 3.2 “Short-term Attention via Segment-wise Sliding Widow” paragraph 1) “We use the simple yet effective sliding window attention to capture fine-grained local correlations, where each query attends to nearby tokens within a fixed-size neighborhood. Similar techniques have also been adopted in [14, 16, 11]. Specifically, we divide the input sequence into disjoint segments with length w[*Examiner notes: corresponds to threshold sequence size] for efficiency reason. All tokens within a segment attend to all tokens within its home segment, as well as w/2 consecutive tokens on the left and right side of its home segment[*Examiner notes: mapped to comparing a length of each sequence to the threshold sequence size] (zero-padding when necessary), resulting in an attention span over a total of 2w key-value pair”
Dai, Chen, Yao, Jonnakuti, Zhu, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transformer of Dai in view of Chen, Yao, and Jonnakuti with the comparing and ordering of sequences taught by Zhu because (Zhu page 2 paragraph 4) “We propose Long-Short Transformer (Transformer-LS), an efficient Transformer that integrates a dynamic projection based attention to model long-range correlations, and a local window attention to capture fine-grained correlations. Transformer-LS can be applied to both autoregressive and bidirectional models with linear time and memory complexity”
Regarding Claim 11:
Dai in view of Chen, Yao, and Jonnakuti teaches:
The computer-implemented method of Claim 1
(see rejection of claim 1)
Dai in view of Chen, Yao, and Jonnakuti does not teach:
wherein the batch of sequences is input into the transformer in an alternating order determined utilizing the threshold sequence-size
However, Zhu teaches:
wherein the batch of sequences is input into the transformer in an alternating order determined utilizing the threshold sequence-size
(page 4 section 3.2 “Short-term Attention via Segment-wise Sliding Widow” paragraph 1) “We use the simple yet effective sliding window attention to capture fine-grained local correlations, where each query attends to nearby tokens within a fixed-size neighborhood. Similar techniques have also been adopted in [14, 16, 11]. Specifically, we divide the input sequence into disjoint segments with length w[*Examiner notes: corresponds to threshold sequence size] for efficiency reason.”; (page 3 figure 1 caption) “Long-short term attention of a single attention head. Here, the sequence length n = 8, hidden dimension d = 3, local window segment size w = 2[*Examiner notes: corresponds to threshold sequence size], and rank of dynamic projection r = 3. Within the figure, K(V ) denotes key K or value V . In the left figure, we virtually replicate K or V ∈ Rn×d into n rows, and highlight the keys and values within the attention span (denoted as K˜ (V˜ )) of all n queries Q for the short-term attention[*Examiner notes: utilizing threshold sequence size]. In the middle figure, all queries attend to the same projected keys K¯ and values V¯ within the long-term attention”; [*Examiner notes: The sequences are arranged in alternating order because the short term attention computation (determined using threshold sequence-size) is followed by the long term attention computation and their results are aggregated. See figure 1 annotated below.]
Dai, Chen, Yao, Jonnakuti, Zhu, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transformer of Dai in view of Chen, Yao, and Jonnakuti with the comparing and ordering of sequences taught by Zhu because (Zhu page 2 paragraph 4) “We propose Long-Short Transformer (Transformer-LS), an efficient Transformer that integrates a dynamic projection based attention to model long-range correlations, and a local window attention to capture fine-grained correlations. Transformer-LS can be applied to both autoregressive and bidirectional models with linear time and memory complexity”
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Dai in view of Chen, Yao, Jonnakuti, and further in view of Wei et al. (PGPUB No. US20220129632A1) herein referred to as Wei, and Zhu.
Regarding Claim 10:
Dai in view of Chen, Yao, and Jonnakuti teaches:
The computer-implemented method of Claim 1
(see rejection of claim 1)
Dai in view of Chen, Yao, and Jonnakuti does not teach:
wherein organizing the batch of sequences includes: comparing lengths of the sequences to the threshold sequence-size, wherein the lengths of at least some of the sequences are different
and arranging the sequences within the batch of sequences in an alternating order based on the comparison
However, Wei teaches:
wherein organizing the batch of sequences includes: comparing lengths of the sequences to the threshold sequence-size, wherein the lengths of at least some of the sequences are different
(paragraph [0008]) “In some embodiments, before recognizing the named entity from the second text data, recognizing the named entity from the first text data, further includes: determining whether a text length of the second text data is greater than a preset text length threshold; using the second text data as the long text in response to determining that the text length of the second text data is greater than the preset text length threshold.”
(paragraph [0011]) “In some embodiments, before recognizing the named entity from the second text data, recognizing the named entity from the first text data, further includes: determining whether a text length of the second text data is greater than a preset text length threshold; using the second text data as the short text in response to determining that the text length of the second text data is less than or equal to the preset text length threshold.”
Dai, Chen, Yao, Jonnakuti, Wei, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transformer of Dai in view of Chen, Yao, and Jonnakuti with the comparing of sequences taught by Wei because (Wei paragraph [0076]) “When the named entity is recognized from the second text data, for the long text and the short text, different methods may be used to recognize the named entity, so as to improve the recognition efficiency.”
And Zhu teaches:
and arranging the sequences in an alternating order based on the comparison
(page 3 figure 1 caption) “Long-short term attention of a single attention head. Here, the sequence length n = 8, hidden dimension d = 3, local window segment size w = 2, and rank of dynamic projection r = 3. Within the figure, K(V ) denotes key K or value V . In the left figure, we virtually replicate K or V ∈ Rn×d into n rows, and highlight the keys and values within the attention span (denoted as K˜ (V˜ )) of all n queries Q for the short-term attention[*Examiner notes: based on comparison]. In the middle figure, all queries attend to the same projected keys K¯ and values V¯ within the long-term attention”; [*Examiner notes: The sequences are arranged I alternating order because the short term attention computation is followed by the long term attention computation and their results are aggregated. See figure 1 annotated below.]
PNG
media_image2.png
199
602
media_image2.png
Greyscale
Dai, Chen, Yao, Jonnakuti, Wei, Zhu, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transformer of Dai in view of Chen, Yao, Jonnakuti, and Wei with the ordering of sequences taught by Zhu because (Zhu page 2 paragraph 4) “We propose Long-Short Transformer (Transformer-LS), an efficient Transformer that integrates a dynamic projection based attention to model long-range correlations, and a local window attention to capture fine-grained correlations. Transformer-LS can be applied to both autoregressive and bidirectional models with linear time and memory complexity”
Claims 6, 9, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Dai in view of Chen, Yao, Jonnakuti, and further in view of Vaswani et al. “Attention is all you need”, herein referred to as Vaswani.
Regarding Claim 6
Dai in view of Chen, Yao, and Jonnakuti teaches:
The computer-implemented method of Claim 1
(see rejection of claim 1)
Dai in view of Chen, Yao, and Jonnakuti does not explicitly teach:
where the layers include VMM blocks and attention- compute blocks
However, Vaswani teaches:
(page 2 section 3.1 paragraph 1) “The encoder is composed of a stack of N = 6 identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism[*Examiner notes: mapped to attention compute blocks], and the second is a simple, positionwise fully connected feed-forward network[*Examiner notes: mapped to VMM block].”; (page 5 section 3.3) “In addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully connected feed-forward network, which is applied to each position separately and identically. This consists of two linear transformations[*Examiner notes: matrix] with a ReLU activation in between. FFN(x) = max(0, xW1 + b1)W2 + b2 (2)”; [*Examiner notes: A feed forward neural network is in essence a series of matrix multiplications. This is evidenced by 3blue1brown’s YouTube video, But what is a neural network? (timestamp 13:37) “Organize all of the activations from one layer into a column as a vector. Then organize all of the weights as a matrix, where each row of that matrix corresponds to the connections between one layer and a particular neuron in the next layer. What that means is that taking the weighted sum of the activations in the first layer according to these weights corresponds to one of the terms in the matrix vector product of everything we have on the left here.”]
PNG
media_image3.png
798
1430
media_image3.png
Greyscale
Dai, Chen, Yao, Jonnakuti, Vaswani, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transformer of Dai in view of Chen, Yao, and Jonnakuti with the layers including VMM blocks and attention blocks of Vaswani because (Vaswani page 2 paragraph 3) “In this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output. The Transformer allows for significantly more parallelization and can reach a new state of the art in translation quality after being trained for as little as twelve hours on eight P100 GPUs. ”
Regarding Claim 9
Dai in view of Chen, Yao, and Jonnakuti teaches:
The computer-implemented method of Claim 1,
(see rejection of claim 1)
Dai further teaches:
the threshold sequence-size results in a second compute time for the sequence by attention- compute blocks of the transformer
[*Examiner notes: The broadest reasonable interpretation of the term “compute time” includes computational cost/complexity. See specification paragraphs [0079]-[0080] and [0060]-[0061]]; (page 12 section “B Efficient Computation of the Attention with Relative Positional Embedding”) "As we discussed in section 3.3, the naive way of computing the Wk,RRi−j for all pairs (i, j) is subject to a quadratic cost. Here, we present a simple method with only a linear cost[*Examiner notes: mapped to second compute time]”; [*Examiner notes: sequences of size L (threshold sequence size) are input to the transformer and thus attention block, and therefore the threshold sequence size results in the second compute time]
and the first compute time equals the second compute time for the threshold sequence-size.
(Page 13 line 3) “Hence, the main computation cost comes from the matrix-vector multiplication de = [Qv]T[*Examiner notes: compute time of attention block equals compute time of vector matrix multiplication] , which is not expensive any more."
Dai in view of Chen, Yao, and Jonnakuti does not explicitly teach:
wherein: the threshold sequence-size results in a first compute time for a sequence by vector matrix multipliers (VMMs) of the transformer where input excitation vectors are multiplied by trained-weight matrices
[*Examiner notes: While Dai does teach “one or more vector matrix multipliers (VMMs) of the transformer” (see rejection of claim 4 above), Dai does not appear to explicitly mention VMMs that are separate from the attention block]
However, Vaswani teaches:
wherein: the threshold sequence-size results in a first compute time for a sequence by vector matrix multipliers (VMMs) of the transformer where input excitation vectors are multiplied by trained-weight matrices
(page 5 section 3.3) “In addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully connected feed-forward network, which is applied to each position separately and identically. This consists of two linear transformations[*Examiner notes: matrix] with a ReLU activation in between. FFN(x) = max(0, xW1 + b1)W2 + b2 (2)”; (page 6 paragraph 1 line 2) “Motivating our use of self-attention we consider three desiderata. One is the total computational complexity per layer[*Examiner notes: combined complexity of VMM and attention block]. Another is the amount of computation that can be parallelized, as measured by the minimum number of sequential operations required.”; [*Examiner notes: The transformer model of Vaswani includes multiple layers which each involve attention blocks and vector matrix multiplication blocks (see rejection of claim 6 above). Thus the per layer complexity includes the complexity of the vector matrix multiplication blocks. max(0, xW1+b1) is the input excitation vector from the previous layer and W2 is the trained weight matrices]
Dai, Chen, Yao, Jonnakuti, Vaswani, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transformer of Dai in view of Chen, Yao, and Jonnakuti with the layers including VMM blocks and attention blocks of Vaswani because (Vaswani page 2 paragraph 3) “In this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output. The Transformer allows for significantly more parallelization and can reach a new state of the art in translation quality after being trained for as little as twelve hours on eight P100 GPUs. ”
Regarding Claim 18
Claim 18 is a computer program product claim corresponding to method claim 6. The only difference is that claim 18 recites a computer program product comprising a computer-readable storage medium as taught in the rejection of claim 13 above. The remaining limitations of the claim are taught by the rejection of claim 6.
Claim 21 is rejected under 35 U.S.C. 103 as being obvious over Dai in view of Chen, Yao, Jonnakuti, and further in view of NPL reference Park et al. “OPTIMUS: OPTIMIZED MATRIX MULTIPLICATION STRUCTURE FOR TRANSFORMER NEURAL NETWORK ACCELERATOR” herein referred to as Park.
Regarding Claim 21
Dai in view of Chen, Yao, and Jonnakuti teaches:
The computer-implemented method of Claim 1
(see rejection of claim 1)
Dai in view of Chen, Yao, and Jonnakuti does not explicitly teach:
wherein the metrics further include an amount of time the internal compute units are kept occupied
and a total latency.
However, Park teaches:
wherein the metrics further include an amount of time the internal compute units are kept occupied
(page 4 column 1 paragraph 2) “The challenge of the limited parallelism in the decoding stage is demonstrated in Fig. 3, where the processing time for the encoding and decoding stages is compared for CPU (multi-thread) and GPU. In the encoding stage, the processing time increases as the sentence length grows for CPU while it is almost constant for GPU.”
and a total latency
(page 1 abstract) “The simulation using the WMT15 (EN-DE) dataset shows that latency of OPTIMUS is 41.62×, 24.23×, 16.01× smaller than that of Intel(R) i7 6900K CPU, NVIDIA Titan Xp GPU, and the baseline custom hardware, respectively”
Dai, Chen, Yao, Jonnakuti, Park, and the instant application are analogous because they are all directed to machine learning.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transformer of Dai in view of Chen, Yao, and Jonnakuti with the computing time and latency parameters taught by Park because (Park page 9 column 1 section 7.3) “In real-time processing applications, latency of single batch processing is one of the most important design parameters. As mentioned in Section 3, most of the computation time is spent on decoding because of the sequence to sequence structure (Fig. 10). The decoding processing time can be reduced by skipping redundant computations.”
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ezra J Baker whose telephone number is (703)756-1087. The examiner can normally be reached Monday - Friday 10:00 am - 8:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/E.J.B./Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights