Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This action is in response to the application filed 26 January 2024 and the preliminary amendment filed 31 July 2026 . Responsive to pre-amendment, claims 15-20 have been cancelled and Claims 21-26 were newly added. Claims 1-14 and 21-26 are now pending and have been examined.
It is noted that the claims as originally filed were under a restriction requirement and the preliminary amendment has been filed in response to a requirement for election by phone. AN Interview Summary documenting the restriction requirement is attached to this Office action.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 31 January 2025 and 21 August 2025 and are being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 14 and 21-25 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claims do not fall within at least one of the four categories of patent eligible subject matter because a "computer storage media storing instructions" is directed to software per se.
Claims 1-14 and 21-26 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1
Step 1
Claim 1 recites a method, and thus the claimed process falls within a statutory category of invention.
Step 2A Prong 1
The claim recites to generate an output, which is a mental process. The claim recites generating a plurality of parameter blocks by partitioning the plurality of parameters along the row dimension and the column dimension according to a number of the plurality of hardware accelerators, which is a mental process. The claim recites determining a ratio of a number of parameters along the row dimension relative to a number of parameters along the column dimension, which is a mental process. The claim recites determining, from the ratio, whether to use row sharding or column sharding with the plurality of hardware accelerators to calculate an output for the fully connected layer and then calculating the output for the fully connected layer using either row sharding or column sharding, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The additional element processing an input through each of a plurality of layers of a neural network ... , wherein the plurality of layers comprise a fully connected layer having a plurality of parameters arranged in a row dimension and a column dimension invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element using a plurality of hardware accelerators invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 2
Step 1
Regarding Claim 2, the rejection of Claim 1 is incorporated.
Step 2A Prong 1
The claim recites generating a partial output for the fully connected layer by determining a multiplication between (i) each parameter block in the first subset of the plurality of parameter blocks and (ii) the corresponding first input vector over one or more hardware cycles while communicating the corresponding first input vectors to other hardware accelerators over an accelerator interconnect network during the one or more hardware cycles, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2
The additional element loading a first subset of the plurality of parameter blocks arranged along the row dimension one after another amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering and inputting"). The additional element into a particular hardware accelerator does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element receiving, for each parameter block in the first subset of the plurality of parameter blocks, a corresponding first input vector amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering and inputting "). The additional element at the particular hardware accelerator does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment").
Step 2B
The additional element loading a first subset of the plurality of parameter blocks arranged along the row dimension one after another is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network"). The additional element into a particular hardware accelerator does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element receiving, for each parameter block in the first subset of the plurality of parameter blocks, a corresponding first input vector is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network"). The additional element at the particular hardware accelerator does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 3
Step 1
Regarding Claim 3, the rejection of Claim 2 is incorporated.
Step 2A Prong 1
Claim 3 recites the abstract ideas of parent Claim 2.
Step 2A Prong 2
The additional element loading a first parameter block in the first subset of the plurality of parameter blocks amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering"). The additional element into the particular hardware accelerator does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element loading a second parameter block in the first subset of the plurality of parameter blocks ..., wherein the second parameter block is an immediate horizontal neighbor of the first parameter block amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering").
Step 2B
The additional element loading a first parameter block in the first subset of the plurality of parameter blocks is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network"). The additional element into the particular hardware accelerator does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element loading a second parameter block in the first subset of the plurality of parameter blocks ..., wherein the second parameter block is an immediate horizontal neighbor of the first parameter block is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 4
Step 1
Regarding Claim 4, the rejection of Claim [X] is incorporated.
Step 2A Prong 1
The claim recites generating the partial output for the fully connected layer by determining a multiplication between (i) each parameter block in the second subset of the plurality of parameter blocks and (ii) the second input vector over one or more hardware cycles while communicating corresponding results of the multiplications to the other hardware accelerators over the accelerator interconnect network during the multiple hardware cycles, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2
The additional element loading a second subset of the plurality of parameter blocks arranged in the column dimension one after another amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering"). The additional element into the particular hardware accelerator does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element receiving a second input vector amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering"). The additional element at the particular hardware accelerator does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment").
Step 2B
The additional element loading a second subset of the plurality of parameter blocks arranged in the column dimension one after another is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network"). The additional element into the particular hardware accelerator does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element receiving a second input vector is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network"). The additional element at the particular hardware accelerator does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 5
Step 1
Regarding Claim 5, the rejection of Claim 4 is incorporated.
Step 2A Prong 1
Claim 5 recites the abstract ideas recited by parent Claim 4.
Step 2A Prong 2
The additional element loading a first parameter block in the second subset of the plurality of parameter blocks amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering"). The additional element into the particular hardware accelerator does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element loading a second parameter block in the second subset of the plurality of parameter blocks ..., wherein the second parameter block is an immediate vertical neighbor of the first parameter block amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering").
Step 2B
The additional element loading a first parameter block in the second subset of the plurality of parameter blocks is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network"). The additional element into the particular hardware accelerator does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element loading a second parameter block in the second subset of the plurality of parameter blocks ..., wherein the second parameter block is an immediate vertical neighbor of the first parameter block is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 6
Step 1
Regarding Claim 6, the rejection of Claim 1 is incorporated.
Step 2A Prong 1
The claim recites determining, from the ratio, whether to use row sharding or column sharding with the plurality of hardware accelerators to calculate an output for the fully connected layer and then calculating the output for the fully connected layer using either row sharding or column sharding (as recited by Claim 1), wherein calculating the output for the fully connected layer comprises computing a summation of the partial outputs for the fully connected layer, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 7
Step 1
Regarding Claim 7, the rejection of Claim 1 is incorporated.
Step 2A Prong 1
The claim recites determining a loss of the output relative to a ground truth output of the input, which is a mental process. The claim recites determining an update to the plurality of parameters of the fully connected layer based on the loss, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 8
Step 1
Regarding Claim 8, the rejection of Claim 7 is incorporated.
Step 2A Prong 1
The claim recites determining an update to the plurality of parameters of the fully connected layer based on the loss (as recited by Claim 7), wherein determining the update to the plurality of parameters comprises computing a backpropagation of the loss through the plurality of parameters using either row sharding or column sharding, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 9
Step 1
Claim 9 recites a system, and thus the claimed machine falls within a statutory category of invention.
Step 2A Prong 1
The claim recites to generate an output, which is a mental process. The claim recites generating a plurality of parameter blocks by partitioning the plurality of parameters along the row dimension and the column dimension according to a number of the plurality of hardware accelerators, which is a mental process. The claim recites determining a ratio of a number of parameters along the row dimension relative to a number of parameters along the column dimension, which is a mental process. The claim recites determining, from the ratio, whether to use row sharding or column sharding with the plurality of hardware accelerators to calculate an output for the fully connected layer and then calculating the output for the fully connected layer using either row sharding or column sharding, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The additional element one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform operations invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element processing an input through each of a plurality of layers of a neural network ..., wherein the plurality of layers comprise a fully connected layer having a plurality of parameters arranged in a row dimension and a column dimension invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element using a plurality of hardware accelerators invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Claims 10-13 and 26, dependent on Claim 9, incorporate the rejection of Claim 9. Claims 10-13 and 26 incorporate substantively all the limitations of Claims 2-5 and 7, respectively, in system form and are rejected under the same rationales.
Regarding Claim 14
Step 1
Claim 14 is directed to non-statutory subject matter because a "computer storage media storing instructions" is directed to software per se. For the purposes of examination, Claim 14 has been interpreted to read "non-transitory computer storage media storing instructions."
Claim 14 recites computer storage media storing instructions, and thus the claimed manufacture falls within a statutory category of invention.
Step 2A Prong 1
The claim recites to generate an output, which is a mental process. The claim recites generating a plurality of parameter blocks by partitioning the plurality of parameters along the row dimension and the column dimension according to a number of the plurality of hardware accelerators, which is a mental process. The claim recites determining a ratio of a number of parameters along the row dimension relative to a number of parameters along the column dimension, which is a mental process. The claim recites determining, from the ratio, whether to use row sharding or column sharding with the plurality of hardware accelerators to calculate an output for the fully connected layer and then calculating the output for the fully connected layer using either row sharding or column sharding, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The additional element instructions that when executed by one or more computers cause the one more computers to perform operations invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element processing an input through each of a plurality of layers of a neural network ..., wherein the plurality of layers comprise a fully connected layer having a plurality of parameters arranged in a row dimension and a column dimension invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element using a plurality of hardware accelerators invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Claims 21-25, dependent on Claim 14, incorporate the rejection of Claim 9. Claims 21-25 incorporate substantively all the limitations of Claims 2-5 and 7, respectively, in computer storage media form and are rejected under the same rationales.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 6, 8, 9, and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Shoeybi, et al., "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism" (hereinafter "Shoeybi") in view of Pope, et al., "Efficiently Scaling Transformer Inference" (hereinafter "Pope").
Regarding Claim 1, Shoeybi teaches:
A method for processing an input ... to generate an output (Shoeybi , p. 5, 4. Setup: "Pretrained language understanding models are central tasks in natural language processing and language understanding. ... In this work we focus on GPT-2 (Radford et al., 2019), a left-to-right generative transformer based language model, and BERT (Devlin et al., 2018), a bi-directional transformer model based on language model masking") ... through each of a plurality of layers of a neural network ... (Shoeybi , p. 1, Abstract: "we present our techniques for training very large transformer models and implement a simple, efficient intra-layer model parallel approach that enables training transformer models with billions of parameters") using a plurality of hardware accelerators (Shoeybi , p. 3, 2.3. Data and Model Parallelism in Deep Learning: "With language models of increasing size and complexity like BERT and GPT-2, neural networks have approached the memory capacity of modern hardware accelerators. ... Our approach is to utilize model parallelism to split the model across multiple accelerators"), wherein the plurality of layers comprise a fully connected layer (Shoeybi, p. 4, 3. Model Parallel Transformers: "We take advantage of the structure of transformer networks to create a simple model parallel implementation .... A transformer layer consists of a self attention block followed by a two-layer, multi-layer perceptron (MLP) as shown in Figure 2" and p. 3, Figure 2, "Transformer Architecture. Purple blocks correspond to fully connected layers. Each blue block represents a single transformer layer that is replicated N times") having a plurality of parameters arranged in a row dimension and a column dimension (Shoeybi, p. 4, 3. Model Parallel Transformers: "We start by detailing the MLP block. The first part of the block is a GEMM followed by a GeLU nonlinearity: [Eq. 1] One option to parallelize the GEMM is to split the weight matrix A along its rows and input
X
along its columns as:
X
=
X
1
,
X
2
,
A
=
A
1
A
2
(2)
... Another option is to split A along its columns
A
=
A
1
,
A
2
," where Shoeybi's weight matrix
A
corresponds to the instant parameters), and wherein the method comprises:
generating a plurality of parameter blocks by partitioning the plurality of parameters along the row dimension and the column dimension (Shoeybi, p. 4, 3. Model Parallel Transformers: "we partition the first GEMM in this column parallel fashion and split the second GEMM along its rows so it takes the output of the GeLU layer directly without requiring any communication as shown in Figure 3a. The output of the second GEMM is then reduced across the GPUs before passing the output to the dropout layer. This approach splits both GEMMs in the MLP block across GPUs," where Shoeybi's GEMM operation matrices correspond to the instant blocks) according to a number of the plurality of hardware accelerators (Shoeybi, p. 4, Figure 3: "Blocks of Transformer with Model Parallelism. (a) MLP," depicting partitioning of matrices as
A
1
/
B
1
and
A
2
/
B
2
per two GPUs); ... ; and
determining ... whether to use row sharding or column sharding with the plurality of hardware accelerators to calculate an output for the fully connected layer and then calculating the output for the fully connected layer using ... row sharding or column sharding (Shoeybi, p. 4, Figure 3, "Blocks of Transformer with Model Parallelism", (a) MLP, depicting row and column sharding of weight matrices A and B to calculate outputs Y and Z, and p. 4, 3. Model Parallel Transformers: "We start by detailing the MLP block. ... One option to parallelize the GEMM is to split the weight matrix A along its rows and input X along its columns .... Another option is to split A along its columns.... This is advantageous as it removes a synchronization point. Hence, we partition the first GEMM in this column parallel fashion and split the second GEMM along its rows so it takes the output of the GeLU layer directly without requiring any communication as shown in Figure 3a").
Shoeybi teaches determining whether to use row sharding or column sharding with the plurality of hardware accelerators.
Shoeybi does not explicitly teach determining a ratio of a number of parameters along the row dimension relative to a number of parameters along the column dimension; and determining, from the ratio, whether to use row sharding or column sharding.
However, Pope teaches:
determining a ratio of a number of parameters along the row dimension relative to a number of parameters along the column dimension; and determining, from the ratio, whether to use row sharding or column sharding (Pope, p. 4, 3.2.2 Feedforward layer, 2D weight-stationary layout: "Overview. For a larger number of chips, a more economical strategy involves partitioning each
E
×
F
weight matrix along both the E and F axes, such that each shard is roughly square. ... This is called 2D weight-stationary. The total compute cost is the same as 1D weight-stationary, but communication is much more efficient .... Therefore, 2D weight-stationary becomes more communication-efficient when
n
c
h
i
p
s
>
d
f
f
d
m
o
d
e
l
. Since typically
d
f
f
=
4
d
m
o
d
e
l
, this occurs when
n
c
h
i
p
s
>
16
," where Pope's E and F correspond to model row and column dimensions, as in p. 4, 3.2.1 Feedforward layer, 1D weight-stationary layout: "Overview. When a model doesn't fit on a single chip, the simplest partitioning strategy is 1D weight-stationary, where each
E
×
F
weight matrix is partitioned (or sharded) among
n
c
h
i
p
s
along the
E
or
F
axis" and p 3, 2.2 Inference Setup: "The model has model (or embed) dimension
d
m
o
d
e
l
(or
E
), feedforward intermediate dimension
d
f
f
(or
F
)").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Shoeybi regarding determining whether to use row sharding or column sharding with the plurality of hardware accelerators with those of Pope regarding determining a ratio of a number of parameters along the row dimension relative to a number of parameters along the column dimension; and determining, from the ratio, whether to use row sharding or column sharding.
The motivation to do so would be to facilitate processing over a varying number of devices with efficient communication (Pope, p. 4, 3.2.2 Feedforward layer, 2D weight-stationary layout: "For a larger number of chips, a more economical strategy involves partitioning each
E
×
F
weight matrix along both the E and F axes.... The total compute cost is the same as 1D weight-stationary, but communication is much more efficient").
Regarding Claim 9, Shoeybi teaches:
A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform operations (Shoeybi , p. 12, Figure 8: "Grouping of GPUs for hybrid model and data parallelism with 8-way model parallel and 64-way data parallel" and p. 3, 2.3. Data and Model Parallelism in Deep Learning: "With language models of increasing size and complexity like BERT and GPT-2, neural networks have approached the memory capacity of modern hardware accelerators. ... Our approach is to utilize model parallelism to split the model across multiple accelerators," where a storage device storing instructions is inherent in a system splits models for acceleration) corresponding precisely those recited by the method of Claim 1. Claim 9 is rejected under the same rationale as Claim 1.
Regarding Claim 14, Shoeybi teaches:
One or more computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform operations (Shoeybi , p. 12, Figure 8: "Grouping of GPUs for hybrid model and data parallelism with 8-way model parallel and 64-way data parallel" and p. 3, 2.3. Data and Model Parallelism in Deep Learning: "Our approach is to utilize model parallelism to split the model across multiple accelerators," where a storage device storing instructions is inherent in a system that splits models for acceleration) corresponding precisely those recited by the method of Claim 1. Claim 14 is rejected under the same rationale as Claim 1.
Regarding Claim 6, the rejection of Claim 1 is incorporated. The Shoeybi/Pope combination teaches:
wherein calculating the output for the fully connected layer comprises computing a summation of the partial outputs for the fully connected layer (Shoeybi, p. 4, 3. Model Parallel Transformers: "One option to parallelize the GEMM is to split the weight matrix A along its rows and input X along its columns .... This partitioning will result in
Y
=
G
e
L
U
X
1
A
1
+
X
2
A
2
" and "Another option is to split A along its columns
A
=
A
1
,
A
2
. ... ¶ This is advantageous as it removes a synchronization point. ... The output of the second GEMM is then reduced across the GPUs before passing the output to the dropout layer. This approach splits both GEMMs in the MLP block across GPUs and requires only a single all-reduce operation in the forward pass (g operator)," where Shoeybi's all-reduce operation reasonably suggests a summation of the partitioned results).
Regarding Claim 8, the rejection of Claim 7 is incorporated. The Shoeybi/Pope/Zheng combination teaches:
wherein determining the update to the plurality of parameters comprises computing a backpropagation of the loss through the plurality of parameters using either row sharding or column sharding (Shoeybi, p. 11, B.1. Hybrid Model and Data Parallelism: "During back propagation we run multiple gradient all-reduce operations in parallel to reduce weight gradients within each distinct data parallel group. ... GPUs within each model parallel group perform all-reduces amongst all GPUs within the group. For data parallelism, each of the all-reduce operations takes place with one of the GPUs from each model parallel group" and p. 4, 3. Model Parallel Transformers: "The subsequent GEMM from the output linear layer (after self attention) is parallelized along its rows and takes the output of the parallel attention layer directly, without requiring communication between the GPUs. This approach for both the MLP and self attention layer fuses groups of two GEMMs, eliminates a synchronization point in between, and results in better scaling. This enables us to perform all GEMMs in a simple transformer layer using only two all-reduces in the forward path and two in the backward path (see Figure 4)," where Shoeybi's backward path corresponds to the instant backpropagation).
Claims 2-4, 10-12, and 21-23 are rejected under 35 U.S.C. 103 as being unpatentable over Shoeybi, et al., "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism" (hereinafter "Shoeybi") in view of Pope, et al., "Efficiently Scaling Transformer Inference" (hereinafter "Pope") in view of Siegl, et al. (US 2020/0081744 A1, hereinafter "Siegl").
Regarding Claim 2, the rejection of Claim 1 is incorporated.
The Shoeybi/Pope combination teaches determining whether to use row sharding or column sharding with the plurality of hardware accelerators to calculate an output for the fully connected layer.
The Shoeybi/Pope combination may not explicitly teach wherein using row sharding comprises: loading a first subset of the plurality of parameter blocks arranged along the row dimension one after another into a particular hardware accelerator, receiving, for each parameter block in the first subset of the plurality of parameter blocks, a corresponding first input vector at the particular hardware accelerator, and generating a partial output for the fully connected layer by determining a multiplication between (i) each parameter block in the first subset of the plurality of parameter blocks and (ii) the corresponding first input vector over one or more hardware cycles while communicating the corresponding first input vectors to other hardware accelerators over an accelerator interconnect network during the one or more hardware cycles.
However, Siegl teaches:
wherein using row sharding comprises: loading a first subset of the plurality of parameter blocks arranged along the row dimension one after another into a particular hardware accelerator (Siegl, Fig. 4B, depicting accelerators 402, 404, and 406 receiving parameters blocks for loading, and [0041]: "Each accelerator 402 , 404 , 406 , in parallel, multi plies one block of matrix B by the corresponding columns of the partition of matrix A fetched from the host 408. The result is accumulated into the local partition of matrix C. This processing continues until each accelerator out of P given accelerators loaded 1/P of matrices A , B and C," where Siegl's parameter matrix A is sharded by rows, as in [0039]: "Matrices A and C are split by row into P partitions , stored in column-major order");
receiving, for each parameter block in the first subset of the plurality of parameter blocks, a corresponding first input vector at the particular hardware accelerator (Siegl, [0040]: "FIG . 4B shows a buffer flow in a heterogeneous system with three accelerators in one embodiment . ... Each accelerator 402, 404, 406, in parallel , fetches one block of matrix A and one block of matrix B , such that they can be multiplied to produce a contribution to the local partition of matrix C," where Siegl's blocks of matrix B correspond to the instant input vectors); and
generating a partial output for the fully connected layer by determining a multiplication between (i) each parameter block in the first subset of the plurality of parameter blocks and (ii) the corresponding first input vector (Siegl, [0040]: "Each accelerator 402, 404, 406, in parallel, fetches one block of matrix A and one block of matrix B , such that they can be multiplied to produce a contribution to the local partition of matrix C," where Siegl's local partition of matrix C corresponds to the instant partial output) over one or more hardware cycles while communicating the corresponding first input vectors to other hardware accelerators ... during the one or more hardware cycles ... (Siegl, [0004]: "The method may further include each of the P hardware accelerators in parallel multiplying one block of the second matrix stored locally by corresponding columns of the partition of the first matrix stored locally and accumulating a result into a local partition of the third matrix," where Siegl's in parallel corresponds to the instant hardware cycles, as in the processing restart of [0042]: "As just 1/P of matrix B was read from the host , each accelerator 402 , 404 , 406 , in parallel , reads the block of matrix B stored on its neighbor in a ring communication pattern . At the same time , each accelerator 402 , 404 , 406 fetches the next block of matrix A. ... If the multiplication is complete, the processing terminates, with the result in matrix C, otherwise the processing restarts with fetching of block of matrix A and a block of matrix B" and [0043]: "FIG . 4C illustrates a communication timeline of three accelerators as an example . Accelerators 402 , 404 , 406 execute in lock step . ... For each further step one further sub-block of matrix A and one further sub - block of matrix B is pushed from the host 408 into each accelerator , as shown at time step 412") over an accelerator interconnect network (Siegl, [0019]: "A heterogeneous system may include multiple host nodes 102a , 102n interconnected by a host interconnect 104, and by multiple accelerators 106a , 106b , 106c , 106n connected by and accelerator interconnect 108").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Shoeybi/Pope combination regarding determining whether to use row sharding or column sharding with the plurality of hardware accelerators to calculate an output for the fully connected layer with those of Siegl regarding wherein using row sharding comprises: loading a first subset of the plurality of parameter blocks arranged along the row dimension one after another into a particular hardware accelerator, receiving, for each parameter block in the first subset of the plurality of parameter blocks, a corresponding first input vector at the particular hardware accelerator, and generating a partial output for the fully connected layer by determining a multiplication between (i) each parameter block in the first subset of the plurality of parameter blocks and (ii) the corresponding first input vector over one or more hardware cycles while communicating the corresponding first input vectors to other hardware accelerators over an accelerator interconnect network during the one or more hardware cycles.
The motivation to do so would be to facilitate improved network and compute utilization during matrix multiplication operations (Siegl, [0049]: "With multiple sub-blocks per partition, it is possible to use two rings operating in parallel in opposite directions for two subsets of sub - blocks . Such bi - directional connections between the accelerators may utilize the full bandwidth available in both directions , for example , pro viding for efficient use of available bandwidth in a network of computer . This optimization also improves the performance of GEMM").
Claims 10 and 21 recite all limitations of Claim 2 in system and computer storage media forms, respectively, and are rejected under the same rationales.
Regarding Claim 3, the rejection of Claim 2 is incorporated. Siegl further teaches:
wherein loading the first subset of the plurality of parameter blocks arranged along the row dimension one after another into the particular hardware accelerator comprises: loading a first parameter block in the first subset of the plurality of parameter blocks into the particular hardware accelerator (Siegl, Fig. 3C, depicting block
A
0
from parameter matrix
A
being loaded from the first row partition into Accelerator 0, and [0038]: "FIG . 3C illustrates a communication timeline of three accelerators as an example . Accelerators 302, 304, 306 execute in lock step . In one embodiment , as no sub-block of matrix A is given , the first or initial step demands the host 308 to push one sub - block of matrix A into each accelerator memory"); and
loading a second parameter block in the first subset of the plurality of parameter blocks into the particular hardware accelerator, wherein the second parameter block is an immediate horizontal neighbor of the first parameter block (Siegl, Fig. 3C, depicting block
A
1
from parameter matrix A being loaded after block
A
0
from the first row partition of A, and [0038]: "FIG . 3C illustrates a communication timeline of three accelerators as an example. ... For each further step , one further sub-block of matrix A is pushed from the host 308 into each accelerator 302 , 304 , 306").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Shoeybi/Pope/Siegl combination regarding wherein loading the first subset of the plurality of parameter blocks arranged along the row dimension one after another into the particular hardware accelerator with the further teachings of Siegl regarding wherein loading the first subset of the plurality of parameter blocks arranged along the row dimension one after another into the particular hardware accelerator comprises: loading a first parameter block in the first subset of the plurality of parameter blocks into the particular hardware accelerator; and loading a second parameter block in the first subset of the plurality of parameter blocks into the particular hardware accelerator, wherein the second parameter block is an immediate horizontal neighbor of the first parameter block
The motivation to do so would be to facilitate improved network and compute utilization during matrix multiplication operations (Siegl, [0049]: "With multiple sub-blocks per partition, it is possible to use two rings operating in parallel in opposite directions for two subsets of sub - blocks . Such bi - directional connections between the accelerators may utilize the full bandwidth available in both directions , for example , pro viding for efficient use of available bandwidth in a network of computer . This optimization also improves the performance of GEMM").
Claims 11 and 22 recite all limitations of Claim 3 in system and computer storage media forms, respectively, and are rejected under the same rationales.
Regarding Claim 4, the rejection of Claim 1 is incorporated. The Shoeybi/Pope combination teaches:
wherein using column sharding comprises: ... the plurality of parameter blocks arranged in the column dimension one after another (Shoeybi, p. 4, 3. Model Parallel Transformers: "Another option is to split A along its columns
A
=
A
1
,
A
2
. ... This is advantageous as it removes a synchronization point. Hence, we partition the first GEMM in this column parallel fashion and split the second GEMM along its rows so it takes the output of the GeLU layer directly without requiring any communication as shown in Figure 3a. The output of the second GEMM is then reduced across the GPUs before passing the output to the dropout layer. This approach splits both GEMMs in the MLP block across GPUs," where Shoeybi's GEMM operation matrices correspond to the instant blocks).
The Shoeybi/Pope combination teaches wherein using column sharding comprises the plurality of parameter blocks arranged in the column dimension one after another.
The Shoeybi/Pope combination may not explicitly teach loading a second subset of the plurality of parameter blocks arranged ... one after another into the particular hardware accelerator receiving a second input vector at the particular hardware accelerator generating the partial output for the fully connected layer by determining a multiplication between (i) each parameter block in the second subset of the plurality of parameter blocks and (ii) the second input vector over one or more hardware cycles while communicating corresponding results of the multiplications to the other hardware accelerators over the accelerator interconnect network during the multiple hardware cycles.
However, Siegl teaches:
... loading a second subset of the plurality of parameter blocks arranged ... one after another into the particular hardware accelerator (Siegl, Fig. 4B, depicting accelerators 402, 404, and 406 receiving parameters blocks for loading, and [0041]: "Each accelerator 402 , 404 , 406 , in parallel, multi plies one block of matrix B by the corresponding columns of the partition of matrix A fetched from the host 408. The result is accumulated into the local partition of matrix C. This processing continues until each accelerator out of P given accelerators loaded 1/P of matrices A , B and C," where Siegl's parameter matrix A is sharded by rows, as in [0039]: "Matrices A and C are split by row into P partitions , stored in column-major order");
receiving a second input vector at the particular hardware accelerator (Siegl, [0040]: "FIG . 4B shows a buffer flow in a heterogeneous system with three accelerators in one embodiment . ... Each accelerator 402, 404, 406, in parallel, fetches one block of matrix A and one block of matrix B , such that they can be multiplied to produce a contribution to the local partition of matrix C," where Siegl's blocks of matrix B correspond to the instant input vectors); and
generating the partial output for the fully connected layer by determining a multiplication between (i) each parameter block in the second subset of the plurality of parameter blocks and (ii) the second input vector (Siegl, [0040]: "Each accelerator 402, 404, 406, in parallel, fetches one block of matrix A and one block of matrix B , such that they can be multiplied to produce a contribution to the local partition of matrix C," where Siegl's local partition of matrix C corresponds to the instant partial output) over one or more hardware cycles while communicating corresponding results of the multiplications to other hardware accelerators ... during the one or more hardware cycles ... (Siegl, [0004]: "The method may further include each of the P hardware accelerators in parallel multiplying one block of the second matrix stored locally by corresponding columns of the partition of the first matrix stored locally and accumulating a result into a local partition of the third matrix," where Siegl's in parallel corresponds to the instant hardware cycles, as in the processing restart of [0042]: "As just 1/P of matrix B was read from the host , each accelerator 402 , 404 , 406 , in parallel , reads the block of matrix B stored on its neighbor in a ring communication pattern" and [0043]: "FIG . 4C illustrates a communication timeline of three accelerators as an example . Accelerators 402 , 404 , 406 execute in lock step . ... For each further step one further sub-block of matrix A and one further sub - block of matrix B is pushed from the host 408 into each accelerator , as shown at time step 412") over the accelerator interconnect network (Siegl, [0019]: "A heterogeneous system may include multiple host nodes 102a , 102n interconnected by a host interconnect 104, and by multiple accelerators 106a , 106b , 106c , 106n connected by and accelerator interconnect 108").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Shoeybi/Pope combination regarding wherein using column sharding comprises the plurality of parameter blocks arranged in the column dimension one after another with those of Siegl regarding loading a second subset of the plurality of parameter blocks arranged ... one after another into the particular hardware accelerator receiving a second input vector at the particular hardware accelerator generating the partial output for the fully connected layer by determining a multiplication between (i) each parameter block in the second subset of the plurality of parameter blocks and (ii) the second input vector over one or more hardware cycles while communicating corresponding results of the multiplications to the other hardware accelerators over the accelerator interconnect network during the multiple hardware cycles.
The motivation to do so would be to facilitate improved network and compute utilization during matrix multiplication operations (Siegl, [0049]: "With multiple sub-blocks per partition, it is possible to use two rings operating in parallel in opposite directions for two subsets of sub - blocks . Such bi - directional connections between the accelerators may utilize the full bandwidth available in both directions , for example , pro viding for efficient use of available bandwidth in a network of computer . This optimization also improves the performance of GEMM").
Claims 12 and 23 recite all limitations of Claim 4 in system and computer storage media forms, respectively, and are rejected under the same rationales.
Claims 5, 13, and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Shoeybi, et al., "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism" (hereinafter "Shoeybi") in view of Pope, et al., "Efficiently Scaling Transformer Inference" (hereinafter "Pope") in view of Siegl, et al. (US 2020/0081744 A1, hereinafter "Siegl") in view of Zheng, et al., "Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning" (hereinafter "Zheng").
Regarding Claim 5, the rejection of Claim 4 is incorporated. Siegl further teaches:
... loading a first parameter block in the second subset of the plurality of parameter blocks into the particular hardware accelerator (Siegl, Fig. 3C, depicting block
A
0
from parameter matrix
A
being loaded from the first row partition into Accelerator 0, and [0038]: "FIG . 3C illustrates a communication timeline of three accelerators as an example . Accelerators 302, 304, 306 execute in lock step . In one embodiment , as no sub-block of matrix A is given , the first or initial step demands the host 308 to push one sub - block of matrix A into each accelerator memory"); and
loading a second parameter block in the second subset of the plurality of parameter blocks into the particular hardware accelerator, wherein the second parameter block is an immediate ... neighbor of the first parameter block (Siegl, Fig. 3C, depicting block A_1 from parameter matrix A being loaded after block
A
0
from the first row partition of A, and [0038]: "FIG . 3C illustrates a communication timeline of three accelerators as an example. ... For each further step , one further sub-block of matrix A is pushed from the host 308 into each accelerator 302 , 304 , 306").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Shoeybi/Pope/Siegl combination regarding loading the second subset of the plurality of parameter blocks arranged in the column dimension one after another into the particular hardware accelerator with the further teachings of Siegl regarding loading a first parameter block in the second subset of the plurality of parameter blocks into the particular hardware accelerator; and loading a second parameter block in the second subset of the plurality of parameter blocks into the particular hardware accelerator, wherein the second parameter block is an immediate neighbor of the first parameter block.
The motivation to do so would be to facilitate improved network and compute utilization during matrix multiplication operations (Siegl, [0049]: "With multiple sub-blocks per partition, it is possible to use two rings operating in parallel in opposite directions for two subsets of sub - blocks . Such bi - directional connections between the accelerators may utilize the full bandwidth available in both directions , for example , pro viding for efficient use of available bandwidth in a network of computer . This optimization also improves the performance of GEMM").
The Shoeybi/Pope/Siegl combination teaches loading a subset of the plurality of parameter blocks one after another into the particular hardware accelerator.
The Shoeybi/Pope/Siegl combination does not explicitly teach loading the second subset of the plurality of parameter blocks arranged in the column dimension one after another into the particular hardware accelerator ... wherein the second parameter block is an immediate vertical neighbor of the first parameter block.
However, Zheng teaches:
loading the second subset of the plurality of parameter blocks arranged in the column dimension one after another into the particular hardware accelerator ... wherein the second parameter block is an immediate vertical neighbor of the first parameter block (Zheng, p. 3, Figure 2: "Common parallelization techniques for training a 2-layer Multi-layer Perceptron (MLP). Only the forward pass is shown. 'x' is the input data. 'w1' and 'w2' are two weight matrices," depicting at "(b) Operator Parallelism" the column-partitioned parameters of weight matrix W1 as immediate vertical neighbors).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Shoeybi/Pope/Siegl combination regarding loading a subset of the plurality of parameter blocks one after another into the particular hardware accelerator with those of Zheng regarding loading the second subset of the plurality of parameter blocks arranged in the column dimension one after another into the particular hardware accelerator ... wherein the second parameter block is an immediate vertical neighbor of the first parameter block.
The motivation to do so would be to facilitate executing models too large to fit into the memory of a single device by parallelizing across devices in a manner with minimal compute and communication costs (Zheng, p. 3, 2.1 Conventional View of ML Parallelism: "Operator parallelism. When the model is too large to fit in one device, operator parallelism is an effective model parallelism option. Operator parallelism refers to approaches that partition the computation of a specific operator (abbreviated as op in the following text), such as matmul shown in Fig. 2b, along non-batch axes, and compute each part of the operator in parallel across multiple devices" and p. 5, 4.1 The Space of Intra-Operator Parallelism: "a matrix multiplication ... corresponds to a three-level for-loop. To parallelize it, we can parallelize the loop i, loop j, loop k, or combinations of them across devices, which would have different computation and communication costs, require different layouts for the input tensors, and result in output tensors with different layouts. ... The goal of the intra-op pass is to pick one parallel algorithm for every operator to minimize the execution time of the entire graph").
Claims 13 and 24 recite all limitations of Claim 5 in system and computer storage media forms, respectively, and are rejected under the same rationales.
Claims 7, 25, and 26 are rejected under 35 U.S.C. 103 as being unpatentable over Shoeybi, et al., "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism" (hereinafter "Shoeybi") in view of Pope, et al., "Efficiently Scaling Transformer Inference" (hereinafter "Pope") in view of Zheng, et al., "Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning" (hereinafter "Zheng").
Regarding Claim 7, the rejection of Claim 1 is incorporated. The Shoeybi/Pope combination teaches:
determining a loss of the output ... (Shoeybi, p. 5, 3. Model Parallel Transformers: "we fuse the output of the parallel
G
E
M
M
Y
1
,
Y
2
with the cross entropy loss which reduces the dimension to
b
×
s
. Communicating scalar losses instead of logits is a huge reduction in communication that improves the efficiency of our model parallel approach"); and
determining an update to the plurality of parameters of the fully connected layer based on the loss (Shoeybi, p. 11, B.1. Hybrid Model and Data Parallelism: "During back propagation we run multiple gradient all-reduce operations in parallel to reduce weight gradients within each distinct data parallel group. ... GPUs within each model parallel group perform all-reduces amongst all GPUs within the group. For data parallelism, each of the all-reduce operations takes place with one of the GPUs from each model parallel group").
The Shoeybi/Pope combination teaches determining a loss of the output and determining an update to the plurality of parameters of the fully connected layer based on the loss.
The Shoeybi/Pope combination may not explicitly teach a loss of the output relative to a ground truth output of the input.
However, Zheng teaches:
a loss of the output relative to a ground truth output of the input (Zheng, p. 4, 3 Overview: "Figure 4: An example to demonstrate Alpa’s API for Jax," line 6, "return jax.numpy.mean((out - batch["y"]) ** 2)," where the the "batch[y]" term of the loss function corresponds to the instant ground truth output of the input training batch).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Shoeybi/Zheng combination regarding determining a loss of the output with the teachings of Zheng regarding a loss of the output relative to a ground truth output of the input.
The motivation to do so would be to facilitate common practice for deep learning model update during training according to model losses (Zheng, p. 2, 2 Background: Distributed Deep Learning: "DL computation is commonly represented by popular ML frameworks [1, 9, 42] as a dataflow graph. ... Training a DL model for one iteration consists of computing a loss by forwarding a batch of data through the graph, deriving the updates via a reverse backward pass, and applying the updates to the parameters via weight update operations" and p. 10, 8.1 End-to-End Performance, Models and training workloads: "To study the ability to train large models, we follow common ML practice to scale the model size along with the number of GPUs, with the parameter range reported in Table 4. ... The gradients are accumulated across micro-batches").
Claims 26 and 25 recite all limitations of Claim 7 in system and computer storage media forms, respectively, and are rejected under the same rationales.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Zhuang, et al., "On Optimizing The Communication Of Model Parallelism," teach a method of cross-mesh resharding, which combines intra-operator and inter-operator parallelism to support executing large models on large clusters. The method optimizes for communication efficiency where a sharded tensor is sent between mesh devices when the tensor may be distributed with varying layouts.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT N DAY whose telephone number is (703)756-1519. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/R.N.D./Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights