DETAILED ACTION
This action is responsive to the claims filed on 01/16/2024. Claims 1-20 are pending for examination.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 03/02/2026 and 08/02/2024 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 2, 6-10, 13-15, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (Li, S., Xue, F., Baranwal, C., Li, Y., & You, Y. (2023, July). Sequence parallelism: Long sequence training from system perspective. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 2391-2404).), hereafter referred to as Li, in view of Dao et al. (Dao, T. (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. arXiv preprint arXiv:2307.08691.), hereafter referred to as Dao.
Claim 1: Li teaches:
A processor, comprising: one or more circuits to cause inferencing of two or more contiguous portions of information to be distributed between two or more respective processing cores (Li, page 2391, col. 2, paragraph 2, “In this paper, we designed and implemented sequence parallelism, which aims at breaking the limitation that we must store the whole sequence in one GPU. The proposed system can train transformer based models with longer sequences and a larger batch size. Specifically, we first split the input sequence into multiple chunks along the sequence dimension and feed each sub-sequence chunk to one corresponding GPU.”, The claimed “information” reads on Li’s input sequence. Li’s chunks are contiguous portions because Li splits along the sequence dimension. The claimed “processing cores” can be read broadly on Li’s GPUs/devices, since each device processes its assigned sub-sequence chunk during transformer attention computation.)
Dao, in the same field of core partitioning, teaches the following which Li fails to teach:
based, at least in part, on locations of the two or more contiguous portions within the information relative to one or more terminating portions of the information. (Dao, page 2, paragraph 1, “FlashAttention has seen wide adoption in large-scale training and inference of Transformers”; Dao, page 6, Algorithm 1, steps 1-2, “Divide Q into
(
T
r
=
⌈
N
B
r
⌉
)
blocks
(
Q
1
,
…
,
Q
T
r
)
” and “divide K, V into
T
c
=
⌈
N
B
c
)
blocks”; Dao, page 6, section entitled “Causal masking,” “for any blocks where all the column indices are more than the row indices . . . we can skip the computation of that block”; Dao, page 8, section 3.2, “the outer loop (over sequence length) is embarrassingly parallel, and we schedule them on different thread blocks,” and Figure 2, “each worker takes care of a block of rows of the attention matrix.”, Dao’s indexed row blocks (
Q
i
) and indexed column blocks (
K
j
) and (
V
j
) correspond to contiguous ranges of positions within the finite input sequence, with the final indexed blocks (
Q
T
r
)
,
(
K
T
c
), and (
V
T
c
) corresponding to terminating portions of the sequence representations. Dao assigns the forward-pass attention computation for the respective row blocks to different worker thread blocks, which are scheduled on GPU multiprocessors. Dao further determines whether attention computation is performed for a pair of blocks according to the locations of those blocks within the sequence: when the column indices of a block are greater than its row indices, the block lies in the later, causally masked portion of the sequence and its computation is skipped. Consequently, the inferencing work performed for each distributed row block depends on the block’s indexed location within the sequence and on the location of the later blocks extending toward the terminating block of the sequence. When Dao’s position-dependent attention partitioning is incorporated into Li’s system, the inferencing of Li’s contiguous sub-sequence chunks is distributed among GPU processing resources based, at least in part, on the locations of the chunks within the ordered sequence relative to the terminating portion of the sequence.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Dao into the system of Li. Li teaches distributing Transformer sequence processing by splitting an input sequence into multiple chunks and feeding each chunk to a corresponding GPU/device, thereby avoiding the need for a single device to hold the whole sequence and enabling distributed self-attention computation. Dao teaches that attention computation for long-sequence Transformers is limited by inefficient work partitioning among GPU thread blocks and warps, and improves attention performance by better partitioning the attention computation across GPU execution resources. A person of ordinary skill in the art implementing Li’s distributed sequence-parallel attention would have been motivated to apply Dao’s teachings regarding attention work partitioning because both references address the same technical problem of efficiently executing Transformer attention for long sequences on GPU-based parallel processing hardware. The combination would have predictably improved Li’s distributed Transformer attention by applying Dao’s indexed sequence-block scheduling and causal-mask position rules to determine the attention work performed for Li’s respective sub-sequence chunks. In particular, because Dao teaches that the computation for an indexed row block depends on which later column blocks remain before the terminating block of the sequence and which later blocks are causally masked, a person of ordinary skill would have been motivated to distribute Li’s sub-sequence inferencing work in view of the ordered locations and corresponding unmasked workloads of the chunks. Doing so would improve GPU occupancy, reduce inefficient work allocation, and increase the execution efficiency of Li’s distributed Transformer system.
Claims 8 and 15 recite limitations substantially similar to claim 1, as such a similar analysis applies.
Claim 2: Li and Dao teaches the limitations of claim 1, Li further teaches:
The processor of claim 1, wherein the one or more circuits are to further cause inferencing of an equal number of portions of the information to be distributed to each of the two or more respective processing cores. (Li, page 2393, col. 2, section 3.1, “To distribute sub-sequences to multiple devices, the main challenge is calculating attention scores across devices. Therefore, we propose Ring Self Attention (RSA) to compute attention output in a distributed setting. There are two steps in RSA to obtain the final output. Please note, we only consider bidirectional self-attention here to introduce RSA succinctly. We treat all heads equally so it can be extended to multi-head attention directly. Given query embeddings {q1 1,q1 2,...,qN embeddings {k1 1,k1 2,...,kN dings {v1 1,v1 2, ..., vN L },key L } and value embed L }, where qns represents the key embedding of the sth token in the sequence which is on nth device. We define all key embed dings on nth device as Kn. In RSA, nth device holds the corresponding query embeddings Qn,key embeddings Kn and value embeddings V n. The embeddings on nth device correspond to the nth chunk whose sub-sequence length is L/N.”, Li divides a sequence of length L across N GPUs so that each GPU holds a corresponding L/N sub-sequence. That teaches equal-sized distributed portions, and under broad reading also teaches equal chunk allocation to each processing device.)
Claim 6: Li and Dao teaches the limitations of claim 1, Li further teaches:
The processor of claim 1, wherein each of the two or more contiguous portions of information comprises a same number of tokens. (Li, page 2393, col. 2, section 3.1, “To distribute sub-sequences to multiple devices, the main challenge is calculating attention scores across devices. Therefore, we propose Ring Self Attention (RSA) to compute attention output in a distributed setting. There are two steps in RSA to obtain the final output. Please note, we only consider bidirectional self-attention here to introduce RSA succinctly. We treat all heads equally so it can be extended to multi-head attention directly. Given query embeddings {q1 1,q1 2,...,qN embeddings {k1 1,k1 2,...,kN dings {v1 1,v1 2, ..., vN L },key L } and value embed L }, where qns represents the key embedding of the sth token in the sequence which is on nth device. We define all key embed dings on nth device as Kn. In RSA, nth device holds the corresponding query embeddings Qn,key embeddings Kn and value embeddings V n. The embeddings on nth device correspond to the nth chunk whose sub-sequence length is L/N.”, Li’s sequence chunks correspond to token spans of the input sequence. Because each sub-sequence is L/N, the chunks are equal length. Since transformer sequences are token sequences, equal sub-sequence length maps to each contiguous portion having the same number of tokens.)
Claim 7: Li and Dao teaches the limitations of claim 1, Li further teaches:
The processor of claim 1, wherein one or more activations based, at least in part, on the two or more contiguous portions of information are to be distributed to the two or more respective processing cores. (Li, page 2392, col. 2, section 2, “For an input sentence (
X
=
{
x
1
,
…
,
x
N
}
) with (N) tokens, we encode every token (x) into three attention embeddings (i.e., query (q), key (k), value (v))”; Li, page 2393, col. 2, section 3.1, “In RSA, nth device holds the corresponding query embeddings (
Q
n
), key embeddings (
K
n
) and value embeddings (
V
n
). The embeddings on nth device correspond to the nth chunk whose sub-sequence length is (
L
N
)”; Li, page 2394, col. 1, section 3.1, “we first transmit the key embeddings among devices to calculate the attention scores (
Q
K
T
) in a circular fashion,” and “we transmit all value embeddings instead of key embeddings in a similar way.”, Li first generates query, key, and value embeddings from the tokens in each contiguous sub-sequence chunk. These embeddings are intermediate neural-network activation values because they are generated from the token inputs and are used in the attention-layer computations. Li expressly states that the embeddings on a particular device correspond to that device’s assigned sub-sequence chunk and then transmits the key and value embeddings among the devices. Thus, Li distributes activations that are based on the respective contiguous portions of information to the GPU devices corresponding to the claimed processing cores.)
Claim 9: Li and Dao teaches the limitations of claim 1, Dao further teaches:
The method of claim 8, wherein the inferencing is to be performed using a large-language model. (Dao, page 6, paragraph 3, “Causal masking. One common use case of attention is in auto-regressive language modeling, where we need to apply a causal mask to the attention matrix S(i.e.,any entry S𝑖𝑗 with 𝑗>𝑖 is set to−∞). 1.As Flash Attention and Flash Attention-2 already operate by blocks, for any blocks where all the column indices are more than the row indices (approximately half of the blocks for large sequence length),we can skip the computation of that block. This leads to around 1.7-1.8× speed up compared to attention without the causal mask.”,
Page 11, section 4.2, “We measure the training throughput of GPT-style models with either 1.3 B or 2.7B parameters,” Dao applies FlashAttention-2 to GPT-style billion-parameter models, which supports the “large-language model” aspect. The attention forward computation in such models is part of inference and is also used during training.)
Claim 10: Li and Dao teaches the limitations of claim 1, Dao further teaches:
The method of claim 8, wherein the inferencing is to train a large-language model. (Dao, page 6, paragraph 3, “Causal masking. One common use case of attention is in auto-regressive language modeling, where we need to apply a causal mask to the attention matrix S(i.e.,any entry S𝑖𝑗 with 𝑗>𝑖 is set to−∞). 1.As Flash Attention and Flash Attention-2 already operate by blocks, for any blocks where all the column indices are more than the row indices (approximately half of the blocks for large sequence length),we can skip the computation of that block. This leads to around 1.7-1.8× speed up compared to attention without the causal mask.”,
Page 11, section 4.2, “We measure the training throughput of GPT-style models with either 1.3 B or 2.7B parameters,” Dao applies FlashAttention-2 to GPT-style billion-parameter models, which supports the “large-language model” aspect. The forward/inference-like attention computation occurs during training of the model.)
Claim 13: Li and Dao teaches the limitations of claim 1, Dao further teaches:
The method of claim 8, wherein each processing core of the two or more respective processing cores receive a same number of portions of the information. (Li, page 2393, col. 2, section 3.1, “To distribute sub-sequences to multiple devices, the main challenge is calculating attention scores across devices. Therefore, we propose Ring Self Attention (RSA) to compute attention output in a distributed setting. There are two steps in RSA to obtain the final output. Please note, we only consider bidirectional self-attention here to introduce RSA succinctly. We treat all heads equally so it can be extended to multi-head attention directly. Given query embeddings {q1 1,q1 2,...,qN embeddings {k1 1,k1 2,...,kN dings {v1 1,v1 2, ..., vN L },key L } and value embed L }, where qns represents the key embedding of the sth token in the sequence which is on nth device. We define all key embeddings on nth device as Kn. In RSA, nth device holds the corresponding query embeddings Qn,key embeddings Kn and value embeddings V n. The embeddings on nth device correspond to the nth chunk whose sub-sequence length is L/N.”, Li divides a sequence of length L across N GPUs so that each GPU holds a corresponding L/N sub-sequence. That teaches equal-sized distributed portions, and under broad reading also teaches equal chunk allocation to each processing device.)
Claim 14: Li and Dao teaches the limitations of claim 1, Li further teaches:
The method of claim 8, wherein the two or more respective processing cores exchange token embeddings so that each of the two or more respective processing cores has a token embedding for each token in the information. (Li, page 2392, col. 2, section 2, “For an input sentence (
X
=
{
x
1
,
…
,
x
N
}) with (N) tokens, we encode every token (x) into three attention embeddings (i.e., query (q), key (k), value (v)”; Li, page 2393, col. 2, section 3.1, “In RSA, nth device holds the corresponding query embeddings (
Q
n
), key embeddings (
K
n
) and value embeddings (
V
n
). The embeddings on nth device correspond to the nth chunk”; Li, page 2394, col. 1, section 3.1, “we first transmit the key embeddings among devices to calculate the attention scores (
Q
K
T
) in a circular fashion,” “Such communication needs to be conducted (N-1) times to make sure the query embeddings of each sub-sequence can multiply all the key embeddings,” and “each device . . . will receive different key embeddings from the previous device”; Li, page 2394, col. 1, section 3.1, “Since computing (
O
n
) requires (
S
n
) and all value embeddings . . . we transmit all value embeddings instead of key embeddings in a similar way.” Li teaches that each token is represented by query, key, and value embeddings and that each GPU initially holds the embeddings corresponding to the tokens in its local chunk. Li’s devices then exchange the token embeddings by repeatedly sending and receiving the key embeddings around the ring and, in the second stage, sending and receiving all value embeddings in the same manner. The (N-1) ring communications ensure that the query embeddings for each device’s sub-sequence are processed against the key embeddings for all tokens in the input sequence. Accordingly, each GPU device receives and has access during the attention computation to a token embedding corresponding to every token in the information, thereby teaching the claimed exchange between the respective processing cores.)
Claim 20: Li and Dao teaches the limitations of claim 1, Li further teaches:
The computer of claim 15, wherein each of the processing core of the two or more respective processing cores has an embedding for each token in the information. (Li, page 2393, col. 2, section 3.1, “To distribute sub-sequences to multiple devices, the main challenge is calculating attention scores across devices. Therefore, we propose Ring Self Attention (RSA) to compute attention output in a distributed setting. There are two steps in RSA to obtain the final output. Please note, we only consider bidirectional self-attention here to introduce RSA succinctly. We treat all heads equally so it can be extended to multi-head attention directly. Given query embeddings {q1 1,q1 2,...,qN embeddings {k1 1,k1 2,...,kN dings {v1 1,v1 2, ..., vN L },key L } and value embed L }, where qns represents the key embedding of the sth token in the sequence which is on nth device. We define all key embed dings on nth device as Kn. In RSA, nth device holds the corresponding query embeddings Qn,key embeddings Kn and value embeddings V n. The embeddings on nth device correspond to the nth chunk whose sub-sequence length is L/N.”, Li teaches that each device receives or accesses key/value embeddings corresponding to tokens outside its own chunk in order to compute attention over the sequence. That satisfies the broad result that each core has embedding information for each token.)
Claims 3, 5, 11-12, 16-17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Dao and (Local Storage Scheme and Block-Cyclic Mapping. Local storage scheme and block-cyclic mapping. (1997, May 13). https://web.archive.org/web/20230722222752/https://www.netlib.org/scalapack/slug/node76.html), hereafter referred to as ScalaPack.
Claim 3: Li and Dao teaches the limitations of claim 1, ScalaPack, in the same field of further teaches:
The processor of claim 1, wherein the information is split into a number of portions, where the number of portions is an even multiple of a number of processing cores that each receives one or more portions of the information. (ScalaPack, paragraphs 1-2, “The block-cyclic distribution scheme is a mapping of a set of blocks onto the processes. The previous section informally described this mapping as well as some of its properties. To be complete, we must now explain how the blocks that are mapped to the same process are arranged and stored in the local process memory. In other words, we shall describe the precise mapping that associates to a matrix entry identified by its global indexes the coordinates of the process that owns it and its local position within that process's memory. Suppose we have an array of length N to be stored on P processes. By convention, the array entries are numbered 1 through N and the processes are numbered 0 through P-1. First, the array is divided into contiguous blocks of size NB. When NB does not divide N evenly, the last block of array elements will only contain
PNG
media_image1.png
26
98
media_image1.png
Greyscale
entries instead of NB. By convention, these blocks are numbered starting from zero and dealt out to the processes like a deck of cards. In other words, if we assume that the process 0 receives the first block, the
PNG
media_image2.png
15
20
media_image2.png
Greyscale
block is assigned to the process of coordinate
PNG
media_image3.png
26
76
media_image3.png
Greyscale
. The blocks assigned to the same process are stored contiguously in memory. ”, ScaLAPACK teaches the claimed block-cyclic allocation technique: divide data into blocks and deal the blocks to processors cyclically. Choosing the number of chunks as 2P, 4P, etc., for P processors is a straightforward balanced implementation of cyclic block assignment, yielding an even multiple of portions relative to cores.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further incorporate the block-cyclic distribution teachings of ScalaPack into the combined system of Li and Dao. As discussed above, Li teaches splitting input sequence information into chunks distributed across GPU devices, while Dao teaches improving attention execution by better partitioning work among GPU processing resources. ScalaPack teaches a known high-performance computing technique in which data is decomposed into blocks and the blocks are distributed across a process grid in a block-cyclic manner to improve load balance and communication characteristics. A person of ordinary skill in the art would have recognized that Li’s sequence chunks are data blocks to be processed in parallel, and that Dao’s attention workload is sensitive to how such blocks are assigned to processing resources. Accordingly, a person of ordinary skill would have been motivated to apply ScalaPack’s block-cyclic distribution technique to the Li-Dao system so that multiple contiguous sequence portions could be assigned across multiple processing cores or devices in a balanced and predictable manner. Doing so would have been no more than the use of a known parallel data-distribution technique for its known purpose—improving load balance and scalability—in the analogous context of distributing sequence-block workloads for parallel Transformer attention.
Claim 5: Li and Dao teaches the limitations of claim 1, ScalaPack, in the same field of further teaches:
The processor of claim 1, wherein at least one portion of the information from a beginning of the information and at least one portion of the information at an end of the information are distributed to a same processing core. (ScalaPack, paragraph 2, “In other words, if we assume that the process 0 receives the first block, the
PNG
media_image2.png
15
20
media_image2.png
Greyscale
block is assigned to the process of coordinate
PNG
media_image3.png
26
76
media_image3.png
Greyscale
. The blocks assigned to the same process are stored contiguously in memory.”,
ScalaPack, page 1, paragraph 2, “First, the array is divided into contiguous blocks of size (NB),” “these blocks are numbered starting from zero and dealt out to the processes like a deck of cards,” and “if we assume that the process 0 receives the first block, the kth block is assigned to the process of coordinate
(
m
o
d
(
k
,
P
)
).”
ScalaPack’s first block has block index (k=0) and is therefore assigned to process (mod(0,P)=0). Due to the disclosed modulo-(P) cyclic mapping, a later block having index (k=P), or more generally (k=mP), is also assigned to process 0. Accordingly, when the information contains (P+1) blocks, the first portion is block 0 and the final or end portion is block (P), and both are assigned to the same process 0. Li’s ordered sub-sequence chunks correspond to ScalaPack’s ordered contiguous blocks, and Li’s GPU devices correspond to ScalaPack’s processes. Applying ScalaPack’s disclosed cyclic ownership rule to Li therefore results in a beginning sub-sequence portion and an end sub-sequence portion being distributed to the same processing core.)
Claim 11: Li and Dao teaches the limitations of claim 8, ScalaPack, in the same field of further teaches:
The method of claim 8, wherein each processing core of the two or more respective processing cores receives at least two portions of the information. (ScalaPack, paragraph 1, “First, the array is divided into contiguous blocks of size NB. When NB does not divide N evenly, the last block of array elements will only contain
PNG
media_image1.png
26
98
media_image1.png
Greyscale
entries instead of NB. By convention, these blocks are numbered starting from zero and dealt out to the processes like a deck of cards.”, In a block-cyclic distribution, once the number of blocks exceeds the number of processors, each processor receives multiple blocks. This suggests that each core is receiving at least two portions when the sequence is over-decomposed into more chunks than cores.)
Claim 12: Li and Dao teaches the limitations of claim 8, ScalaPack, in the same field of further teaches:
The method of claim 8, wherein each processing core of the two or more respective processing cores receives an even number of portions of the information. (ScalaPack, paragraph 6, “In the two-dimensional case, assuming the matrix is partitioned in
PNG
media_image4.png
22
83
media_image4.png
Greyscale
blocks and that the first block is given to the process of coordinates (RSRC, CSRC), the analytical formula given above for the one-dimensional case are simply reused independently in each dimension of the
PNG
media_image5.png
25
57
media_image5.png
Greyscale
process grid. For example, the matrix entry (I,J) is thus to be found in the process of coordinates
PNG
media_image6.png
26
50
media_image6.png
Greyscale
within the local (l,m) block at the position (x,y) given by:
PNG
media_image7.png
67
660
media_image7.png
Greyscale
These formula specify how an
PNG
media_image8.png
12
38
media_image8.png
Greyscale
by
PNG
media_image9.png
12
35
media_image9.png
Greyscale
matrix A is mapped and stored on the process grid. It is first decomposed into
PNG
media_image10.png
12
52
media_image10.png
Greyscale
by
PNG
media_image11.png
12
49
media_image11.png
Greyscale
blocks starting at its upper left corner. These blocks are then uniformly distributed across the process grid in a cyclic manner.”, Cyclic distribution assigns an even count of chunks to each core. For example, 4P chunks over P cores gives four chunks per core. This is a predictable balanced variant of known block-cyclic allocation.)
Claim 17: Li and Dao teaches the limitations of claim 8, ScalaPack, in the same field of further teaches:
The computer system of claim 15, wherein each of the one or more respective processing cores has a same workload that results from processing portions of the information distributed to that respective processing core. (ScalaPack, paragraph 6, “In the two-dimensional case, assuming the matrix is partitioned in
PNG
media_image4.png
22
83
media_image4.png
Greyscale
blocks and that the first block is given to the process of coordinates (RSRC, CSRC), the analytical formula given above for the one-dimensional case are simply reused independently in each dimension of the
PNG
media_image5.png
25
57
media_image5.png
Greyscale
process grid. For example, the matrix entry (I,J) is thus to be found in the process of coordinates
PNG
media_image6.png
26
50
media_image6.png
Greyscale
within the local (l,m) block at the position (x,y) given by:
PNG
media_image7.png
67
660
media_image7.png
Greyscale
These formula specify how an
PNG
media_image8.png
12
38
media_image8.png
Greyscale
by
PNG
media_image9.png
12
35
media_image9.png
Greyscale
matrix A is mapped and stored on the process grid. It is first decomposed into
PNG
media_image10.png
12
52
media_image10.png
Greyscale
by
PNG
media_image11.png
12
49
media_image11.png
Greyscale
blocks starting at its upper left corner. These blocks are then uniformly distributed across the process grid in a cyclic manner.”, Cyclic distribution assigns an even count of chunks to each core. It is interpreted by the examiner that a uniformly distribution of processes across cores will result in the processing cores having the same workload.)
Claim 16: Li and Dao teaches the limitations of claim 15, ScalaPack, in the same field of further teaches:
The computer system of claim 15, wherein the instructions, if performed by the one or more processors, are to further cause inferencing of an even number of portions of the information to be distributed to each of the two or more respective processing cores. (ScalaPack, paragraph 6, “In the two-dimensional case, assuming the matrix is partitioned in
PNG
media_image4.png
22
83
media_image4.png
Greyscale
blocks and that the first block is given to the process of coordinates (RSRC, CSRC), the analytical formula given above for the one-dimensional case are simply reused independently in each dimension of the
PNG
media_image5.png
25
57
media_image5.png
Greyscale
process grid. For example, the matrix entry (I,J) is thus to be found in the process of coordinates
PNG
media_image6.png
26
50
media_image6.png
Greyscale
within the local (l,m) block at the position (x,y) given by:
PNG
media_image7.png
67
660
media_image7.png
Greyscale
These formula specify how an
PNG
media_image8.png
12
38
media_image8.png
Greyscale
by
PNG
media_image9.png
12
35
media_image9.png
Greyscale
matrix A is mapped and stored on the process grid. It is first decomposed into
PNG
media_image10.png
12
52
media_image10.png
Greyscale
by
PNG
media_image11.png
12
49
media_image11.png
Greyscale
blocks starting at its upper left corner. These blocks are then uniformly distributed across the process grid in a cyclic manner.”, Cyclic distribution assigns an even count of chunks to each core. For example, 4P chunks over P cores gives four chunks per core. This is a predictable balanced variant of known block-cyclic allocation.)
Claim 19: Li and Dao teach the limitations of claim 15, and ScalaPack, in the same field of distributing portions of data among parallel processing resources, further teaches:
The computer system of claim 15, wherein a first portion of the information and a last portion of the information are distributed to a same processing core of the two or more processing cores. (ScalaPack, page 1, paragraph 2, “First, the array is divided into contiguous blocks of size (NB),” “these blocks are numbered starting from zero and dealt out to the processes like a deck of cards,” and “if we assume that the process 0 receives the first block, the (k)th block is assigned to the process of coordinate mod(k,P)”; ScalaPack, page 3, paragraph 5, “In figure 4.5, the process of coordinates (0,0) owns four blocks. The matrix entries of the global columns 1, 2 and 5 are contiguously stored in that process’s memory.” ScalaPack teaches cyclically assigning ordered contiguous blocks to processes such that the first block is assigned to process 0 and a later final block having an index that is a multiple of (P) is also assigned to process 0; its illustrated example likewise shows process ((0,0)) owning portions from both the first and last global columns. Applying this block-cyclic assignment to Li’s ordered sub-sequence chunks therefore results in a first portion and a last portion of the information being distributed to the same GPU processing core.)
Claims 4 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Dao and Chatterjee et al., (US 2006/0179267 A1), hereafter referred to as Chatterjee.
Claim 4: Li and Dao teaches the limitations of claim 1, Chatterjee in the same field of further teaches:
The processor of claim 1, wherein at least two of the two or more terminating portions of information are distributed to a same processing core. (Chatterjee, paragraph 43, “array element (i,j) maps to logical processor (i div D mod P, j div D mod Q).”; Chatterjee, paragraph 45, “a ‘skew’ is applied in one of the factors of the (Q) dimension . . . so that processor holding block ((R,C)) also holds blocks ((R+P,C-1)), ((R+2P,C-2)), etc.”; Chatterjee, paragraph 58, “the result (j/D) is the block number, and ‘mod (Q)’ tells which logical processor column the block cyclic wrapping would take it to.” The claimed “terminating portions” are interpreted as the boundary blocks at the two opposing ends of an ordered row of information, namely the block in column 0 at one end and the block in column (Q-1) at the other end. Under Chatterjee’s skewed cyclic mapping, when (C=0), the processor that holds boundary block ((R,0)) also holds block ((R+P,C-1)). Because Chatterjee expressly applies modulo-(Q) block-cyclic wrapping in the column dimension, the column coordinate (C-1=-1) wraps to column (Q-1). Thus, the same processor holds both block ((R,0)), located at one terminating boundary of the ordered row, and block ((R+P,Q-1)), located at the opposite terminating boundary. When this skewed cyclic block-assignment rule is applied to Li’s ordered sequence chunks, the two terminating sequence portions are distributed to the same GPU processing core.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further incorporate the skewed block-cyclic distribution teachings of Chatterjee into the combined system of Li and Dao. Li teaches dividing an input sequence into chunks and distributing those chunks to corresponding GPU devices for distributed Transformer attention, while Dao teaches that attention performance can be improved through better work partitioning across GPU processing resources. Chatterjee teaches distributing contiguous blocks of array data to specific processors using cyclic and skewed block-cyclic patterns, including arrangements in which a processor holding one block also holds additional offset blocks, and teaches that such distribution can address processing imbalance by spreading workload over a larger number of processors. A person of ordinary skill in the art would have recognized that the sequence chunks in Li are analogous to Chatterjee’s contiguous data blocks, and that Dao provides the attention-specific reason to distribute such chunks in a manner that accounts for workload differences associated with sequence position. Therefore, a person of ordinary skill would have been motivated to apply Chatterjee’s cyclic/skewed block-distribution teachings to the Li-Dao system to assign multiple sequence portions, including separated or terminal sequence portions, to selected processing cores in a predictable pattern that improves workload balance and utilization of available processors. The modification would have involved applying a known processor-block mapping technique to a known distributed sequence-processing system to achieve the predictable result of improved parallel workload distribution.
Claim 18 recites the substantially similar limitations to that of claim 4, as such a similar analysis applies.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Korthikanti, V. A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., & Catanzaro, B. (2023). Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 341-353.
Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., ... & Zaharia, M. (2021, November). Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the international conference for high performance computing, networking, storage and analysis (pp. 1-15).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HYUNGJUN B YI whose telephone number is (703)756-4799. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Jung can be reached on (571) 270-3779. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/H.B.Y./Examiner, Art Unit 2124
/USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146
Prosecution Insights