DETAILED ACTION
This Office Action is responsive to the Applicant’s submission, filed on March 2, 2026, amending claims 1 and 11. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 2, 5-7, 9, 10 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over the article entitled, “Zebra: Memory Bandwidth Reduction for CNN Accelerators With Zero Block Regularization of Activation Maps” by Shih et al. (“Shih”), over U.S. Patent Application Publication No. 2018/0300606 to Corkery et al. (“Corkery”), over U.S. Patent Application Publication No. 2021/0266565 to Zhou et al. (“Zhou”), and also over U.S. Patent No. 11,797,830 to Gunnam et al. (“Gunnam”).
Regarding claim 1, Shih describes a “dynamic activation map reduction method to reduce memory bandwidth” (section I “Introduction”). Like claimed, Shih particularly teaches:
receiving outputs from a layer of a neural network (Shih discloses that the dynamic activation map reduction method comprises dividing an activation map into a plurality of non-overlapping blocks, and forcing all values in “unimportant blocks” to zero:
With above observations, we propose a dynamic activation map reduction method to reduce memory bandwidth. Inspired by [11], we prune blocks of zeros in the maps instead of whole maps in [11] or fine grained case as introduced by ReLUs. If all values in one block are all zero, we consider it as background or unimportant one and thus we can prune it in run-time. However, the zero block percentage after ReLUs is still quite low as shown in Table. I. To force the network learns more ”zero blocks”, we propose a regularization method called Zero Block regularization of activation maps (Zebra) to achieve dynamic activation map reduction. Our method divides the activation maps into several non-overlapping blocks, then lets the model to dynamically learn unimportant blocks through Zebra, and simply forces all values in those blocks to zero to achieve simple run-time structured activation maps pruning. The inference has very low overhead and achieves 70% of memory bandwidth reduction within 1% of accuracy loss.
(Section I “Introduction”; emphasis added).
The activation map understandably comprises outputs of a neural network layer, e.g. a convolutional neural network layer, as is known in the art:
In [11], the authors analyze the sparsity of activation maps in different layers. The result shows that the deeper activation maps have higher zero ratio.
(Section 1 “Introduction”; emphasis added).
As to the memory bandwidth overhead of Zebra, the results are shown in Table. V. The required bandwidth means the total activation maps size for one image and the bandwidth overhead is the need of block index storage. In this calculation, we assume a layer-by-layer hardware processing that will store the activation maps to external DRAM for each convoluitonal layer processing. This overhead is totally negligible compared with the amount of reduction in activation maps.
(Section III.B “Results”).
Accordingly, Shih’s dynamic activation map reduction method necessitates receiving outputs, e.g. an activation map, from a layer of neural network so as to divide the activation map into blocks and force values in unimportant blocks to zero.);
partitioning the outputs into a plurality of partitions, wherein sizes of each of the plurality of partitions are defined prior to receiving the outputs (As noted above, Shih discloses that the dynamic activation map reduction method comprises dividing an activation map into a plurality of non-overlapping blocks. Shi further teaches that the sizes of each of the blocks is manually chosen based on various factors including activation map sparsity and location within the neural network:
In Zebra, one important parameter is the block size, which depends on the size of activation maps and datasets. Theoretically, small block size will have higher accuracy and higher sparsity since small block means more fine-grained partitioned. However, small block size will need more index parameters to store the zero conditions of the blocks which leads to memory bandwidth overhead.
(Section II.C “Design Parameters Analysis of Zebra”).
The proposed method is evaluated on two image classification datasets, CIFAR-10 [12] and Tiny-Imagenet that is the subset of Imagenet [13]. CIFAR-10 consists of 10 classes of natural images with resolution 32×32. Empirically, we choose block size as 4 by considering both activation maps sparsity and the zero blocks index overhead. Note that we set block size as 2 when the size of activation maps in deeper layers goes to 2 × 2 in VGG and MobileNets. Tiny-Imagenet contains 200 classes with resolution 64 × 64. We set block size to 8 for better activation maps reduction and the index storage overhead trade-off.
(Section III.A “Settings”; emphasis added).
As the division of the activation maps into such manually-sized blocks occurs during both training and inference – see e.g. Section II.A “Model Training” and Section II.B “Model Inference” – it is apparent that the sizes of each of the plurality of blocks are defined prior to receiving the activation maps during the training and inference. Accordingly, Shih teaches partitioning the outputs, i.e. activation map, into a plurality of partitions, i.e. blocks, wherein sizes of each of the plurality of partitions are defined prior to receiving the outputs.); and
identifying first partitions in the plurality of partitions that can be treated as having zero values (As noted above, Shih discloses that the dynamic activation map reduction method comprises dividing an activation map into a plurality of non-overlapping blocks, and forcing all values in unimportant blocks to zero. In particular, Shih discloses that if the maximum value of a block is less than a threshold, the block is regarded as an unimportant block, and all values therein are zeroed:
Fig. 1 shows the flow of Zebra to train zero blocks. Activation maps are first divided into non-overlapped blocks. Zebra learns zero blocks or unimportant blocks through regularization. The importance of a block is represented by the maximum value in each block. If the maximum value of a block is smaller than a learned threshold, this block will be regarded as a zero block. The blocks in the same channel use the same threshold. The threshold is learned by feeding activation maps to a small network with a global average pooling layer and a fully-connected layer.
(Section II.A “Model Training”).
Accordingly, Shih teaches identifying first partitions, i.e. unimportant blocks, in the plurality of partitions that can be treated as having zero values.).
Shih thus teaches a method similar to that of claim 1. Furthermore, Shi mentions that “index parameters…store the zero conditions of the blocks” (Section II.C “Design Parameters Analysis of Zebra”). Shih, however, does not explicitly teach generating an encoding that identifies locations of the first partitions (i.e. the unimportant/zero blocks) among remaining second partitions in the plurality of partitions, as is required by claim 1. Moreover, Shi does not explicitly teach dividing the second partitions into subsets of partitions, and sending the encoding and each of the subsets of partitions to corresponding execution units in a subsequent layer in the neural network, wherein the encoding is generated by a partitioning circuit located downstream of an output buffer of the layer and upstream of an on-chip memory, as is further required by claim 1.
Corkery describes a deep neural network (DNN) module that can compress and decompress activation data to reduce the utilization of memory bus bandwidth (see e.g. paragraph 0008). Similar to Shih, Corkery particularly teaches:
receiving outputs from a layer of a neural network (see e.g. paragraph 0009: Corkery discloses that the DNN module comprises a compression unit that receives an uncompressed chunk of data generated by one or more of the neurons of the neural network. The DNN module thus receives outputs, i.e. an uncompressed chunk of data, from a layer of the neural network, i.e. from one or more of the neurons of the neural network.);
partitioning the outputs into a plurality of partitions (see e.g. paragraph 0009: Corkery discloses that the uncompressed chunk of data includes a fixed number of bytes, e.g. 64 bytes. Each of the bytes can be considered a partition of the chunk.); and
identifying first partitions in the plurality of partitions that can be treated as having zero values (see e.g. paragraphs 0010-0011 and 0036-0037: Corkery discloses that the DNN module generates a compressed output chunk from the received uncompressed chunk, wherein the compressed output chunk comprises a mask portion that includes a number of bits equal to the number of bytes in the uncompressed chunk and wherein each bit corresponds to a particular byte in the uncompressed chunk. Corkery further discloses that the DNN module sets to zero each bit in the mask portion that has a corresponding byte in the uncompressed chunk containing all zeros, and sets to one each bit in the mask portion that has a corresponding byte in the uncompressed chunk that is non-zero – see e.g. paragraphs 0012 and 0037. Thus to generate the mask portion of the compressed chunk, the DNN module necessarily identifies bytes in the uncompressed chunk that can be treated as having zero values, i.e. that comprise all zeros.).
Further regarding the claimed invention, Corkery particularly teaches generating an encoding that identifies locations of the first partitions among remaining second partitions in the plurality of partitions (see e.g. paragraphs 0010-0012 and 0036-0037: as noted above, Corkery discloses that the DNN module generates a compressed output chunk that comprises a mask portion that includes a number of bits equal to the number of bytes in the uncompressed chunk; each bit in the mask portion corresponds to a byte in the uncompressed chunk, with the bit being set to zero if the corresponding byte is zero and set to one if the corresponding byte is non-zero. The mask portion is considered an encoding that identifies the locations of the first partitions, i.e. the locations of bytes that are zero, among second partitions, i.e. among bytes that are non-zero.). Corkery further suggests that the encoding is generated by a partitioning circuit located downstream of an output buffer of the layer and upstream of an on-chip memory (see e.g. paragraphs 0009-0010: like noted above, Corkery discloses that the DNN module comprises a compression unit that receives an uncompressed chunk of data generated by a layer of the neural network, and then generates the compressed output chunk from the received uncompressed chunk. In particular, Corkery discloses that the compression unit receives the uncompressed chunk of output data from one or more neurons on the DNN module, and after compressing the chunk, provides the compressed chunk to an on-board BaSRAM memory – see e.g. paragraphs 0059-0060, 0067-0069 and 0075. The compression unit is thus considered a partitioning circuit like claimed; the compression unit generates the encoding, i.e. the mask portion of the compressed chunk, and is located downstream of an output buffer of the layer, i.e. downstream of an output buffer necessary for the neurons implementing the layer, and upstream of an on-chip memory, e.g. upstream of the BaSRAM.).
It would have been obvious to one of ordinary skill in the art, having the teachings of Shih and Corkery before the effective filing date of the claimed invention, to modify the method taught by Shih so as to generate an encoding that identifies locations of the first partitions among remaining second partitions in the plurality of partitions, wherein the encoding is generated by a partitioning circuit located downstream of an output buffer of the layer and upstream of an on-chip memory, like done by Corkery. It would have been advantageous to one of ordinary skill to utilize such an encoding because it provides a compressed version of activation data that reduces utilization of memory bus bandwidth, as is taught by Corkery (see e.g. paragraphs 0008-0010). It would have particularly been obvious to utilize a compression circuit located downstream of an output buffer of the layer and upstream of an on-chip memory, because this would enable compressed activation data to be stored in memory (and thus require less memory and memory bandwidth than if uncompressed activation data was stored in memory), as is further evident from Corkery (see e.g. paragraphs 0059-0060, 0067-0069 and 0075).
Similar to Shih and Corkery, Zhou teaches receiving outputs (i.e. a feature map) from a layer of a neural network, compressing the outputs, and storing the compressed outputs (see e.g. paragraphs 0005, 0038-0048 and 0068-0069). Regarding the claimed invention, Zhou also particularly teaches sending the compressed outputs to a subsequent layer in the neural network (see e.g. paragraphs 0043-0044 and 0069-0071).
It would have been obvious to one of ordinary skill in the art, having the teachings of Shih, Corkery and Zhou before the effective filing date of the claimed invention, to modify the method taught by Shih and Corkery such that the compressed output data (i.e. the encoding and the second partitions) is sent to a subsequent layer in the neural network like taught by Zhou. It would have been advantageous to one of ordinary skill to utilize such a combination because, like taught by Zhou, it would enable the subsequent layer to use the output data, as is necessary in neural networks (see e.g. paragraphs 0020-0021 and 0028-0032).
Gunnam generally teaches, after compressing a feature map, dividing the feature map into subsets (i.e. sub-feature maps) and sending each subset to a corresponding execution unit (i.e. a sparse tensor compute unit) for a subsequent layer of a neural network (see e.g. column 38, lines 43-59; and column 39, line 18 – column 40, line 25).
It would have been obvious to one of ordinary skill in the art, having the teachings of Shih, Corkery, Zhou and Gunnam before the effective filing date of the claimed invention, to modify the method taught by Shih, Corkery and Zhou such that the compressed feature map (i.e. the encoding and second partitions) is divided into subsets (i.e. subsets of partitions), and wherein each subset (i.e. the encoding and a subset of partitions) is sent to a corresponding execution unit in the subsequent layer of the neural network like taught by Gunnam. It would have been advantageous to one of ordinary skill to utilize such a combination because it would enable each of the subsets to be processed in parallel, as is taught by Gunnam (see e.g. column 1, lines 10-34; and column 39, lines 32-36). Accordingly, Shih, Corkery, Zhou and Gunnam are considered to teach, to one of ordinary skill in the art, a method like that of claim 1 for inducing sparsity for outputs of a neural network layer.
As per claim 2, it would have been obvious, as is described above, to modify the method taught by Shih so as to generate an encoding like taught by Corkery that identifies locations of the first partitions (i.e. the unimportant/zero blocks) among remaining second partitions, and to send such compressed output data (i.e. the encoding and second partitions) to a subsequent layer in the neural network like taught by Zhou. Zhou particularly teaches receiving the compressed output data (i.e. a compressed feature map) at the subsequent layer in the neural network and decompressing the compressed output data (see e.g. paragraphs 0043-0045 and 0069-0070). Corkery teaches that the compressed output data comprises an encoding (i.e. in a mask portion) and the second partitions (i.e. in a data portion), and that decompressing the compressed output data comprises arranging the second partitions based on the encoding (see e.g. paragraphs 0016 and 0105-0109). As further described above, it would have been obvious to modify the method taught by Shih, Corkery and Zhou such that the second partitions are divided into subsets of partitions, and wherein each subset is sent to a corresponding execution unit in the subsequent layer of the neural network like taught by Gunnam. Accordingly, with the combination of Shih, Corkery and Zhou, it follows the compressed output data (i.e. subsets of the second partitions and the encoding) is received at the subsequent layer in the neural network, whereat the compressed output data is decompressed in part by arranging subsets of the second partitions based on the encoding. Shih, Corkery, Zhou and Gunnam thus further teach a method like that of claim 2.
As per claim 5, Shih suggests that the outputs (i.e. activation maps) from a layer of a neural network can comprise a three-dimensional (i.e. C × W × H) array of outputs, wherein the array of outputs comprises a dimension (i.e. C) for different channels in the neural network (see e.g. Section II.C “Design Parameters Analysis of Zebra”). The plurality of partitions (i.e. blocks) of such an array of outputs are thus considered three-dimensional partitions. Accordingly, the above-described combination of Shih, Corkery, Zhou and Gunnam further teaches a method like that of claim 5.
As per claim 6, Shih teaches that the first partitions (i.e. the unimportant/zero blocks) need not be contiguous in the plurality of partitions (see e.g. Section II.A “Model Training:” it is apparent that unimportant blocks, i.e. blocks having a maximum value less than a threshold and which are thus zeroed, need not be contiguous). Corkery similarly teaches that first partitions (i.e. bytes containing all zeros) need not be contiguous in the plurality of partitions (see e.g. paragraphs 0077-0078 and FIG. 3). Accordingly, the above-described combination of Shih, Corkery, Zhou and Gunnam further teaches a method like that of claim 6.
As per claim 7, Shih further teaches that identifying the first partitions in the plurality of partitions that can be treated as having zero values comprises receiving a criterion from a design environment, and applying the criterion to each of the plurality of partitions (see e.g. Section II.A “Model Training:” like noted above, Shih teaches that if the maximum value of a block is less than a threshold, the block is regarded as an unimportant block and all values therein are zeroed. Shih particularly discloses that the threshold is user-defined – see e.g. Section II.A “Model Training” and Section II.B “Model Inference.” The environment in which the user defines the threshold can be considered a design environment, and therefore, Shih teaches receiving a criterion, e.g. the threshold, from a design environment and applying the criterion to each of the plurality of partitions, i.e. blocks.). Accordingly, the above-described combination of Shih, Corkery, Zhou and Gunnam further teaches a method like that of claim 7.
As per claim 9, Shih further teaches that the criterion is sent as a runtime function from the design environment (see e.g. Section II.A “Model Training” and Section II.B “Model Inference:” like noted above, Shih teaches that if the maximum value of a block is less than a user-defined threshold, the block is regarded as an unimportant block and all values therein are zeroed. The threshold is applied during runtime, e.g. during model training and/or inference, and is thus considered a runtime function sent from the design environment.). Accordingly, the above-described combination of Shih, Corkery, Zhou and Gunnam further teaches a method like that of claim 9.
As per claim 10, Shih further teaches that the criterion is encoded as part of a graph representing the neural network (see e.g. Section II.A “Model Training” and Section II.B “Model Inference:” like noted above, Shih teaches that if the maximum value of a block is less than a user-defined threshold, the block is regarded as an unimportant block and all values therein are zeroed. The threshold is applied during runtime to blocks of the activation maps produced by the neural network, e.g. during neural network model training and/or inference – see e.g. Section II.A “Model Training” and Section II.B “Model Inference.” Consequently, the threshold can be considered a criterion encoded as a part of a graph representing the neural network.). Accordingly, the above-described combination of Shih, Corkery, Zhou and Gunnam further teaches a method like that of claim 10.
As per claim 21, Shih further suggests that defining the sizes of each of the plurality of partitions prior to receiving the outputs comprises processing sample data using the execution unit of the layer of a neural network, and defining sizes of a plurality of partitions based on results of processing sample data using the execution unit (Like noted above, Shi teaches that the sizes of each of the blocks is manually chosen based on various factors including activation map sparsity and location within the neural network:
In Zebra, one important parameter is the block size, which depends on the size of activation maps and datasets. Theoretically, small block size will have higher accuracy and higher sparsity since small block means more fine-grained partitioned. However, small block size will need more index parameters to store the zero conditions of the blocks which leads to memory bandwidth overhead.
(Section II.C “Design Parameters Analysis of Zebra”).
The proposed method is evaluated on two image classification datasets, CIFAR-10 [12] and Tiny-Imagenet that is the subset of Imagenet [13]. CIFAR-10 consists of 10 classes of natural images with resolution 32×32. Empirically, we choose block size as 4 by considering both activation maps sparsity and the zero blocks index overhead. Note that we set block size as 2 when the size of activation maps in deeper layers goes to 2 × 2 in VGG and MobileNets. Tiny-Imagenet contains 200 classes with resolution 64 × 64. We set block size to 8 for better activation maps reduction and the index storage overhead trade-off.
(Section III.A “Settings”; emphasis added).
The activation map sparsity is understandably determined by executing the neural network, i.e. by processing sample data. Accordingly, Shih teaches processing sample data using the execution unit of the layer of a neural network, and defining sizes of a plurality of partitions, i.e. blocks, based on results, e.g. the determined sparsity, of processing sample data using the execution unit.). Accordingly, the above-described combination of Shih, Corkery, Zhou and Gunnam is further considered to teach a method like that of claim 21.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Shih, Corkery, Zhou and Gunnam, which is described above, and also over U.S. Patent Application Publication No. 2022/0309124 to Mei et al. (“Mei”).
As described above, Shih, Corkery, Zhou and Gunnam teach a method like that of claim 2, which includes receiving outputs from a layer of a neural network, identifying first partitions of the output that can be treated as having zero values, dividing remaining second partitions into subsets of partitions, and sending an encoding and each of the subsets of partitions to corresponding execution units in a subsequent layer in the neural network. Shih suggests that a subsequent layer can perform a multiplication operation (e.g. as required by convolution; see e.g. Section II.C. “Design Parameters Analysis of Zebra”), as does Corkery (see e.g. paragraph 0049) and Zhou (see e.g. paragraph 0030). Shih, Corkery, Zhou and Gunnam however do not teach or suggest that when one of the corresponding execution units in the subsequent layer performs a multiplication operation, the first partitions can be discarded as a multiply-by-zero operation, as is required by claim 3.
Mei nevertheless generally teaches performing a multiplication operation, e.g. at a layer of a neural network, whereby zero value elements are discarded as a multiply-by-zero operation (see e.g. paragraphs 0001, 0064, 0135 and 0198-0199).
It would have been obvious to one of ordinary skill in the art, having the teachings of Shih, Corkery, Zhou, Gunnam and Mei before the effective filing date of the claimed invention, to modify the method taught by Shih, Corkery, Zhou and Gunnam such that when one of the execution units in the subsequent layer performs the multiplication operation, the zero value elements (i.e. the first partitions) are discarded as a multiply-by-zero operation, as is taught by Mei. It would have been advantageous to one of ordinary skill to utilize such a combination because it can save power consumption and increase computer throughput, as is taught by Mei (see e.g. paragraphs 0001 and 0198-0199). Accordingly, Shih, Corkery, Zhou, Gunnam and Mei are considered to teach, to one of ordinary skill in the art, a method like that of claim 3.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Shih, Corkery, Zhou and Gunnam, which is described above, and also U.S. Patent Application Publication No. 2021/0125071 to Ren et al. (“Ren”).
Regarding claim 8, Shih, Corkery, Zhou and Gunnam teach a method like that of claim 7, as is described above, which comprises receiving a criterion from a design environment and applying the criterion to each of a plurality of partitions. Shih, Corkery, Zhou and Gunnam however do not teach that the criterion comprises a relative magnitude function that calculates an aggregate for the values in a partition and sets the values in the partition to zero if the aggregate is less than a threshold, as is required by claim 8.
Ren generally describes a block-wise row and column pruning (BRCP) algorithm that can be applied to an input weight pattern, whereby the input weight patterns is partitioned into a number of partitions (e.g. sub-columns, sub-rows) and particular partitions are identified as partitions that can be treated as having zero values (i.e. that can be pruned/set as zeros) (see e.g. paragraphs 0004, 0020 and 0052). Ren suggests that identifying the partitions that can be treated as having zero values comprises receiving a criterion from a design environment and applying the criterion to each of the plurality of partitions (see e.g. paragraphs 0050-0051: Ren mentions that the BRCP algorithm is a “designated” algorithm, and so it is apparent is selected or implemented via a design environment. Ren also suggests that various characteristics of the BRCP algorithm such as sub-column and sub-row thresholds are based on criteria, e.g. a predetermined desired sparsity, that would necessarily be received from the design environment – see e.g. paragraphs 0054-0056, 0070 and 0094). Like claimed, Ren particularly teaches that the criterion can comprise a relative magnitude function that calculates an aggregate (i.e. a norm) for the values in a partition and sets the values in the partition to zero (i.e. prunes a sub-column and/or sub-row) if the aggregate is less than a threshold (see e.g. paragraphs 0054-0056 and 0058).
It would have been obvious to one of ordinary skill in the art, having the teachings of Shih, Corkery, Zhou, Gunnam and Ren before the effective filing date of the claimed invention, to modify the method taught by Shih, Corkery, Zhou and Gunnam so as to employ a criterion like taught by Ren to identify first partitions in the plurality of partitions that can be treated as having zero values, wherein the criterion comprises a relative magnitude function that calculates an aggregate for the values in a partition and sets the values in the partition to zero if the aggregate is less than a threshold. It would have been advantageous to one of ordinary skill to utilize such a combination because it would enable the identification of relatively unimportant partitions, as is evident from Ren (see e.g. paragraphs 0018-0019, 0054-0056 and 0058). Accordingly, Shih, Corkery, Zhou, Gunnam and Ren are considered to teach, to one of ordinary skill in the art, a method like that of claim 8.
Claims 11 and 13-19 are rejected under 35 U.S.C. 103 as being unpatentable over the article entitled, “Zebra: Memory Bandwidth Reduction for CNN Accelerators With Zero Block Regularization of Activation Maps” by Shih et al. (“Shih”), over U.S. Patent Application Publication No. 2018/0300606 to Corkery et al. (“Corkery”), over U.S. Patent Application Publication No. 2021/0266565 to Zhou et al. (“Zhou”), over U.S. Patent No. 12,518,165 to Xiao et al. (“Xiao”), and also over U.S. Patent Application Publication No. 2022/0101118 to Yan et al. (“Yan”).
Regarding claim 11, Shih describes a “dynamic activation map reduction method to reduce memory bandwidth” (section I “Introduction”). Like claimed, Shih particularly teaches:
implementing a layer of a neural network and generating outputs from the layer (Shih discloses that the dynamic activation map reduction method comprises dividing an activation map into a plurality of non-overlapping blocks, and forcing all values in “unimportant blocks” to zero:
With above observations, we propose a dynamic activation map reduction method to reduce memory bandwidth. Inspired by [11], we prune blocks of zeros in the maps instead of whole maps in [11] or fine grained case as introduced by ReLUs. If all values in one block are all zero, we consider it as background or unimportant one and thus we can prune it in run-time. However, the zero block percentage after ReLUs is still quite low as shown in Table. I. To force the network learns more ”zero blocks”, we propose a regularization method called Zero Block regularization of activation maps (Zebra) to achieve dynamic activation map reduction. Our method divides the activation maps into several non-overlapping blocks, then lets the model to dynamically learn unimportant blocks through Zebra, and simply forces all values in those blocks to zero to achieve simple run-time structured activation maps pruning. The inference has very low overhead and achieves 70% of memory bandwidth reduction within 1% of accuracy loss.
(Section I “Introduction”; emphasis added).
The activation map understandably comprises outputs of a neural network layer, e.g. a convolutional neural network layer, as is known in the art:
In [11], the authors analyze the sparsity of activation maps in different layers. The result shows that the deeper activation maps have higher zero ratio.
(Section 1 “Introduction”; emphasis added).
As to the memory bandwidth overhead of Zebra, the results are shown in Table. V. The required bandwidth means the total activation maps size for one image and the bandwidth overhead is the need of block index storage. In this calculation, we assume a layer-by-layer hardware processing that will store the activation maps to external DRAM for each convoluitonal layer processing. This overhead is totally negligible compared with the amount of reduction in activation maps.
(Section III.B “Results”).
Accordingly, Shih teaches implementing a layer of a neural network and generating outputs, i.e. an activation map, from the layer.);
receiving outputs from the layer of a neural network (see e.g. Section I “Introduction:” like noted above, Shih discloses that the dynamic activation map reduction method comprises dividing an activation map into a plurality of non-overlapping blocks, and forcing all values in “unimportant blocks” to zero. Like further noted above, the activation map understandably comprises outputs of a neural network layer, e.g. of a convolutional neural network layer – see e.g. Section 1 “Introduction” and Section III.B “Results.” Accordingly, Shih’s dynamic activation map reduction method necessitates receiving outputs, e.g. an activation map, from a layer of neural network so as to divide the activation map into blocks and force values in unimportant blocks to zero.);
partitioning the outputs into a plurality of partitions, wherein sizes of each of the plurality of partitions are defined prior to receiving the outputs and based on a runtime selection of a partition size and a runtime selection of a criterion (As noted above, Shih discloses that the dynamic activation map reduction method comprises dividing an activation map into a plurality of non-overlapping blocks. Shi further teaches that the sizes of each of the blocks is manually chosen based on various factors including activation map sparsity and location within the neural network:
In Zebra, one important parameter is the block size, which depends on the size of activation maps and datasets. Theoretically, small block size will have higher accuracy and higher sparsity since small block means more fine-grained partitioned. However, small block size will need more index parameters to store the zero conditions of the blocks which leads to memory bandwidth overhead.
(Section II.C “Design Parameters Analysis of Zebra”).
The proposed method is evaluated on two image classification datasets, CIFAR-10 [12] and Tiny-Imagenet that is the subset of Imagenet [13]. CIFAR-10 consists of 10 classes of natural images with resolution 32×32. Empirically, we choose block size as 4 by considering both activation maps sparsity and the zero blocks index overhead. Note that we set block size as 2 when the size of activation maps in deeper layers goes to 2 × 2 in VGG and MobileNets. Tiny-Imagenet contains 200 classes with resolution 64 × 64. We set block size to 8 for better activation maps reduction and the index storage overhead trade-off.
(Section III.A “Settings”; emphasis added).
As the division of the activation maps into such manually-sized blocks occurs during both training and inference – see e.g. Section II.A “Model Training” and Section II.B “Model Inference” – it is apparent that the sizes of each of the plurality of blocks are defined prior to receiving the activation maps during the training and inference. Shih also discloses that if the maximum value of a block is less than a threshold, the block is regarded as an unimportant block, and all values therein are zeroed:
Fig. 1 shows the flow of Zebra to train zero blocks. Activation maps are first divided into non-overlapped blocks. Zebra learns zero blocks or unimportant blocks through regularization. The importance of a block is represented by the maximum value in each block. If the maximum value of a block is smaller than a learned threshold, this block will be regarded as a zero block. The blocks in the same channel use the same threshold. The threshold is learned by feeding activation maps to a small network with a global average pooling layer and a fully-connected layer.
(Section II.A “Model Training”).
After training, we examine the threshold values obtained by Zebra regularization. To our surprise, the learned threshold values are almost converged to the given Tobj as shown in Fig. 2. Therefore, during inference we can remove the small network and just use the user-defined Tobj as the threshold in run-time as shown in Fig. 2.
(Section II. B “Model Inference”).
The threshold can be considered a “criterion” like claimed. Shih discloses that the unimportant blocks are pruned – see e.g. section I “Introduction,” which recites “[i]f all values in one block are all zero, we consider it as a background or unimportant one and thus we can prune it in run-time.” Accordingly, Shih teaches partitioning the outputs, i.e. activation map, into a plurality of partitions, i.e. blocks, wherein sizes of each of the plurality of partitions are defined prior to receiving the outputs and based on a selection of a partition size and a selection of a criterion. As the partition size and threshold are applied during runtime, they can be considered “runtime” selections like claimed.); and
identifying first partitions in the plurality of partitions that can be treated as having zero values by applying the runtime selection of the criterion (As noted above, Shih discloses that the dynamic activation map reduction method comprises dividing an activation map into a plurality of non-overlapping blocks, and forcing all values in unimportant blocks to zero. As further described above, Shih discloses that if the maximum value of a block is less than a threshold, the block is regarded as an unimportant block, and all values therein are zeroed:
Fig. 1 shows the flow of Zebra to train zero blocks. Activation maps are first divided into non-overlapped blocks. Zebra learns zero blocks or unimportant blocks through regularization. The importance of a block is represented by the maximum value in each block. If the maximum value of a block is smaller than a learned threshold, this block will be regarded as a zero block. The blocks in the same channel use the same threshold. The threshold is learned by feeding activation maps to a small network with a global average pooling layer and a fully-connected layer.
(Section II.A “Model Training”).
After training, we examine the threshold values obtained by Zebra regularization. To our surprise, the learned threshold values are almost converged to the given Tobj as shown in Fig. 2. Therefore, during inference we can remove the small network and just use the user-defined Tobj as the threshold in run-time as shown in Fig. 2.
(Section II. B “Model Inference”).
Accordingly, Shih teaches identifying first partitions, i.e. unimportant blocks, in the plurality of partitions that can be treated as having zero values by applying the runtime selection of the criterion, i.e. the threshold.).
Shih teaches that the above-described tasks can be implemented on a neural network accelerator (see e.g. Section II.C “Design Parameters Analysis of Zebra,” which states “[t]he required hardware implementation is also small enough to be easily integrated with current accelerators after activation functions,” and Section IV “Conclusion,” which states “[i]n this paper, we propose a regularization method, Zebra, to help reduce the large memory bandwidth in modern CNN accelerators.”). The portion of the neural network accelerator circuitry necessary for implementing a layer of a neural network and generating outputs from the layer like taught by Shih as noted above is considered a “compute node” like claimed. The portion of the neural network accelerator circuitry necessary for receiving the outputs, partitioning the outputs, and identifying the first partitions like taught by Shih as noted above is considered a “partitioning circuit” like claimed. Accordingly, Shih teaches a neural network accelerator similar to that of claim 11. Furthermore, Shi mentions that “index parameters…store the zero conditions of the blocks” (Section II.C “Design Parameters Analysis of Zebra”). Shih, however, does not disclose that the neural network accelerator partitioning circuit generates an encoding that identifies locations of the first partitions (i.e. the unimportant/zero blocks) among remaining second partitions in the plurality of partitions, wherein a memory of the neural network accelerator stores the encoding and the second partitions for a subsequent layer in the neural network, and wherein the partitioning circuit is configured to overwrite values in the first partitions with zero values in an output buffer prior to storage in the memory, as is required by claim 11. Moreover, Shih does not explicitly disclose that the runtime selection of the partition size and the runtime selection of the criterion are specific to an execution unit of the layer of the neural network, as is further required by claim 11.
Corkery describes a deep neural network (DNN) module that can compress and decompress activation data to reduce the utilization of memory bus bandwidth (see e.g. paragraph 0008). Similar to Shih, Corkery particularly teaches:
receiving outputs from a layer of a neural network (see e.g. paragraph 0009: Corkery discloses that the DNN module comprises a compression unit that receives an uncompressed chunk of data generated by one or more of the neurons of the neural network. The DNN module thus receives outputs, i.e. an uncompressed chunk of data, from a layer of the neural network, i.e. from one or more of the neurons of the neural network.);
partitioning the outputs into a plurality of partitions (see e.g. paragraph 0009: Corkery discloses that the uncompressed chunk of data includes a fixed number of bytes, e.g. 64 bytes. Each of the bytes can be considered a partition of the chunk.); and
identifying first partitions in the plurality of partitions that can be treated as having zero values (see e.g. paragraphs 0010-0011 and 0036-0037: Corkery discloses that the DNN module generates a compressed output chunk from the received uncompressed chunk, wherein the compressed output chunk comprises a mask portion that includes a number of bits equal to the number of bytes in the uncompressed chunk and wherein each bit corresponds to a particular byte in the uncompressed chunk. Corkery further discloses that the DNN module sets to zero each bit in the mask portion that has a corresponding byte in the uncompressed chunk containing all zeros, and sets to one each bit in the mask portion that has a corresponding byte in the uncompressed chunk that is non-zero – see e.g. paragraphs 0012 and 0037. Thus to generate the mask portion of the compressed chunk, the DNN module necessarily identifies bytes in the uncompressed chunk that can be treated as having zero values, i.e. that comprise all zeros.).
Further regarding the claimed invention, Corkery also teaches generating an encoding that identifies locations of the first partitions among remaining second partitions in the plurality of partitions (see e.g. paragraphs 0010-0012 and 0036-0037: as noted above, Corkery discloses that the DNN module generates a compressed output chunk that comprises a mask portion that includes a number of bits equal to the number of bytes in the uncompressed chunk; each bit in the mask portion corresponds to a byte in the uncompressed chunk, with the bit being set to zero if the corresponding byte is zero and set to one if the corresponding byte is non-zero. The mask portion is considered an encoding that identifies the locations of the first partitions, i.e. the locations of bytes that are zero, among second partitions, i.e. among bytes that are non-zero.).
It would have been obvious to one of ordinary skill in the art, having the teachings of Shih and Corkery before the effective filing date of the claimed invention, to modify the partitioning circuit taught by Shih so as to generate an encoding that identifies locations of the first partitions among remaining second partitions in the plurality of partitions, like done by Corkery. It would have been advantageous to one of ordinary skill to utilize such an encoding because it provides a compressed version of activation data that reduces utilization of memory bus bandwidth, as is taught by Corkery (see e.g. paragraphs 0008-0010).
Similar to Shih and Corkery, Zhou teaches receiving outputs (i.e. a feature map) from a layer of a neural network and compressing the outputs (see e.g. paragraphs 0005, 0038-0048 and 0068-0069). Regarding the claimed invention, Zhou also particularly teaches storing the compressed outputs in on-chip memory for a subsequent layer in the neural network (see e.g. paragraphs 0043-0044, 0046, 0048, 0069-0071 and 0075-0076).
It would have been obvious to one of ordinary skill in the art, having the teachings of Shih, Corkery and Zhou before the effective filing date of the claimed invention, to modify the neural network accelerator taught by Shih and Corkery so as to comprise memory to store the compressed output data (i.e. the encoding and the second partitions) for a subsequent layer in the neural network like taught by Zhou. It would have been advantageous to one of ordinary skill to utilize such a combination because, like taught by Zhou, it would enable the subsequent layer to use the output data, as is necessary in neural networks (see e.g. paragraphs 0020-0021 and 0028-0032).
Xiao generally teaches receiving outputs from a layer of a neural network, partitioning the outputs into a plurality of partitions (i.e. banks), and identifying outputs (i.e. outputs that are not within the top-K output values) within the partitions that can be treated as having zero values (see e.g. column 1, line 47-62; and column 9, lines 2-46). Regarding the claimed invention, Xiao suggests overwriting values in the partitions with zero values in an output buffer prior to providing the output values to a subsequent layer in the neural network (see e.g. column 1, line 47-62; and column 9, lines 16-46; column 10, line 60 – column 11, line 11; and column 11, lines 19-36).
It would have been obvious to one of ordinary skill in the art, having the teachings of Shih, Corkery, Zhou and Xiao before the effective filing date of the claimed invention, to modify the partitioning circuit taught by Shih, Corkery and Zhou so as to overwrite the output values that are to be treated as zero (i.e. the first partitions) with zero values in an output buffer prior to providing the output values to the subsequent layer in the neural network, as is done by Xiao. It would have been advantageous to one of ordinary skill to utilize such a combination because it would enable the zero values (i.e. pruned values) to be readily identified, as is evident from Xiao (see e.g. column 1, line 47-62; and column 9, lines 16-46; column 10, line 60 – column 11, line 11; and column 11, lines 19-36). As noted above, it would have been obvious to modify the neural network accelerator taught by Shih and Corkery so as to comprise memory like taught by Zhou to store the output data for access by the subsequent layer in the neural network. It thus follows that the partitioning circuit would overwrite the values in the first partitions with zero values in the output buffer prior to storage in the memory for the subsequent layer in the neural network.
Similar to Shih, Yan teaches receiving outputs (i.e. activation feature maps) from a layer of a neural network, partitioning the outputs into a plurality of partitions (i.e. banks), wherein sizes of each of the plurality of partitions are defined prior to receiving the outputs based on a runtime selection of a partition size, and wherein values in the plurality of partitions are treated as zeros based on a runtime selection of a criterion (e.g. a sparsity rate or threshold value) (see e.g. paragraphs 0025, 0028-0030 and 0038-0040). Yan further teaches that the partition size and criterion can be specific to an execution unit of a layer of the neural network (see e.g. paragraphs 0029-0030 and 0048: Yan discloses that the bank size and target sparsity can be selected based on hyperparameters/types/dimensions of tensors of the neural network, and can be individually specified for each layer of the neural network. The bank size and target sparsity can thus be considered to be specific to an execution unit of the layer of the neural network.).
It would have been obvious to one of ordinary skill in the art, having the teachings of Shih, Corkery, Zhou, Xiao and Yan before the effective filing date of the claimed invention, to modify the neural network accelerator taught by Shih, Corkery, Zhou and Xiao such that the partition size and criterion are specific to an execution unit of the layer of the neural network like taught by Yan. It would have been advantageous to one of ordinary skill to utilize such a combination because it can provide for more optimal execution of the neural network, as is evident from Yan (see e.g. paragraphs 0029-0030 and 0048). Accordingly, Shih, Corkery, Zhou, Xiao and Yan are considered to teach, to one of ordinary skill in the art, a neural network accelerator like that of claim 11.
As per claim 13, it would have been obvious, as is described above, to modify the neural network accelerator taught by Shih so as to generate an encoding like taught by Corkery that identifies locations of the first partitions (i.e. the unimportant/zero blocks) among remaining second partitions, and to store in memory such compressed output data (i.e. the encoding and second partitions) for a subsequent layer in the neural network like taught by Zhou. Zhou particularly teaches receiving the compressed output data (i.e. a compressed feature map) at the subsequent layer in the neural network and decompressing the compressed output data (see e.g. paragraphs 0043-0045 and 0069-0070). Corkery teaches that the compressed output data comprises an encoding (i.e. in a mask portion) and the second partitions (i.e. in a data portion), and that decompressing the compressed output data comprises arranging the second partitions based on the encoding (see e.g. paragraphs 0016 and 0105-0109). Accordingly, with the combination of Shih, Corkery, Zhou, Xiao and Yan, it follows the compressed output data (i.e. the second partitions and the encoding) is received at the subsequent layer in the neural network, whereat the compressed output data is decompressed in part by arranging the second partitions based on the encoding. The neural network accelerator circuitry for necessary for performing such operations can be considered a “sequencer circuit” like claimed. Shih, Corkery, Zhou, Xiao and Yan thus further teach a neural network accelerator like that of claim 13.
As per claim 14, Shih suggests that the layer of the neural network comprises executing a convolution core (see e.g. Section II.C “Design Parameters Analysis of Zebra”). Corkery provides a similar teaching (see e.g. paragraphs 0052-0053), as does Zhou (see e.g. paragraphs 0026-0032), Yan (see e.g. paragraph 0027) and Xiao (see e.g. column 1, lines 47-62). Accordingly, the above-described combination of Shih, Corkery, Zhou, Xiao and Yan is further considered to teach a neural network accelerator like that of claim 14.
As per claim 15, it would have been obvious, as is described above, to modify the neural network accelerator taught by Shih and Corkery so as to comprise memory like taught by Zhou to store the compressed output data (i.e. the encoding and the second partitions). Zhou particularly teaches that the memory can comprise an on-chip static random-access memory (SRAM) (see e.g. paragraphs 0038, 0046, 0048 and 0075-0076). Corkery provides a similar teaching (see e.g. paragraphs 0058-0059 and 0069). Accordingly, the above-described combination of Shih, Corkery, Zhou, Xiao and Yan is further considered to teach a neural network accelerator like that of claim 15.
As per claim 16, Shih does not explicitly disclose that the partitioning circuit is not used when training the neural network as is claimed. Corkery nevertheless suggests a partitioning circuit that is not used when training a neural network (see e.g. paragraphs 0046-0047: Corkery discloses that training for a neural network implemented by a DNN module can be performed offline, e.g. in a server farm or data center. Accordingly, as the training is performed away from the DNN module, it is apparent that a partitioning circuit, i.e. compression unit, of the DNN module would not be used when training the neural network). It would have been obvious to one of ordinary skill in the art, having the teachings of Shih, Corkery, Zhou, Xiao and Yan before the effective filing date of the claimed invention, to further modify the neural network accelerator taught by Shih, Corkery, Zhou, Xiao and Yan so that the partitioning circuit is not used when training the neural network like taught by Corkery. It would have been advantageous to one of ordinary skill to utilize such a combination because it would enable the training of the neural network to be performed remotely in a suitable computing environment, as is evident from Corkery (see e.g. paragraphs 0046-0047). Accordingly, Shih, Corkery, Zhou, Xiao and Yan are further considered to teach, to one of ordinary skill in the art, a neural network accelerator like that of claim 16.
As per claim 17, Shih teaches that a number of partitions in the plurality of partitions can be determined during training of the neural network (see e.g. Section II.A “Model Training”). Accordingly, the above-described combination of Shih, Corkery, Zhou, Xiao and Yan is further considered to teach a neural network accelerator like that of claim 17.
As per claim 18, Shih further teaches that identifying the first partitions in the plurality of partitions that can be treated as having zero values comprises receiving a criterion from a design environment, and applying the criterion to each of the plurality of partitions (see e.g. Section II.A “Model Training:” like noted above, Shih teaches that if the maximum value of a block is less than a threshold, the block is regarded as an unimportant block and all values therein are zeroed. Shih particularly discloses that the threshold is user-defined – see e.g. Section II.A “Model Training” and Section II.B “Model Inference.” The environment in which the user defines the threshold can be considered a design environment, and therefore, Shih teaches receiving a criterion, e.g. the threshold, from a design environment and applying the criterion to each of the plurality of partitions, i.e. blocks.). Accordingly, the above-described combination of Shih, Corkery, Zhou, Xiao and Yan further teaches a neural network accelerator like that of claim 18.
As per claim 19, Shih suggests that the outputs (i.e. activation maps) from a layer of a neural network can comprise a three-dimensional (i.e. C × W × H) array of outputs, wherein the array of outputs comprises a dimension (i.e. C) for different channels in the neural network (see e.g. Section II.C “Design Parameters Analysis of Zebra”). The plurality of partitions (i.e. blocks) of such an array of outputs are thus considered three-dimensional partitions. Accordingly, the above-described combination of Shih, Corkery, Zhou, Xiao and Yan further teaches a neural network accelerator like that of claim 19.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Shih, Corkery, Zhou, Xiao and Yan, which is described above, and also over the article entitled, “Simba: Scaling Deep-Learning Inference with Multi-Chip-Module-Based Architecture” by Shao et al. (“Shao”).
Regarding claim 12, Shih, Corkery, Zhou, Xiao and Yan teach a neural network accelerator like in claim 11, as is described above, which comprises a compute node configured to implement a layer of a neural network and a memory configured to store an encoding and second partitions for a subsequent layer in the neural network. Shih, Corkery, Zhou, Xiao and Yan, however, do not explicitly teach that the network accelerator comprises a plurality of chiplets, wherein the compute node is implemented on a first chiplet in the plurality of chiplets, and wherein the subsequent layer is implemented on a second chiplet in the plurality of chiplets, as is required by claim 12.
Shao nevertheless teaches implementing a deep neural network on multi-chip-modules comprising a plurality of chiplets (see e.g. the “Abstract” and section 1 “Introduction”). Regarding the claimed invention, Shao particularly teaches executing the neural network via cross-layer pipelining, wherein separate layers of the neural network are implemented on different chiplets (see e.g. section 5.3 “Cross-Layer Pipelining”).
It would have been obvious to one of ordinary skill in the art, having the teachings of Shih, Corkery, Zhou, Xiao, Yan and Shao before the effective filing date of the claimed invention, to modify the network accelerator taught by Shih, Corkery, Zhou, Xiao and Yan such that it comprises a plurality of chiplets like taught by Shao, wherein different layers are implemented on different chiplets (i.e. the compute node associated with a first layer is implemented on a first chiplet of the plurality of chiplets and the subsequent layer is implemented on a second chiplet of the plurality of chiplets). It would have been advantageous to one of ordinary skill to utilize such a combination because it can provide increased throughput, as is suggested by Shao (see e.g. section 1 “Introduction” and section 5.3 “Cross-Layer Pipelining”). Accordingly, Shih, Corkery, Zhou, Xiao, Yan and Shao are considered to teach, to one of ordinary skill in the art, a network accelerator like that of claim 12.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over the article entitled, “Zebra: Memory Bandwidth Reduction for CNN Accelerators With Zero Block Regularization of Activation Maps” by Shih et al. (“Shih”), over U.S. Patent Application Publication No. 2018/0300606 to Corkery et al. (“Corkery”), over U.S. Patent Application Publication No. 2021/0266565 to Zhou et al. (“Zhou”), and also over U.S. Patent Application Publication No. 2022/0101118 to Yan et al. (“Yan”).
Regarding claim 20, Shih describes a “dynamic activation map reduction method to reduce memory bandwidth” (section I “Introduction”). Like claimed, Shih particularly teaches:
receiving outputs from a layer of a neural network (Shih discloses that the dynamic activation map reduction method comprises dividing an activation map into a plurality of non-overlapping blocks, and forcing all values in “unimportant blocks” to zero:
With above observations, we propose a dynamic activation map reduction method to reduce memory bandwidth. Inspired by [11], we prune blocks of zeros in the maps instead of whole maps in [11] or fine grained case as introduced by ReLUs. If all values in one block are all zero, we consider it as background or unimportant one and thus we can prune it in run-time. However, the zero block percentage after ReLUs is still quite low as shown in Table. I. To force the network learns more ”zero blocks”, we propose a regularization method called Zero Block regularization of activation maps (Zebra) to achieve dynamic activation map reduction. Our method divides the activation maps into several non-overlapping blocks, then lets the model to dynamically learn unimportant blocks through Zebra, and simply forces all values in those blocks to zero to achieve simple run-time structured activation maps pruning. The inference has very low overhead and achieves 70% of memory bandwidth reduction within 1% of accuracy loss.
(Section I “Introduction”; emphasis added).
The activation map understandably comprises outputs of a neural network layer, e.g. a convolutional neural network layer, as is known in the art:
In [11], the authors analyze the sparsity of activation maps in different layers. The result shows that the deeper activation maps have higher zero ratio.
(Section 1 “Introduction”; emphasis added).
As to the memory bandwidth overhead of Zebra, the results are shown in Table. V. The required bandwidth means the total activation maps size for one image and the bandwidth overhead is the need of block index storage. In this calculation, we assume a layer-by-layer hardware processing that will store the activation maps to external DRAM for each convoluitonal layer processing. This overhead is totally negligible compared with the amount of reduction in activation maps.
(Section III.B “Results”).
Accordingly, Shih’s dynamic activation map reduction method necessitates receiving outputs, e.g. an activation map, from a layer of neural network so as to divide the activation map into blocks and force values in unimportant blocks to zero.);
partitioning the outputs into a plurality of partitions, wherein each of the plurality of partitions comprises a plurality of the outputs, and sizes of each of the plurality of partitions are defined prior to receiving the outputs (As noted above, Shih discloses that the dynamic activation map reduction method comprises dividing an activation map into a plurality of non-overlapping blocks. Shi further teaches that the sizes of each of the blocks is manually chosen based on various factors including activation map sparsity and location within the neural network:
In Zebra, one important parameter is the block size, which depends on the size of activation maps and datasets. Theoretically, small block size will have higher accuracy and higher sparsity since small block means more fine-grained partitioned. However, small block size will need more index parameters to store the zero conditions of the blocks which leads to memory bandwidth overhead.
(Section II.C “Design Parameters Analysis of Zebra”).
The proposed method is evaluated on two image classification datasets, CIFAR-10 [12] and Tiny-Imagenet that is the subset of Imagenet [13]. CIFAR-10 consists of 10 classes of natural images with resolution 32×32. Empirically, we choose block size as 4 by considering both activation maps sparsity and the zero blocks index overhead. Note that we set block size as 2 when the size of activation maps in deeper layers goes to 2 × 2 in VGG and MobileNets. Tiny-Imagenet contains 200 classes with resolution 64 × 64. We set block size to 8 for better activation maps reduction and the index storage overhead trade-off.
(Section III.A “Settings”; emphasis added).
As the division of the activation maps into such manually-sized blocks occurs during both training and inference – see e.g. Section II.A “Model Training” and Section II.B “Model Inference” – it is apparent that the sizes of each of the plurality of blocks are defined prior to receiving the activation maps during the training and inference. Accordingly, Shih teaches partitioning the outputs, i.e. activation map, into a plurality of partitions, i.e. blocks, wherein each of the plurality of partitions comprises a plurality of the outputs, and sizes of each of the plurality of partitions are defined prior to receiving the outputs.); and
identifying first partitions in the plurality of partitions that satisfy a criterion indicating that values in the first partitions may be set to zero (As noted above, Shih discloses that the dynamic activation map reduction method comprises dividing an activation map into a plurality of non-overlapping blocks, and forcing all values in unimportant blocks to zero. In particular, Shih discloses that if the maximum value of a block is less than a threshold, the block is regarded as an unimportant block, and all values therein are zeroed:
Fig. 1 shows the flow of Zebra to train zero blocks. Activation maps are first divided into non-overlapped blocks. Zebra learns zero blocks or unimportant blocks through regularization. The importance of a block is represented by the maximum value in each block. If the maximum value of a block is smaller than a learned threshold, this block will be regarded as a zero block. The blocks in the same channel use the same threshold. The threshold is learned by feeding activation maps to a small network with a global average pooling layer and a fully-connected layer.
(Section II.A “Model Training”).
Accordingly, Shih teaches identifying first partitions, i.e. unimportant blocks, in the plurality of partitions that satisfy a criterion, i.e. that have a maximum value less than a threshold, the criterion indicating that values in the first partitions may be set to zero.).
Shih thus teaches a method similar to that of claim 20. Furthermore, Shi mentions that “index parameters…store the zero conditions of the blocks” (Section II.C “Design Parameters Analysis of Zebra”). Shih, however, does not explicitly teach generating an encoding that identifies locations of the first partitions (i.e. the unimportant/zero blocks) among remaining second partitions in the plurality of partitions, and sending the encoding and the second partitions to a subsequent layer in the neural network and discarding the first partitions, as is required by claim 20. Shih also does not explicitly teach receiving the second partitions at the subsequent layer in the neural network, arranging the second partitions with zero values based on the encoding, and executing the subsequent layer in the neural network, as is further required by claim 20. Moreover, Shih does not explicitly disclose that the sizes of each of the plurality of partitions is based on a width of an execution unit of the layer of the neural network, as is further required by claim 20.
Corkery describes a deep neural network (DNN) module that can compress and decompress activation data to reduce the utilization of memory bus bandwidth (see e.g. paragraph 0008). Similar to Shih, Corkery particularly teaches:
receiving outputs from a layer of a neural network (see e.g. paragraph 0009: Corkery discloses that the DNN module comprises a compression unit that receives an uncompressed chunk of data generated by one or more of the neurons of the neural network. The DNN module thus receives outputs, i.e. an uncompressed chunk of data, from a layer of the neural network, i.e. from one or more of the neurons of the neural network.);
partitioning the outputs into a plurality of partitions (see e.g. paragraph 0009: Corkery discloses that the uncompressed chunk of data includes a fixed number of bytes, e.g. 64 bytes. Each of the bytes can be considered a partition of the chunk.); and
identifying first partitions in the plurality of partitions that satisfy a criterion (see e.g. paragraphs 0010-0011 and 0036-0037: Corkery discloses that the DNN module generates a compressed output chunk from the received uncompressed chunk, wherein the compressed output chunk comprises a mask portion that includes a number of bits equal to the number of bytes in the uncompressed chunk and wherein each bit corresponds to a particular byte in the uncompressed chunk. Corkery further discloses that the DNN module sets to zero each bit in the mask portion that has a corresponding byte in the uncompressed chunk containing all zeros, and sets to one each bit in the mask portion that has a corresponding byte in the uncompressed chunk that is non-zero – see e.g. paragraphs 0012 and 0037. Thus to generate the mask portion of the compressed chunk, the DNN module necessarily identifies bytes in the uncompressed chunk that satisfy a criterion, i.e. that comprise all zeros.).
Further regarding the claimed invention, Corkery also teaches generating an encoding that identifies locations of the first partitions among remaining second partitions in the plurality of partitions (see e.g. paragraphs 0010-0012 and 0036-0037: as noted above, Corkery discloses that the DNN module generates a compressed output chunk that comprises a mask portion that includes a number of bits equal to the number of bytes in the uncompressed chunk; each bit in the mask portion corresponds to a byte in the uncompressed chunk, with the bit being set to zero if the corresponding byte is zero and set to one if the corresponding byte is non-zero. The mask portion is considered an encoding that identifies the locations of the first partitions, i.e. the locations of bytes that are zero, among second partitions, i.e. among bytes that are non-zero.). In addition, Corkery teaches arranging the second partitions with zero values based on the encoding (see e.g. paragraphs 0016 and 0093-0097: Corkery discloses that the DNN module comprises a decompression unit that can decompress chunks of data that have been compressed and does so by, in part, arranging the non-zero bytes indicated in the data portion of the compressed chunk with zero values based on the locations of the non-zero bytes indicated in the mask portion.).
It would have been obvious to one of ordinary skill in the art, having the teachings of Shih and Corkery before the effective filing date of the claimed invention, to modify the method taught by Shih so as to generate an encoding that identifies locations of the first partitions among remaining second partitions in the plurality of partitions, like done by Corkery, and whereby the second partitions are arranged with zero values based on the encoding. It would have been advantageous to one of ordinary skill to utilize such an encoding because it provides a compressed version of activation data that reduces utilization of memory bus bandwidth, as is taught by Corkery (see e.g. paragraphs 0008-0010).
Similar to Shih and Corkery, Zhou teaches receiving outputs (i.e. a feature map) from a layer of a neural network, compressing the outputs, and storing the compressed outputs (see e.g. paragraphs 0005, 0038-0048 and 0068-0069). Regarding the claimed invention, Zhou also particularly teaches sending the compressed outputs to a subsequent layer in the neural network, whereby the subsequent layer receives the compressed outputs and the subsequent layer is executed (see e.g. paragraphs 0043-0045 and 0069-0071).
It would have been obvious to one of ordinary skill in the art, having the teachings of Shih, Corkery and Zhou before the effective filing date of the claimed invention, to modify the method taught by Shih and Corkery so as to: send the compressed output (i.e. the encoding and the second partitions) to a subsequent layer in the neural network (thus discarding the first partitions); receiving the compressed output (i.e. second partitions) at the subsequent layer in the neural network; and execute the subsequent layer in the neural network, as is taught by Zou. It would have been advantageous to one of ordinary skill to utilize such a combination because, like taught by Zhou, it would enable the subsequent layer to use the output data, as is necessary in neural networks (see e.g. paragraphs 0020-0021 and 0028-0032).
Similar to Shih, Yan teaches receiving outputs (i.e. activation feature maps) from a layer of a neural network, and partitioning the outputs into a plurality of partitions (i.e. banks), wherein sizes of each of the plurality of partitions are defined prior to receiving the outputs (see e.g. paragraphs 0025, 0028-0030 and 0038-0040). Yan further suggests that the sizes of each of the plurality of partitions can be based on a width of an execution unit of the layer of the neural network (see e.g. paragraphs 0029-0030 and 0048: Yan discloses that the bank size can be selected based on hyperparameters/types/dimensions of tensors of the neural network, and can be individually specified for each layer of the neural network. The bank size can thus be selected based on a width of an execution unit, i.e. based on a width of the tensor, for the layer of the neural network.).
It would have been obvious to one of ordinary skill in the art, having the teachings of Shih, Corkery, Zhou and Yan before the effective filing date of the claimed invention, to modify the neural network accelerator taught by Shih, Corkery and Zhou such that the partition size is based on a width of an execution unit of the layer of the neural network like taught by Yan. It would have been advantageous to one of ordinary skill to utilize such a combination because it can provide for more optimal execution of the neural network, as is evident from Yan (see e.g. paragraphs 0029-0030 and 0048). Accordingly, Shih, Corkery, Zhou and Yan are considered to teach, to one of ordinary skill in the art, a method like that of claim 20.
Response to Arguments
The Examiner acknowledges the Applicant’s amendments to claims 1 and 11. Regarding the 35 U.S.C. § 103 rejection for claim 1, the Applicant argues that Shih, Corkery, Zhou and Gunnam fail to teach a partitioning circuit positioned downstream of an output buffer and upstream of an on-chip memory, as is now claimed.
The Examiner however respectfully disagrees. Like noted above, Corkery describes a DNN module that comprises a compression unit that receives an uncompressed chunk of data generated by a layer of a neural network, and then generates a compressed output chunk from the uncompressed chunk (see e.g. paragraphs 0009-0010). The compression unit particularly receives an uncompressed chunk of data that is output from one or more neurons on the DNN module, and after compressing the chunk, provides the compressed chunk to an on-board BaSRAM memory for storage therein (see e.g. paragraphs 0059-0060, 0067-0069 and 0075). The compression unit is thus located downstream from the neurons that output the chunk of uncompressed data of the neural network layer, and upstream from the on-board storage in which the compressed chunk is stored. The neurons would necessarily comprise a buffer to temporarily store the uncompressed output chunk for provision to the compression unit.
Corkery further teaches that generating the compressed chunk comprises partitioning the uncompressed chunk and identifying partitions that can be treated as having zero values. Particularly, Corkery discloses that the uncompressed chunk of data includes a number of bytes, e.g. 64 bytes, and that the compression unit generates a compressed output chunk that comprises a mask portion that includes a number of bits equal to this number of bytes, with each bit corresponding to a particular byte in the uncompressed chunk (see e.g. paragraphs 0009-0011 and 0036-0037). Each such byte in the uncompressed chunk is a type of partition of the chunk. Corkery further discloses that the compression unit sets to zero each bit in the mask portion that has a corresponding byte in the uncompressed chunk containing all zeros, and sets to one each bit in the mask portion that has a corresponding byte in the uncompressed chunk that is non-zero (see e.g. paragraphs 0012 and 0037). The compression unit can be considered a “partitioning circuit” like claimed; as it identifies partitions (i.e. bytes) within the outputs from a neural network layer that can be treated as having zero values. Analogously, like noted above, Shih also teaches compressing outputs (i.e. an activation map) from a layer of a neural network by partitioning the outputs into a plurality of partitions (i.e. blocks), and identifying partitions that can be treated as having all zero values (see e.g. Section II.A “Model Training,” which recites: “Activation maps are first divided into non-overlapped blocks…The importance of a block is represented by the maximum value in each block. If the maximum value of block is smaller than a learned threshold, this block will be regarded as a zero block.”).
Like described above, it would have been obvious to one of ordinary skill in the art, having the teachings of Shih and Corkery before the effective filing date of the claimed invention, to modify the method taught by Shih so as to analogously generate an encoding (i.e. a mask portion comprising zeros representing the blocks that have all zero values, and comprising ones representing the blocks that have non-zero values) like taught by Corkery when compressing the output of the neural network layer, wherein the encoding is generated by a compression unit (i.e. partitioning circuit) located downstream of an output buffer of the layer and upstream of an on-chip memory, like done by Corkery. It would have been advantageous to one of ordinary skill to utilize such an encoding because it provides a compressed version of activation data that reduces utilization of memory bus bandwidth, as is taught by Corkery (see e.g. paragraphs 0008-0010). It would have particularly been obvious to utilize a compression circuit located downstream of an output buffer of the layer and upstream of an on-chip memory, because this would enable compressed activation data to be stored in memory (and thus require less memory and memory bandwidth than if uncompressed activation data was stored in memory), as is further evident from Corkery (see e.g. paragraphs 0059-0060, 0067-0069 and 0075). As noted, both Shih and Corkery teach that compressing the layer output comprises partitioning the output, and identifying partitions that can be treated as having zero values, inter alia. The compression unit can thus be considered a “partitioning circuit” like claimed. Accordingly, the Examiner respectfully maintains that at least the combination of Shih and Corkery teaches the limitation reciting, “wherein the encoding is generated by a partitioning circuit located downstream of an output buffer of the layer and upstream of an on-chip memory,” as is now claimed.
Regarding the 35 U.S.C. § 103 rejection for claim 11, the Applicant argues that Shih, Corkery, Zhou, Gunnam and Yan fail to teach a partitioning circuit configured to overwrite values in an output buffer with zeros prior to storage, as is now claimed. These arguments have been considered, but are moot in view of the new grounds of rejection (i.e. that incorporate U.S. Patent No. 12,518,165 to Xiao et al.) presented above, which are required by the Applicant’s amendments.
Further regarding claim 11, the Applicant argues that the cited references fail to teach “partitioning the outputs into a plurality of partitions, where sizes…are defined…based on a runtime selection of a partition size and a runtime selection of a criterion that are specific to an execution unit of the layer of the neural network,” as is claimed.
The Examiner, however, respectfully disagrees. Yan discloses that “the bank size/layout and the sparsity rate are auto-configurable based on one or more hyperparameters/types/ dimensions of tensors of the neural networks.” (Paragraph 0028; emphasis added). Similarly, Yan further discloses that “[t]he bank size/layout/target sparsity can be individually specified for each layer of the DNN” and that “the bank size/layout derives a number of banks that is divisible by a tensor dimension of an activation tensor of the DNN model.” (Paragraph 0048). As noted above, such banks are analogous to the claimed partitions. Yan thus teaches sizes of partitions (i.e. banks) that are based on a runtime selection of a partition size and a runtime selection of a criterion that are specific to an output tensor of a layer of a neural network.
An output tensor can be considered “an execution unit” of the layer of the neural network, when given the broadest reasonable interpretation of “execution unit.” A tensor is a unit of data that is processed (e.g. by subsequent layers). The Applicant submits that an execution unit “is a physical component of the accelerator hardware.” (Applicant’s Remarks, page 9). However, no such limitation is provided in the claims. Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). Accordingly, the Examiner respectfully maintains that Yan teaches “partitioning the outputs into a plurality of partitions, where sizes…are defined…based on a runtime selection of a partition size and a runtime selection of a criterion that are specific to an execution unit of the layer of the neural network.”
Regarding the 35 U.S.C. § 103 rejection for claim 20, the Applicant argues that the cited prior art fails to teach or suggest defining partition sizes “based on a width of an execution unit of the layer of the neural network,” as is claimed.
The Examiner, however, respectfully disagrees. Like noted above with respect to claim 11, Yan teaches defining sizes of partitions (i.e. banks) based on the dimensions of tensors output by neural network layers. As further noted above, an output tensor can be considered “an execution unit” of the layer of the neural network, when given the broadest reasonable interpretation of “execution unit.” The Examiner thus respectfully maintains that Yan teaches defining partition (i.e. bank) sizes based on a width of an execution unit (i.e. tensor) of the layer of the neural network.
Regarding the 35 U.S.C. § 103 rejection for claim 21, the Applicant argues that the cited references fail to teach or suggest defining partition sizes based on results of processing sample data using an execution unit.
The Examiner, however, respectfully disagrees. Shih discloses, “Empirically, we choose block size as 4 by considering both activation maps sparsity and the zero blocks index overhead.” (Section III.A “Settings”; emphasis added). Shih thus suggests that the sizes of the partitions (i.e. blocks) is selected based on results of processing sample data, as would be necessary to empirically choose a block size (e.g. based on trial-and-error, with different block sizes and observing the results). Accordingly, the Examiner respectfully maintains that Shih teaches, “defining sizes of a plurality of partitions based on results of processing sample data using the execution unit,” as is recited in claim 21.
Conclusion
The prior art made of record on form PTO-892 and not relied upon is considered pertinent to applicant’s disclosure. The applicant is required under 37 C.F.R. §1.111(C) to consider these references fully when responding to this action. In particular, the U.S. Patent Application Publication to Singh et al. cited therein describes a processing apparatus comprising compute logic to generate neural network data for a convolutional neural network (CNN), wherein the compute logic comprises a direct memory access controller to read the neural network data from a memory buffer and encode the neural network data.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BLAINE T BASOM whose telephone number is (571)272-4044. The examiner can normally be reached Monday-Friday, 9:00 am - 5:30 pm, EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Ell can be reached at (571)270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BTB/
5/28/2026
/MATTHEW ELL/Supervisory Patent Examiner, Art Unit 2141
Prosecution Insights