DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-5, 8-11, 13-16, 18, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kun Xu et al. (hereinafter Xu) (“US 20220318604 A1” 10/06/2022) in view of Zhi-Gang Liu et al. (hereinafter Liu) (“US 20220164663 A1”, 05/26/2022), further in view of Georgescu Bogdan et al. (hereinafter Bogdan) (US 9730643 B2, 08/15/2017).
Regarding claim 1, Xu teaches;
A method of executing a layer of a deep neural network (DNN), comprising:
([0056] At block 712, the decompressed weight tensor is written to a target memory (e.g., a state buffer) in the accelerator. Upon receiving the decompressed weight tensor, the accelerator can load the weight tensor into a processing engine array and begin performing computations using the decompressed weight tensor. The computations can be part of a continuing training process, or can be for an inference operation.)
NOTE: The methods disclosed by Xu pertain to compressing and decompressing weight tensors of a layer which are used in the execution of that layer.
([0014] Weight models used in machine learning, and in particular for deep learning, can be large. For example, some weight tensors can take up 32 GB of storage space. When the neural network model size is greater than the external memory capacity of a computing device, the model cannot fit in a single device… [0015] Although weight tensors used in machine learning can be large, most weight tensors are also sparse, since many of the weight values are zero. To reduce the storage size of weight tensors and speed up loading of weight tensors from system memory, a compression technique can be employed to remove zero values from a weight tensor before storing the weight tensor in system memory.)
NOTE: The methods disclosed by Xu address a problem commonly encountered in deep learning, thus, the aforementioned layer can be a layer of a DNN.
storing compressed data in a datastore,
([Abstract] To reduce the storage size of weight tensors and speed up loading of weight tensors from system memory, a compression technique can be employed to remove zero values from a weight tensor before storing the weight tensor in system memory.)
the compressed data comprising one or more nonzero-valued data elements that are a subset of an
([Abstract] To reduce the storage size of weight tensors and speed up loading of weight tensors from system memory, a compression technique can be employed to remove zero values from a weight tensor before storing the weight tensor in system memory.)
NOTE: Teaches the compressed data comprising one or more nonzero-valued data elements that are a subset of a weight operand of the layer (zero values of the layers weight tensor are removed, meaning the remaining non-zero valued data elements are a subset of the weight tensor of that layer), the weight operand comprising a plurality of data elements (the term 'removing values' is plural, indicating a plurality of data elements in the tensor)
determining whether the
([0028] For example, compression header 220 may include a bit map 222 to provide index information indicating locations of the zero weight values (or non-zero weight values) in the decompressed weight tensor.)
NOTE: Teaches determining whether the weight operand comprises any zero-valued data element based on a sparsity bitmap of the weight operand (bit map indicates the location of zero valued data in the weight tensor).
the sparsity bitmap comprising a plurality of bits, each bit corresponding to a respective data element in the
[fig. 2]
PNG
media_image1.png
394
736
media_image1.png
Greyscale
([0028] Weight compression 266 can be performed to reduce the storage size of initial weight tensor 202 by removing any zero values from modified weight tensor 204 (or initial weight tensor 202)… For example, compression header 220 may include a bit map 222 to provide index information indicating locations of the zero weight values (or non-zero weight values) in the decompressed weight tensor.)
NOTE: Teaches the sparsity bitmap comprising a plurality of bits (plurality of bits in bit map 222 as pictured in fig. 2), each bit corresponding to a respective data element in the weight operand (in fig. 2, each bit of bit map 222 corresponds to a data element in the weight tensor 204) and indicating whether the respective data element is zero or nonzero (0 bit indicates zero element and 1 bit indicates nonzero element).
after determining that the
[fig. 2]
PNG
media_image1.png
394
736
media_image1.png
Greyscale
([0030] To regenerate the weight tensor (e.g., for loading into a systolic array, and/or for use in subsequent training or inference operations), weight expansion operation 268 can be performed on compressed weight tensor 266 to re-insert zero values in the proper locations in the weight tensor. For example, bit map 222 containing the index information indicating locations of the zero weight values can be read, and the non-zero weight values can be placed, in the order as the weight values appear in compressed weight tensor 226, into the decompressed weight tensor 250 at the corresponding indexes of the ‘1’ values in bit map 222. Thus, W0 is placed at index 0, W3 is placed at index 3, W5 is placed at index 5, and W7 is placed at index 7 of decompressed weight tensor 250. The remaining weight values of decompressed weight tensor 250 are set to zero.)
NOTE: Teaches after determining that the weight operand comprises a zero-valued data element, generating uncompressed data (as shown in fig 2, generation of the decompressed data / weight expansion is performed after determining that the weight tensor comprises zero-valued data elements) by inserting the zero-valued data element into the compressed data based on a position of the zero-valued data element in the weight operand.
generating a ([0028] a bit map 222 to provide index information indicating locations of the zero weight values (or non-zero weight values) in the decompressed weight tensor) …
transmitting the uncompressed data to a processing element, the processing element configured to compute an output operand based on the uncompressed data
([0056] At block 712, the decompressed weight tensor is written to a target memory (e.g., a state buffer) in the accelerator. Upon receiving the decompressed weight tensor, the accelerator can load the weight tensor into a processing engine array and begin performing computations using the decompressed weight tensor. The computations can be part of a continuing training process, or can be for an inference operation.)
NOTE: Teaches transmitting the uncompressed data to a processing element (load the decompressed weight tensor to a processing engine), the processing element configured to compute an output operand (the result of the inference operation) based on the uncompressed data (performing computations using the decompressed weight tensor).
process the zero-valued data element for computing the output operand ([0055] zero weight values are inserted into the set of non-zero weight values to decompress the compressed weight tensor [0056] Upon receiving the decompressed weight tensor, the accelerator can load the weight tensor into a processing engine array and begin performing computations ... for an inference operation).
Xu fails to teach but Liu teaches;
Input operand
([Abstract] The processor is configured to generate, based on an input tensor, a number of basic block matrices, each basic block matrix including a number of elements; for each basic block matrix: prune, based on a sparsity value, the elements of the basic block matrix, generate a mask for the basic block matrix, each mask including a number of bits, each bit corresponding to a different element of the basic block matrix, and compress the basic block matrix to generate a compressed basic block matrix having fewer elements than the basic block matrix. The MMA is configured to multiply, based on the masks, the compressed basic block matrices and a weight matrix to generate an output matrix.)
NOTE: Teaches compressing an input tensor / input operand.
transmitting the ([0224] PE 450 includes data selection circuit 451, data selectors or multiplexers [0225] Data selection circuit ... receives a mask or index, associated with a compressed basic block matrix ... decode the mask by applying a bitmask, ... and generate and send appropriate selection signals to multiplexers), the sparsity acceleration logic configured to enable, based on the ([0232] data selection circuit 451 receives mask “1010” from scalar register 424. Data selection circuit 451 determines that the first element, w.sup.1.sub.1, should be selected by multiplexer 452.sup.2 based on the first bit of the mask being set to 1 ... then generates and sends the appropriate selection signals to multiplexers [0233] Multiplexer ... receives a selection signal ... to generate intermediate product ... and sends ... to adder circuit ... Adder circuit 457 adds intermediate products ... then outputs the resulting data value)
OBVIOUSNESS TO COMBINE XU AND LIU:
Xu and Liu are both analogous art to each other and to the present invention as they both pertain to compressing operands in a neural network. Xu pertains to a method of compressing (to save resources) and decompressing weight tensors (when they need to be used) while Liu pertains to an activation compression method. Additionally, Lu indicates that a large portion input feature map activations are zero valued, and that compressing input feature maps greatly lowers memory footprint, reduces power consumption, and increases throughput; “[0013] ANN accelerators maintain the input feature maps in a dense or uncompressed form, even though the input feature maps typically contain a significant amount of activations that have a value of zero (e.g., greater than 60%) … [0114] Embodiments of the present disclosure advantageously compress the input feature maps during inference by dynamically removing or pruning smaller value activations (including zero values), lower the memory footprint of the activation memory by factor greater than 2×, reduce the power consumption related to activation memory accesses (e.g., DRAM, SRAM, cache, etc.), and increase the effective processing throughput by factor greater than 2× for a MAC array.” Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to perform Xu’s compressing and decompressing methods on input feature map data to address the issue of large amounts of zero activations in input feature maps, predictably lowering memory footprint, reducing power consumption, and increasing throughput.
Xu and Liu fail to teach but Bogdan teaches;
generating a new sparsity bitmap …, the new sparsity bitmap comprising one or more bits, each of which has a value of one
PNG
media_image2.png
94
539
media_image2.png
Greyscale
([col. 27, lines 10-12] At 1504, the sparsity map s is initialized with all ones, representing that all of the weights remain)
OBVIOUSNESS TO COMBINE BOGDAN WITH XU AND LIU:
Xu teaches that zero values can be removed to reduce storage size, then restored when the original tensor is needed, Liu teaches sparsity controlled processing circuitry in which a bitmap is provided to a processing element, and a bit value of one causes the corresponding data element of the input operand to be selected for processing, while Bogdan teaches initializing a sparsity map with all ones to indicate that all corresponding values remain active. It would have been obvious to one of ordinary skill in the art, before the effective filing date, to use Bogdan’s known all-one sparsity map representation for Xu’s reconstructed tensor so that all reconstructed positions, including restored zero-valued positions, are set as active and can be selected for processing when the full tensor is required, while retaining the ability to employ Liu’s sparsity aware compression and processing techniques when sparse processing is appropriate, predictably preserving the storage, memory, and processing benefits associated with sparse representations (as previously taught by Liu), while permitting restored zero-value positions to be processed when the full reconstructed tensor is required (as taught by Xu).
Regarding claim 3, Xu in view of Liu teach
The method of claim 1
(Using the same reasoning from claim 1)
Xu fails to teach but Liu teaches;
wherein the layer is selected from a group consisting of a depthwise convolution layer, a group convolution layer, an elementwise layer, and a pooling layer.
([0035] Pooling layer 30-2 is locally-connected to convolutional layer 30-1, and includes a plurality of nodes that are connected to local regions in the input volume (not depicted for clarity). Pooling layer 30-2 also produces an output volume that is provided as the input volume to the subsequent layer, such as, for example, another convolutional layer 30-1, a flatten layer 40, etc. In certain embodiments, convolutional layer 30-1 and pooling layer 30-2 form a single hidden layer 30. Similarly, in certain embodiments, convolutional layer 30-1, a ReLU layer and pooling layer 30-2 form a single hidden layer 30. Generally, the output volumes of the convolutional and pooling layers may be described as feature maps, and one or more single hidden layers 30 form a feature learning portion of CNN 15.)
NOTE: Liu teaches a pooling layer being executed to produce an output operand (output feature map).
Xu and Liu both teach a layer of a neural network being executed to produce an output operand, thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to substitute the layer of Xu with the pooling layer of Liu, predictably resulting in an output operand.
OBVIOUSNESS:
Using the reasoning from claim one, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to perform the process of compressing and decompressing data taught by Xu on input feature map data as taught by Liu to address the issue of large amounts of zero activations in input feature maps to lower memory footprint, reduce power consumption, and increase throughput.
Regarding claim 4, Xu in view of Liu teach;
The method of claim 1
(Using the same reasoning from claim 1)
Xu fails to teach but Liu teaches;
wherein the input operand is a part of an input feature map of the layer,
([0041] Input feature maps 204 include four channels and one input data matrix for each channel, i.e., input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4... Convolutional layer calculation 200 convolves filter 202 with input feature maps 204 to produce output feature maps 206.)
([0042] Generally, input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4 form an input tensor)
NOTE: Teaches the input operand (input tensor) is a part of an input feature map (the input tensor is formed from the input data matrices of the input feature map) of the layer.
the input feature map comprises one or more channels,
([0041] Input feature maps 204 include four channels and one input data matrix for each channel, i.e., input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4)
and the one or more data elements in the input operand is in different channels of the one or more channels.
(([0041] Input feature maps 204 include four channels and one input data matrix for each channel, i.e., input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4... Convolutional layer calculation 200 convolves filter 202 with input feature maps 204 to produce output feature maps 206.)
([0042] Generally, input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4 form an input tensor)
NOTE: Teaches the one or more data elements in the input operand is in different channels of the one or more channels (the input tensor includes data matrices for each channel, ‘each’ indicating a plurality of channels).
OBVIOUSNESS:
Using the reasoning from claim one, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to perform the process of compressing and decompressing data taught by Xu on input feature map data as taught by Liu to address the issue of large amounts of zero activations in input feature maps to lower memory footprint, reduce power consumption, and increase throughput.
Regarding claim 5, Xu in view of Liu teaches;
The method of claim 1
(Using the same reasoning from claim 1)
Xu fails to teach but Liu teaches;
wherein the output operand comprises data elements in two or more of the different channels.
([0035] Generally, the output volumes of the convolutional and pooling layers may be described as feature maps)
([0041] Output feature maps 206 include four channels and one output data matrix for each filter or weight set, i.e., output data matrices 206.sup.1, 206.sup.2, 206.sup.3 and 206.sup.4.)
NOTE: Teaches the output operand (the output feature map of the NN layer) comprises data elements in two or more of the different channels (output feature map includes 4 channels where each output data matrix of the feature map corresponds to a channel).
OBVIOUSNESS:
Using the reasoning from claim one, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to perform the process of compressing and decompressing data taught by Xu on input feature map data as taught by Liu to address the issue of large amounts of zero activations in input feature maps to lower memory footprint, reduce power consumption, and increase throughput.
Regarding claim 8, Xu in view of Liu teaches;
The method of claim 1
(Using the same reasoning from claim 1)
Xu teaches;
wherein determining whether the
([0028] A value of ‘1’ in bit map 222 indicates that the corresponding element in the weight tensor is a non-zero value, and a value of ‘0’ in bit map 222 indicates that the corresponding element in the weight tensor has a zero value.)
NOTE: Teaches determining whether a bit in the sparsity bitmap is zero (bitmap indicates zero values with a '0' bit).
Xu fails to teach but Liu teaches;
Input operand
([Abstract] The processor is configured to generate, based on an input tensor, a number of basic block matrices, each basic block matrix including a number of elements; for each basic block matrix: prune, based on a sparsity value, the elements of the basic block matrix, generate a mask for the basic block matrix, each mask including a number of bits, each bit corresponding to a different element of the basic block matrix, and compress the basic block matrix to generate a compressed basic block matrix having fewer elements than the basic block matrix. The MMA is configured to multiply, based on the masks, the compressed basic block matrices and a weight matrix to generate an output matrix.)
NOTE: Teaches compressing an input tensor / input operand.
OBVIOUSNESS:
Using the reasoning from claim one, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to perform the process of compressing and decompressing data taught by Xu on input feature map data as taught by Liu to address the issue of large amounts of zero activations in input feature maps to lower memory footprint, reduce power consumption, and increase throughput.
Regarding claim 9, Xu in view of Liu teaches;
The method of claim 1
(Using the same reasoning from claim 1)
Xu teaches;
further comprising: determining the position of the zero-valued data element in the
[fig. 2]
PNG
media_image1.png
394
736
media_image1.png
Greyscale
([0028] For example, compression header 220 may include a bit map 222 to provide index information indicating locations of the zero weight values (or non-zero weight values) in the decompressed weight tensor. Bit map 222 can be a binary bit map containing a number of bits being equal to the number of elements or weight values in the weight tensor.)
NOTE: Teaches determining the position of the zero-valued data element in the weight operand based on a position of a bit in the sparsity bitmap that corresponds to the zero-valued data element (the bit map indicates zero or non-zero data values in the weight tensor based on the position of the bit corresponding to the data value).
wherein the plurality of bits in the sparsity bitmap is in a sequence.
[fig. 2]
PNG
media_image1.png
394
736
media_image1.png
Greyscale
NOTE: The bitmap 222 is an ordered set of bits corresponding to the order of the bits in the weight tensor, which is considered a sequence. Thus, Xu teaches the plurality of bits in the sparsity bitmap being in a sequence.
Xu fails to teach but Liu teaches;
Input operand
([Abstract] The processor is configured to generate, based on an input tensor, a number of basic block matrices, each basic block matrix including a number of elements; for each basic block matrix: prune, based on a sparsity value, the elements of the basic block matrix, generate a mask for the basic block matrix, each mask including a number of bits, each bit corresponding to a different element of the basic block matrix, and compress the basic block matrix to generate a compressed basic block matrix having fewer elements than the basic block matrix. The MMA is configured to multiply, based on the masks, the compressed basic block matrices and a weight matrix to generate an output matrix.)
NOTE: Teaches compressing an input tensor / input operand.
OBVIOUSNESS:
Using the reasoning from claim one, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to perform the process of compressing and decompressing data taught by Xu on input feature map data as taught by Liu to address the issue of large amounts of zero activations in input feature maps to lower memory footprint, reduce power consumption, and increase throughput.
Regarding claim 10, Xu in view of Liu teaches;
The method of claim 1
(Using the same reasoning from claim 1)
Xu teaches;
further comprising: storing the sparsity bitmap in the datastore,
([0028] For example, compression header 220 may include a bit map 222 to provide index information indicating locations of the zero weight values (or non-zero weight values) in the decompressed weight tensor.)
([0029] Compression header 220 can be stored together with compressed weight tensor 226 (e.g., in system memory) such that a component (e.g., a DMA engine) reading the compressed weight tensor 226 can access compression header 220 to obtain information on how to expand or decompress the compressed weight tensor 226.)
NOTE: The compression header comprises the bitmap and is stored in memory. Thus, Xu teaches storing the sparsity bitmap in the datastore (memory).
wherein determining whether the
([0030] To regenerate the weight tensor (e.g., for loading into a systolic array, and/or for use in subsequent training or inference operations), weight expansion operation 268 can be performed on compressed weight tensor 266 to re-insert zero values in the proper locations in the weight tensor. For example, bit map 222 containing the index information indicating locations of the zero weight values can be read, and the non-zero weight values can be placed, in the order as the weight values appear in compressed weight tensor 226, into the decompressed weight tensor 250 at the corresponding indexes of the ‘1’ values in bit map 222. Thus, W0 is placed at index 0, W3 is placed at index 3, W5 is placed at index 5, and W7 is placed at index 7 of decompressed weight tensor 250. The remaining weight values of decompressed weight tensor 250 are set to zero.)
NOTE: Teaches determining whether the weight operand comprises the zero-valued data element (bitmap indicates locations of the zero valued data in the weight tensor) after the compressed data and the sparsity bitmap are stored in the datastore (at the time bit map is read, the compressed weight tensor and the compression header containing the sparsity bitmap have already been stored in memory).
Xu fails to teach but Liu teaches;
Input operand
([Abstract] The processor is configured to generate, based on an input tensor, a number of basic block matrices, each basic block matrix including a number of elements; for each basic block matrix: prune, based on a sparsity value, the elements of the basic block matrix, generate a mask for the basic block matrix, each mask including a number of bits, each bit corresponding to a different element of the basic block matrix, and compress the basic block matrix to generate a compressed basic block matrix having fewer elements than the basic block matrix. The MMA is configured to multiply, based on the masks, the compressed basic block matrices and a weight matrix to generate an output matrix.)
NOTE: Teaches compressing an input tensor / input operand.
OBVIOUSNESS:
Using the reasoning from claim one, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to perform the process of compressing and decompressing data taught by Xu on input feature map data as taught by Liu to address the issue of large amounts of zero activations in input feature maps to lower memory footprint, reduce power consumption, and increase throughput.
Regarding claim 11,
Claim 11 is a machine claim that is substantially similar to method claim 1, except for the preamble, which is taught by Xu;
A DNN accelerator configured to execute a layer of a deep neural network (DNN), the compute block comprising:
([0017] FIG. 1 illustrates an example of a computing system 100. Computing system 100 includes a DMA engine 150, a system memory 120, and an accelerator 130. Computing system 100 may include other components not specifically shown, such as a host processor. Accelerator 130 can be a neural network accelerator (e.g., a neural network processor, tensor processing unit, etc.), and may include a processing engine array 132 (e.g., a systolic array), a state buffer 134, and a result buffer 136.)
NOTE: Indicates a NN accelerator for performing the methods disclosed by Xu, said methods including execution of a layer of a DNN (as taught in claim 1). Thus, Xu teaches a DNN accelerator configured to execute a layer of a deep neural network (DNN).
The remaining limitations are substantially similar and are taught using the same reasoning as in claim 1.
Regarding claim 13;
Claim 13 is a machine claim that is substantially similar to method claim 3, and is rejected using the same reasoning as in claim 3.
Regarding claim 14;
Xu in view of Liu teach;
The DNN accelerator of claim 11
(Using the same reasoning from claim 11)
Xu fails to teach but Liu teaches;
wherein the input operand is a part of an input feature map of the layer,
([0041] Input feature maps 204 include four channels and one input data matrix for each channel, i.e., input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4... Convolutional layer calculation 200 convolves filter 202 with input feature maps 204 to produce output feature maps 206.)
([0042] Generally, input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4 form an input tensor)
NOTE: Teaches the input operand (input tensor) is a part of an input feature map (the input tensor is formed from the input data matrices of the input feature map) of the layer.
the input feature map comprises a plurality of channels,
([0041] Input feature maps 204 include four channels and one input data matrix for each channel, i.e., input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4)
and the plurality of data elements in the input operand is in different channels of the plurality of channels.
(([0041] Input feature maps 204 include four channels and one input data matrix for each channel, i.e., input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4... Convolutional layer calculation 200 convolves filter 202 with input feature maps 204 to produce output feature maps 206.)
([0042] Generally, input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4 form an input tensor)
NOTE: Teaches the plurality of data elements in the input operand (four data matrices in the input tensor / operand) is in different channels of the plurality of channels (the input tensor / operand includes data matrices for each channel, ‘each’ indicating a plurality of channels).
OBVIOUSNESS:
Using the reasoning from claim one, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to perform the process of compressing and decompressing data taught by Xu on input feature map data as taught by Liu to address the issue of large amounts of zero activations in input feature maps to lower memory footprint, reduce power consumption, and increase throughput.
Regarding claim 15;
Claim 15 is a machine claim that is substantially similar to method claim 5, and is rejected using the same reasoning as in claim 5.
Regarding claim 16,
Claim 16 is a CRM claim that is substantially similar to method claim 1, except for the preamble, which is taught by Xu;
One or more non-transitory computer-readable media storing instructions executable to perform operations for executing a layer of a deep neural network (DNN), the operations comprising:
([0096] Storage devices, the DRAM 1030, and any other memory component in the host system 1000 are examples of computer-readable storage media. Computer-readable storage media are physical mediums that are capable of storing data in a format that can be read by a device such as the host processor 1072. Computer-readable storage media can be non-transitory.)
([0097] In various examples, the data stored on computer-readable storage media can include program instructions, data structures, program modules, libraries, other software program components, and/or other data that can be transmitted within a data signal, such as a carrier wave or other transmission. The computer-readable storage media can, additionally or alternatively, include documents, images, video, audio, and other data that can be operated on or manipulated through the use of a software program.)
NOTE: Teaches non-transitory computer readable media storing instructions executable to perform the methods disclosed by Xu, said methods including execution of a layer of a DNN (as taught in claim 1).
The remaining limitations are substantially similar and are taught using the same reasoning as in claim 1.
Regarding claim 18;
Claim 18 is a CRM claim that is substantially similar to machine claim 14 except for one added limitation, which is taught by Liu;
and the output operand comprises data elements in two or more of the different channels.
([0035] Generally, the output volumes of the convolutional and pooling layers may be described as feature maps)
([0041] Output feature maps 206 include four channels and one output data matrix for each filter or weight set, i.e., output data matrices 206.sup.1, 206.sup.2, 206.sup.3 and 206.sup.4.)
NOTE: Teaches the output operand (the output feature map of the NN layer) comprises data elements in two or more of the different channels (output feature map includes 4 channels where each output data matrix of the feature map corresponds to a channel).
The remaining limitations are substantially similar to claim 14 and are taught using the same reasoning.
Regarding claim 20;
Claim 20 is a CRM claim that is substantially similar to method claim 8, and is rejected using the same reasoning as in claim 8.
Claim(s) 6-7, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu (“US 20220318604 A1” 10/06/2022) in view of Liu (“US 20220164663 A1”, 05/26/2022) further in view of Alessandro Aimar et al. (hereinafter Aimar) (“NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps”, 03/06/2018).
Regarding claim 6, Xu in view of Liu teaches;
The method of claim 1,
(Using the same reasoning as claim 1)
Xu and Liu fail to teach but Aimar teaches;
further comprising: storing the output operand in the datastore;
([pg. 2] The output feature maps produced by the current layer are streamed off-chip to the external memory.)
NOTE: Teaches storing the output operand (output feature maps) in the datastore (external memory).
and writing a subset of the output operand from the datastore to a memory, the subset of the output operand comprising one or more nonzero-valued data elements in the output operand.
([pg. 3] NullHop uses a novel sparse matrix compression algorithm... The coding uses two elements: a Sparsity Map (SM) and a Non-Zero Value List (NZVL)... The SM is used to reconstruct the positions of the non-zero pixels. These non-zero pixel values are stored as the NZVL: an ordered, variable-length list containing all the non-zero values in the feature maps.)
NOTE: The compressed format of the feature maps contains only the non-zero values (‘values’ is plural, indicating a plurality of non-zero values) of the feature maps, i.e., a subset. Thus, teaches the subset of the output operand (the aforementioned output feature map) comprising a plurality of nonzero-valued data elements in the output operand.
([pg. 3] They are then streamed back to the accelerator SRAM when the accelerator has finished processing the current layer. The feature maps are always stored in a compressed format)
NOTE: Teaches writing a subset of the output operand (the compressed feature maps) from the datastore to a memory (streamed from the external memory to the SRAM).
OBVIOUSNESS TO COMBINE AIMAR WITH XU AND LIU:
Aimar is analogous to the present disclosure as it pertains to a neural network accelerator based on sparse representations of feature maps.
Additionally, Aimar teaches exploiting output activation sparsity and storing / outputting feature maps in compressed form to reduce memory requirements and external memory transfers.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to write only the nonzero-valued portion of the computed output operand as taught by Aimar in order to reduce the amount of data written to memory.
Regarding claim 7, Xu in view of Liu teach;
The method of claim 1,
(Using the same reasoning as in claim 1)
Xu and Liu fail to teach but Aimar teaches;
Further comprising: generating a new sparsity bitmap for the output operand, the new sparsity bitmap comprising one or more bits, each of which corresponds to a respective data element in the output operand and indicates whether the respective data element in the output operand is zero or nonzero.
([pg. 3] NullHop uses a novel sparse matrix compression algorithm... The coding uses two elements: a Sparsity Map (SM) and a Non-Zero Value List (NZVL). The SM is a 3D mask, having the same number of entries as number of pixels in the feature maps. Each binary entry in the SM is 1 if the corresponding pixel is non-zero, and 0 otherwise:)
NOTE: Teaches generating a new sparsity bitmap for the output operand (a sparsity map is generated for the feature maps, which includes the aforementioned output feature map), the new sparsity bitmap comprising one or more bits, each of which corresponds to a respective data element in the output operand and indicates whether the respective data element in the output operand is zero or nonzero (the sparsity map has bits corresponding to each pixel of the output feature map indicating if the pixel is zero or nonzero).
Regarding claim 19, Xu in view of Liu teach;
The one or more non-transitory computer-readable media of claim 16,
(Using the same reasoning as in claim 16)
Xu and Liu fail to teach but Aimar teaches;
wherein the operations further comprise: generating a new sparsity bitmap for the output operand, the new sparsity bitmap comprising a plurality of bits, each of which corresponds to a respective data element in the output operand and indicates whether the respective data element in the output operand is zero or nonzero.
([pg. 3] NullHop uses a novel sparse matrix compression algorithm... The coding uses two elements: a Sparsity Map (SM) and a Non-Zero Value List (NZVL). The SM is a 3D mask, having the same number of entries as number of pixels in the feature maps. Each binary entry in the SM is 1 if the corresponding pixel is non-zero, and 0 otherwise:)
NOTE: Teaches generating a new sparsity bitmap for the output operand (a sparsity map is generated for the feature maps, which includes the aforementioned output feature map), the new sparsity bitmap comprising a plurality of bits (‘each binary entry’ is plural, and indicates a plurality of bits), each of which corresponds to a respective data element in the output operand and indicates whether the respective data element in the output operand is zero or nonzero (the sparsity map has bits corresponding to each pixel of the output feature map indicating if the pixel is zero or nonzero).
Response to Arguments
Applicant’s arguments, filed 08/05/2026, with respect to the rejections under 35 U.S.C. § 112 have been fully considered and are persuasive. The 35 U.S.C. § 112(b) rejections of claims 5, 11, and 15 have been withdrawn.
Applicant’s arguments, filed 08/05/2026, with respect to the rejections under 35 U.S.C. § 101 have been fully considered and are persuasive. The 35 U.S.C. § 101 abstract idea rejections of the claims have been withdrawn.
Applicant's arguments filed 08/05/2026 with respect to the rejections under 35 U.S.C. § 103 have been fully considered but they are not persuasive. Starting page 4, the applicant states; “Bogdan describes a sparsity map that is initiated with all ones during a DNN training process. Bogdan's sparsity map is changed to replace some of the ones with zeros when the corresponding weight values is lower than a threshold value. Thus, Bogdan merely describes a method of changing ones in a sparsity map to zeros based on weight values for training a DNN. Bogdan is silent regarding generating a new sparsity bitmap after an input operand is uncompressed and transmitting the new sparsity bitmap to a sparsity acceleration logic associated with the processing element, not to mention that the sparsity acceleration logic is configured to enable, based on the new sparsity bitmap, the processing element to process the zero-valued data element for computing the output operand. The other reference cannot remedy the deficiency of Bogdan as they are all silent regarding the generating and transmitting step. Therefore, amended claim 1 is not obvious over the combination of the cited references. The other independent claims are amended to recite similar limitations as amended claim 1 and therefore, are also patentable. The dependent claims are patentable at least due to their dependency on the amended independent claims. Accordingly, withdrawal of the § 103 rejections is respectfully requested.” Examiner respectfully disagrees. As reflected by the present office action, the rejection does not rely on Bogdan alone for these limitations. Xu teaches decompressing a compressed tensor by reinserting zero values at their proper locations for subsequent operations and further teaches bitmap information corresponding to the locations of zero and non-zero values in the decompressed tensor. Liu teaches that a data selection circuit associated with a processing element receives a mask, and that a corresponding data element from an input operand is selected when its mask bit is set to one and is thereafter supplied to a multiplier for computation. Bogdan further teaches the known representation of initializing a sparsity map with all ones, thereby indicating that all corresponding values remain active. Thus, in the proposed combination, after Xu reconstructs the full tensor including the restored zero-valued positions, Bogdan’s all-one sparsity map representation is used to designate all reconstructed positions as active, and Liu’s mask-controlled processing circuitry uses that bitmap to select the corresponding positions for processing. The predictable result is that restored zero-valued positions are selected when the full reconstructed tensor is required. Accordingly, Applicant’s assertion that the cited references are silent regarding the generating and transmitting steps does not address the combined teaching relied upon in the rejection. Obviousness is determined based on what the combined teachings of the references would have suggested to one of ordinary skill in the art, rather than whether a single reference independently discloses the complete claimed arrangement.
For the above reasons, the 35 U.S.C. § 103 rejections for amended claim 1, and similar amended independent claims 11 and 16, stand. Dependent claims 3-10, 13-15, and 18-19 also remain rejected under 35 U.S.C. § 103 at least by virtue of their dependency to claims 1, 11, and 16.
CONCLUSION
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Matthew Alan Cady whose telephone number is (571) 272-7229. The examiner can normally be reached Monday - Friday, 7:30 am - 5:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached on (571)272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC)
at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW ALAN CADY/ Examiner, Art Unit 2145
/CESAR B PAULA/ Supervisory Patent Examiner, Art Unit 2145
