DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the arguments filed on 03/05/2026. Claims 1-20 are pending in the application and have been considered below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-5, 7-13 and 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Burger et al. (US 2016/0379115 A1, hereinafter referred to as Burger), in view of et al. (US 2018/0189633 A1, hereinafter referred to as HENRY), and further in view of Han et al. (“EIE: Efficient Inference Engine on Compressed Deep Neural Network,” hereinafter referred to as Han) and Shafiee et al. (“ISAAC: A Convolutional Neural Network Accelerator with In-Situ Analog Arithmetic in Crossbars,” hereinafter referred to as Shafiee).
As to claim 1, Burger teaches a method for performing the operations of a neural network using a neural network accelerator comprising a plurality of tiles (paragraphs [0128], [0163], the acceleration component may have different configurable domains (e.g., tiles), some of which are being currently used and others which are not being currently used);
wherein each tile of the plurality of tiles comprises a plurality of operators, wherein each operator is configured to perform neural network computations (paragraphs [0248]-[0251] The next two acceleration components (3606, 3608) perform more complex computations, referred to herein as "free form expressions" (FFE), compared to the computations performed by head component 3604; [0259] FFE acceleration components 3606 and 3608 may perform mathematical computations using feature values determined by head component 3604; [0261] FIG. 42, multiplication, addition and division operations);
wherein each tile of the plurality of tiles has a narrow memory bank and a separate wide memory bank shared by the plurality of operators of the tile, wherein the narrow memory bank includes one or more memory units, wherein the narrow memory bank is configured to store and distribute input activations to the plurality of operators (Fig. 58 elements 5804 (input memory (activations) (interpreted by Examiner as “narrow memory;” the activation memory is interpreted as the narrow memory)) at a first memory bandwidth, wherein the wide memory bank includes one or more memory units, and wherein the wide memory bank is configured to store and provide access to weights of the neural network to the plurality of operators at a second memory bandwidth that is greater than the first memory bandwidth (Fig. 58 elements 5804 (input memory (activations) (The (input memory (activations) is interpreted by Examiner as “narrow memory”)) and 5008 (input memory (weights) (interpreted by Examiner as “wide memory”)); paragraphs [0329], provide access to high-bandwidth memory that can be used to store the weight matrices…; [0330], first memory 5804 interpreted by Examiner as “narrow memory”) is used to buffer input activations data and third memory 5808 (interpreted by Examiner as “wide memory”) is used to buffer weights data; Examiner interpretation is based on paragraphs [0007] and [0056] of the original disclosure (specification); [0342]-[0343], configuring the acceleration component includes storing the weights, input activations and errors in the memory stack; and streaming the weights, input activations and errors to the storage elements of the neural engines; [0278] Server unit component 4602 optionally may include memory 4624 coupled to local acceleration component 4606. In an implementation, memory 4624 is external to (i.e., not included in the same integrated circuit package) local acceleration component 4606 and includes a low bandwidth memory, such as DIMMS 4626(interpreted by Examiner as “first memory bandwidth”; Examiner interpretation is based on paragraph [0013] of the original disclosure.). As used herein, a "low bandwidth memory" is a memory having a bandwidth less than about 50 GB/sec) (interpreted by Examiner as “first memory bandwidth”; Examiner interpretation is based on paragraph [0013] of the original disclosure). In other implementations, memory 4624 also may include a high bandwidth (interpreted by Examiner as “second memory bandwidth,” Examiner interpretation is based on paragraph [0013] of the original disclosure), high power memory technology, such as HMC 4628. As used herein, a high bandwidth, high power memory" is a memory having a memory bandwidth greater than about 50 GB/sec (interpreted by Examiner as “second memory bandwidth,” Examiner interpretation is based on paragraph [0013] of the original disclosure) and a power efficiency of less than about 20 MB/sec/mW.; [0279] Thus, server unit component 4602 includes acceleration component 4606 with high bandwidth, low power memory (memory stack 4620). In a hybrid implementation, server unit component 4602 includes acceleration component 4606 with both high bandwidth, low power memory (memory stack 4620) and low bandwidth memory (DIMMS 4626). In another hybrid implementation, server unit component 4602 includes acceleration component 4606 with both high bandwidth, low power memory (memory stack 4620), low bandwidth memory (DIMMS 4626), and high bandwidth, high power memory (HMC 4628)).
Burger discloses (Fig. 58 elements 5804 (input memory (activations) (“narrow memory”) and 5008 (input memory (weights) (“wide memory”); wherein Examiner interpretation is based on paragraphs [0007] and [0056] of the specification). Burger further discloses weights that are distributed and pinned across multiple neural engines.
However, Burger fails to explicitly teach the method comprising:
distributing the weights of the neural network across the plurality of tiles such that each tile stores, in its wide memory bank, weights required to perform the computations of one or more respective layers of the neural network; and
executing, by each tile, the operations of one or more respective layers of the neural network including distributing the input activations stored in the narrow memory bank to the plurality of operators and accessing, by the tile to the one or more operators of the tile, respective weights required to perform the operations stored in the wide memory bank memory banks of the tile.
However, Burger fails to explicitly teach:
distributing the weights of the neural network across the plurality of tiles such that each tile stores, in its wide memory bank, weights required to perform the computations of one or more respective layers of the neural network; and
executing, by each tile, the operations of one or more respective layers of the neural network including distributing the input activations stored in the narrow memory bank to the plurality of operators and accessing, by the tile to the one or more operators of the tile, respective weights required to perform the operations stored in the wide memory bank memory banks of the tile.
Han, in combination with Burger, teaches:
distributing the weights of the neural network across the plurality of tiles such that each tile stores, in its wide memory bank, weights required to perform the computations of one or more respective layers of the neural network (page 252, left column, VIII. COMPARISON WITH RELATED WORK, wherein Examiner interprets “distributes weights on 16 tiles, each tile with 4 eDRAM banks, thus has a peak memory bandwidth of 16 × 4 × (1024bit/8) × 606MHz = 4964GB/s”).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the system of Burger to add weights distribution on tiles to the system of Burger, as taught by Han above. The modification would have been obvious because one of ordinary skill would be motivated to have performance estimation based on the peak memory bandwidth, as suggested by Han ((page 252, left column, VIII. COMPARISON WITH RELATED WORK).
However, Burger and Han fail to explicitly teach:
executing, by each tile, the operations of one or more respective layers of the neural network including distributing the input activations stored in the narrow memory bank to the plurality of operators and obtaining, by the tile to the one or more operators of the tile, respective weights required to perform the operations stored in the wide memory bank memory banks of the tile.
Shafiee, in combination with Burger and Han, teaches:
executing, by each tile, the operations of one or more respective layers of the neural network including distributing the input activations stored in the narrow memory bank to the plurality of operators and accessing, by the tile to the one or more operators of the tile, respective weights required to perform the operations stored in the wide memory bank memory banks of the tile (I. INTRODUCTION, connected chips (nodes), each made up of 16 tiles. The dominant data structures in CNNs and DNNs are the synaptic weight matrices that define each neuron layer. These are distributed across several eDRAM banks on multiple tiles/nodes. The computations involving these weights are brought to the eDRAM banks and executed on adjacent NFUs, thus achieving near data processing; page 2/13 II. BACKGROUND, right column, activation function, page 3/13, C. The DaDianNao Architecture A single DaDianNao [9] chip (node) is made up of 16 tiles and two central eDRAM banks connected by an on-chip fat-tree network. A tile is made up of a neural functional unit (NFU) and four eDRAM banks. An NFU has a pipeline with multiple parallel multipliers, a tree of adders, and a transfer function. Tiling is used to maximize data reuse and reduce transfers in/out of eDRAM banks. Synaptic weights are distributed across all nodes/tiles and feed their local NFUs; page 174/13, IV. THE ISAAC PIPELINE …The synaptic weights for that layer are therefore scattered across eDRAM banks in all tiles…weights from the eDRAM banks to its SRAM buffers; tiles 0-3 may be assigned to layer 0, tiles 4-11 may be assigned to layer 1, and so on. In this case, tiles 0-3 would store all weights for layer 0 and perform all layer 0 computations in parallel; Figs 2 and 3).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Burger and Han to add memory banks weights storage to the combination system of Burger and Han, as taught by Shafiee above. The modification would have been obvious because one of ordinary skill would be motivated to perform the required operations for one layer in parallel and maximize throughput for one layer, as suggested by Shafiee (page 4/13, IV. THE ISAAC PIPELINE, first paragraph).
As to claim 2, which incorporates the rejection of claim1, Burger teaches: providing, by each tile to the plurality of operators of the tile, input activations output from a previous layer or another tile (paragraphs [0314]-[0316]…using an activation function and generates output activations of Layer. Typically, the activation function is a weighted sum of products, taking the input activation of each neuron and scaling it by a tunable weight parameter).
As to claim 3, which incorporates the rejection of claim1, Burger teaches: wherein executing the operations of the one or more respective layers of the neural network comprises executing the operations of the neural network using only the weights distributed across the plurality of tiles (paragraphs [0332]-[0333], controller component 5704 tiles the matrix multiplication independently across multiple neural engines (5712, 5714, 5716, 5718, 5720). Each of neural engines (5712, 5714, 5716, 5718, 5720) reads in a subset of the weight matrix; the weights are distributed and pinned across multiple neural engines (5712, 5714, 5716, 5718, 5720)).
As to claim 4, which incorporates the rejection of claim 3, Burger, Henry and Han fail to explicitly teach: wherein executing the operations of the one or more respective layers of the neural network comprises executing the operations of the neural network without retrieving weights from a memory that is not local to any of the one or more tiles.
Shafiee, in combination with Burger and Han, teaches wherein executing the operations of the one or more respective layers of the neural network comprises executing the operations of the neural network without retrieving weights from a memory that is not local to any of the one or more tiles (I. INTRODUCTION, right column, first paragraph, The dominant data structures in CNNs and DNNs are the synaptic weight matrices that define each neuron layer. These are distributed across several eDRAM banks on multiple tiles/nodes. The computations involving these weights are brought to the eDRAM banks and executed on adjacent NFUs, thus achieving near data processing. This requires moving the outputs of the previous neuron layer to the relevant tiles so they can merge with co-located synaptic weights to produce the outputs of the current layer. The outputs are then routed to appropriate eDRAM banks so they can serve as inputs to the next layer. Most of the chip area is used to store synaptic weights in eDRAM).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Burger and Han to add an execution of the operations of the neural network to the combination system of Burger and Han, as taught by Shafiee above. The modification would have been obvious because one of ordinary skill would be motivated to perform the required operations for one layer in parallel and maximize throughput for one layer, as suggested by Shafiee (page 4/13, IV. THE ISAAC PIPELINE, first paragraph).
As to claim 5, which incorporates the rejection of claim 1, Burger and Han fail to explicitly teach wherein the narrow memory bank comprises one or more SRAM memory banks.
Shafiee, in combination with Burger and Han, teaches wherein the narrow memory bank comprises one or more SRAM memory banks (page 244, left column,
We present the first accelerator for sparse and weight sharing neural networks. Operating directly on compressed networks enables the large neural network models
to fit in on-chip SRAM, which results in 120× better energy savings compared to accessing from external DRAM; page 246, we store pointers in two SRAM banks and use the LSB of the address to select between banks. Pj and pj+1 will always be in different banks.)
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Burger and Han to add SRAM memory banks to the combination system of Burger and Han, as taught by Shafiee above. The modification would have been obvious because one of ordinary skill would be motivated to perform the required operations for one layer in parallel and maximize throughput for one layer, as suggested by Shafiee (page 4/13, IV. THE ISAAC PIPELINE, first paragraph).
As to claim 7, which incorporates the rejection of claim1, Burger, teaches wherein the tiles are logically arranged in a ring (paragraph [0095], a ring in that a series of acceleration components are traversed in a first direction to arrive at a final result).
However, Burger and Han fail to explicitly teach:
wherein executing the operations of the one or more respective layers of the neural network comprises providing an output of one tile as an input activation to another tile.
Shafiee, in combination with Burger and Han, teaches:
wherein executing the operations of the one or more respective layers of the neural network comprises providing an output of one tile as an input activation to another tile (page 4/13, IV. THE ISAAC PIPELINE, left column, first paragraph, “All the NFUs in the system are leveraged to perform the required operations for one layer in parallel. The synaptic weights for that layer are therefore scattered across eDRAM banks in all tiles. The outputs are stored in eDRAM banks and serve as inputs when the next layer begins its operation).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Burger and Han to add an input activation to another tile to the combination system of Burger and Han, as taught by Shafiee above. The modification would have been obvious because one of ordinary skill would be motivated to maximize throughput for one layer, as suggested by Shafiee (page 4/13, IV. THE ISAAC PIPELINE, first paragraph).
As to claim 8, which incorporates the rejection of claim1, Burger and Han fail to explicitly teach: wherein distributing the weights of the neural network comprises distributing all weights of the neural network across the wide memory bank of the plurality of tiles.
Shafiee, in combination with Burger and Han, teaches wherein distributing the weights of the neural network comprises distributing all weights of the neural network across the wide memory bank of the plurality of tiles (page 4/13, IV. THE ISAAC PIPELINE, left column, first paragraph, “All the NFUs in the system are leveraged to perform the required operations for one layer in parallel. The synaptic weights for that layer are therefore scattered across eDRAM banks in all tiles. The outputs are stored in eDRAM banks and serve as inputs when the next layer begins its operation).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Burger and Han to add weights’ distributing to the combination system of combination system of Burger and Han, as taught by Shafiee above. The modification would have been obvious because one of ordinary skill would be motivated to maximize throughput for one layer, as suggested by Shafiee (page 4/13, IV. THE ISAAC PIPELINE, first paragraph).
As to claim 9, Burger teaches a neural network accelerator comprising a plurality of tiles (paragraphs [0128], [0163], the acceleration component may have different configurable domains (e.g., tiles), some of which are being currently used and others which are not being currently used);
wherein each tile of the plurality of tiles comprises a plurality of operators, wherein each operator is configured to perform neural network computations
(paragraphs [0248]-[0251] The next two acceleration components (3606, 3608) perform more complex computations, referred to herein as "free form expressions" (FFE), compared to the computations performed by head component 3604; [0259] FFE acceleration components 3606 and 3608 may perform mathematical computations using feature values determined by head component 3604; [0261] FIG. 42, multiplication, addition and division operations);
wherein each tile of the plurality of tiles has a narrow memory bank and a separate wide memory bank shared by the plurality of operators of the tile, wherein the narrow memory bank includes one or more memory units, wherein the narrow memory bank is configured to store and distribute input activations to the plurality of operators at a first memory bandwidth (Fig. 58 elements 5804 (input memory (activations) (interpreted by Examiner as “narrow memory;” the activation memory is interpreted as the narrow memory because it stores the input activation)), wherein the wide memory bank includes one or more memory units and wherein the wide memory bank is configured to store and provide access to weights of the neural network to the plurality of operators at a second memory bandwidth that is greater than the first memory bandwidth(Fig. 58 elements 5804 (input memory (activations) (The (input memory (activations) is interpreted by Examiner as “narrow memory” because it stores the activations)) and 5008 (input memory (weights) (interpreted by Examiner as “wide memory” because it stores the weights); paragraphs [0330], first memory 5804 interpreted by Examiner as “narrow memory”)is used to buffer input activations data and third memory 5808 (interpreted by Examiner as “wide memory”) is used to buffer weights data; [0342]-[0343], configuring the acceleration component includes storing the weights, input activations and errors in the memory stack; and streaming the weights, input activations and errors to the storage elements of the neural engines; [0278] Server unit component 4602 optionally may include memory 4624 coupled to local acceleration component 4606. In an implementation, memory 4624 is external to (i.e., not included in the same integrated circuit package) local acceleration component 4606 and includes a low bandwidth memory, such as DIMMS 4626(interpreted by Examiner as “first memory bandwidth”; Examiner interpretation is based on paragraph [0013] of the original disclosure.). As used herein, a "low bandwidth memory" is a memory having a bandwidth less than about 50 GB/sec) (interpreted by Examiner as “first memory bandwidth”; Examiner interpretation is based on paragraph [0013] of the original disclosure). In other implementations, memory 4624 also may include a high bandwidth (interpreted by Examiner as “second memory bandwidth,” Examiner interpretation is based on paragraph [0013] of the original disclosure), high power memory technology, such as HMC 4628. As used herein, a high bandwidth, high power memory" is a memory having a memory bandwidth greater than about 50 GB/sec (interpreted by Examiner as “second memory bandwidth,” Examiner interpretation is based on paragraph [0013] of the original disclosure) and a power efficiency of less than about 20 MB/sec/mW.; [0279] Thus, server unit component 4602 includes acceleration component 4606 with high bandwidth, low power memory (memory stack 4620). In a hybrid implementation, server unit component 4602 includes acceleration component 4606 with both high bandwidth, low power memory (memory stack 4620) and low bandwidth memory (DIMMS 4626). In another hybrid implementation, server unit component 4602 includes acceleration component 4606 with both high bandwidth, low power memory (memory stack 4620), low bandwidth memory (DIMMS 4626), and high bandwidth, high power memory (HMC 4628).
Burger discloses (Fig. 58 elements 5804 (input memory (activations) (“narrow memory”) and 5008 (input memory (weights) (“wide memory”); wherein Examiner interpretation is based on paragraphs [0007] and [0056] of the specification). Burger further discloses weights that are distributed and pinned across multiple neural engines.
However, Burger fails to explicitly teach the method comprising:
distributing the weights of the neural network across the plurality of tiles such that each tile stores, in its wide memory bank, weights required to perform the computations of one or more respective layers of the neural network; and
executing, by each tile, the operations of one or more respective layers of the neural network including distributing the input activations stored in the narrow memory bank to the plurality of operators and accessing, by the tile to the one or more operators of the tile, respective weights required to perform the operations stored in the wide memory bank memory banks of the tile..
Han, in combination with Burger, teaches:
distributing the weights of the neural network across the plurality of tiles such that each tile stores, in its wide memory bank, weights required to perform the computations of one or more respective layers of the neural network (page 252, left column, VIII. COMPARISON WITH RELATED WORK, wherein Examiner interprets “distributes weights on 16 tiles, each tile with 4 eDRAM banks, thus has a peak memory bandwidth of 16 × 4 × (1024bit/8) × 606MHz = 4964GB/s”).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Burger and Henry to add weights distribution on tiles to the combination system of Burger and Henry, as taught by Shafiee above. The modification would have been obvious because one of ordinary skill would be motivated to have performance estimation based on the peak memory bandwidth, as suggested by Han ((page 252, left column, VIII. COMPARISON WITH RELATED WORK).
However, Burger and Han fail to explicitly teach:
executing, by each tile, the operations of one or more respective layers of the neural network including distributing the input activations stored in the narrow memory bank to the plurality of operators and accessing, by the tile to the one or more operators of the tile, respective weights required to perform the operations stored in the wide memory bank memory banks of the tile.
Shafiee, in combination with Burger and Han, teaches:
executing, by each tile, the operations of one or more respective layers of the neural network including distributing the input activations stored in the narrow memory bank to the plurality of operators and accessing, by the tile to the one or more operators of the tile, respective weights required to perform the operations stored in the wide memory bank memory banks of the tile (I. INTRODUCTION, connected chips (nodes), each made up of 16 tiles. The dominant data structures in CNNs and DNNs are the synaptic weight matrices that define each neuron layer. These are distributed across several eDRAM banks on multiple tiles/nodes. The computations involving these weights are brought to the eDRAM banks and executed on adjacent NFUs, thus achieving near data processing; page 2/13 II. BACKGROUND, right column, activation function, page 3/13, C. The DaDianNao Architecture A single DaDianNao [9] chip (node) is made up of 16 tiles and two central eDRAM banks connected by an on-chip fat-tree network. A tile is made up of a neural functional unit (NFU) and four eDRAM banks. An NFU has a pipeline with multiple parallel multipliers, a tree of adders, and a transfer function.
Tiling is used to maximize data reuse and reduce transfers in/out of eDRAM banks. Synaptic weights are distributed across all nodes/tiles and feed their local NFUs;
page 174/13, IV. THE ISAAC PIPELINE …The synaptic weights for that layer are therefore scattered across eDRAM banks in all tiles…weights from the eDRAM banks to its SRAM buffers; tiles 0-3 may be assigned to layer 0, tiles 4-11 may be assigned
to layer 1, and so on. In this case, tiles 0-3 would store all weights for layer 0 and perform all layer 0 computations in parallel; Figs 2 and 3).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Burger and Han to add memory banks weights storage to the combination system of Burger and Han, as taught by Shafiee above. The modification would have been obvious because one of ordinary skill would be motivated to perform the required operations for one layer in parallel and maximize throughput for one layer, as suggested by Shafiee (page 4/13, IV. THE ISAAC PIPELINE, first paragraph).
As to claim 10, which incorporates the rejection of claim 9, Burger teaches: providing, by each tile to the plurality of operators of the tile, input activations output from a previous layer or another tile (paragraphs [0314]-[0316]…using an activation function and generates output activations of Layer i. Typically, the activation function is a weighted sum of products, taking the input activation of each neuron and scaling it by a tunable weight parameter).
As to claim 11, which incorporates the rejection of claim 9, Burger teaches: wherein executing the operations of the one or more respective layers of the neural network comprises executing the operations of the neural network using only the weights distributed across the plurality of tiles (paragraphs [0332]-[0333], controller component 5704 tiles the matrix multiplication independently across multiple neural engines (5712, 5714, 5716, 5718, 5720). Each of neural engines (5712, 5714, 5716, 5718, 5720) reads in a subset of the weight matrix; the weights are distributed and pinned across multiple neural engines (5712, 5714, 5716, 5718, 5720)).
As to claim 12, which incorporates the rejection of claim 11, Burger and Han to explicitly teach: wherein executing the operations of the one or more respective layers of the neural network comprises executing the operations of the neural network without retrieving weights from a memory that is not local to any of the one or more tiles.
Shafiee, in combination with Burger and Han, teaches wherein executing the operations of the one or more respective layers of the neural network comprises executing the operations of the neural network without retrieving weights from a memory that is not local to any of the one or more tiles (I. INTRODUCTION, right column, first paragraph, The dominant data structures in CNNs and DNNs are the synaptic weight matrices that define each neuron layer. These are distributed across several eDRAM banks on multiple tiles/nodes. The computations involving these weights are brought to the eDRAM banks and executed on adjacent NFUs, thus achieving near data processing. This requires moving the outputs of the previous neuron layer to the relevant tiles so they can merge with co-located synaptic weights to produce the outputs of the current layer. The outputs are then routed to appropriate eDRAM banks so they can serve as inputs to the next layer. Most of the chip area is used to store synaptic weights in eDRAM…).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Burger and Han to add an execution of the operations of the neural network to the combination system of Burger and Han, as taught by Shafiee above. The modification would have been obvious because one of ordinary skill would be motivated to perform the required operations for one layer in parallel and maximize throughput for one layer, as suggested by Shafiee (page 4/13, IV. THE ISAAC PIPELINE, first paragraph).
As to claim 13, which incorporates the rejection of claim 9, Burger and Han fail to explicitly teach wherein the narrow memory bank comprises one or more SRAM memory banks.
Shafiee, in combination with Burger and Han, teaches wherein the narrow memory bank comprises one or more SRAM memory banks (page 244, left column,
We present the first accelerator for sparse and weight sharing neural networks. Operating directly on compressed networks enables the large neural network models
to fit in on-chip SRAM, which results in 120× better energy savings compared to accessing from external DRAM; page 246, we store pointers in two SRAM banks and use the LSB of the address to select between banks…different banks.)
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Burger and Han to add SRAM memory banks to the combination system of Burger and Han, as taught by Shafiee above. The modification would have been obvious because one of ordinary skill would be motivated to perform the required operations for one layer in parallel and maximize throughput for one layer, as suggested by Shafiee (page 4/13, IV. THE ISAAC PIPELINE, first paragraph).
As to claim 15, which incorporates the rejection of claim 9, Burger teaches wherein the tiles are logically arranged in a ring (paragraph [0095], a ring in that a series of acceleration components are traversed in a first direction to arrive at a final result).
However, Burger and Han fail to explicitly teach:
wherein executing the operations of the one or more respective layers of the neural network comprises providing an output of one tile as an input activation to another tile.
Shafiee in combination with Burger and Han, teaches:
wherein executing the operations of the one or more respective layers of the neural network comprises providing an output of one tile as an input activation to another tile (page 4/13, IV. THE ISAAC PIPELINE, left column, first paragraph, “All the NFUs in the system are leveraged to perform the required operations for one layer in parallel. The synaptic weights for that layer are therefore scattered across eDRAM banks in all tiles. The outputs are stored in eDRAM banks and serve as inputs when the next layer begins its operation).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Burger and Han to add an input activation to another tile to the combination system of Burger and Han, as taught by Shafiee above. The modification would have been obvious because one of ordinary skill would be motivated to maximize throughput for one layer, as suggested by Shafiee (page 4/13, IV. THE ISAAC PIPELINE, first paragraph).
As to claim 16, which incorporates the rejection of claim 9, Burger and Han fail to explicitly teach: wherein distributing the weights of the neural network comprises distributing all weights of the neural network across the wide memory bank of the plurality of tiles.
Shafiee, in combination with Burger and Han, teaches wherein distributing the weights of the neural network comprises distributing all weights of the neural network across the wide memory bank of the plurality of tiles (page 4/13, IV. THE ISAAC PIPELINE, left column, first paragraph, “All the NFUs in the system are leveraged to perform the required operations for one layer in parallel. The synaptic weights for that layer are therefore scattered across eDRAM banks in all tiles. The outputs are stored in eDRAM banks and serve as inputs when the next layer begins its operation).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Burger and Han to add weights’ distributing to the combination system of Burger and Han, as taught by Shafiee above. The modification would have been obvious because one of ordinary skill would be motivated to maximize throughput for one layer, as suggested by Shafiee (page 4/13, IV. THE ISAAC PIPELINE, first paragraph).
As to claim 17, Burger teaches one or more non-transitory computer storage media encoded with instructions that, when executed by a neural network accelerator comprising a plurality of tiles (paragraphs [0128], [0163], the acceleration component may have different configurable domains (e.g., tiles), some of which are being currently used and others which are not being currently used);
wherein each tile of the plurality of tiles comprises a plurality of operators, wherein each operator is configured to perform neural network computations (paragraphs [0248]-[0251] The next two acceleration components (3606, 3608) perform more complex computations, referred to herein as "free form expressions" (FFE), compared to the computations performed by head component 3604; [0259] FFE acceleration components 3606 and 3608 may perform mathematical computations using feature values determined by head component 3604; [0261] FIG. 42, multiplication, addition and division operations);
wherein each tile of the plurality of tiles has a narrow memory bank and a separate wide memory bank shared by the plurality of operators of the tile, wherein the narrow memory bank includes one or more memory units, wherein the narrow memory bank is configured to store and distribute input activations to the plurality of operators at a first memory bandwidth (Fig. 58 elements 5804 (input memory (activations) (interpreted by Examiner as “narrow memory;” the activation memory is interpreted as the narrow memory because it stores the input activation)), wherein the wide memory bank includes one or more memory units and wherein the wide memory bank is configured to store and provide access weights of the neural network to the plurality of operators at a second memory bandwidth that is greater than the first memory bandwidth, (Fig. 58 elements 5804 (input memory (activations) (The (input memory (activations) is interpreted by Examiner as “narrow memory” because it stores the activations)) and 5008 (input memory (weights) (interpreted by Examiner as “wide memory”) because it stores the weights); paragraphs [0330], first memory 5804 interpreted by Examiner as “narrow memory”)is used to buffer input activations data and third memory 5808 (interpreted by Examiner as “wide memory”) is used to buffer weights data; [0342]-[0343], configuring the acceleration component includes storing the weights, input activations and errors in the memory stack; and streaming the weights, input activations and errors to the storage elements of the neural engines; [0278] Server unit component 4602 optionally may include memory 4624 coupled to local acceleration component 4606. In an implementation, memory 4624 is external to (i.e., not included in the same integrated circuit package) local acceleration component 4606 and includes a low bandwidth memory, such as DIMMS 4626(interpreted by Examiner as “first memory bandwidth”; Examiner interpretation is based on paragraph [0013] of the original disclosure.). As used herein, a "low bandwidth memory" is a memory having a bandwidth less than about 50 GB/sec) (interpreted by Examiner as “first memory bandwidth”; Examiner interpretation is based on paragraph [0013] of the original disclosure). In other implementations, memory 4624 also may include a high bandwidth (interpreted by Examiner as “second memory bandwidth,” Examiner interpretation is based on paragraph [0013] of the original disclosure), high power memory technology, such as HMC 4628. As used herein, a high bandwidth, high power memory" is a memory having a memory bandwidth greater than about 50 GB/sec (interpreted by Examiner as “second memory bandwidth,” Examiner interpretation is based on paragraph [0013] of the original disclosure) and a power efficiency of less than about 20 MB/sec/mW.; [0279] Thus, server unit component 4602 includes acceleration component 4606 with high bandwidth, low power memory (memory stack 4620). In a hybrid implementation, server unit component 4602 includes acceleration component 4606 with both high bandwidth, low power memory (memory stack 4620) and low bandwidth memory (DIMMS 4626). In another hybrid implementation, server unit component 4602 includes acceleration component 4606 with both high bandwidth, low power memory (memory stack 4620), low bandwidth memory (DIMMS 4626), and high bandwidth, high power memory (HMC 4628).
Burger discloses (Fig. 58 elements 5804 (input memory (activations) (“narrow memory”) and 5008 (input memory (weights) (“wide memory”); wherein Examiner interpretation is based on paragraphs [0007] and [0056] of the specification). Burger further discloses weights that are distributed and pinned across multiple neural engines.
However, Burger fails to explicitly teach the method comprising:
distributing the weights of the neural network across the plurality of tiles such that each tile stores, in its wide memory bank, weights required to perform the computations of one or more respective layers of the neural network; and
executing, by each tile, the operations of one or more respective layers of the neural network including distributing the input activations stored in the narrow memory bank to the plurality of operators and accessing, by the tile to the one or more operators of the tile, respective weights required to perform the operations stored in the wide memory bank memory banks of the tile.
However, Burger and Henry fail to explicitly teach:
distributing the weights of the neural network across the plurality of tiles such that each tile stores, in its wide memory bank, weights required to perform the computations of one or more respective layers of the neural network; and
executing, by each tile, the operations of one or more respective layers of the neural network including distributing the input activations stored in the narrow memory bank to the plurality of operators and accessing, by the tile to the one or more operators of the tile, respective weights required to perform the operations stored in the wide memory bank memory banks of the tile.
Han, in combination with Burger, teaches:
distributing the weights of the neural network across the plurality of tiles such that each tile stores, in its wide memory bank, weights required to perform the computations of one or more respective layers of the neural network (page 252, left column, VIII. COMPARISON WITH RELATED WORK, wherein Examiner interprets “distributes weights on 16 tiles, each tile with 4 eDRAM banks, thus has a peak memory bandwidth of 16 × 4 × (1024bit/8) × 606MHz = 4964GB/s”).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Burger to add weights distribution on tiles to the combination system of Burger, as taught by Han above. The modification would have been obvious because one of ordinary skill would be motivated to have performance estimation based on the peak memory bandwidth, as suggested by Han ((page 252, left column, VIII. COMPARISON WITH RELATED WORK).
However, Burger and Han fail to explicitly teach:
executing, by each tile, the operations of one or more respective layers of the neural network including distributing the input activations stored in the narrow memory bank to the plurality of operators and accessing, by the tile to the one or more operators of the tile, respective weights required to perform the operations stored in the wide memory bank memory banks of the tile.
Shafiee, in combination with Burger and Han, teaches:
executing, by each tile, the operations of one or more respective layers of the neural network including distributing the input activations stored in the narrow memory bank to the plurality of operators and accessing, by the tile to the one or more operators of the tile, respective weights required to perform the operations stored in the wide memory bank memory banks of the tile (I. INTRODUCTION, connected chips (nodes), each made up of 16 tiles. The dominant data structures in CNNs and DNNs are the synaptic weight matrices that define each neuron layer. These are distributed across several eDRAM banks on multiple tiles/nodes. The computations involving these weights are brought to the eDRAM banks and executed on adjacent NFUs, thus achieving near data processing; page 2/13 II. BACKGROUND, right column, activation function, page 3/13, C. The DaDianNao Architecture A single DaDianNao [9] chip (node) is made up of 16 tiles and two central eDRAM banks connected by an on-chip fat-tree network. A tile is made up of a neural functional unit (NFU) and four eDRAM banks. An NFU has a pipeline with multiple parallel multipliers, a tree of adders, and a transfer function. Tiling is used to maximize data reuse and reduce transfers in/out of eDRAM banks. Synaptic weights are distributed across all nodes/tiles and feed their local NFUs; page 174/13, IV. THE ISAAC PIPELINE …The synaptic weights for that layer are therefore scattered across eDRAM banks in all tiles…weights from the eDRAM banks to its SRAM buffers; tiles 0-3 may be assigned to layer 0, tiles 4-11 may be assigned to layer 1, and so on. In this case, tiles 0-3 would store all weights for layer 0 and perform all layer 0 computations in parallel; Figs 2 and 3).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Burger and Han to add memory banks weights storage to the combination system of Burger and Han, as taught by Shafiee above. The modification would have been obvious because one of ordinary skill would be motivated to perform the required operations for one layer in parallel and maximize throughput for one layer, as suggested by Shafiee (page 4/13, IV. THE ISAAC PIPELINE, first paragraph).
As to claim 18, which incorporates the rejection of claim17, Burger teaches:
providing, by each tile to the plurality of operators of the tile, input activations output from a previous layer or another tile (paragraphs [0314]-[0316]…using an activation function and generates output activations of Layer i. Typically, the activation function is a weighted sum of products, taking the input activation of each neuron and scaling it by a tunable weight parameter).
As to claim 19, which incorporates the rejection of claim17, Burger teaches: wherein executing the operations of the one or more respective layers of the neural network comprises executing the operations of the neural network using only the weights distributed across the plurality of tiles (paragraphs [0332]-[0333], controller component 5704 tiles the matrix multiplication independently across multiple neural engines (5712, 5714, 5716, 5718, 5720). Each of neural engines (5712, 5714, 5716, 5718, 5720) reads in a subset of the weight matrix; the weights are distributed and pinned across multiple neural engines (5712, 5714, 5716, 5718, 5720)).
As to claim 20, which incorporates the rejection of claim 11, Burger and Han fail to explicitly teach: wherein executing the operations of the one or more respective layers of the neural network comprises executing the operations of the neural network without retrieving weights from a memory that is not local to any of the one or more tiles.
Shafiee, in combination with Burger and Han, teaches wherein executing the operations of the one or more respective layers of the neural network comprises executing the operations of the neural network without retrieving weights from a memory that is not local to any of the one or more tiles (I. INTRODUCTION, right column, first paragraph, The dominant data structures in CNNs and DNNs are the synaptic weight matrices that define each neuron layer. These are distributed across several eDRAM banks on multiple tiles/nodes. The computations involving these weights are brought to the eDRAM banks and executed on adjacent NFUs, thus achieving near data processing. This requires moving the outputs of the previous neuron layer to the relevant tiles so they can merge with co-located synaptic weights to produce the outputs of the current layer. The outputs are then routed to appropriate eDRAM banks so they can serve as inputs to the next layer. Most of the chip area is used to store synaptic weights in eDRAM).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Burger and Han to add an execution of the operations of the neural network to the combination system of Burger and Han, as taught by Shafiee above. The modification would have been obvious because one of ordinary skill would be motivated to perform the required operations for one layer in parallel and maximize throughput for one layer, as suggested by Shafiee (page 4/13, IV. THE ISAAC PIPELINE, first paragraph).
Claims 6 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Burger et al. (US 2016/0379115 A1, hereinafter referred to as Burger), in view of et Han et al. (“EIE: Efficient Inference Engine on Compressed Deep Neural Network,” hereinafter referred to as Han), and further in view of and Shafiee et al. (“ISAAC: A Convolutional Neural Network Accelerator with In-Situ Analog Arithmetic in Crossbars,” hereinafter referred to as Shafiee)), and Brothers et al. (US 2016/0350645 A1, hereinafter referred to as Brothers).
As to claim 6, which incorporates the rejection of claim1, Burger, Han and Shafiee fail to explicitly teach wherein distributing the weights of the neural network comprises distributing more than 100,000, more than 1,000,000, or more than 100,000,000 weights across the wide memory bank.
Bothers, in combination with Burger, Han and Shafiee, teaches wherein distributing the weights of the neural network comprises distributing more than 100,000, more than 1,000,000, or more than 100,000,000 weights across the wide memory bank (paragraph [0025], a typical neural network may perform on the order of gigaops per image, utilize 100s of millions to billions of weights).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Burger, Han and Shafiee to add weights distribution to the combination system of Burger, Han and Shafiee, as taught by Bothers above. The modification would have been obvious because one of ordinary skill would be motivated to improve computational efficiency of a neural network, as suggested by Bothers ([0025]).
As to claim 14, which incorporates the rejection of claim 9, Burger, Han and Shafiee fail to explicitly teach wherein distributing the weights of the neural network comprises distributing more than 100,000, more than 1,000,000, or more than 100,000,000 weights across the wide memory bank.
Bothers, in combination with Burger, Han and Shafiee, teaches wherein distributing the weights of the neural network comprises distributing more than 100,000, more than 1,000,000, or more than 100,000,000 weights across the wide memory bank (paragraph [0025], a typical neural network may perform on the order of gigaops per image, utilize 100s of millions to billions of weights).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Burger, Han and Shafiee to add weights distribution to the combination system of Burger, Han and Shafiee, as taught by Bothers above. The modification would have been obvious because one of ordinary skill would be motivated to improve computational efficiency of a neural network, as suggested by Bothers ([0025]).
Response to Applicant’s arguments
Applicant's arguments on file on 03/05/2026 with respect to prior art rejection of claims 1-16 have been considered and are not persuasive.
Rejections under 35 U.S.C. § 103
Argument (page 1):
Claim 1 as amended recites that "the narrow memory bank is configured to store and
distribute input activations to the plurality of operators at a first memory bandwidth, wherein the wide memory bank includes one or more memory units, and wherein the wide memory bank is configured to store and provide access to weights of the neural network to the plurality of operators at a second memory bandwidth that is greater than the first memory bandwidth."
Examiner’s response:
Examiner respectfully disagrees. Burger, Han, and Shafiee, teach this combination of features as explained in the rejection above.
Argument (page 2)
Applicant respectfully submits that the cited portion of Burger does not teach or suggest
"that "the narrow memory bank is configured to store and distribute input activations to the plurality of operators at a first memory bandwidth" or that "the wide memory bank is configured to store and provide access to weights of the neural network to the plurality of operators at a second memory bandwidth that is greater than the first memory bandwidth."
Rather, the cited portion of Burger merely describes distributing and pinning weights
"across multiple neural engines" and exchanging activations in a "shift-register-like fashion."
Examiner’s response:
Examiner respectfully disagrees. First, Burger teaches, in paragraph [0278], a low bandwidth memory, such as DIMMS 4626(interpreted by Examiner as “first memory bandwidth”; Examiner interpretation is based on paragraph [0013] of the original disclosure.). As used herein, a "low bandwidth memory" is a memory having a bandwidth less than about 50 GB/sec) (interpreted by Examiner as “first memory bandwidth”; Examiner interpretation is based on paragraph [0013] of the original disclosure). In other implementations, memory 4624 also may include a high bandwidth (interpreted by Examiner as “second memory bandwidth,” Examiner interpretation is based on paragraph [0013] of the original disclosure).
Second, Burger teaches, in [0003]-[0004], “The system further includes a plurality of neural engines configured on acceleration component die. The neural engines include logic to implement forward propagation and backpropagation stages of the deep neural network.” As taught in the art, forward propagation and backpropagation rely on specialized operator kernels to execute layer computations. Both forward and backpropagation are entirely based on mathematical operators and are defined by a sequence of mathematical operations.
Finally, Burger teaches, in [0342]-[0343], “configuring the acceleration component includes storing the weights, input activations and errors in the memory stack; and streaming the weights, input activations and errors to the storage elements of the neural engines.”
Argument (page 3)
Applicant respectfully submits that the cited portion of Henry does not teach or suggest
"that "the narrow memory bank is configured to store and distribute input activations to the plurality of operators at a first memory bandwidth" or that "the wide memory bank is configured to store and provide access to weights of the neural network to the plurality of operators at a second memory bandwidth that is greater than the first memory bandwidth."
The cited portion of Henry merely describes that, "in the normal mode, the weight RAM
124 has W rows ... and is 4096 words wide," and that "the weight RAM 124 concurrently
outputs 4096 weight words ... and the data RAM 122 concurrently outputs 4096 data words, to the 4096 wide NPU array 126," with mux-registers operating "as a single 4096-word rotator," which concerns memory depth, word width, and concurrent output to an NPU array."
Examiner’s response:
Examiner respectfully disagrees. Henry is no longer used due to the amendment of the independent claims with the striking of “each having a size or width that is greater than a size or width of the one or more memory units of the narrow memory bank.”
Argument (pages 3-4)
Applicant respectfully submits that the cited portion of Han does not teach or suggest
"distributing input activations stored in the narrow memory bank to the plurality of operators and obtaining, by the plurality of operators of the tile, respective weights required to perform the operations stored in the wide memory bank of the tile." Han also does not describe that "the narrow memory bank is configured to store and distribute input activations to the plurality of operators at a first memory bandwidth" or that "the wide memory bank is configured to store and provide access to weights of the neural network to the plurality of operators at a second memory bandwidth that is greater than the first memory bandwidth."
Rather, Han merely states that "DaDianNao distributes weights on 16 tiles, each tile with 4 eDRAM banks, thus has a peak memory bandwidth of 16 x 4 x (1024bit=8) x 606MHz = 4964GB/s," and that "[i ]ts performance on Mx Vis estimated based on the peak memory bandwidth because Mx Vis completely memory bound," contrasting this with EIE where "all weights fit in on-chip SRAM."."
Examiner’s response:
Examiner respectfully disagrees. Han, in combination with Burger, teaches:
"distributing input activations stored in the narrow memory bank to the plurality of operators and obtaining, by the plurality of operators of the tile, respective weights required to perform the operations stored in the wide memory bank of the tile" as explained in the rejection above. In page 244, left column, Han further teaches a method of both distributed storage and distributed computation to parallelize a sparsified layer across multiple PEs, which achieves load balance and good scalability.
Examiner interprets the “distributed computation to parallelize a sparsified layer across multiple PEs” to be performed by tile operators.
Argument (pages 4-5)
Applicant respectfully submits that the cited portion of Shafiee does not teach or suggest that "the narrow memory bank is configured to store and distribute input activations to the plurality of operators at a first memory bandwidth," nor that "the wide memory bank is configured to store and provide access to weights of the neural network to the plurality of operators at a second memory bandwidth that is greater than the first memory bandwidth," as recited in amended claim 1.
However, Shafiee does not disclose or suggest that "each tile of the plurality of tiles has a narrow memory bank and a separate wide memory bank shared by the plurality of operators of the tile," nor that "the narrow memory bank is configured to store and distribute input activations to the plurality of operators at a first memory bandwidth," nor that "the wide memory bank is configured to store and provide access to weights of the neural network to the plurality of operators at a second memory bandwidth that is greater than the first memory bandwidth," as expressly recited in amended claim 1. Although Shafiee discusses "peak memory bandwidth" and that weights are "scattered across eDRAM banks," it does not disclose two distinct memory banks within a tile having different relative bandwidths, much less a configuration in which the
"second memory bandwidth" is "greater than the first memory bandwidth," as recited in
amended claim 1.
Accordingly, Applicant respectfully submits that claim 1 and its dependent claims are in
condition for allowance. Independent claims 9 and 17 and their respective dependent claims are allowable for corresponding reasons.
Examiner’s response:
Examiner respectfully disagrees. Burger does disclose or suggest that "each tile of the plurality of tiles has a narrow memory bank and a separate wide memory bank shared by the plurality of operators of the tile, as shown in the rejection above.
Examiner respectfully submits that claim 1 and its dependent claims are not in
condition for allowance. Independent claims 9 and 17 and their respective dependent claims are not allowable for corresponding reasons.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ABABACAR SECK whose telephone number is (571)270-7146. The examiner can normally be reached Monday-Friday 8:00 A.M.-6:00 P.M..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Lamardo Viker be reached on 5712705871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ABABACAR SECK/Examiner, Art Unit 2122
/VIKER A LAMARDO/Supervisory Patent Examiner, Art Unit 2147
Prosecution Insights