Detailed Action
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This Office action is Non-Final and is in response to claims filed on 09/19/2023 via amendment. Claims 1-11 are pending for examination. Claims 1-9 are as originally filed. Claim 10 is currently amended. Claim 11 is newly presented.
Claim Objections
Claims 5 and 10 are objected to because of the following informalities:
Claim 5 recites “and wherein compute cores are configured to compute a plurality of output feature map values simultaneously without reloading weight values”. This should be changed to “and wherein the compute cores are configured to compute a plurality of output feature map values simultaneously without reloading weight values”.
Claim 10 recites “a plurality of adjacent current input feature map values from the one or more memory devices into respective multiply and accumulate units: performing corresponding multiply and accumulate operations using”. The colon should be changed to a semicolon, “a plurality of adjacent current input feature map values from the one or more memory devices into respective multiply and accumulate units; performing corresponding multiply and accumulate operations using”
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 9 is rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract ideas without significantly more.
With regards to claim 9, at step 1, the claim is directed to a method, which is a statutory category of invention.
At Step 2A Prong 1, the examiner notes that the claim is directed to mental processes and/or mathematical concepts. The claim language has been reproduced below:
A method comprising: (mental process, evaluation)
receiving, by a plurality of multi-accumulator multiply-and-accumulate (MAC) units, a first matrix and a second matrix; (mental process, evaluation; mathematical relationship)
multiplying, by the plurality of multi-accumulator multiply-and-accumulate (MAC) units, element values of the first matrix with a plurality of element values of the second matrix to generate a corresponding plurality of partial products; and (mathematical calculation)
accumulating, by the plurality of multi-accumulator multiply-and-accumulate (MAC) units, the plurality of partial products in respective accumulators of the multi-accumulator multiply-and-accumulate (MAC) units (mathematical calculation).
Each of the non-bolded limitations are mental processes and/or mathematical calculations. The “A method comprising” limitation is an evaluation mental process that can be performed by choosing what the method comprises. The “a first matrix and a second matrix” limitation is an evaluation mental process and mathematical relationship that can be performed by choosing what the MAC units receive. The “multiplying” limitation is a mathematical calculation that can be performed by multiplying the element values by hand using pen and paper. The “accumulating” limitation is a mathematical calculation that can be performed by accumulating the partial products by hand using pen and paper.
At step 2A Prong 2, the additional elements are bolded above. The “receiving” limitations, as claimed under BRI, are additional elements that are insignificant extra-solution activity. The “receiving” in the context of the claim encompasses mere data gathering. The remaining additional elements amount to no more than components comprising mere instructions to apply the exception and do not integrate the judicial exception into a practical application. See MPEP 2106.05(f). At Step 2B, the claim recites “receiving, by a plurality of multi-accumulator multiply-and-accumulate (MAC) units,”, and, per MPEP 2106.05(d) (Il), the courts have recognized the following computer functions as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity:
i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); and
iv. Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93.
Claims 1-8 and 10-11 integrate the abstract ideas into a practical application.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 5, 7, and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Knag et al. (US 20200097807 A1), hereinafter Knag in view of Srinivasa et al. (US 20220101091 A1), hereinafter Srinivasa.
With regards to claim 5, Knag teaches A memory processing unit (MPU) comprising: a first memory including a plurality of memory regions, (Knag [0035]: The Binary Neural Network accelerator 102 is implemented as a two level hierarchy with a static Random Access Memory 110 and controller 118 surrounded by an interleaved memory compute 112 that includes a latch-based Compute Near Memory array 204 interleaved with wide vector inner product execution units 202; Knag Fig. 2: Shows a plurality of latch memories)
wherein the plurality of memory regions are configured in corresponding pluralities of memory blocks; (Knag [0035]: The Binary Neural Network accelerator 102 is implemented as a two level hierarchy with a static Random Access Memory 110 and controller 118 surrounded by an interleaved memory compute 112 that includes a latch-based Compute Near Memory array 204 interleaved with wide vector inner product execution units 202; Knag Fig. 2: Shows the latch memory with a plurality of memory blocks)
and a plurality of processing regions interleaved between the plurality of regions of the first memory, (Knag [0035]: The Binary Neural Network accelerator 102 is implemented as a two level hierarchy with a static Random Access Memory 110 and controller 118 surrounded by an interleaved memory compute 112 that includes a latch-based Compute Near Memory array 204 interleaved with wide vector inner product execution units 202; Knag Fig. 2: Shows the vector inner product units between the latch memories)
wherein the processing regions include a [plurality of core groups] of compute cores configurable in one or more clusters, (Knag [0035]: The Binary Neural Network accelerator 102 is implemented as a two level hierarchy with a static Random Access Memory 110 and controller 118 surrounded by an interleaved memory compute 112 that includes a latch-based Compute Near Memory array 204 interleaved with wide vector inner product execution units 202)
wherein the [plurality of core groups] of respective ones of the plurality of processing regions are coupled between adjacent ones of the plurality of memory regions of the first memory (Knag [0035]: The Binary Neural Network accelerator 102 is implemented as a two level hierarchy with a static Random Access Memory 110 and controller 118 surrounded by an interleaved memory compute 112 that includes a latch-based Compute Near Memory array 204 interleaved with wide vector inner product execution units 202; Knag Fig. 2: Shows the vector inner product units coupled between the latch memories)
and between adjacent core groups coupled to respective adjacent memory regions, (Knag [0035]: The Binary Neural Network accelerator 102 is implemented as a two level hierarchy with a static Random Access Memory 110 and controller 118 surrounded by an interleaved memory compute 112 that includes a latch-based Compute Near Memory array 204 interleaved with wide vector inner product execution units 202; Knag Fig. 2: Shows that the vector inner product units are coupled between the latch memory)
and wherein compute cores are configured to compute a plurality of output feature map values simultaneously without reloading weight values (Knag [0035]: A set of weights is stored in vector latches 208, 210 located in the inner product execution units 202. The set of weights can be reused many times over the course of the convolution operation; Knag [0040]: The energy of data movement is reduced by decreasing average data bandwidth between memory and execution units. The highly parallel design with wide vector inner product execution units and two sets of local weights per execution unit has high input and weight reuse which in turn decreases the required bandwidth between the memory and execution units).
Knag fails to teach plurality of core groups.
However, Srinivasa teaches plurality of core groups (Srinivasa [0025]: embodiments of the present invention relate to DNN accelerators capable of near memory sparse matrix computation. An example DNN accelerator includes a multiplication controller, a buffer, two switches, and an array of process elements (PEs)).
Therefore, it would have been obvious before the effective filing date of the claimed invention for one of ordinary skill in the art to combine the teaching Knag with the plurality of core groups as taught by Srinivasa. One of ordinary skill in the art would be motivated to make this combination because the multiplication controller and switches can improve the efficiency of the DNN accelerator in matrix computation and avoids waste of computation resources on meaningless matrix computation operations as taught by Srinivasa (Srinivasa [0026]).
With Regards to claim 7, Knag in view of Srinivasa teaches all of the limitations of claim 5 above. Knag further teaches wherein a number of the plurality of compute cores in one or more of the [core groups] each include a multiply-and-accumulate (MAC) unit (Knag [0041]: FIG. 3 is a block diagram illustrating an embodiment of one of the wide vector inner product execution units 202 shown in FIG. 2. An inner product is a method to multiply two vectors, with the result of the multiplication being a scalar. The scaler result is the dot product of the co-ordinates of two vectors; Knag [0042]: In the example shown in FIG. 3, 1024 bit data vector X and 1024 bit weight vector W are multiplied by performing an Exclusive NOR operation in Exclusive NOR gates 302 and reducing the result of the Exclusive NOR operations outputs in reduction circuitry 304. The result of the reduction of the Exclusive NOR operations from reduction circuitry 304 is stored in accumulate circuitry 306)
including a multiplier and a [plurality] of accumulators (Knag [0041]: FIG. 3 is a block diagram illustrating an embodiment of one of the wide vector inner product execution units 202 shown in FIG. 2. An inner product is a method to multiply two vectors, with the result of the multiplication being a scalar. The scaler result is the dot product of the co-ordinates of two vectors; Knag [0042]: In the example shown in FIG. 3, 1024 bit data vector X and 1024 bit weight vector W are multiplied by performing an Exclusive NOR operation in Exclusive NOR gates 302 and reducing the result of the Exclusive NOR operations outputs in reduction circuitry 304. The result of the reduction of the Exclusive NOR operations from reduction circuitry 304 is stored in accumulate circuitry 306).
Knag fails to teach the core groups and a plurality [of accumulators].
However, Srinivasa teaches the core groups (Srinivasa [0025]: Embodiments of the present invention relate to DNN accelerators capable of near memory sparse matrix computation. An example DNN accelerator includes a multiplication controller, a buffer, two switches, and an array of process elements (PEs))
a plurality [of accumulators] (Srinivasa [0053]: FIG. 3 illustrates an architecture of an example PE 300, in accordance with various embodiments. The PE 300 is an embodiment of the PE 280 in FIG. 2. The PE 300 may be used in embodiments of the sparse DNN accelerator 220 where the compression module 230 compresses IFMs. As shown in FIG. 3, the PE 300 includes multipliers 310A-C (collectively referred to as “multipliers 310” or “multiplier 310”), a demultiplexer 320, and accumulators 330A-C (collectively referred to as “accumulators 330” or “accumulator 330”)).
Therefore, it would have been obvious before the effective filing date of the claimed invention for one of ordinary skill in the art to combine the teaching Knag in view of Srinivasa with the plurality of core groups and plurality of accumulators as taught by Srinivasa. One of ordinary skill in the art would be motivated to make this combination because the multiplication controller and switches can improve the efficiency of the DNN accelerator in matrix computation and avoids waste of computation resources on meaningless matrix computation operations as taught by Srinivasa (Srinivasa [0026]).
With regards to claim 9, Knag teaches A method comprising: receiving, by a plurality of [multi-accumulator] multiply-and-accumulate (MAC) units, a first matrix and a second matrix; (Knag [0041]; Knag [0042]: FIG. 3 is a block diagram illustrating an embodiment of one of the wide vector inner product execution units 202 shown in FIG. 2. An inner product is a method to multiply two vectors, with the result of the multiplication being a scalar. The scaler result is the dot product of the co-ordinates of two vectors; Knag [0045]: In the example shown in FIG. 3, 1024 bit data vector X and 1024 bit weight vector W are multiplied by performing an Exclusive NOR operation in Exclusive NOR gates 302 and reducing the result of the Exclusive NOR operations outputs in reduction circuitry 304. The result of the reduction of the Exclusive NOR operations from reduction circuitry 304 is stored in accumulate circuitry 306; FIGS. 4A-E illustrate an example of a striding operation in the image classifier 100 shown in FIG. 1. In the example shown, the stride is one and a 2×2 convolutional kernel is moved by the stride using a raster scan memory access pattern over a 3×3 input image from left to right. The intersection of the weight kernel with the image represents a single vector dot product with a scalar output; Knag [0060]: The non-linear sign activation function can be used to reduce output bandwidth further by quantizing the 16 bit accumulator value to a single bit. An embodiment with 128 parallel execution units and inner product width of 1024 bits balances input and output memory bandwidth)
multiplying, by the plurality of [multi-accumulator] multiply-and-accumulate (MAC) units, element values of the first matrix with a plurality of element values of the second matrix to generate a corresponding plurality of partial products; (Knag [0041]: FIG. 3 is a block diagram illustrating an embodiment of one of the wide vector inner product execution units 202 shown in FIG. 2. An inner product is a method to multiply two vectors, with the result of the multiplication being a scalar. The scaler result is the dot product of the co-ordinates of two vectors; Knag [0042]: In the example shown in FIG. 3, 1024 bit data vector X and 1024 bit weight vector W are multiplied by performing an Exclusive NOR operation in Exclusive NOR gates 302 and reducing the result of the Exclusive NOR operations outputs in reduction circuitry 304. The result of the reduction of the Exclusive NOR operations from reduction circuitry 304 is stored in accumulate circuitry 306; Knag [0045]: FIGS. 4A-E illustrate an example of a striding operation in the image classifier 100 shown in FIG. 1. In the example shown, the stride is one and a 2×2 convolutional kernel is moved by the stride using a raster scan memory access pattern over a 3×3 input image from left to right. The intersection of the weight kernel with the image represents a single vector dot product with a scalar output)
and accumulating, by the plurality of [multi-accumulator] multiply-and-accumulate (MAC) units, the plurality of partial products in respective accumulators of the [multi-accumulator] multiply-and-accumulate (MAC) units (Knag [0041]: FIG. 3 is a block diagram illustrating an embodiment of one of the wide vector inner product execution units 202 shown in FIG. 2. An inner product is a method to multiply two vectors, with the result of the multiplication being a scalar. The scaler result is the dot product of the co-ordinates of two vectors; Knag [0042]: In the example shown in FIG. 3, 1024 bit data vector X and 1024 bit weight vector W are multiplied by performing an Exclusive NOR operation in Exclusive NOR gates 302 and reducing the result of the Exclusive NOR operations outputs in reduction circuitry 304. The result of the reduction of the Exclusive NOR operations from reduction circuitry 304 is stored in accumulate circuitry 306; Knag [0045]: FIGS. 4A-E illustrate an example of a striding operation in the image classifier 100 shown in FIG. 1. In the example shown, the stride is one and a 2×2 convolutional kernel is moved by the stride using a raster scan memory access pattern over a 3×3 input image from left to right. The intersection of the weight kernel with the image represents a single vector dot product with a scalar output; Knag [0060]: The non-linear sign activation function can be used to reduce output bandwidth further by quantizing the 16 bit accumulator value to a single bit. An embodiment with 128 parallel execution units and inner product width of 1024 bits balances input and output memory bandwidth).
Knag fails to teach the multi-accumulator [multiply-and-accumulate (MAC) units,].
However, Srinivasa teaches the multi-accumulator [multiply-and-accumulate (MAC) units,] (Srinivasa [0053]: FIG. 3 illustrates an architecture of an example PE 300, in accordance with various embodiments. The PE 300 is an embodiment of the PE 280 in FIG. 2. The PE 300 may be used in embodiments of the sparse DNN accelerator 220 where the compression module 230 compresses IFMs. As shown in FIG. 3, the PE 300 includes multipliers 310A-C (collectively referred to as “multipliers 310” or “multiplier 310”), a demultiplexer 320, and accumulators 330A-C (collectively referred to as “accumulators 330” or “accumulator 330”)).
Therefore, it would have been obvious before the effective filing date of the claimed invention for one of ordinary skill in the art to combine the teaching Knag with the multi-accumulators as taught by Srinivasa. One of ordinary skill in the art would be motivated to make this combination because the multiplication controller and switches can improve the efficiency of the DNN accelerator in matrix computation and avoids waste of computation resources on meaningless matrix computation operations as taught by Srinivasa (Srinivasa [0026]).
Claims 1-2, 6, 8, and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Knag in view of Srinivasa further in view of Lovell et al. (US 20230108883 A1), hereinafter Lovell.
With regards to claim 1, Knag teaches A memory processing unit (MPU) comprising: a first memory including a plurality of memory regions; (Knag [0035]: The Binary Neural Network accelerator 102 is implemented as a two level hierarchy with a static Random Access Memory 110 and controller 118 surrounded by an interleaved memory compute 112 that includes a latch-based Compute Near Memory array 204 interleaved with wide vector inner product execution units 202; Knag Fig. 2: Shows a plurality of latch memories)
and a plurality of processing regions interleaved between the plurality of regions of the first memory, (Knag [0035]: The Binary Neural Network accelerator 102 is implemented as a two level hierarchy with a static Random Access Memory 110 and controller 118 surrounded by an interleaved memory compute 112 that includes a latch-based Compute Near Memory array 204 interleaved with wide vector inner product execution units 202; Knag Fig. 2: Shows the vector inner product units between the latch memories)
wherein the processing regions include a plurality of computer cores, (Knag [0035]: The Binary Neural Network accelerator 102 is implemented as a two level hierarchy with a static Random Access Memory 110 and controller 118 surrounded by an interleaved memory compute 112 that includes a latch-based Compute Near Memory array 204 interleaved with wide vector inner product execution units 202; Knag Fig. 2: Shows plurality of vector inner products)
wherein the plurality of compute cores are coupled between adjacent ones of the plurality of memory regions, (Knag [0035]: The Binary Neural Network accelerator 102 is implemented as a two level hierarchy with a static Random Access Memory 110 and controller 118 surrounded by an interleaved memory compute 112 that includes a latch-based Compute Near Memory array 204 interleaved with wide vector inner product execution units 202; Knag Fig. 2: Shows that the vector inner product units are coupled between the latch memory)
wherein the compute cores include [multi-accumulator] multiply-and-accumulate (MAC) unit configured to computer matrix dot products (Knag [0041]: FIG. 3 is a block diagram illustrating an embodiment of one of the wide vector inner product execution units 202 shown in FIG. 2. An inner product is a method to multiply two vectors, with the result of the multiplication being a scalar. The scaler result is the dot product of the co-ordinates of two vectors; Knag [0042]: In the example shown in FIG. 3, 1024 bit data vector X and 1024 bit weight vector W are multiplied by performing an Exclusive NOR operation in Exclusive NOR gates 302 and reducing the result of the Exclusive NOR operations outputs in reduction circuitry 304. The result of the reduction of the Exclusive NOR operations from reduction circuitry 304 is stored in accumulate circuitry 306)
wherein element values of a given matrix are reused by [time division multiplexing] computations of the matrix dot product (Knag [0035]: A set of weights is stored in vector latches 208, 210 located in the inner product execution units 202. The set of weights can be reused many times over the course of the convolution operation).
Knag fails to teach the multi-accumulator [multiply-and-accumulate (MAC) units,].
However, Srinivasa teaches the multi-accumulator [multiply-and-accumulate (MAC) units,] (Srinivasa [0053]: FIG. 3 illustrates an architecture of an example PE 300, in accordance with various embodiments. The PE 300 is an embodiment of the PE 280 in FIG. 2. The PE 300 may be used in embodiments of the sparse DNN accelerator 220 where the compression module 230 compresses IFMs. As shown in FIG. 3, the PE 300 includes multipliers 310A-C (collectively referred to as “multipliers 310” or “multiplier 310”), a demultiplexer 320, and accumulators 330A-C (collectively referred to as “accumulators 330” or “accumulator 330”)).
Therefore, it would have been obvious before the effective filing date of the claimed invention for one of ordinary skill in the art to combine the teaching Knag with the multi-accumulators as taught by Srinivasa. One of ordinary skill in the art would be motivated to make this combination because the multiplication controller and switches can improve the efficiency of the DNN accelerator in matrix computation and avoids waste of computation resources on meaningless matrix computation operations as taught by Srinivasa (Srinivasa [0026]).
Knag in view of Srinivasa fails to teach time division multiplexing.
However, Lovell teaches time division multiplexing (Lovell [0035]: data may be parallelly processed in hardware and then combined in the time domain, e.g., via time-division multiplexing or other time slicing methods).
Therefore, it would have been obvious before the effective filing date of the claimed invention for one of ordinary skill in the art to combine the teaching Knag in view of Srinivasa with the time division multiplexing as taught by Lovell. One of ordinary skill in the art would be motivated to make this combination because faster processing not only allows for reduced execution times, it also reduces leakage losses, advantageously, leading to a better utilization of computational resources while reducing power consumption as taught by Lovell (Lovell [0037]).
With regards to claim 2, Knag in view of Srinivasa further in view of Lovell teaches all of the limitations of claim 1 above. Knag further teaches wherein each [multi-accumulator] multiply-and-accumulate unit includes a multiplier (Knag [0041]: IG. 3 is a block diagram illustrating an embodiment of one of the wide vector inner product execution units 202 shown in FIG. 2. An inner product is a method to multiply two vectors, with the result of the multiplication being a scalar. The scaler result is the dot product of the co-ordinates of two vectors; Knag [0042]: In the example shown in FIG. 3, 1024 bit data vector X and 1024 bit weight vector W are multiplied by performing an Exclusive NOR operation in Exclusive NOR gates 302 and reducing the result of the Exclusive NOR operations outputs in reduction circuitry 304. The result of the reduction of the Exclusive NOR operations from reduction circuitry 304 is stored in accumulate circuitry 306)
and a [plurality] of accumulators configured to accumulate corresponding partial products of the matrix dot product computed by the multiplier in corresponding ones of a [plurality] of accumulators (Knag [0041]: FIG. 3 is a block diagram illustrating an embodiment of one of the wide vector inner product execution units 202 shown in FIG. 2. An inner product is a method to multiply two vectors, with the result of the multiplication being a scalar. The scaler result is the dot product of the co-ordinates of two vectors; Knag [0042]: In the example shown in FIG. 3, 1024 bit data vector X and 1024 bit weight vector W are multiplied by performing an Exclusive NOR operation in Exclusive NOR gates 302 and reducing the result of the Exclusive NOR operations outputs in reduction circuitry 304. The result of the reduction of the Exclusive NOR operations from reduction circuitry 304 is stored in accumulate circuitry 306).
Knag fails to teach the multi-accumulator [multiply-and-accumulate unit] and a plurality of accumulators.
However, Srinivasa teaches the multi-accumulator [multiply-and-accumulate unit] (Srinivasa [0053]: FIG. 3 illustrates an architecture of an example PE 300, in accordance with various embodiments. The PE 300 is an embodiment of the PE 280 in FIG. 2. The PE 300 may be used in embodiments of the sparse DNN accelerator 220 where the compression module 230 compresses IFMs. As shown in FIG. 3, the PE 300 includes multipliers 310A-C (collectively referred to as “multipliers 310” or “multiplier 310”), a demultiplexer 320, and accumulators 330A-C (collectively referred to as “accumulators 330” or “accumulator 330”)).
a plurality of accumulators (Srinivasa [0053]: FIG. 3 illustrates an architecture of an example PE 300, in accordance with various embodiments. The PE 300 is an embodiment of the PE 280 in FIG. 2. The PE 300 may be used in embodiments of the sparse DNN accelerator 220 where the compression module 230 compresses IFMs. As shown in FIG. 3, the PE 300 includes multipliers 310A-C (collectively referred to as “multipliers 310” or “multiplier 310”), a demultiplexer 320, and accumulators 330A-C (collectively referred to as “accumulators 330” or “accumulator 330”)).
Therefore, it would have been obvious before the effective filing date of the claimed invention for one of ordinary skill in the art to combine the teaching Knag in view of Srinivasa further in view of Lovell with the multi-accumulators as taught by Srinivasa. One of ordinary skill in the art would be motivated to make this combination because the multiplication controller and switches can improve the efficiency of the DNN accelerator in matrix computation and avoids waste of computation resources on meaningless matrix computation operations as taught by Srinivasa (Srinivasa [0026]).
With regards to claim 6, Knag in view of Srinivasa teaches all of the limitations of claim 5 above. Knag further teaches wherein at least one of the [plurality of core groups] include a plurality of near memory (M) compute cores, (Knag [0035]: The Binary Neural Network accelerator 102 is implemented as a two level hierarchy with a static Random Access Memory 110 and controller 118 surrounded by an interleaved memory compute 112 that includes a latch-based Compute Near Memory array 204 interleaved with wide vector inner product execution units 202)
wherein the plurality of near memory (M) compute cores each comprise a multiplier and a [plurality] of accumulators configured to [time multiplex] computation of a dot product of a first matrix and a second matrix (Knag [0041]: FIG. 3 is a block diagram illustrating an embodiment of one of the wide vector inner product execution units 202 shown in FIG. 2. An inner product is a method to multiply two vectors, with the result of the multiplication being a scalar. The scaler result is the dot product of the co-ordinates of two vectors; Knag [0042]: In the example shown in FIG. 3, 1024 bit data vector X and 1024 bit weight vector W are multiplied by performing an Exclusive NOR operation in Exclusive NOR gates 302 and reducing the result of the Exclusive NOR operations outputs in reduction circuitry 304. The result of the reduction of the Exclusive NOR operations from reduction circuitry 304 is stored in accumulate circuitry 306; Knag [0045]: FIGS. 4A-E illustrate an example of a striding operation in the image classifier 100 shown in FIG. 1. In the example shown, the stride is one and a 2×2 convolutional kernel is moved by the stride using a raster scan memory access pattern over a 3×3 input image from left to right. The intersection of the weight kernel with the image represents a single vector dot product with a scalar output)
wherein element values of the first matrix are reused (Knag [0035]: set of weights is stored in vector latches 208, 210 located in the inner product execution units 202. The set of weights can be reused many times over the course of the convolution operation; Knag [0040]: The energy of data movement is reduced by decreasing average data bandwidth between memory and execution units. The highly parallel design with wide vector inner product execution units and two sets of local weights per execution unit has high input and weight reuse which in turn decreases the required bandwidth between the memory and execution units).
Knag fails to teach the plurality of core groups and a plurality [of accumulators].
However, Srinivasa teaches the plurality of core groups (Srinivasa [0025]: Embodiments of the present invention relate to DNN accelerators capable of near memory sparse matrix computation. An example DNN accelerator includes a multiplication controller, a buffer, two switches, and an array of process elements (PEs))
a plurality [of accumulators] (Srinivasa [0053]: FIG. 3 illustrates an architecture of an example PE 300, in accordance with various embodiments. The PE 300 is an embodiment of the PE 280 in FIG. 2. The PE 300 may be used in embodiments of the sparse DNN accelerator 220 where the compression module 230 compresses IFMs. As shown in FIG. 3, the PE 300 includes multipliers 310A-C (collectively referred to as “multipliers 310” or “multiplier 310”), a demultiplexer 320, and accumulators 330A-C (collectively referred to as “accumulators 330” or “accumulator 330”)).
Therefore, it would have been obvious before the effective filing date of the claimed invention for one of ordinary skill in the art to combine the teaching Knag in view of Srinivasa with the plurality of core groups and plurality of accumulators as taught by Srinivasa. One of ordinary skill in the art would be motivated to make this combination because the multiplication controller and switches can improve the efficiency of the DNN accelerator in matrix computation and avoids waste of computation resources on meaningless matrix computation operations as taught by Srinivasa (Srinivasa [0026]).
Knag in view of Srinivasa fails to teach time multiplex [computation].
However, Lovell teaches time multiplex [computation] (Lovell [0035]: data may be parallelly processed in hardware and then combined in the time domain, e.g., via time-division multiplexing or other time slicing methods).
Therefore, it would have been obvious before the effective filing date of the claimed invention for one of ordinary skill in the art to combine the teaching Knag in view of Srinivasa with the time division multiplexing as taught by Lovell. One of ordinary skill in the art would be motivated to make this combination because faster processing not only allows for reduced execution times, it also reduces leakage losses, advantageously, leading to a better utilization of computational resources while reducing power consumption as taught by Lovell (Lovell [0037]).
With regards to claim 8, Knag in view of Srinivasa further in view of Lovell teaches all of the limitations of claim 6 above. Knag further teaches wherein: the multiplier is configured to compute corresponding partial products of the matrix dot product reusing a given element of first matrix; (Knag [0041]: FIG. 3 is a block diagram illustrating an embodiment of one of the wide vector inner product execution units 202 shown in FIG. 2. An inner product is a method to multiply two vectors, with the result of the multiplication being a scalar. The scaler result is the dot product of the co-ordinates of two vectors; Knag [0042]: In the example shown in FIG. 3, 1024 bit data vector X and 1024 bit weight vector W are multiplied by performing an Exclusive NOR operation in Exclusive NOR gates 302 and reducing the result of the Exclusive NOR operations outputs in reduction circuitry 304. The result of the reduction of the Exclusive NOR operations from reduction circuitry 304 is stored in accumulate circuitry 306; Knag [0035]: A set of weights is stored in vector latches 208, 210 located in the inner product execution units 202. The set of weights can be reused many times over the course of the convolution operation; Knag [0040]: The energy of data movement is reduced by decreasing average data bandwidth between memory and execution units. The highly parallel design with wide vector inner product execution units and two sets of local weights per execution unit has high input and weight reuse which in turn decreases the required bandwidth between the memory and execution units)
and the [plurality] of accumulators are configured to corresponding partial products of the matrix dot product computed by the multiplier (Knag [0041]: FIG. 3 is a block diagram illustrating an embodiment of one of the wide vector inner product execution units 202 shown in FIG. 2. An inner product is a method to multiply two vectors, with the result of the multiplication being a scalar. The scaler result is the dot product of the co-ordinates of two vectors; Knag [0042]: In the example shown in FIG. 3, 1024 bit data vector X and 1024 bit weight vector W are multiplied by performing an Exclusive NOR operation in Exclusive NOR gates 302 and reducing the result of the Exclusive NOR operations outputs in reduction circuitry 304. The result of the reduction of the Exclusive NOR operations from reduction circuitry 304 is stored in accumulate circuitry 306).
Knag fails to teach the plurality [of accumulators].
However, Srinivasa teaches the plurality [of accumulators] (Srinivasa [0053]: FIG. 3 illustrates an architecture of an example PE 300, in accordance with various embodiments. The PE 300 is an embodiment of the PE 280 in FIG. 2. The PE 300 may be used in embodiments of the sparse DNN accelerator 220 where the compression module 230 compresses IFMs. As shown in FIG. 3, the PE 300 includes multipliers 310A-C (collectively referred to as “multipliers 310” or “multiplier 310”), a demultiplexer 320, and accumulators 330A-C (collectively referred to as “accumulators 330” or “accumulator 330”)).
Therefore, it would have been obvious before the effective filing date of the claimed invention for one of ordinary skill in the art to combine the teaching Knag in view of Srinivasa further in view of Lovell with the plurality of core groups and plurality of accumulators as taught by Srinivasa. One of ordinary skill in the art would be motivated to make this combination because the multiplication controller and switches can improve the efficiency of the DNN accelerator in matrix computation and avoids waste of computation resources on meaningless matrix computation operations as taught by Srinivasa (Srinivasa [0026]).
With regards to claim 11, Knag in view of Srinivasa further in view of Lovell teaches all of the limitations of claim 1 above. Knag fails to teach wherein the time division multiplexing increases the number of channels of the multi-accumulator multiply-and-accumulate units without increasing the number of multi-accumulator multiply-and-accumulate units.
However, Lovell teaches wherein the time division multiplexing increases the number of channels of the multi-accumulator multiply-and-accumulate units without increasing the number of multi-accumulator multiply-and-accumulate units (Lovell [0035]: data may be parallelly processed in hardware and then combined in the time domain, e.g., via time-division multiplexing or other time slicing methods. As an example, one physical processor 215 that comprises 64 physical local processors, each processor communicatively coupled with memory 216, may process 64 channels of data. In embodiments, processor 215 may use 64 physical local processors to process up to 16 channels of data over 16 cycles of time, thereby, creating 16 virtual processors. It is understood that, processors 215 may comprise local memory and share the same basic architecture and logic that may reused over time and repurposed for different output channels).
Therefore, it would have been obvious before the effective filing date of the claimed invention for one of ordinary skill in the art to combine the teaching Knag in view of Srinivasa further in view of Lovell with the time division multiplexing as taught by Lovell. One of ordinary skill in the art would be motivated to make this combination because faster processing not only allows for reduced execution times, it also reduces leakage losses, advantageously, leading to a better utilization of computational resources while reducing power consumption as taught by Lovell (Lovell [0037]).
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Knag in view of Srinivasa further in view of Lovell further in view of Li et al. (US 11507814 B1), hereinafter Li.
With regards to claim 3, Knag in view of Srinivasa further in view of Lovell teaches all of the limitations of claim 2 above. Knag further teaches wherein the number of accumulators of each multi-accumulator multiply-and-accumulate unit [is configured based on a number of physical channels] (Knag [0041]: FIG. 3 is a block diagram illustrating an embodiment of one of the wide vector inner product execution units 202 shown in FIG. 2. An inner product is a method to multiply two vectors, with the result of the multiplication being a scalar. The scaler result is the dot product of the co-ordinates of two vectors; Knag [0042]: In the example shown in FIG. 3, 1024 bit data vector X and 1024 bit weight vector W are multiplied by performing an Exclusive NOR operation in Exclusive NOR gates 302 and reducing the result of the Exclusive NOR operations outputs in reduction circuitry 304. The result of the reduction of the Exclusive NOR operations from reduction circuitry 304 is stored in accumulate circuitry 306).
Knag fails to teach [wherein the number of accumulators of each multi-accumulator multiply-and-accumulate unit] is configured based on a number of physical channels.
However, Li teaches [wherein the number of accumulators of each multi-accumulator multiply-and-accumulate unit] is configured based on a number of physical channels (Li Column 11 Lines 34-36: The outputs of each column of PEs can be summed or added at an adder circuitry of the respective column).
Therefore, it would have been obvious before the effective filing date of the claimed invention for one of ordinary skill in the art to combine the teaching Knag in view of Srinivasa further in view of Lovell with the number of accumulators based on the number of physical channels as taught by Li. One of ordinary skill in the art would be motivated to make this combination because such frequent toggles or transitions of states of logic circuits can cause a large overall power consumption. In one aspect, to minimize the number of toggles, a sequence or order of computations can be rearranged as taught by Li (Li Column 13 lines 56-59). Also, this would increase the efficiency of the system, as it would not have too little or too many accumulators, which could slow down operations or increase power consumption.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Knag in view of Srinivasa further in view of Lovell further in view of Li further in view of Botimer et al. (US 20210011732 A1), hereinafter Botimer.
With regards to claim 4, Knag in view of Srinivasa further in view of Lovell further in view of Li teaches all of the limitations of claim 3 above. Knag fails to teach wherein the number of physical channels and the number of the plurality of computer cores configured to compute the matrix dot product are selected based on a specified computation utilization of the plurality of compute cores.
However, Botimer teaches wherein the number of physical channels and the number of the plurality of computer cores configured to compute the matrix dot product are selected based on a specified computation utilization of the plurality of compute cores (Botimer [0040]: the number of multiply and accumulate units and the number of buffer elements in each subset of the serial shift buffer can be based on the parameters of the weights matrix).
Therefore, it would have been obvious before the effective filing date of the claimed invention for one of ordinary skill in the art to combine the teaching Knag in view of Srinivasa further in view of Lovell further in view of Li with the number of channels and compute cores being based on the computation utilization as taught by Botimer. One of ordinary skill in the art would be motivated to make this combination because the data reuse embodiments can also advantageously reduce power consumption by the memory devices and or processing units as taught by Botimer (Botimer [0052]). Also, this would increase the efficiency of the system as it would not have too little or too many units which could slow down operations or increase power consumption.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Knag in view of Srinivasa further in view of Botimer.
With regards to claim 10, Knag in view of Srinivasa teaches all of the limitations of claim 9 above. Knag further teaches further comprising: loading a current weight value from the one or more memory devices into a plurality of multiply and accumulate units, (Knag [0035]: The Binary Neural Network accelerator 102 is implemented as a two level hierarchy with a static Random Access Memory 110 and controller 118 surrounded by an interleaved memory compute 112 that includes a latch-based Compute Near Memory array 204 interleaved with wide vector inner product execution units 202. The latch-based Compute Near Memory array 204 can also be referred to as a near memory latch array. A set of weights is stored in vector latches 208, 210 located in the inner product execution units 202. The set of weights can be reused many times over the course of the convolution operation. Storing two sets of weights in vector latches 208, 210 allows for 2 times more input data reuse compared to a single set of weights, and also reduces the input energy/switch-activity by a factor of 2; Knag [0041]: FIG. 3 is a block diagram illustrating an embodiment of one of the wide vector inner product execution units 202 shown in FIG. 2. An inner product is a method to multiply two vectors, with the result of the multiplication being a scalar. The scaler result is the dot product of the co-ordinates of two vectors; Knag [0042]: In the example shown in FIG. 3, 1024 bit data vector X and 1024 bit weight vector W are multiplied by performing an Exclusive NOR operation in Exclusive NOR gates 302 and reducing the result of the Exclusive NOR operations outputs in reduction circuitry 304. The result of the reduction of the Exclusive NOR operations from reduction circuitry 304 is stored in accumulate circuitry 306)
and a [plurality] of adjacent current input feature map values from the one or more memory devices into respective multiply and accumulate units: (Knag [0035]: The Binary Neural Network accelerator 102 is implemented as a two level hierarchy with a static Random Access Memory 110 and controller 118 surrounded by an interleaved memory compute 112 that includes a latch-based Compute Near Memory array 204 interleaved with wide vector inner product execution units 202. The latch-based Compute Near Memory array 204 can also be referred to as a near memory latch array. A set of weights is stored in vector latches 208, 210 located in the inner product execution units 202. The set of weights can be reused many times over the course of the convolution operation. Storing two sets of weights in vector latches 208, 210 allows for 2 times more input data reuse compared to a single set of weights, and also reduces the input energy/switch-activity by a factor of 2; Knag [0041]: FIG. 3 is a block diagram illustrating an embodiment of one of the wide vector inner product execution units 202 shown in FIG. 2. An inner product is a method to multiply two vectors, with the result of the multiplication being a scalar. The scaler result is the dot product of the co-ordinates of two vectors; Knag [0042]: In the example shown in FIG. 3, 1024 bit data vector X and 1024 bit weight vector W are multiplied by performing an Exclusive NOR operation in Exclusive NOR gates 302 and reducing the result of the Exclusive NOR operations outputs in reduction circuitry 304. The result of the reduction of the Exclusive NOR operations from reduction circuitry 304 is stored in accumulate circuitry 306)
performing corresponding multiply and accumulate operations using the current weight value and corresponding ones of the plurality current input feature map values to generate corresponding current accumulated values by the respective multiply and accumulate units; (Knag [0035]: The Binary Neural Network accelerator 102 is implemented as a two level hierarchy with a static Random Access Memory 110 and controller 118 surrounded by an interleaved memory compute 112 that includes a latch-based Compute Near Memory array 204 interleaved with wide vector inner product execution units 202. The latch-based Compute Near Memory array 204 can also be referred to as a near memory latch array. A set of weights is stored in vector latches 208, 210 located in the inner product execution units 202. The set of weights can be reused many times over the course of the convolution operation. Storing two sets of weights in vector latches 208, 210 allows for 2 times more input data reuse compared to a single set of weights, and also reduces the input energy/switch-activity by a factor of 2; Knag [0041]: FIG. 3 is a block diagram illustrating an embodiment of one of the wide vector inner product execution units 202 shown in FIG. 2. An inner product is a method to multiply two vectors, with the result of the multiplication being a scalar. The scaler result is the dot product of the co-ordinates of two vectors; Knag [0042]: In the example shown in FIG. 3, 1024 bit data vector X and 1024 bit weight vector W are multiplied by performing an Exclusive NOR operation in Exclusive NOR gates 302 and reducing the result of the Exclusive NOR operations outputs in reduction circuitry 304. The result of the reduction of the Exclusive NOR operations from reduction circuitry 304 is stored in accumulate circuitry 306)
iterating through corresponding input [channels] of input feature map and corresponding input [channels] of weights; (Knag [0053]: FIGS. 6A-E illustrate an example of a striding operation with a snake pattern in the image classifier 100 shown in FIG. 1. In order to reduce input memory access, the Binary Neural Network accelerator takes advantage of data reuse in the striding operation. In the example shown, the stride is one and a 2×2 convolutional kernel is moved by the stride using a snake memory access pattern over the input image. A portion of the input image having a width of 6 and height of 4 is shown in FIGS. 6A-6D. The intersection of the weight kernel with the input image represents a single vector dot product with a scalar output)
iterating through [kernel height and kernel width of weights,] and corresponding map width and map height in the input feature map; (Knag [0053]: FIGS. 6A-E illustrate an example of a striding operation with a snake pattern in the image classifier 100 shown in FIG. 1. In order to reduce input memory access, the Binary Neural Network accelerator takes advantage of data reuse in the striding operation. In the example shown, the stride is one and a 2×2 convolutional kernel is moved by the stride using a snake memory access pattern over the input image. A portion of the input image having a width of 6 and height of 4 is shown in FIGS. 6A-6D. The intersection of the weight kernel with the input image represents a single vector dot product with a scalar output)
outputting corresponding current accumulated values as corresponding [output feature map] values; (Knag [0040]: Incorporating very wide execution units to reduce the required output bandwidth by using accumulation to reduce the total number of outputs; Knag [0053]: FIGS. 6A-E illustrate an example of a striding operation with a snake pattern in the image classifier 100 shown in FIG. 1. In order to reduce input memory access, the Binary Neural Network accelerator takes advantage of data reuse in the striding operation. In the example shown, the stride is one and a 2×2 convolutional kernel is moved by the stride using a snake memory access pattern over the input image. A portion of the input image having a width of 6 and height of 4 is shown in FIGS. 6A-6D. The intersection of the weight kernel with the input image represents a single vector dot product with a scalar output)
[resetting the corresponding current accumulated values] and iterating through map width and map height of input feature map, and [corresponding kernel height and kernel width of weights;] (Knag [0053]: FIGS. 6A-E illustrate an example of a striding operation with a snake pattern in the image classifier 100 shown in FIG. 1. In order to reduce input memory access, the Binary Neural Network accelerator takes advantage of data reuse in the striding operation. In the example shown, the stride is one and a 2×2 convolutional kernel is moved by the stride using a snake memory access pattern over the input image. A portion of the input image having a width of 6 and height of 4 is shown in FIGS. 6A-6D. The intersection of the weight kernel with the input image represents a single vector dot product with a scalar output).
Knag fails to teach a plurality of adjacent current input feature map, iterating through corresponding input channels of input feature map and corresponding input channels of weights, iterating through kernel height and kernel width of weights, and corresponding map width and map height in the input feature map, outputting corresponding current accumulated values as corresponding output feature map values, resetting the corresponding current accumulated values [and iterating through map width and map height of input feature map, and] corresponding kernel height and kernel width of weights, and iterating through filters of weights.
However, Botimer teaches a plurality of adjacent current input feature map (Botimer [0033]: The operation can begin with loading a current weight value (0,0,0) and a plurality of adjacent input feature map values (0,0,0) and (0,1,0) from memory into respective multiply and accumulate units)
iterating through corresponding input channels of input feature map and corresponding input channels of weights; (Botimer [0036]: the operations at 710 and 720 can be iterated through corresponding input channels of the input feature map and corresponding input channels of the weights)
iterating through kernel height and kernel width of weights, and corresponding map width and map height in the input feature map; (Botimer [0036]: the operations at 710-730 can be iterated through the kernel height and kernel width of the weights, and the corresponding map width and map height in the input feature map)
outputting corresponding current accumulated values as corresponding output feature map values; (Botimer [0037]: the accumulated value in the second multiply and accumulate unit 610 can be output as a corresponding output feature map value)
resetting the corresponding current accumulated values [and iterating through map width and map height of input feature map, and] corresponding kernel height and kernel width of weights; (Botimer [0038]: At 760, the current accumulated values in the respective multiply and accumulate units 605, 610 can be reset and the operations at 710-750 can be iterated through the map width and map height of the input feature map and the corresponding kernel height and kernel width of the weights)
and iterating through filters of weights (Botimer [0038]: At 770, the operations at 710-760 can be iterated through the filters of the weights to generate the complete output feature map 625).
Therefore, it would have been obvious before the effective filing date of the claimed invention for one of ordinary skill in the art to combine the teaching Knag in view of Srinivasa with plurality of adjacent feature maps, the channels, the kernel height and width, the output feature map, resetting the accumulated values, and iterating through the filters as taught by Botimer. One of ordinary skill in the art would be motivated to make this combination because the data reuse embodiments can also advantageously reduce power consumption by the memory devices and or processing units as taught by Botimer (Botimer [0052]). Also, this would increase the efficiency of the system as it could reset the accumulators to allow for more data to be processed.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jakob O Gudas whose telephone number is (571)272-0695. The examiner can normally be reached Monday-Thursday: 7:30AM-5:00PM Friday: 7:30AM-4:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, James Trujillo can be reached at (571) 272-3677. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.O.G./Examiner, Art Unit 2151
/James Trujillo/Supervisory Patent Examiner, Art Unit 2151
Prosecution Insights