Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Remarks
This Office Action is responsive to Applicants' Amendment filed on July 27, 2026, in which claims 1, 9, 10, 12, and 14-16 are currently amended. Claims 1-20 are currently pending.
Specification
Applicant's amendments made to the specification are acknowledged. Examiner’s objection to the specification are hereby withdrawn, as necessitated by Applicant’s amendments made to the specification.
Response to Arguments
The rejections to claims 15-20 under 35 U.S.C. § 112(b) are hereby withdrawn, as necessitated by applicant's amendments and remarks made to the rejections.
Applicant’s arguments with respect to rejection of claims 1-20 under 35 U.S.C. 101 based on amendment have been considered and are persuasive. The rejections to claims 1-20 under 35 U.S.C. § 101 are hereby withdrawn, as necessitated by applicant's amendments and remarks made to the rejections
Applicant’s arguments with respect to rejection of claims 1-20 under 35 U.S.C. 102/103 based on amendment have been considered and are persuasive. The arguments are moot in view of a new ground of rejection set forth below.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 15, 16, and 19 are rejected under U.S.C. §102(a)(1) as being anticipated by Kuang (“ESSA: Design of a programmable efficient sparse spiking neural network accelerator”, 2022).
PNG
media_image1.png
804
1998
media_image1.png
Greyscale
FIG. 4 of Kuang
PNG
media_image2.png
352
1004
media_image2.png
Greyscale
FIG. 6 of Kuang
Regarding claim 15, Kuang teaches A system, comprising: a mask generation circuit configured to generate, according to a plurality of activation values corresponding to a plurality of spikes, location information to a first memory circuit;([p. 2] "Vj(t) denotes the membrane potential of the jth neuron at the tth time step, Lj is the negative linear leakage, Ij is the potential integration of received input spikes, Vr is the reset voltage, Vth is the neuronal firing threshold, and yj is the spike output of the jth neuron" The generated spikes are packaged into spike packets and then sent to local input channel in PR. Two main optimization techniques are used to improve the execution performance of NSB. First, unutilized neurons are skipped. Second, double memory storage for membrane voltages works as a ping pong buffer, which actualizes simultaneous Vj(t − 1) READ operation of the jth neuron and Vk(t) WRITE operation of the kth neuron to realize pipeline execution with the highest throughput in NSB. [p. 5] "The masked WVj is assigned to the adder as the other operand. The weight mask MUX works as a filter to screen out the invalid weights (e.g., in the compressed weight mode if one row has less than 32 nonzero elements" Activation interpreted as membrane potential. See also FIG. 1. Kuang provides two reasonable readings of mask generation: masked zero inputs which are explicitly skipped in the NSB execution, and masked invalid weights. Compressed spike storage of ISV interpreted as first memory circuit)
second and third memory circuits that are configured as a ping-pong buffer to store first and second portions of a plurality of activation values respectively; ([p. 4] "The generated spikes are packaged into spike packets and then sent to local input channel in PR. Two main optimization techniques are used to improve the execution performance of NSB. First, unutilized neurons are skipped. Second, double memory storage for membrane voltages works as a ping pong buffer, which actualizes simultaneous Vj(t − 1) READ operation of the jth neuron and Vk(t) WRITE operation of the kth neuron to realize pipeline execution with the highest throughput in NSB.")
a counter configured as an activation address calculator to analyze, based on the location information, addresses of memory cells that store non-zero values in the second and third memory circuits;([p. 4] "The generated spikes are packaged into spike packets and then sent to local input channel in PR. Two main optimization techniques are used to improve the execution performance of NSB. First, unutilized neurons are skipped. Second, double memory storage for membrane voltages works as a ping pong buffer, which actualizes simultaneous Vj(t − 1) READ operation of the jth neuron and Vk(t) WRITE operation of the kth neuron to realize pipeline execution with the highest throughput in NSB." [p. 6] "If the value of the jth spike counter is equal to P = 512/8 = 64, it indicates that the compressed spikes have filled the allocated storage of the jth ISV" Kuang's incoming spikes are nonzero activation events. The IDU counts nonzero spike activations of the corresponding time step and that count is then used to tell the input buffer how many spike entries exist and therefore how many stored sparse activations the IB should process)
a plurality of processing circuits configured to perform, based on the location information, an activation operation on the first portion stored in the second memory circuit and on the second portion stored in the third memory circuit in alternation, each with a plurality of weight values to generate a plurality of first membrane potentials and a plurality of second membrane potentials to be stored in a fourth memory circuit, ([p. 4] "3-KB neuron memory (NM)in NSB stores neuron states, soma parameters, and other configurable parameters. NSB performs the residual LIF operations of postsynaptic neurons in pipeline manner. The generated spikes are packaged into spike packets and then sent to local input channel in PR. Two main optimization techniques are used to improve the execution performance of NSB. First, unutilized neurons are skipped. Second, double memory storage for membrane voltages works as a ping pong buffer, which actualizes simultaneous Vj(t − 1) READ operation of the jth neuron and Vk(t) WRITE operation of the kth neuron to realize pipeline execution with the highest throughput in NSB." 3-KB neuron memory NM interpreted as fourth memory circuit)
wherein the second memory circuit is further configured to store a plurality of first output values corresponding to the plurality of second membrane potentials generated based on the second portion of the plurality of activation values stored in the third memory circuit([p. 4] "3-KB neuron memory (NM)in NSB stores neuron states, soma parameters, and other configurable parameters. NSB performs the residual LIF operations of postsynaptic neurons in pipeline manner. The generated spikes are packaged into spike packets and then sent to local input channel in PR. Two main optimization techniques are used to improve the execution performance of NSB. First, unutilized neurons are skipped. Second, double memory storage for membrane voltages works as a ping pong buffer, which actualizes simultaneous Vj(t − 1) READ operation of the jth neuron and Vk(t) WRITE operation of the kth neuron to realize pipeline execution with the highest throughput in NSB.")
and the third memory circuit is further configured to store a plurality of second output values corresponding to the plurality of first membrane potentials generated based on the first portion of the plurality of activation values stored in the second memory circuit([p. 4] "3-KB neuron memory (NM)in NSB stores neuron states, soma parameters, and other configurable parameters. NSB performs the residual LIF operations of postsynaptic neurons in pipeline manner. The generated spikes are packaged into spike packets and then sent to local input channel in PR. Two main optimization techniques are used to improve the execution performance of NSB. First, unutilized neurons are skipped. Second, double memory storage for membrane voltages works as a ping pong buffer, which actualizes simultaneous Vj(t − 1) READ operation of the jth neuron and Vk(t) WRITE operation of the kth neuron to realize pipeline execution with the highest throughput in NSB.").
Regarding claim 16, Kuang teaches The system of claim 15, further comprising: a controller circuit configured to send a control signal associated with locations to the plurality of processing circuits to read non-zero values in the plurality of activation values as the first and second portions of the plurality of activation values.(Kuang [p. 4] "If a row has a nonzero element in the input spike vector (ISV), IB actives the weight accesses of this row. Otherwise, IB skips the memory access [...] 3-KB neuron memory (NM)in NSB stores neuron states, soma parameters, and other configurable parameters. NSB performs the residual LIF operations of postsynaptic neurons in pipeline manner. The generated spikes are packaged into spike packets and then sent to local input channel in PR. Two main optimization techniques are used to improve the execution performance of NSB. First, unutilized neurons are skipped. Second, double memory storage for membrane voltages works as a ping pong buffer, which actualizes simultaneous Vj(t − 1) READ operation of the jth neuron and Vk(t) WRITE operation of the kth neuron to realize pipeline execution with the highest throughput in NSB.").
Regarding claim 19, Kuang teaches The system of claim 15, wherein the plurality of weight values are divided into N groups, and the plurality of processing circuits are further configured to perform the activation operation on the first portion of the plurality of activation values with one group, in the N groups, of weight values to generate the plurality of first membrane potentials, (Kuang [p. 2] "ESSA supports both compressed and uncompressed weight computations, thus optimizing the performance of SNNs in terms of their weight sparsity features. Compressed weight matrices are stored and computed in the format of compressed sparse row (CSR) in order to match ESSA’s event-driven manner." See also FIG. 1 and FIG. 2)
wherein the system further comprises: a neuron core circuit configured to generate an output spike based on the plurality of first membrane potentials. (Kuang [p. 2] "yj is the spike output of the jth neuron").
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 4, 5, 7, 8, 9, 10, and 14 are rejected under U.S.C. §103 as being unpatentable over the combination of Sommer (“Efficient Hardware Acceleration of Sparsely Active Convolutional Spiking Neural Networks”, 2022) and Tapiador-Morales (“Neuromorphic LIF Row-by-Row Multiconvolution Processor for FPGA”, 2019).
Regarding claim 1, Sommer teaches A method, comprising: receiving an input in an input layer of a spiking neural network during a plurality of time steps, wherein the input includes a plurality of spikes;([p. 1 §1] "the outputs of the neurons (activations) are not encoded with real-valued scalars like in standard NNs, but with sequences of binary events called spikes" [p. 2] "A binary spike from the previous layer l−1 arrives at the synapse i of a neuron j and is weighted with the synaptic weight wi" [p. 6] "an SNN is processed layer by layer. Each layer of an SNN needs to be simulated for multiple time steps T [...] for t<-0 to T do Simulate layer for all time steps T" Sommer explicitly uses binary spikes (spike = 1))
generating, based on a plurality of activation values corresponding to the plurality of spikes, location information including a plurality of count numbers each corresponding to non-zero values, in the plurality of activation values, in one of a plurality of rows of the input;([p. 3] "Each output of a neuron (i.e. activation) is a pixel of the resulting 2D output image called a feature map (fmap). A convolutional layer l typically generates a number of Cl output fmaps called channels which can be interpreted as a third dimension. The channels of the input fmap Cl−1 dictate the number of channels of each kernel Kl. The number of kernels kl determine the number of output channels Cl, i.e. Cl = kl. For standard spiking convolutional layers, the binary input fmap [...] is convolved with a number of kl kernels" [pp. 4-5] "The AER of a spike is simply the spike's (i,j) coordinates in the 2D fmap [...] A 2D input fmap is stored as a queue of address events (the AEQ)" [p. 6] "The queues in the 9 columns can be filled in parallel. The 9 parallel write accesses are necessary since thresholding is performed in a 3×3 window. The write logic features 9 write counters, one for each column" Sommer's device generates and stores location information for nonzero spikes by converting binary spike activations into address events. Each spike is represented by its "(i,j) coordinates" and these address events are stored in the AEQ as the compressed representation of the activation feature map. The write counters in Sommer explicitly track non-zero values (spikes))
performing, based on the location information, a matrix multiplication with the non-zero values in a first number of rows in the plurality of rows with a first group of filters of weight values to generate a plurality of first membrane potentials, ([p. 3] "Each output of a neuron (i.e. activation) is a pixel of the resulting 2D output image called a feature map (fmap). A convolutional layer l typically generates a number of Cl output fmaps called channels which can be interpreted as a third dimension. The channels of the input fmap Cl−1 dictate the number of channels of each kernel Kl. The number of kernels kl determine the number of output channels Cl, i.e. Cl = kl. For standard spiking convolutional layers, the binary input fmap [...] is convolved with a number of kl kernels [...] This results in the following equations, with ∗ denoting the convolution operation [See Eqn. 3]" While Sommer discloses that they can optimize out multiplication they explicitly teach that this is only because with the binary spikes simply summing the spikes is mathematically equivalent to performing multiplication ([p. 2] "The spikes that encode neuron activations are binary in nature. Weighting the binary activations ∈ {1,0} does not require an actual multiplication, as the multiplication reduces to: 1 · w = w and 0 · w = 0". In other words Sommer explicitly teaches that their implementation is not an approximation but rather the exact multiplicative convolution formula performed in the instant. Multiplication by 1 optimization as performed in Sommer is still mathematically multiplication.)
stored in a membrane buffer memory circuit,([p. 3771] "The Membrane Potential memory (MemPot) for storing the Vm of all neurons" [p. 3772] "Dual-port RAMs (such as BRAMs present on most FPGA chips" [p. 3773] "(Read MemPot): The convolution unit reads the 9 membrane potentials from the calculated addresses. Also, the permutated kernel will be selected here")
for outputting a first output spike at a first time;([p. 3768] "Once a membrane potential Vm exceeds a given threshold Vt, the neuron fires a spike itself […] SNNs operate in an event-driven manner. They update their internal state Vm only at those time steps t when an event (i.e., a spike) is presented at one of its inputs")
performing, based on the location information, the matrix multiplication with the non-zero values in a second number of rows in the plurality of rows with the first group of filters of weight values to generate a plurality of second membrane potentials for outputting a second output spike at a second time after the first time.([p. 5] "For each address event, 9 neurons (highlighted in white) can be updated in parallel, due to the 3x3 neighbourhood of the kernel […] all spike events are located inside the AEQ [...] the respective weights of the kernel can be added to the neuron potentials by rotating the kernel [...] a 3x3 kernel requires updating the neuron potentials at position (i,j) and all 8 neighboring membrane potentials" [p. 2] "When a membrane potential Vm exceeds the threshold Vt, then the neuron fires a spike itself and Vm is reset to 0. The membrane potential of a neuron j at layer l at each time step t" [p. 6] "an SNN is processed layer by layer. Each layer of an SNN needs to be simulated for multiple time steps T [...] for t<-0 to T do Simulate layer for all time steps T" Sommer's disclosed operation repeats across algorithmic time steps t. The device continues applying the same kernel weights K via the convolution unit and continues thresholding Vm to generate output spikes indexed by time t where t+1 corresponds to a second time after the first time (or similarly t corresponds to a second time after the first time t-1 [See Eqn. 1]).).
However, Sommer does not explicitly teach releasing a row-allocated portion of the membrane buffer memory circuit that stores the plurality of first membrane potentials after the first output spike is output and prior to completing the matrix multiplication for all groups of filters of weight values in the layer.
Tapiador-Morales, in the same field of endeavor, teaches releasing a row-allocated portion of the membrane buffer memory circuit that stores the plurality of first membrane potentials after the first output spike is output and prior to completing the matrix multiplication for all groups of filters of weight values in the layer ([p. 161] "BRAM is divided into four different memory banks: membrane potential (MP) […] Memory banks for MP, LT and RT are organized in multiple blocks that store values in rows of 8 pixels [...] X address selects memory blocks to be accessed, using a decoder to enable or disable the banks" [p. 164] "In the convolution operation, when applying a kernel value to a neuron, its membrane potential may reach the threshold. In that case, this neuron must fire, resetting its membrane potential value" [p. 161] "the membrane potential and kernel values are read and processed row by row until the entire kernel is applied" Tapiador-Morales discloses a BRAM-based membrane state buffer whose membrane-potential values are organized in row-based portions, the MemPot being one of the portions which is selectively disactivated/released, and where when a neuron in the processed row fires its output address is placed the output FIFO and its stored membrane potential value is reset/released).
Sommer as well as Tapiador-Morales are directed towards BRAM based spiking neural networks. Therefore, Sommer as well as Tapiador-Morales are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Sommer with the teachings of Tapiador-Morales by using and releasing BRAM based MemPot rows in operation of the spiking neural network. Tapiador-Morales provides as additional motivation for combination ([p. 161] "The main novelty of the design presented in this paper is that data from memory is accessed row by row, reducing the latency to convolve a square kernel"). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 4, the combination of Sommer and Tapiador-Morales teaches The method of claim 1, wherein the location information further includes a plurality of channel numbers and a plurality of height numbers, wherein each of the channel numbers and a corresponding height number correspond to a location of the one of the plurality of rows.(Sommer [pp. 4-5] "the AER of a spike is simply the spike's (i,j) coordinates in the 2D fmap" [p. 6] "Each element is addressed uniquely by its address (i,j) and its column s ∈ (0,...,8)" 2D fmap being a channel having height coordinate i.).
Regarding claim 5, the combination of Sommer and Tapiador-Morales teaches The method of claim 1, further comprising: performing, based on the location information, the matrix multiplication with the non-zero values in the first number of rows in the plurality of rows with a second group of filters of weight values to generate a plurality of third membrane potentials for outputting a third output spike at a third time after the second time.(Sommer [p. 5] "For each address event, 9 neurons (highlighted in white) can be updated in parallel, due to the 3x3 neighbourhood of the kernel […] all spike events are located inside the AEQ [...] the respective weights of the kernel can be added to the neuron potentials by rotating the kernel [...] a 3x3 kernel requires updating the neuron potentials at position (i,j) and all 8 neighboring membrane potentials" [p. 2] "When a membrane potential Vm exceeds the threshold Vt, then the neuron fires a spike itself and Vm is reset to 0. The membrane potential of a neuron j at layer l at each time step t" [p. 6] "an SNN is processed layer by layer. Each layer of an SNN needs to be simulated for multiple time steps T [...] for t<-0 to T do Simulate layer for all time steps T" Sommer explicitly supports multiple kernels/filters selected by channel and channel-wise sequential SNN processing).
Regarding claim 7, the combination of Sommer, and Tapiador-Morales teaches 7. The method of claim 5, further comprising: performing, based on the location information, the matrix multiplication with the non-zero values in the first number of rows in the plurality of rows with a third group of filters of weight values to generate a plurality of fourth membrane potentials for outputting a fourth output spike at a fourth time after the third time.(Sommer [p. 5] "For each address event, 9 neurons (highlighted in white) can be updated in parallel, due to the 3x3 neighbourhood of the kernel […] all spike events are located inside the AEQ [...] the respective weights of the kernel can be added to the neuron potentials by rotating the kernel [...] a 3x3 kernel requires updating the neuron potentials at position (i,j) and all 8 neighboring membrane potentials" [p. 2] "When a membrane potential Vm exceeds the threshold Vt, then the neuron fires a spike itself and Vm is reset to 0. The membrane potential of a neuron j at layer l at each time step t" [p. 6] "an SNN is processed layer by layer. Each layer of an SNN needs to be simulated for multiple time steps T [...] for t<-0 to T do Simulate layer for all time steps T" Sommer explicitly supports multiple kernels/filters selected by channel and channel-wise sequential SNN processing).
Regarding claim 8, the combination of Sommer, and Tapiador-Morales teaches The method of claim 1, further comprising: generating a neural network result, based on the first and second output spikes, for an image recognition operation of the input.(Sommer [p. 2] "To perform inference on a single sample (e.g. an input image)" [p. 3] " the SNN should provide a satisfactory classification" image classification interpreted as synonymous with an image recognition operation).
Regarding claim 9, Sommer teaches A non-transitory computer-readable medium for storing computer-executable instructions, the computer-executable instructions when executed by a processor implementing a method comprising the following steps:([p. 4] "We start by providing a top-level overview of the hardware architecture and then proceed to show how it can be implemented efficiently on either FPGAs or ASICs")
(a) reading a plurality of input feature maps corresponding to a plurality of time steps from a memory device;([p. 6] "The output fmap of each channel is represented by its own AEQ. These AEQs can be implemented in a single dual-port RAM since each individual AEQ is processed sequentially […] Vm<-ConvolutionUnit(AEQ[cin,l-1,t],K[cout,cin,l], Vm" Sommer stores feature maps in AEQs and explicitly states the AEQs are implementable in RAM (memory). The dataflow in Algorithm 1 explicitly reads AEQ indexed by time step t, i.e., feature-map content corresponding to time steps is retrieved from memory for processing)
(b) performing an activation operation with one of groups of filters of weight values and a number of layers in the plurality of input feature maps to generate corresponding membrane potentials in a plurality of membrane potentials([p. 5] "The convolution core updates the MemPot memory depending on the address events. […] Note that K refers to all kernels of the SNN, thus for each convolution the correct kernel must be selected depending on the current layer l, the current input channel cin and output channel cout." [p. 6] "Vm<-ConvolutionUnit(AEQ[cin,l-1,t],K[cout,cin,l], Vm"" Sommers "activation operation" is the event-based convolution update that updates membrane potentials (MemPot) using kernel weights K. Sommer is explicit that kernels (filters) exist across the SNN and the correct kernel is selected based on layer and channels and Algorithm 1 shows membrane potential generation and update via ConvolutionUnit)
stored in a membrane buffer memory circuit,([p. 3771] "The Membrane Potential memory (MemPot) for storing the Vm of all neurons" [p. 3772] "Dual-port RAMs (such as BRAMs present on most FPGA chips" [p. 3773] "(Read MemPot): The convolution unit reads the 9 membrane potentials from the calculated addresses. Also, the permutated kernel will be selected here")
wherein the one of groups of filters of weight values and the number of the layers in the plurality of input feature maps correspond to one of the plurality of time steps;([p. 6] "Each layer of an SNN needs to be simulated for multiple time steps T")
(c) when the one of the plurality of time steps in step (b) is not an initial time step in the plurality of time steps, updating the corresponding membrane potentials in step (b) by adding up with membrane potentials corresponding to a previous time step;([p. 2] "A binary spike from the previous layer l−1 arrives at the synapse i of a neuron j and is weighted with the synaptic weight wi. The weighted spike is then integrated (i.e. added) into the neurons membrane potential V l mj . When a membrane potential Vm exceeds the threshold Vt, then the neuron fires a spike itself and Vm is reset to 0. The membrane potential of a neuron j at layer l at each time step t is described as [See Eqn. 1]" See also Algorithm 1 where Vm<-ConvolutionUnit is inside the "for t ← 0 to T do" loop. Sommers neuron model is explicitly time-discrete and additive where the membrane potential at the current step is formed from the previous membrane potential plus added weight spikes.)
(d) generating one of a plurality of output spikes to provide a neural network result; and([Abstract] "neuronal outputs (i.e. activations) are not encoded with real-valued activations but with sequences of binary spikes")
(f) repeating steps (b) to (e) until the activation operation is performed to all groups of filters of weight values and all of the layers in the plurality of input feature maps.([p. 6] "Here, an SNN is processed layer by layer. Each layer of an SNN needs to be simulated for multiple time steps T. […] Each output channel of a layer is simulated for all time steps t, one channel after the other." [p. 6] "The sparse output activations are stored in the AEQ [...] The output fmap of each channel is represented by its own AEQ").
However, Sommer does not explicitly teach (e) eliminating a row-allocated portion of the corresponding membrane potentials from the membrane buffer memory circuit after generating the one of the plurality of output spikes and before the activation operation is performed to all groups of filters of weight values; .
Tapiador-Morales, in the same field of endeavor, teaches (e) eliminating a row-allocated portion of the corresponding membrane potentials from the membrane buffer memory circuit after generating the one of the plurality of output spikes and before the activation operation is performed to all groups of filters of weight values; ([p. 161] "BRAM is divided into four different memory banks: membrane potential (MP) […] Memory banks for MP, LT and RT are organized in multiple blocks that store values in rows of 8 pixels [...] X address selects memory blocks to be accessed, using a decoder to enable or disable the banks" [p. 164] "In the convolution operation, when applying a kernel value to a neuron, its membrane potential may reach the threshold. In that case, this neuron must fire, resetting its membrane potential value" [p. 161] "the membrane potential and kernel values are read and processed row by row until the entire kernel is applied" Tapiador-Morales discloses a BRAM-based membrane state buffer whose membrane-potential values are organized in row-based portions, the MemPot being one of the portions which is selectively disactivated/released, and where when a neuron in the processed row fires its output address is placed the output FIFO and its stored membrane potential value is reset/released).
Sommer as well as Tapiador-Morales are directed towards BRAM based spiking neural networks. Therefore, Sommer as well as Tapiador-Morales are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Sommer with the teachings of Tapiador-Morales by using and releasing BRAM based MemPot rows in operation of the spiking neural network. Tapiador-Morales provides as additional motivation for combination ([p. 161] "The main novelty of the design presented in this paper is that data from memory is accessed row by row, reducing the latency to convolve a square kernel"). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 10, the combination of Sommer and Tapiador-Morales teaches The non-transitory computer-readable medium of claim 9, wherein the method further comprises the following steps: (g) generating, based on the plurality of input feature maps, location information; and(Sommer [p. 3] "Each output of a neuron (i.e. activation) is a pixel of the resulting 2D output image called a feature map (fmap). A convolutional layer l typically generates a number of Cl output fmaps called channels which can be interpreted as a third dimension. The channels of the input fmap Cl−1 dictate the number of channels of each kernel Kl. The number of kernels kl determine the number of output channels Cl, i.e. Cl = kl. For standard spiking convolutional layers, the binary input fmap [...] is convolved with a number of kl kernels" [pp. 4-5] "The AER of a spike is simply the spike's (i,j) coordinates in the 2D fmap [...] A 2D input fmap is stored as a queue of address events (the AEQ)" [p. 6] "The queues in the 9 columns can be filled in parallel. The 9 parallel write accesses are necessary since thresholding is performed in a 3×3 window. The write logic features 9 write counters, one for each column" Sommer's device generates and stores location information for nonzero spikes by converting binary spike activations into address events. Each spike is represented by its "(i,j) coordinates" and these address events are stored in the AEQ as the compressed representation of the activation feature map. The write counters in Sommer explicitly track non-zero values (spikes))
(h) reading non-zero values in locations indicated by the location information for step (b).(Sommer [p. 5] "The AEQ stores the address events that are read by the convolution core. The convolution core updates the MemPot memory depending on the address events.").
Regarding claim 14, the combination of Sommer and Tapiador-Morales teaches The non-transitory computer-readable medium of claim 9, wherein the method further comprises a step: (g) eliminating the membrane potentials corresponding to the number of the layers associated with a time step before the previous time step.(Sommer [p. 9] "If any of these two condition is true, then the spike indicator bit of the respective neuron is set to 1. It is only set back to 0 if a new sample has to be processed" Setting back to zero (eliminating the membrane potential) when a new sample (later timestep) has to be processed).
Claims 2, 3, and 6 are rejected under U.S.C. §103 as being unpatentable over the combination of Sommer and Tapiador-Morales and in further view of Chen (“Skydiver: A Spiking Neural Network Accelerator Exploiting Spatio-Temporal Workload Balance”, 2022).
Regarding claim 2, the combination of Sommer and Tapiador-Morales teaches The method of claim 1.
However, the combination of Sommer and Tapiador-Morales doesn't explicitly teach, wherein the first number is different from the second number.
Chen, in the same field of endeavor, teaches The method of claim 1, wherein the first number is different from the second number. ([p. 5733] "arithmetic operations can be saved if connections without a spike are skipped. Moreover, the spike rate differs across timesteps, indicating that the proportion of active neurons varies over time").
The combination of Sommer and Tapiador-Morales as well as Chen are directed towards spiking neural networks for convolution. Therefore, the combination of Sommer and Tapiador-Morales as well as Chen are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Sommer and Tapiador-Morales with the teachings of Chen by building cluster list C corresponding to different numbers of non-zero activations. Chen provides as additional motivation for combination ([p. 5735] “When CBWS has applied alone, this design only achieved a 54.37% balance ratio. After combining with the APRC mechanism, the ratio was improved to 95.69%”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 3, the combination of Sommer and Tapiador-Morales teaches The method of claim 1.
However, the combination of Sommer and Tapiador-Morales doesn't explicitly teach, wherein the first number is smaller than the second number.
Chen, in the same field of endeavor, teaches The method of claim 1, wherein the first number is smaller than the second number. ([p. 5733] "arithmetic operations can be saved if connections without a spike are skipped. Moreover, the spike rate differs across timesteps, indicating that the proportion of active neurons varies over time" While one of ordinary skill in the art would recognize that if the two numbers are different, one must necessarily be smaller than the other, FIG. 4 of Chen explicitly shows a second subsequent output channel producing a smaller number of spikes (2) over the same inference window as a first output channel producing 6 spikes.).
The combination of Sommer and Tapiador-Morales as well as Chen are directed towards spiking neural networks for convolution. Therefore, the combination of Sommer and Tapiador-Morales as well as Chen are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Sommer and Tapiador-Morales with the teachings of Chen by building cluster list C corresponding to different numbers of non-zero activations. Chen provides as additional motivation for combination ([p. 5735] “When CBWS has applied alone, this design only achieved a 54.37% balance ratio. After combining with the APRC mechanism, the ratio was improved to 95.69%”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 6, the combination of Sommer and Tapiador-Morales teaches The method of claim 5.
However, the combination of Sommer and Tapiador-Morales doesn't explicitly teach wherein a number of filters in the first group equals to a number of filters in the second group.
Chen, in the same field of endeavor, teaches The method of claim 5, wherein a number of filters in the first group equals to a number of filters in the second group. (See Algorithm 1. list C which is bounded by number of SPEs in a cluster, where one filter per iteration (for i=0;i<K) is operated on.).
The combination of Sommer and Tapiador-Morales as well as Chen are directed towards spiking neural networks for convolution. Therefore, the combination of Sommer and Tapiador-Morales as well as Chen are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Sommer and Tapiador-Morales with the teachings of Chen by building cluster list C corresponding to different numbers of non-zero activations. Chen provides as additional motivation for combination ([p. 5735] “When CBWS has applied alone, this design only achieved a 54.37% balance ratio. After combining with the APRC mechanism, the ratio was improved to 95.69%”). This motivation for combination also applies to the remaining claims which depend on this combination.
Claim 11 is rejected under U.S.C. §103 as being unpatentable over the combination of Sommer and Tapiador-Morales and Pal (“OuterSPACE: An Outer Product based Sparse Matrix Multiplication Accelerator”, 2018).
Regarding claim 11, the combination of Sommer and Tapiador-Morales teaches The non-transitory computer-readable medium of claim 10.
However, the combination of Sommer and Tapiador-Morales doesn't explicitly teach, wherein the location information includes a plurality of count numbers each corresponding to the non-zero values in one of a plurality of rows,
a plurality of channel numbers and a plurality of height numbers,
wherein each of the channel numbers and a corresponding height number correspond to a location of the one of the plurality of rows.
Pal, in the same field of endeavor, teaches The non-transitory computer-readable medium of claim 10, wherein the location information includes a plurality of count numbers each corresponding to the non-zero values in one of a plurality of rows, ([p. 3 §3.2] "The vals array consists of the non-zero elements of the matrix in row-major order")
a plurality of channel numbers and a plurality of height numbers, ([p. 3 §3.2] "the cols array contains the column indices of the elements in vals")
wherein each of the channel numbers and a corresponding height number correspond to a location of the one of the plurality of rows.([p. 3 §3.2] "the row-ptrs array contains pointers to the start of each row of the matrix").
The combination of Sommer and Tapiador-Morales as well as Pal are directed towards sparse matrix multiplication acceleration. Therefore, the combination of Sommer and Tapiador-Morales as well as Pal are reasonably pertinent analogous art. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Sommer and Tapiador-Morales with the teachings of Pal by using compressed sparse row (CSR) format for the convolution filters. While CSR is well established in the art and would be an obvious design choice for one of ordinary skill in the art, this is explicitly reinforced by Pal who provides as additional motivation for combination ([p. 10] “greater use of local memory during the merge phase”).
Claims 12 and 13 are rejected under U.S.C. §103 as being unpatentable over the combination of Sommer and Tapiador-Morales and Weaver (“Memory-Side Acceleration and Sparse Compression for Quantized Packed Convolutions”, 2022).
Regarding claim 12, the combination of Sommer and Tapiador-Morales teaches The non-transitory computer-readable medium of claim 10.
However, the combination of Sommer and Tapiador-Morales doesn't explicitly teach wherein the step (g) comprises a step: (i) counting the non-zero values in a plurality of rows of the plurality of input feature maps to generate a plurality of count numbers included in the location information.
Weaver, in the same field of endeavor, teaches The non-transitory computer-readable medium of claim 10, wherein the step (g) comprises a step: (i) counting the non-zero values in a plurality of rows of the plurality of input feature maps to generate a plurality of count numbers included in the location information.([p. 2] "Any compression scheme involves at the very least storing the non-zero values along with at least one index to compute it’s original position within the dense matrix. CSR, for example, stores the column index of a value in a row as well as the cumulative number of non-zero values in each row" [p. 3] "the nnz array stores the number of values in each partition. Therefore, while the lengths of both the vals and offset arrays correspond to the total number of non-zero values in the dense matrix" See also Algorithm 2 where filter is explicitly packed).
The combination of Sommer and Tapiador-Morales as well as Weaver are directed towards acceleration of convolutional neural networks. Therefore, the combination of Sommer and Tapiador-Morales as well as Weaver are reasonably pertinent analogous art. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Sommer and Tapiador-Morales with the teachings of Weaver by using the PSR format for the filters. Weaver provides as additional motivation for combination ([p. 6] “As the filter size increases, however, sparse CONV with PSR performs increasingly better than the dense version”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 13, the combination of Sommer, Tapiador-Morales, and Weaver teaches The non-transitory computer-readable medium of claim 12, wherein in step (b) the activation operation is performed in a number of cycles on rows including the non-zero values,(Sommer [p. 3] "Each output of a neuron (i.e. activation) is a pixel of the resulting 2D output image called a feature map (fmap). A convolutional layer l typically generates a number of Cl output fmaps called channels which can be interpreted as a third dimension. The channels of the input fmap Cl−1 dictate the number of channels of each kernel Kl. The number of kernels kl determine the number of output channels Cl, i.e. Cl = kl. For standard spiking convolutional layers, the binary input fmap [...] is convolved with a number of kl kernels [...] This results in the following equations, with ∗ denoting the convolution operation [See Eqn. 3]" [p. 5] "Because all spike events are located inside the AEQ, the clock cycles required to perform the convolution scale directly with the number of spikes, i.e., one clock cycle per event" the binary spikes (the spike being nonzero) simply summing the spikes is mathematically equivalent to performing multiplication ([p. 2] "The spikes that encode neuron activations are binary in nature. Weighting the binary activations ∈ {1,0} does not require an actual multiplication, as the multiplication reduces to: 1 · w = w and 0 · w = 0".)
wherein the number of cycles is associated with count numbers corresponding to the rows including the non-zero values.(Sommer [p. 5] "Because all spike events are located inside the AEQ, the clock cycles required to perform the convolution scale directly with the number of spikes, i.e., one clock cycle per event" [p. 6] "The AEQ is not only a data structure but also features two independent circuits: one for writing and one for reading the queue columns").
Claims 17, 18, and 20 are rejected under U.S.C. §103 as being unpatentable over the combination of Kuang and Sommer.
Regarding claim 17, Kuang teaches The system of claim 16.
However, Kuang doesn't explicitly teach wherein the first portion of the plurality of activation values are included in two rows of a plurality of input feature maps corresponding to one of a plurality of time steps, and
the second portion of the plurality of activation values is included in two rows of the plurality of input feature maps corresponding to another time step of the plurality of time steps.
Sommer, in the same field of endeavor, teaches the first portion of the plurality of activation values are included in two rows of a plurality of input feature maps corresponding to one of a plurality of time steps, and([pp. 4-5] "The AER of a spike is simply the spike's (i,j) coordinates in the 2D fmap [...] A 2D input fmap is stored as a queue of address events (the AEQ)" [p. 6] "The AEQ is not only a data structure but also features two independent circuits: one for writing and one for reading the queue columns" The portions included in any two rows of the fmap (shown as 5x5 in FIG. 7) interpreted as the portion included in two rows of a plurality of fmaps corresponding to one of time steps t to T.)
the second portion of the plurality of activation values is included in two rows of the plurality of input feature maps corresponding to another time step of the plurality of time steps.([pp. 4-5] "The AER of a spike is simply the spike's (i,j) coordinates in the 2D fmap [...] A 2D input fmap is stored as a queue of address events (the AEQ)" [p. 6] "The AEQ is not only a data structure but also features two independent circuits: one for writing and one for reading the queue columns" The portions included in any two rows of the fmap (shown as 5x5 in FIG. 7) interpreted as the portion included in two rows of a plurality of fmaps corresponding to one of time steps t to T.).
Kuang as well as Sommer are directed towards spiking neural network circuits. Therefore, Kuang as well as Sommer are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Kuang with the teachings of Sommer by applying the circuit in Kuang to convolutional neural networks as disclosed in Sommer. Sommer provides as additional motivation for combination ([p. 3277] “our approach is agnostic to the CSNN’s architecture and can be implemented on an ASIC as well. The FPGA prototype developed in this work shows a very promising performance”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 18, Kuang teaches The system of claim 16.
However, Kuang doesn't explicitly teach, wherein the first portion of the plurality of activation values are included in three rows of a plurality of input feature maps corresponding to one of a plurality of time steps, and the second portion of the plurality of activation values is included in three rows of the plurality of input feature maps corresponding to another time step of the plurality of time steps.
Sommer, in the same field of endeavor, teaches the first portion of the plurality of activation values are included in three rows of a plurality of input feature maps corresponding to one of a plurality of time steps, and the second portion of the plurality of activation values is included in three rows of the plurality of input feature maps corresponding to another time step of the plurality of time steps.([p. 4] "the proposed architecture is optimized for 3 × 3 kernels" [p. 6] "The 9 parallel write accesses are necessary since thresholding is performed in a 3×3 window" [p. 6] "Each layer of an SNN needs to be simulated for multiple time steps T […] Each layer of an SNN needs to be simulated for multiple time steps T" See also FIG. 7 which explicitly shows the feature map interlacing scheme).
Kuang as well as Sommer are directed towards spiking neural network circuits. Therefore, Kuang as well as Sommer are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Kuang with the teachings of Sommer by applying the circuit in Kuang to convolutional neural networks as disclosed in Sommer. Sommer provides as additional motivation for combination ([p. 3277] “our approach is agnostic to the CSNN’s architecture and can be implemented on an ASIC as well. The FPGA prototype developed in this work shows a very promising performance”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 20, Kuang teaches The system of claim 19.
However, Kuang doesn't explicitly teach wherein each of the N groups includes M number of filters, and a number of the plurality of processing circuits is associated with a product of the number M and a dimension of the filters.
Sommer, in the same field of endeavor, teaches The system of claim 19, wherein each of the N groups includes M number of filters, and a number of the plurality of processing circuits is associated with a product of the number M and a dimension of the filters.([p. 2] "the multiplication reduces to: 1 · w = w and 0 · w = 0" [p. 3] "Each output of a neuron (i.e. activation) is a pixel of the resulting 2D output image called a feature map (fmap). A convolutional layer l typically generates a number of Cl output fmaps called channels which can be interpreted as a third dimension. The channels of the input fmap Cl−1 dictate the number of channels of each kernel Kl. The number of kernels kl determine the number of output channels Cl, i.e. Cl = kl. For standard spiking convolutional layers, the binary input fmap [...] is convolved with a number of kl kernels [...] This results in the following equations, with ∗ denoting the convolution operation [See Eqn. 3]" Examiner notes that M is not restricted such that it would be very reasonable to interpret M=1.).
Kuang as well as Sommer are directed towards spiking neural network circuits. Therefore, Kuang as well as Sommer are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Kuang with the teachings of Sommer by applying the circuit in Kuang to convolutional neural networks as disclosed in Sommer. Sommer provides as additional motivation for combination ([p. 3277] “our approach is agnostic to the CSNN’s architecture and can be implemented on an ASIC as well. The FPGA prototype developed in this work shows a very promising performance”). This motivation for combination also applies to the remaining claims which depend on this combination.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SIDNEY VINCENT BOSTWICK whose telephone number is (571)272-4720. The examiner can normally be reached M-F 7:30am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached on (571)270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SIDNEY VINCENT BOSTWICK/Examiner, Art Unit 2124
