DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment/Argument
Applicant’s arguments filed on March 30, 2026 in regards to the rejection under 35. U.S.C. 103 have been fully considered but they are not persuasive. Applicant argues in the Remarks (pages 6-7) that the amended claim 1 recites a hybrid spiking neural network architecture that is not disclosed, taught, or reasonably suggested by the combination of Van der Made and Ngyen.
With respect to the Applicant’s first argument, Applicant argues that the amended claim 1 requires a neural core module including an input layer comprising an input neuron and a fully-parallel layer composed of neurons that processes input spikes in parallel. Applicant further argues that Van der Made’s inbound filter fails to disclose such an input layer and fully-parallel layer and instead merely performs spatial selection of incoming spikes. Applicant’s argument is not persuasive.
As set forth in the rejection, Van der Made teaches a spiking neural network architecture that receives and processes spike/events during neural network computations. Van der Made further teaches filtering spikes using “an inbound filter configured to selected relevant spikes from the generated spikes,” thereby reducing the number of spikes/events processed within the architecture. Van der Made additionally teaches parallel neural processing because each NP/CNP includes multiple parallel neural processing engines that update neurons and synapses in parallel and run in parallel across neural-network computations.
Further, claim 1 broadly recites a neural core module that, during a “first processing stage,” “filters the flow of spikes to generate a reduced number of spikes.” Claim 1 does not recite any particular filtering algorithm or exclude spike-selection filtering, spatial filtering, gating, or selective forwarding of spikes. Under the broadest reasonable interpretation, Van der Made’s inbound spike filtering reasonably corresponds to the claimed first-stage spike filtering because selecting relevant spikes from generated spikes necessarily filters the flow of spikes and results in a reduced number of spikes/events being further processed downstream within the neural network architecture.
Applicant’s argument is further not persuasive because the rejection does not rely on the inbound filter alone as teaching the entirety of the claimed neural core module. Rather, as set forth in the rejection, Van der Made is relied upon for spike/event input processing, filtering/selecting relevant spikes, and parallel neural processing architecture includes multiple neural processing engines that process neurons and synapses in parallel. Accordingly, under BRI, Van der Made’s parallel neural processing architecture reasonably corresponds to the claimed “fully-parallel layer composed of neurons that processes the input spikes in parallel.”
With respect to Applicant’s second argument, Applicant argues that Van der Made’s packet collection block fails to disclose the claimed neural processing unit including “a plurality of hidden time-multiplexed layers to sequentially process the reduced number of spikes” and further argues that the packet collection block merely performs formatting operations. Applicant’s argument is not persuasive.
Applicant’s argument improperly attacks the packet collection block in isolation rather than the combined teachings relied upon in the rejection. As set forth in the rejection, Van der Made is relied upon for teaching a spiking neural network architecture comprising multiple hidden neural-network layers between input and output layers, event-based spike processing, and activations/events being delivered between neural network layers. Van der Made further teaches that spikes/events are processed only when activations occur, thereby reducing processed events/spikes within the neural network architecture.
Ngyen is relied upon for the time-multiplexed sequential neural processing technique, including time multiplexing using reserved neuron time slots, round-robin neural processing, time-division multiplexing, and serial communication between intermediate neural network nodes.
Accordingly, the rejection does not rely on Van der Made’s packet collection block alone as corresponding to the entirety of the claimed neural processing unit. Rather, under BRI, the rejection relies on the combined teachings of Van der Made’s hidden multi-layer spiking neural network architecture together with Ngyen’s time-multiplexed sequential neural processing techniques to teach or at least suggest the claimed “neural processing unit” comprising a “plurality of hidden time-multiplexed layers to sequentially process the reduced number of spikes,” where Van der Made further teaches the claimed output layer within the neural network architecture.
With respect to Applicant’s third argument on pages 8-12 of the Remarks, Applicant argues that one of ordinary skill in the art would not have combined the teachings of Van der Made and Ngyen because Van der Made already includes multiple parallel neural-processing engines, whereas Ngyen employs time multiplexing to “mimic parallelism” on CGRA hardware. Applicant’s argument is not persuasive.
As set forth in the rejection, Ngyen is relied upon for teaching time-multiplexed sequential neural network processing, including reserved neuron time slots, round-robin neural processing, time-division multiplexing, and serial communications between neural network nodes. Although Ngyen states that time-multiplexing may be used to “mimic parallelism in neural networks,” time-division multiplexing fundamentally involves sequential time-shared processing in which multiple operations share hardware resources across different time slots, thereby creating the appearance of parallel processing rather than simultaneous parallel execution. Accordingly, Ngyen’s disclosure of employing time multiplexing to “mimic parallelism” in neural networks directly supports the claimed “plurality of hidden time-multiplexed layers to sequentially process the reduced number of spikes.
Applicant’s argument improperly assumes that Ngyen’s time-multiplexing techniques would only be applicable in the absence of parallel neural processing. However, neither Ngyen nor the rejection requires replacing or eliminating Van der Made’s parallel neural processing architecture. Rather, as set forth in the rejection, Van der Made’s is relied upon for the disclosed parallel neural processing architecture including multiple parallel neural processing engines, while Ngyen is relied upon for the additional teaching of sequential time-multiplexed neural network processing.
Accordingly, the rejection does not propose replacing Van der Made’s parallel neural processing architecture with Ngyen’s time-multiplexed processing. Rather, under BRI, the rejection relies on the combined teachings of Van der Made and Ngyen to teach or at least suggest the claimed hybrid architecture including both parallel neural processing and time-multiplexed sequential neural network processing stages. One of ordinary skill in the art would have recognized that Ngyen’s time-multiplexed sequential neural processing techniques could be incorporated into downstream neural network processing stages of Van der Made’s parallel spiking neural-network architecture in order to efficiently utilize hardware resources while preserving the parallel neural processing functionality of Van der Made.
With respect to Applicant’s third argument on pages 8-12 of the Remarks, Applicant argues that one of ordinary skill in the art would not have combined Van der Made with Ngyen because Van der Made already teaches parallel neural processing engines while Ngyen allegedly uses time multiplexing only to mimic parallelism on constrained hardware. Applicant further argues that Ngyen’s disclosed time multiplexing is merely a specific time-slot reservation scheme that allegedly would not result in the claimed “plurality of hidden time-multiplexed layers,” that the Office failed to provide sufficient motivation to combine, and that Van der Made allegedly teaches away from the claimed architecture through fused computations across layers. Applicant’s arguments have been fully considered but are not persuasive for the reasons set forth below.
Applicant argues that Van der Made already includes “8 parallel engines” and therefore that one of ordinary skill in the art would not have incorporated Ngyen’s disclosed time-multiplexed processing techniques into Van der Made’s architecture. Applicant further argues that Ngyen’s disclosed time multiplexing merely “mimics parallelism” on constrained CGRA hardware. Applicant’s arguments are not persuasive. As set forth in the rejection, Ngyen is relied upon for teaching time-multiplexed sequential neural network processing including reserved neuron time slots, round-robin neural processing, time-communications between neural network nodes. Although Ngyen states that time multiplexing may be used to “mimic parallelism in neural networks,” time-division multiplexing fundamentally involves sequential time-shared processing in which operations are processed across different time slots rather than simultaneously in parallel. Accordingly, Ngyen’s disclosed time-division multiplexing and round-robin neural processing directly correspond to the claim language reciting “time-multiplexed layers to sequentially process the reduced number of spikes.”
Applicant’s argument improperly assumes Ngyen’s disclosed time-multiplexing processing techniques would only be applicable in the complex absence of parallel neural processing. However, neither Ngyen nor the rejection requires replacing or eliminating Van der Made’s disclosed parallel neural-processing architecture. Rather, the rejection relies on Van der Made for the broader spiking neural network architecture including hidden layers, output layers, event/spike processing between layers, and parallel neural processing operations, while Ngyen is relied upon for the additional teaching of sequential time-multiplexed neural network processing techniques. Accordingly, the rejection does not rely on replacing Van der Made’s parallel neural processing architecture with Ngyen’s time-multiplexed processing. Rather, the rejection relies on the combined teachings of Van der Made and Ngyen to teach or at least suggest the claimed hybrid architecture including both parallel neural processing and sequential time-multiplexed neural network processing stages.
Applicant additionally argues that Ngyen’s disclosed time multiplexing is not merely “some scheduling exists,” but rather a particular time-slot reservation and round-robin time division multiplexing scheme that serializes operations and communications in time. Applicant’s argument is not persuasive. Claim 1 broadly recites “a plurality of hidden time-multiplexed layers to sequentially process the reduced number of spikes” and does not recite any particular scheduling architecture, asynchronous processing technique, or specific multiplexing implementation beyond sequential time-multiplexed neural network processing. As set forth in the rejection, Ngyen teaches reserved time slots, round-robin processing, time-division multiplexing, and serial neuron communications. Under BRI, such teachings directly correspond to the broadly recited “time-multiplexed layers to sequentially process the reduced number of spikes.”
Applicant further argues that Van der Made already describes coordination of event handling and parallel execution using ping-pong buffers and multiple parallel engines and therefore allegedly undermines the Office’s rationale for incorporating Ngyen’s disclosed time-multiplexed processing techniques. Applicant’s argument is not persuasive because the rejection does not rely on Ngyen to replace Van der Made’s event handling, buffering operations, or parallel neural processing architecture. Rather, one of ordinary skill in the art would have recognized that Ngyen’s disclosed time-multiplexed sequential processing techniques could be incorporated into downstream neural network processing stages within Van der Made’s broader spiking neural network architecture in order to efficiently utilize hardware resources while preserving the parallel neural processing functionality of Van der Made. Further, the rejection is not limited to resolving a singular contention mechanism or bottleneck within Van der Made. Rather, the rejection relies on the predictable use of Ngyen’s disclosed time-multiplexed sequential processing techniques within the broader spiking neural network architecture taught by Van der Made.
Applicant further argues that the Office failed to establish the claimed “plurality of hidden time-multiplex layers” and further argues that the Office’s interview position allegedly is inconsistent because Ngyen purportedly is “not relied upon to modify the architecture,” while the rejection nonetheless relies on Ngyen’s disclosed time-multiplexed processing techniques. Applicant’s argument is not persuasive. Claim 1 broadly recites “a plurality of hidden time-multiplexed layers to sequentially process the reduced number of spikes.” As set forth in the rejection, Van der Made teaches a spiking neural network architecture comprising multiple hidden layers between input and output layers, while Ngyen teaches time-multiplexed sequential neural-network processing including reserved time slots, round-robin processing, time-division multiplexing, and serial neural communications. Under BRI, the combined teachings reasonably correspond to the claimed hidden time-multiplexed layers that sequentially process spikes. Accordingly, the rejection consistently relies on Van der Made for the broader spiking neural network architecture and relies on Ngyen for the additional teaching of the time-multiplexed sequential neural network processing techniques.
Applicant also argues that Van der Made teaches away from the claimed sequential time-multiplexed layering by teaching “fusing” multiple layers together and that the rejection of claim 4 is inconsistent with the rejection of claim 1. Applicant’s argument is not persuasive. As set forth in the rejection, Van der Made disclosed fusion of computations across layers is directed to restructuring computations across layers in order to reduce intermediate memory storage and retrieval operations. However, claim 1 broadly recites “a plurality of hidden time-multiplexed layers to sequentially process the reduced number of spikes” and does not exclude fused computations across layers, prohibit computations across multiple layers from being performed together, or require all computations be performed exclusively sequentially. Further, the rejection relies on Ngyen for the disclosed time-multiplexed sequential process techniques and additionally relies on Van der Made’s disclosed fused multilayer computations as teaching computations performed across multiple neural network layers. Accordingly, Van der Made’s disclosed fusion of computations across layers, do not criticize, discredit, or otherwise discourage the incorporation of Ngyen’s disclosed time-multiplexed sequential processing techniques within portions of the broader spiking neural-network architecture.
With respect to Applicant’s fourth argument on pages 12-14 of the Remarks, Applicant argues that Van der Made allegedly fails to disclose or suggest “where only spiking events are processed by each layer of the hardware architecture” because Van der Made purportedly performs packetization, row-by-row processing, and organization by spikes into more complex data structures rather than processing only spiking events. Applicant further argues that the claimed invention allegedly processes events “in any order,” whereas Van der Made allegedly processes data in a predefined order row-by-row order. Applicant’s arguments are not persuasive.
As set forth in the rejection, Van der Made expressly teaches event-based spike processing within a spiking neural network architecture. Specifically, Van der Made teaches that “event-based convolution is implemented in a Spiking Neural Network (SNN) using event-based rank-coding,” that “the spike event is transmitted over a bus to the next layer of the network,” and that “spikes occur only when there are activations, and since events are processed only when spikes occur (rather than upon every convolutional stride), there are fewer events.” Accordingly, Van der Made expressly teaches that spike/events are transmitted between neural network layers and that processing occurs only when spike/events occur.
Applicant’s arguments regarding inbound filtering, packet collection, row-by-row processing, and organization of events within memory are not persuasive because claim 1 broadly recites “wherein the flow of spikes is an event based data where only spiking events are processed by each layer of the hardware architecture” and does not recite any exclusion of packetization, buffering, event organization, row-based processing, or intermediate event handling. Further, claim 1 does not recite that spikes/events must arrive “in any order,” prohibit ordered event processing, or require the absence of predefined processing arrangements within portions of the architecture.
Applicant additionally argues that Van der Made allegedly operates on more complex data structures than “simple spiking events.” However, as set forth in the office action, Van der Made expressly teaches that the underlying processing data are spike/events within an event-base spiking neural network architecture and further teaches that events are processed only when spikes occur. The additional used of packet collection, buffering, row ordering, or event organization within portions of the architecture does not negate Van der Made’s disclosed event based spike processing. Under BRI, Van der Made’s disclosed event-based spike/event processing architecture reasonably corresponds to the claimed “flow of spikes is event-based data where only spiking events are processed by each layer of the hardware architecture.”
Applicant argues that claim 11 recites features similar to, but not necessarily coextensive with, those of claim 1 and therefore allegedly is patentable for reasons similar to those discussed with respect to claim 1. Applicant’s argument is not persuasive. As set forth above, the rejection of claim 1 is maintained because the combined teachings of Van der Made and Ngyen teach or at least suggest the limitations of claim 1 under BRI. Claim 11 recites limitations similar in scope to those addressed with respect to claim 1, and Applicant has no identified any additional limitation in claim 11 that would render the claim separately patentable over the cited combination. Accordingly, claim 11 remains unpatentable for substantially the same reasons discussed above with claim 1.
Applicant argues that claims 9, 10, and 12 are patentable based on their respective dependencies from claim 1 and 11. Applicant further argues that Kawao and Chaturvedi allegedly were cited only for the additional limitations of dependent claims 9, 10, and 12 and purportedly do not cure the alleged deficiencies of Van der Made and Ngyen. Applicant’s arguments are not persuasive.
As discussed above, the rejection of independent claim 1 and 11 is maintained because the combined teachings of Van der Made and Ngyen teach or at least suggest the recited limitations under BRI. Kawao and Chaturvedi are additionally relied upon for the further limitations specifically recited in dependent claims 9, 10, and 12. Applicant has not separately traversed the specific teachings relied upon from Kawao and Chaturvedi with respect to the additional limitations of claims 9, 10, and 12. Accordingly, because claims 9, 10, and 12 depend from unpatentable base claims and because the additional references teach or at least suggest the further recited dependent limitations as set forth in the rejection, the rejections of claims 9, 10, and 12 are maintained.
Accordingly, for at least the reasons set forth above, the rejections of claims 1-5 and 7-13 under 35 U.S.C. 103 are maintained.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 7-8, 11 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Van der Made et al. (Pat. No. US 11227210 B2, filed on July 25th 2019, hereinafter Van der Made) in view of Ngyen et al. (NPL, FIST: A Framework to Interleave Spiking Neural Networks on CGRAs, 2015 23rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, 2015, hereinafter Ngyen).
Regarding claim 1, Van der Made teaches the following limitations:
A hardware architecture for spiking neural networks, the hardware architecture enabling the implementation of a classification of an image and comprising: spike generator module that receives an input pixel of the image and generates a flow of spikes (Van der Made, paragraph [0007] “Embodiments include a system that includes a spike converter configured to generate spikes from the digital input data” [0049] “In the present embodiment, event-based convolution is implemented in a Spiking Neural Network (SNN) using event-based rank-coding “ [0077] “The frame-based image must be converted into an event-based image first. This can be done by preprocessing or by a dedicated spike encoding area on the chip using the new event-based convolution algorithm.” [0080] “Prior to these blocks there is a spike converter that takes the input data in its native format, and converts it to spikes that may be processed by the subsequent blocks. “ – Van der Made teaches a hardware architecture for spiking neural networks configured for image classification using event-based convolution in a spiking neural network. Van der Made further teaches a spike converter that receives digital image input data and converts the image into spikes/event-based image data for subsequent neural network processing. Under BRI, Van der Made’s spike converter receiving native input image data and converting the image data into spikes corresponds to the claimed “spike generator module that receives an input pixel of the image and generates a flow of spikes.)
a neural core module that, during a first processing stage, receives the flow of spikes and filters the flow of spikes to generate a reduced number of spikes, the neural core module comprising an input layer comprising an input neuron to receive the flow of spikes, and a fully parallel layer composed of neurons that processes the input spikes in parallel (Van der Made, [0007] “ an inbound filter configured to select relevant spikes from the generated spikes.” [0031] “After neurons are fully processed, they output a neuronal activation…These activations are events that are delivered to the next layer, which is stored on a different CNP.” [0049] “ In the present embodiment, event-based convolution is implemented in a Spiking Neural Network (SNN) using event-based rank-coding” [0056] “Each NP has 8 parallel engines that update a plurality of neurons and a plurality of synapses in parallel.” [0063] “Conventional implementations of neural networks perform the computations in the network by following its structure, one layer at a time. This approach produces a large amount of intermediate data that are steadily output to memory as the computation progresses. Upon completion of a layer, the intermediate data are sent back to the same computational hardware, and the process repeats until computation in all layers has concluded. The amount of intermediate data that must be transported between the computational hardware and memory increases with increasing neural network size. In an additional advancement, the inventors have employed fusion of the computation across multiple layers. Rather than processing each layer to completion before proceeding to the next layer, the inventors have restructured the computation such that multiple layers are computed together, which avoids the need to store or retrieve the intermediate data from the memory. In short, fusing takes the operations of two or more layers and fuses them into a single operation.” [0068] “each CNP has 8 neural processing engines that run in parallel.” – Van der Made teaches spiking a neural network architecture that receives and processes spikes/events during neural network computations. Van der Made further teaches filtering spikes using an inbound filter configured to select relevant spikes from generated spikes, thereby reducing the number of spikes/events processed within the architecture. Van der Made additionally teaches restructuring neural network computations such that operations across multiple layers are fused together into a single computation rather than processed strictly layer-by-layer. Further, Van der Made teaches parallel neural processing because each NP/CNP includes multiple parallel neural processing engines that update neurons and synapses in parallel and run in parallel across neural network computations. Under the broadest reasonable interpretation, Van der Made’s parallel neural processing architecture reasonably corresponds to the claimed “fully parallel layer composed of neurons that processes input spikes in parallel.)
wherein the flow of spikes is an event-based data where only spiking events are processed by each layer of the hardware architecture (Van der Made, [0031] “ After neurons are fully processed, they output a neuronal activation…These activations are events that are delivered to the next layer.” [0049] “In the present embodiment, event-based convolution is implemented in a Spiking Neural Network (SNN) using event-based rank-coding” [0051] “ The spike event is transmitted over an internal bus to the next layer of the network.” [0060] “ In various embodiments, the approach is event-based, and the spikes are encoded, rather than every single convolution stride. Spikes occur only when there are activations, and since events are processed only when spikes occur (rather than upon every convolution stride), there are far fewer events.” – Van der Made teaches event-based spike processing within a spiking neural network architecture. Van der Made further teaches that spikes/events are transmitted between neural network layers and that events are processed only when spikes occur. Under BRI, Van der Made’s event-based spike/event processing architecture corresponds to the claimed “flow of spikes is an event based data where only spiking events are processed by each layer of the hardware architecture.)
However, Van der Made does not teach, but Van der Made in view of Ngyen teaches the following limitation:
a neural processing unit that, during a second processing stage subsequent to the first processing stage, processes the reduced number of spikes, the neural processing unit comprising a plurality of hidden time-multiplexed layers to sequentially process the reduced number of spikes, the plurality of hidden time-multiplexed layers comprising an output layer (Van der Made, paragraph [0031] “ After neurons are fully processed, they output a neuronal activation…These activations are events that are delivered to the next layer” [0049] “A deep neural network (DNN) is defined as an artificial neural network that has multiple hidden layers between its input and output layers, each layer comprising a perceptron…In the present embodiment, event-based convolution is implemented in a Spiking Neural Network (SNN) using event-based rank-coding “ [0060] “Spikes occur only when there are activations, and since events are processed only when spikes occur (rather than upon every convolution stride), there are far fewer events.” [0063] “Rather than processing each layer to completion before proceeding to the next layer, the inventors have restructured the computation such that multiple layers are computed together, which avoids the need to store or retrieve the intermediate data from the memory. In short, fusing takes the operations of two or more layers and fuses them into a single operation.” [0065] “These activations are events that are delivered to the next layer in the neural network, which is stored on a different CNP.” Nguyen, [page 753] “To mimic parallelism in neural networks, we have employed time multiplexing. Where a time slot is reserved for each transmitting neuron (connected to the receiving neuron)… In every cycle, the neuroDPU receives an input from one of the three registers (representing the neurons) in round robin fashion.” [page 754] “The one-to-many connectivity was realized by using time division multiplexing. “ & “the intermediate nodes communicate with each other
serially to ensure one to many connectivity.”– Van der Made teaches a spiking neural network architecture comprising multiple hidden layers between input and output layers. Van der Made further teaches event-based spike processing in which spikes/events are processed only when activation occurs, resulting in fewer processed event/spikes, and teaches activations/events being delivered between neural network layers. Van der Made additionally teaches multi-layer neural network processing in which computations across multiple layers may be fused together into a single operation. Ngyen is relied upon for the time-multiplexing sequential neural processing technique, teaching time multiplexing using reserved neuron time slots, round-robin neural processing, time-division multiplexing, and serial neural network communications in order to mimic neural network parallelism. Under BRI, applying Nygen’s time-multiplexing neural processing techniques to Van der Made’s hidden spiking neural network layers reasonably corresponds to the claimed “neural processing unit” comprising “a plurality of hidden time-multiplexed layers to sequentially process the reduced number of spikes, where Van der Made’s neural network architecture further includes the claimed output layer.); and
a classification module that is connected to the output layer of the plurality of hidden time multiplexed layers and determines a winner class from the output layer (Van der Made, paragraph [0065] “ After neurons are fully processed, they output a neuronal activation… These activations are events that are delivered to the next layer in the neural network, which is stored on a different CNP.” [0135] “Winner Takes All (WTA) picks at most one winning feature for each location. In a large field of features, too much relevant information may be discarded. Without WTA, the best fitting features at a location will still spike, but they will also be accompanied by spikes of lesser fitting features. Those lesser fitting features cost power and processing time. By applying WTA in strategically chosen groups, the lesser fitting features can be eliminated while retaining all of the best fits.” [0136] “The purpose of the present invention is to reduce the number of propagated spikes by strategically grouping the filters from each location in the input stream and applying a WTA technique on those groups” [0138] “For some features, it is only necessary to know that any one of the potentials in a group is above threshold.” Nguyen, [page 753] “To mimic parallelism in neural networks, we have employed time multiplexing. Where a time slot is reserved for each transmitting neuron (connected to the receiving neuron)… In every cycle, the neuroDPU receives an input from one of the three registers (representing the neurons) in round robin fashion.” – Van der Made teaches activation/events being delivered between neural network layers within a spiking neural network architecture. Van der Made further teaches a Winner-Takes-All (WTA) classification architecture that selects a winning feature from grouped neural network outputs and determines winners based on whether neuron/filters potentials exceed a threshold. Ngyen is relied upon for the known technique of time-multiplexed neural processing, where neuron processing is performed using reserved time slots and round-robin execution to mimic parallelism in neural networks. It would have been obvious to a person of ordinary skill in the art to implement Ngyen’s time-multiplexing processing techniques in the hidden layers of Van der Made’s spiking neural network in order to efficiently reuse neural processing hardware resources while preserving neural network functionality. Accordingly, in the combination, Van der Made’s output layer/outputs from the neural network layers are provided to Van der Made’s WTA classification architecture, and that WTA classification architecture determines the winning feature/class from those outputs. Under BRI, this corresponds to the claimed “classification module that is connected to the output layer of the plurality of hidden time-multiplexed layers and determines a winner class from the output layer.)
Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having the combination of Van der Made and Ngyen before them, to incorporate the time-multiplexing and round-robin processing techniques in the spiking neural network architecture of Van der Made. One would have been motivated to make such a combination in order to more efficiently use available processing resources while still maintaining parallel neural-network functionality. Further, both Van der Made and Ngyen are directed to spiking neural networks and spike-based neural processing, thereby making the references highly analogous and compatible for combination.
Regarding claim 2, Van der Made in view of Ngyen, as outlined above, all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1, mutatis mutandis. Van der Made in view of Ngyen further teaches:
wherein the spike generator is implemented as a neural coding function (Van der Made, paragraph [0049] describes rate coding but teaches rank coding which are neural coding functions.)
Regarding claim 3, Van der Made in view of Ngyen, as outlined above, all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1, mutatis mutandis. Van der Made in view of Ngyen further teaches:
wherein the neural core module further comprises a control module that sequentially reads output spikes from the fully-parallel layer and stores the output spikes in an output FiFo buffer. (Van der Made, paragraph [0080] “Prior to these blocks there is a spike converter that takes the input data in its native format, and converts it to spikes that may be processed by the subsequent blocks.” [0056] “Input events are placed in a dual buffer (ping-pong buffer) whereby one buffer is processed while the other buffer is filled. Each NP has 8 parallel engines that update a plurality of neurons and a plurality of synapses in parallel… The address of the neuron is placed in a packet on the bus when a neuron generates a spike or transmits a potential to the next NP. At the receiving NP, the packet is received into the Input Events Buffer SRAM also named the Ping-Pong buffer for processing by that NP. The t+n (whereby n is a value) in the image indicate the timing of events; t+30 indicates that the information is received in the buffer at time t+30 cycles , where the values are stored in the buffer at time t.” Ngyen, [page 753] “To mimic parallelism in neural networks, we have employed time multiplexing….In every cycle, the neuroDPU receives an input from one of the three registers (representing the neurons) in round robin fashion.” – Van der Made teaches a fully-parallel neural-processing architecture in which spikes generated from neurons processed in parallel are communicated as spike-event information and stored in event buffers for subsequent processing. Van der Made further teaches that spike-event generated from neurons is received into a Ping-Pong buffer and processed across processing cycles. Nguyen teaches time-multiplexed round-robin neural processing in which neuron outputs are sequentially processed during processing cycles. Under BRI, the combination of Van der Made and Nguyen teaches sequentially reading spike-event outputs from the fully parallel layer and storing the outputs in a buffer structure corresponding to the claimed output FiFo buffer.)
Regarding claim 4, Van der Made in view of Ngyen, as outlined above, all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1, mutatis mutandis. Van der Made in view of Ngyen further teaches:
wherein each of the plurality of hidden time-multiplexed layers comprises neural processing unit modules that emulate the time-multiplexed layers (Ngyen (pg. 753, Fig.3, col. 2, last paragraph teaches the time-multiplexing layers. Van der Made, paragraph [0063] teaches rather than processing each layer to completion before processing the next layer, the inventors have restructured the computation such that multiple layers are computed together, which avoids the need to store or retrieve the intermediate data from the memory. In short, fusing takes the operation of two or more layers and fuses them into a single operation (emulates the time-multiplexed layers)).
Regarding claim 5, Van der Made in view of Ngyen, as outlined above, all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1, mutatis mutandis. Van der Made in view of Ngyen further teaches:
wherein the classification module is a Terminate Delta module (Van der Made , paragraphs [0136]-[0137] teaches a Winner-Takes-All (WTA) classification architecture in which winning features are selected from the neural network outputs while lesser fitting features are eliminated. Under BRI, a “Terminate Delta Module” reasonably corresponds to classification decision logic that determines/selects a winner class from computing neural network outputs. Therefore, the WTA classification architecture of Van der Made reasonably corresponds to the claim “Terminate Delta Module.”)
Regarding claim 7, Van der Made in view of Ngyen, as outlined above, all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1, mutatis mutandis. Van der Made in view of Ngyen further teaches:
wherein the flow of spikes is a frame-based data where every ‘0’ and ‘1’ in an input frame is processed by each layer of the hardware architecture (Van der Made, Table 1, paragraph [0117]-[0118] teaches each neuron ‘processes’ a packet that means each input frame is processed by each layer. To form a packet, you allow N unique events to be placed in to a bit-array of length M and represent the presence of an event with a ‘1’ entry and the absence of an event with a ‘0’ entry as shown in Table 1.).
Regarding claim 8, Van der Made in view of Ngyen, as outlined above, all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1, mutatis mutandis. Van der Made in view of Ngyen further teaches:
wherein the spiking neural networks are fully-connected based spiking neural networks or spiking convolutional neural networks (Van de Made, paragraph [0049] teaches that spiking neural networks can be implemented as Spiking Convolutional Neural Networks (SNNs). Since the prior art already discloses a spiking convolutional neural network, it obviously meets the limitation of claim 8, which recites that the spiking neural networks are fully-connected based SNNs or spiking CNNs. A person of ordinary skill in the art would recognize that the described spiking structure obviously includes both fully-connected and convolutional configurations, making the claim distinction an obvious implementation (abstract).).
Regarding claim 11, Van der Made in view of Ngyen teaches the method for processing spiking neural networks are recited. As explained in the rejection of claim 1, the combined teachings already disclose or suggest receiving an input pixel and generating a flow of spikes, during a first processing stage receiving and filtering the flow of spikes by a neural core module to generate a reduced number of spikes using a fully-parallel layer composed of neurons that process input spikes in parallel, and during a second processing stage subsequent to the first processing stage sequentially processing the reduced number of spikes using time-multiplexed neural-network layers. The combined teachings further disclose or suggest determining, by a classification module connected to the output layer of the hidden time-multiplexed layers, a winner class from the output layer, where the flow of spikes is event-based data in which only spiking events are processed by the hardware architecture. Since Claim 11 does not introduce any new limitations that distinguish it from the previously rejected subject matter, it is likewise unpatentable under 35 U.S.C. 103 for the same reasons stated in the rejection of claim 1.
Regarding claim 13, Van der Made in view of Ngyen, as outlined above, all the elements of claim 2, therefore is rejected for the same reasons as those presented for claim 2, mutatis mutandis. Van der Made in view of Ngyen further teaches:
wherein the neural coding function is rate coding or spike select (Van der Made, col. 6, lines 16-24 describes rate coding but teaches rank coding which is a neural coding function.).
Claims 9 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Van der Made et al. (Pat. No. US 11227210 B2, filed on July 25th 2019, hereinafter Van der Made) in view of Ngyen et al. (NPL, FIST: A Framework to Interleave Spiking Neural Networks on CGRAs, 2015 23rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, 2015, hereinafter Ngyen) and further in view of Kawao et al. (NPL, Spiking neural network simulation on FPGAs with automatic and intensive pipelining, 2016 International Symposium on Nonlinear Theory and Its Applications, NOLTA2016. VOL 11., 2016, hereinafter Kawao).
Regarding claim 9, Van der Made in view of Ngyen, as outlined above, all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1, mutatis mutandis. Van der Made in view of Ngyen does not teach, but Kawao teaches the following limitation:
A Field Programmable Gate Array (FPGA) (Kawao (abstract) further teaches implementing spiking neural networks on FPGAs and executing them in a pipelined manner for improved performance.)
It would have been obvious to a person of ordinary skill in the art of neural network hardware acceleration before the effective filing date, having the combination of Van der Made and Ngyen before them, to incorporate implementation of spiking neural networks on FPGAs and execute them in a pipelined manner, as taught by Kawao. One would have been motivated to make such a combination in order to enhance computational efficiency and better manage workload distribution across hardware resources (abstract).
Regarding claim 12, Van der Made in view of Ngyen, as outlined above, all the elements of claim 11, therefore is rejected for the same reasons as those presented for claim 11, mutatis mutandis. Van der Made in view of Ngyen does not teach, but Kawao teaches the following limitation:
wherein the steps are executed in a pipelined way (Kawao (abstract) further teaches implementing spiking neural networks on FPGAs and executing them in a pipelined manner for improved performance.).
It would have been obvious to a person of ordinary skill in the art of neural network hardware acceleration before the effective filing date, having the combination of Van der Made and Ngyen before them, to incorporate implementation of spiking neural networks on FPGAs and execute them in a pipelined manner, as taught by Kawao. One would have been motivated to make such a combination in order to enhance computational efficiency and better manage workload distribution across hardware resources (abstract).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Van der Made et al. (Pat. No. US 11227210 B2, filed on July 25th 2019, hereinafter Van der Made) in view of Ngyen et al. (NPL, FIST: A Framework to Interleave Spiking Neural Networks on CGRAs, 2015 23rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, 2015, hereinafter Ngyen) and further in view of Chaturvedi et al. (NPL, ASIC Implementation for Improved Character Recognition and Classification using SNN Model, Procedia Computer Science, Volume 62, Pages 151-158, ISSN 1877-0509, 2015, hereinafter Chaturvedi).
Regarding claim 10, Van der Made in view of Ngyen, as outlined above, all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1, mutatis mutandis. Van der Made in view of Ngyen does not teach, but Chaturvedi teaches the following limitation:
An Application Specific Integrated Circuit (ASIC) (Chaturvedi (abstract) teaches the implementation of spiking neural networks on an Application Specific Integrated Circuit (ASIC), demonstrating that ASICs are a well-known hardware platform for improving the efficiency and scalability of neural network computations.)
It would have been obvious to a person of ordinary skill in the art of neural network hardware acceleration before the effective filing date to implement spiking neural networks on an Application Specific Integrated Circuit (ASIC), as taught by Chaturvedi (abstract), to improve efficiency and scalability. Since Chaturvedi (pg. 155, first paragraph underneath Fig. 7) explicitly teaches the ASIC implementations of spiking neural networks enhance performance by reducing power consumption and improving simulation efficiency, one would have been motivated to make such a combination to achieve improved efficiency, power consumption, and large-scale processing capabilities.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Daravanh Phakousonh whose telephone number is (571)272-6324. The examiner can normally be reached Mon - Thurs 7 AM - 5 PM, Every other Friday 7 AM - 4PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at 571-272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Daravanh Phakousonh/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights