DETAILED ACTION
1. This office action is in response to the Application No. 17246219 filed on 05/28/2026. Claims 1-25 are presented for examination and are currently pending.
Notice of Pre-AIA or AIA Status
2. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
3. A request for continued examination under 37 CFR 1.114, including the fee set
forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this
application is eligible for continued examination under 37 CFR 1.114, and the fee set
forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action
has been withdrawn pursuant to 37 CFR 1.114. Applicant’s submission filed on
05/28/2026 has been entered.
Allowable Subject Matter
4. Claims 11, 14 and 18 are objected to as being dependent upon a rejected base
claim, but would be allowable if rewritten in independent form including all of the
limitations of the base claim.
Response to Arguments
5. The Applicants argument regarding the prior art have been considered and the Examiner is withdrawing the rejections in the previous Office action because Applicant’s amendment necessitated new grounds of rejection presented in this Office Action.
It is noted that arguments regarding independent claims 1, 2 and 20 have been considered but are moot because new references have now been used to remap the independent claims 1, 2 and 20.
Furthermore, Smith, Lee, Ruckauer, Rhodes, Chen, Tang, Rouhani, Bazhenov, Numaoka and Deng which were applied in the previous Office Action are still relevant to the instant dependent claims. As a result, their teachings have been used in this Office Action.
The dependent claims 3-10, 12, 13,15-17 and 19-25 which depend directly or
indirectly from independent claims 1, 2 and 20 are not allowable because the instant claims are still obvious over the prior art of record.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
6. Claims 1, 2 and 19 are rejected under 35 U.S.C 102(a)(1) as being anticipated by Voelker et al. ("A spike in performance: Training hybrid-spiking neural networks with quantized activation functions." arXiv:2002.03553v2 [cs.LG] 1 Mar 2021).
Regarding claim 1, Voelker teaches a method for using a spiking neural network with improved efficiency (We have presented a new algorithm and accompanying methods that allow interpolation between spiking and non spiking networks. This allows the training of hSNNs, which can have mixtures of activity quantization, leading to computationally efficient neural network implementations. We have also shown how to incorporate standard SNN assumptions, such as the presence of a synaptic filter, pg. 8, right col., third para.),
the method comprising: receiving image data (Figure 1: Visualizing the output (at) of Algorithm 1 given an MNIST digit as input (xt), pg. 3);
providing the image data (given an MNIST digit as input (xt), Fig. 1, pg. 3. The Examiner notes MNIST digits are grayscale images of handwritten numbers) to a trained spiking neural network (SNN) (The network is trained using the categorical cross entropy loss function (fused with softmax), pg. 5, left col., first para.),
the SNN comprising a plurality of neurons (Thus, the hidden neurons in the fully trained networks are conventional (1-bit) spiking neurons, pg. 5, left col., third para.),
each of the plurality of neurons associated with a respective initialization value V0 (The voltage is initialized to v0 = 0.5, pg. 16, Fig. 4; According to instant specification: “if a global values is used, a value V₀ = 0.5, which can be referred to as a "warm start" initial model state, can represents a more natural state and can improve performance”[0124]) of a plurality of initialization values (We recommend initializing v0 ∼ U[0,1) independently for each neuron, pg. 2, right col., third para. The Examiner notes U[0,1) indicates a continuous uniform distribution where values are spread evenly from 0 up to close to 1, but not including 1. The bracket [ indicates 0 is included, and the parenthesis ) indicates 1 is excluded),
wherein a first layer of the trained SNN comprises a first subset of the plurality of neurons, and a second layer of the trained SNN comprises a second subset of the plurality of neurons (SNNs commonly apply a synapse model to the weighted summation of spike-trains. This filters the input to each neuron over time to reduce the amount of spike noise (pg. 4, left col., last para.); To summarize, the architecture that we train includes a nonlinear layer (h) and a linear layer (m), each of which has synaptic filters (pg. 4, right col., third para.); The recurrent network consists of one-bit spiking LIF neurons (representing ht) coupled with multi-bit spiking IF neurons (representing mt), pg. 7, Fig. 3. The Examiner notes nonlinear layer is the first layer and linear layer is a second layer), and
wherein a mean of the plurality of initialization values is within 10% of 0.5, and a standard deviation of the initialization values is at least 0.05 (We recommend initializing v0 ∼ U[0,1) independently for each neuron (pg. 2, right col., third para.) The Examiner notes that for continuous uniform distribution interval U[0, 1), the mean (average) is 0.5. The numbers 0.45 and 0.55 (inclusive) is within 10% of 0.5, as a result mean 0.5 is within 10% of 0.5. The calculated standard deviation is approximately 0.2887);
receiving output from the trained SNN at a time step τ , wherein the output is based on activations of neurons in an output layer of the trained SNN (The ideal output of the activation function at step t is defined as at = f(xt), pg. 12, first para.), and
wherein τ is in a range of 1 to T (Let xt be the input to the activation function at a discrete time-step, t > 0, such that the ideal output (i.e., with unlimited precision) is at = f(xt), pg. 2, right col., third para.); and
classifying the image data based on output of the trained SNN at time step τ (An example of each hybrid-spiking Legendre Memory Unit (hsLMU) network producing the correct classification given a test digit for the sMNIST task (Top; see Table 1) and the psMNIST task(Bottom;seeTable2) ... Classifications are obtained by taking an argmax of the output layer on the final time-step of each sequence, Fig. 3, pg. 7).
Regarding claim 2, Voelker teaches a method for using a spiking neural network with improved efficiency (We have presented a new algorithm and accompanying methods that allow interpolation between spiking and non spiking networks. This allows the training of hSNNs, which can have mixtures of activity quantization, leading to computationally efficient neural network implementations. We have also shown how to incorporate standard SNN assumptions, such as the presence of a synaptic filter, pg. 8, right col., third para.),
the method comprising: receiving image data (Figure 1: Visualizing the output (at) of Algorithm 1 given an MNIST digit as input (xt), pg. 3);
providing the image data (given an MNIST digit as input (xt), Fig. 1, pg. 3. The Examiner notes MNIST digits are grayscale images of handwritten numbers) to a trained spiking neural network (SNN) (The network is trained using the categorical cross entropy loss function (fused with softmax), pg. 5, left col., first para.),
the SNN comprising a plurality of neurons (Thus, the hidden neurons in the fully trained networks are conventional (1-bit) spiking neurons, pg. 5, left col., third para.),
each of the plurality of neurons associated with a respective initialization value V0 (The voltage is initialized to v0 = 0.5, pg. 16, Fig. 4; According to instant specification: “if a global values is used, a value V₀ = 0.5, which can be referred to as a "warm start" initial model state, can represents a more natural state and can improve performance”[0124]) of a plurality of initialization values (We recommend initializing v0 ∼ U[0,1) independently for each neuron, pg. 2, right col., third para. The Examiner notes U[0,1) indicates a continuous uniform distribution where values are spread evenly from 0 up to close to 1, but not including 1. The bracket [ indicates 0 is included, and the parenthesis ) indicates 1 is excluded),
wherein a first layer of the trained SNN comprises a first subset of the plurality of neurons, and a second layer of the trained SNN comprises a second subset of the plurality of neurons (SNNs commonly apply a synapse model to the weighted summation of spike-trains. This filters the input to each neuron over time to reduce the amount of spike noise (pg. 4, left col., last para.); To summarize, the architecture that we train includes a nonlinear layer (h) and a linear layer (m), each of which has synaptic filters (pg. 4, right col., third para.); The recurrent network consists of one-bit spiking LIF neurons (representing ht) coupled with multi-bit spiking IF neurons (representing mt), pg. 7, Fig. 3. The Examiner notes nonlinear layer is the first layer and linear layer is a second layer), and
wherein a mean of the plurality of initialization values is within 10% of 0.5, and a standard deviation of the initialization values is at least 0.05 (We recommend initializing v0 ∼ U[0,1) independently for each neuron (pg. 2, right col., third para.) The Examiner notes that for continuous uniform distribution interval U[0, 1), the mean (average) is 0.5. The numbers 0.45 and 0.55 (inclusive) is within 10% of 0.5, as a result mean 0.5 is within 10% of 0.5. The calculated standard deviation is approximately 0.2887);
receiving output from the trained SNN at a time step τ, wherein the output is based on activations of neurons in an output layer of the trained SNN (The ideal output of the activation function at step t is defined as at = f(xt), pg. 12, first para.), and
wherein τ is in a range of 1 to T (Let xt be the input to the activation function at a discrete time-step, t > 0, such that the ideal output (i.e., with unlimited precision) is at = f(xt), pg. 2, right col., third para.); and
performing a task associated with the data based on output of the trained SNN at time step τ (An example of each hybrid-spiking Legendre Memory Unit (hsLMU) network producing the correct classification given a test digit for the sMNIST task (Top; see Table 1) and the psMNIST task(Bottom;seeTable2) ... Classifications are obtained by taking an argmax of the output layer on the final time-step of each sequence, Fig. 3, pg. 7).
Regarding claim 19, Voelker teaches the method of claim 2, Voelker teaches wherein the data comprises time-series data (For sMNIST, the pixels are supplied sequentially in a time-series of length 282, pg. 4, right col., last para.)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
7. Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Voelker et al. ("A spike in performance: Training hybrid-spiking neural networks with quantized activation functions." arXiv:2002.03553v2 [cs.LG] 1 Mar 2021) in view of Lee et al. ("Enabling spike-based backpropagation for training deep neural network architectures." Frontiers in neuroscience 14 (2020): 497482).
Regarding claim 3, Voelker teaches the method of claim 2, Voelker does not explicitly teach wherein the time step τ is determined based on a latency-optimized timing schedule established during refinement of the trained SNN, and wherein the output is indicative of a neuron in the output layer that had the most activations up to time T
Lee teaches wherein the time step τ is determined based on a latency-optimized timing schedule (To obtain the optimal #time-steps required for our proposed training method, we trained VGG9 networks on CIFAR-10 dataset using different time-steps ranging from 10 to 120 (shown in Figure 6A) (pg. 12, right col., last para. to pg. 13, left col,. first para.); Next, we construct our networks by leveraging frequently used architectures such as VGG and ResNet. To the best of our knowledge, this is the first work that demonstrates spike-based supervised BP learning for SNNs containing more than 10 trainable layers, pg. 16, left col., second para.) established during refinement of the trained SNN (The Figure 9 shows the relationship between inference accuracy, latency and #spikes/inference for ResNet11 networks trained on CIFAR-10 dataset, pg. 16, right col., second to the last para.), and
wherein the output is indicative of a neuron in the output layer that had the most activations up to time T (Whenever the membrane potential exceeds the firing threshold (Vth), the post-neuron in the output feature map spikes, pg. 4, Fig. 2)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker to incorporate the teachings of Lee for the benefit of training deep convolutional SNNs directly (with input spike events) using spike-based backpropagation which achieves the best classification accuracies in MNIST, SVHN, and CIFAR-10 datasets (Lee, abstract).
8. Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Voelker et al. ("A spike in performance: Training hybrid-spiking neural networks with quantized activation functions." arXiv:2002.03553v2 [cs.LG] 1 Mar 2021) in view of Lee et al. ("Enabling spike-based backpropagation for training deep neural network architectures." Frontiers in neuroscience 14 (2020): 497482) and further in view of Ruckauer et al. (US20190122110)
Regarding claim 4, Voelker and Lee teaches the method of claim 3, Voelker teaches wherein the data comprises image data (given an MNIST digit as input (xt), Fig. 1, pg. 3. The Examiner notes MNIST digits are grayscale images of handwritten numbers),
wherein the task comprises a computer vision task that includes classification of the image data (the network outputs a classification at the end of each input sequence (pg. 5, left col., first para.); given an MNIST digit as input (xt), Fig. 1, pg. 3. The Examiner notes MNIST digits are grayscale images of handwritten numbers), and
wherein the neuron in the output layer up to time step τ corresponds to a first class of a plurality of classes (The output layer also uses a 10 time-step lowpass filter (pg. 5, left col. last para.); the network outputs a classification
at the end of each input sequence, pg. 5, left col., first para.)
Voelker and Lee does not explicitly teach wherein the neuron in the output layer that had the most activations up to time step τ.
Ruckauer teaches wherein the neuron in the output layer that had the most activations up to time step τ (and neurons with activations greater than or equal to a predetermined threshold are taken into consideration [0093]; To determine first timing information ti (0) about a timing at which the i-th neuron first fires, a neuron potential at ti (0) is set to a neuron potential threshold θ, which is expressed as ui(ti (0)) =. Accordingly, for ti (0), Equation 4 is expressed as shown in Equation 5 below. [0075]; … the output layer outputs a sufficiently accurate value or label [0055]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker and Lee to incorporate the teachings of Ruckauer for the benefit of a neural network conversion operation, in which an activation of a neuron of an ANN is matched to a reciprocal of information about a timing at which a first spike is to be generated by a neuron of an SNN, thus advantageously reducing the number of computations performed by the SNN (Ruckauer [0061])
9. Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Voelker et al. ("A spike in performance: Training hybrid-spiking neural networks with quantized activation functions." arXiv:2002.03553v2 [cs.LG] 1 Mar 2021) in view of Smith et al (US20170286828) and further in view of Rhodes et al. ("Real-time cortical simulation on neuromorphic hardware." Philosophical Transactions of the Royal Society A 378.2164 (2020): 20190160).
Regarding claim 5, Voelker teaches the method of claim 2, Voelker does not explicitly teach the limitations of claim 5.
Smith teaches further comprising: receiving output from the trained SNN at a time step τ' subsequent to time step τ (Then the normalized output time volley is Zn=<t1−tmin, t2−tmin, . . . tn−tmin>t. [0177]; The Examiner notes that Zn=<t1−tmin, t2−tmin, . . . tn−tmin>t as output at different time steps and t2−tmin as time step τ' subsequent to t1−tmin (τ));
performing the task based on output of the trained SNN at step time τ' (In one application, the spiking neural network composed of CCs may perform classification” [0069]; As shown in FIG. 4, a CC is further composed of an Excitatory Column, (EC) [0074]; This process of 2×2 reduction proceeds until a single classifier remains. The output of this classifier is the final output, with 10 lines, where the first line to spike indicates the class to which the input pattern belongs [0218]; In one implementation of training, all weights are initially set at zero. The first volley of the training set is applied, and an output spike is forced at all neuron outputs with a spike time close to, and before, the maximum time T.sub.max [0112]; A related task is classification, where training patterns have labels which indicate a pre-specified cluster (or class) to which a given training pattern belongs. Then, after training, arbitrary patterns can be evaluated by the SNN to determine the class to which they belong [0093]; Then the normalized output time volley is Zn=<t1−tmin, t2−tmin, . . . tn−tmin>t. [0177]; The Examiner notes that Zn=<t1−tmin, t2−tmin, . . . tn−tmin>t as output at different time steps and t2−tmin as time step τ' subsequent to t1−tmin (τ)).
wherein the time step τ and the time step τ' are each determined based on confidence thresholds (At the time the sum of the spike responses first reaches a threshold value (denoted as θ), the neuron emits an output spike [0097]) corresponding to such time steps (The threshold is crossed at time t, then the spike in the preliminary output volley is assigned to be t. [0183]) in a predefined latency-accuracy curve (Figure 23; The next time this input pattern is re-presented, firing threshold will be reached sooner which implies a slight decrease of the post-synaptic spike latency….By iteration, it follows that upon repeated presentation of the same input spike pattern, the post-synaptic spike latency will tend to stabilize at a minimal value [0121])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker to incorporate the teachings of Smith for the benefit of a computationally faster training patterns (Smith [0028])
Voelker and Smith does not explicitly teach and wherein T is a time step at which the SNN settles at a steady state, and outputs provided at time steps T and T' represent outputs from the SNN while the SNN is in a transient state during which firing rates and accuracy change over time before the steady state is achieved at T;
Rhodes teaches wherein T is a time step at which the SNN settles at a steady state (Simulations are performed for a total of 10s with a simulation time step of t=0.1ms for accuracy of produced spike times. Simulation output is split into an initial transient followed by steady-state activity, pg. 3, last para.; Note that for SpiNNaker simulations of the cortical microcircuit recorded data comprises output spike times of all model neurons (pg. 10, last para.); the total spikes produced for each simulation time step are plotted in figure 2 (both plots produced from baseline NEST simulation results). This demonstrates significant variations in counts of model spikes per time step from the initial transient phase of the simulation through to steady-state, pg. 6, first para.)), and
outputs provided at time steps T and T' represent outputs from the SNN (Figure 2. Analysis of cortical microcircuit output activity simulated with NEST: total, excitatory and inhibitory, spikes produced per simulation timestep. Left inset shows an initial transient response, while right inset details steady-state oscillations (Online version in colour.), pg. 6; The right inset of figure 2 shows results from 1145<t<1170ms, pg. 6, first para.) while the SNN is in a transient state during which firing rates (while figure 1c shows the mean layer-wise firing rates for the model, the total spikes produced for each simulation time step are plotted in figure 2 (both plots produced from baseline NEST simulation results). This demonstrates significant variations in counts of model spikes per time step from the initial transient phase of the simulation through to steady-state, pg. 6, first para.)
and accuracy change over time (Simulations are performed for a total of 10s with a simulation time step of t=0.1ms for accuracy of produced spike times, pg. 3, last para.) before the steady state is achieved at T (Figure 1. Cortical microcircuit model: … (b)0.4s of steady state output spikes (5% of total spikes plotted for clarity), pg. 3);
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker and Smith to incorporate the teachings of Rhodes for the benefit of improving system efficiency and spike processing capacity (Rhodes, pg. 9, last para.)
10. Claims 6 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Voelker et al. ("A spike in performance: Training hybrid-spiking neural networks with quantized activation functions." arXiv:2002.03553v2 [cs.LG] 1 Mar 2021) in view of Smith et al (US20170286828)
Regarding claim 6, Voelker teaches the method of claim 2, Voelker teaches wherein data comprises image data comprising an array of pixels each associated with a value (a latency of 112 additional time-steps to accumulate the classification after the image has been presented (resulting in a total of 2 × 282 + 112 = 1680 steps ... The authors state that the input to the LSNN is preprocessed using 80 more neurons that fire whenever the pixel crosses over a fixed value associated with each neuron, to obtain “some what better performance.”, pg. 5, right col. last para.)), and
providing the data to the trained SNN comprises: generating, for each pixel, a spike train based on the value associated with the pixel (input to the LSNN is preprocessed using 80 more neurons that fire whenever the pixel crosses over a fixed value associated with each neuron (pg. 5, right col., last para.); SNNs commonly apply a synapse model to the weighted summation of spike-trains, pg. 4, left col., last para.),
wherein spikes are generated at a rate that is proportional to the value associated with the pixel; and providing, to each neuron of a plurality of neurons in an input layer of the trained SNN (the architecture that we train includes a nonlinear layer (h) and a linear layer (m), each of which has synaptic filters, pg. 4, right col., third para.; This filters the input to each neuron over time to reduce the amount of spike noise, pg. 4, left col., last para.),
Voelker does not explicitly teach a spike train associated with a respective pixel of the plurality of pixels.
Smith teaches a spike train associated with a respective pixel of the plurality of pixels (Fig. 1 denotes spike trains; If there are a total of n input elements (e.g., pixels in a grayscale image), then the value of each of the elements is translated into to spikes belonging to a volley [0206]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker to incorporate the teachings of Smith for the benefit of a computationally faster training patterns (Smith [0028])
Regarding claim 20, claim 20 is similar to claim 1. It is rejected in the same manner and reasoning applying. Further, Voelker teaches a system for using a spiking neural network with improved efficiency (We have presented a new algorithm and accompanying methods that allow interpolation between spiking and non spiking networks. This allows the training of hSNNs, which can have mixtures of activity quantization, leading to computationally efficient neural network implementations. We have also shown how to incorporate standard SNN assumptions, such as the presence of a synaptic filter, pg. 8, right col., third para.), but does not explicitly teach
the system comprising: at least one processor that is configured to:
Smith teaches the system comprising: at least one processor that is configured to:
(The structures and methods as described herein can be implemented in a number of ways: directly in special purpose hardware, in programmable hardware as an FPGA, or via software on a general purpose computer or graphics processor [0221]):
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker to incorporate the teachings of Smith for the benefit of a computationally faster training patterns (Smith [0028])
11. Claims 7 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Voelker et al. ("A spike in performance: Training hybrid-spiking neural networks with quantized activation functions." arXiv:2002.03553v2 [cs.LG] 1 Mar 2021) in view of Smith et al (US20170286828)in view of Chen et al. (US20210011161 filed 07/09/2019) and further in view of Tang et al. ("Spike counts based low complexity SNN architecture with binary synapse." IEEE Transactions on Biomedical Circuits and Systems 13.6 (2019): 1664-1677).
Regarding claim 7, Voelker teaches the method of claim 2, Smith teaches wherein the data comprises image data comprising a plurality of spike streams (As has been stated, similar spike volleys represent similar input data patterns [0212]; As one example, the input could be visual images. These images may undergo some form of filtering or edge detection, with the output of the translation stage as spike volleys [0204]; For example, the pattern may be a raw image expressed as pixels or it may be a pre-processed image)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker to incorporate the teachings of Smith for the benefit of a computationally faster training patterns (Smith [0028])
Voelker and Smith does not explicitly teach generated by an imaging device, but wherein the activations in the output layer of the trained SNN due to the plurality of spike streams does not dynamically determine time step r during operation of the SNN.
Chen teaches generated by an imaging device (The vehicle 200 can include a camera, possibly at a location inside sensor unit 202. The camera can be a photosensitive instrument, such as a still camera, a video camera, etc., that is configured to capture a plurality of images of the environment of the vehicle 200 [0073]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker and Smith to incorporate the teachings of Chen for the benefit of a LIDAR device which may determine the distances by projecting light pulses onto the environment and detecting corresponding return light pulses reflected from the various points within the environment, the intensity of each of the return light pulses may be measured by the LIDAR device and represented as a waveform that indicates the intensity of detected light over time (Chen [0024]).
Voelker, Smith and Chen does not explicitly teach but wherein the activations in the output layer of the trained SNN due to the plurality of spike streams does not dynamically determine time step τ during operation of the SNN.
Tang teaches but wherein the activations in the output layer of the trained SNN due to the plurality of spike streams does not dynamically determine time step τ during operation of the SNN (The basic idea is that since the precise timings of output spikes are not critical to make classification outputs in a rate-coding based SNN, pg. 1669, left col., first para.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker, Smith and Chen to incorporate the teachings of Tang for the benefit of a SNN processor in learning mode, processes 79 input spikes per each time step in 3,024 clock cycles on average and consumes average 1.42 pJ/SOP with a throughput of 4.0 GSOP/s which is one of the lowest energy/ classification and energy/SOP (Tang, pg. 1675, left col., last para.)
Regarding claim 8, Voelker, Smith, Chen and Tang teaches the method of claim 7, Chen teaches wherein the imaging device (The vehicle 200 can include a camera, possibly at a location inside sensor unit 202. The camera can be a photosensitive instrument, such as a still camera, a video camera, etc., that is configured to capture a plurality of images of the environment of the vehicle 200 [0073]) comprises
a light detection and ranging (LiDAR) device (These sensors may include a light detection and ranging (LIDAR) device [0024])
The same motivation to combine dependent claim 7 applies here.
12. Claims 9 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Voelker et al. ("A spike in performance: Training hybrid-spiking neural networks with quantized activation functions." arXiv:2002.03553v2 [cs.LG] 1 Mar 2021) in view of Ruckauer et al. (US20190122110)
Regarding claim 9, Voelker teaches the method of claim 2, Voelker does not explicitly teach the limitations of claim 9.
Ruckauer teaches wherein the trained SNN was generated based on a trained analog neural network (ANN) (acquiring connection weight of an analog neural network (ANN) node of a pre-trained ANN [0005]; In an example implementation of an SNN of the present disclosure a pre-trained ANN may be converted to the SNN resulting in the SNN of the present disclosure which may thereby reflect a neural network having been successfully trained based on larger-scale data sets, e.g., as in the training of the ANN, and therefore the SNN of the present disclosure has improved accuracy in its results and/or outputs when compared to the typical SNNs. For example, through such conversion, the SNN may implement the trained objective of the ANN [0050]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker to incorporate the teachings of Ruckauer for the benefit of a neural network conversion operation, in which an activation of a neuron of an ANN is matched to a reciprocal of information about a timing at which a first spike is to be generated by a neuron of an SNN, thus advantageously reducing the number of computations performed by the SNN (Ruckauer [0061])
Regarding claim 17, Voelker teaches the teaches the method of claim 2, Voelker does not explicitly teach the limitations of 17.
Ruckauer teaches wherein the ANN is a convolutional neural network (CNN) (A typical analog neural network (ANN) is a deep neural network including a plurality of hidden layers, and includes, for example, a convolutional neural network (CNN) [0044]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker to incorporate the teachings of Ruckauer for the benefit of a neural network conversion operation, in which an activation of a neuron of an ANN is matched to a reciprocal of information about a timing at which a first spike is to be generated by a neuron of an SNN, thus advantageously reducing the number of computations performed by the SNN (Ruckauer [0061])
13. Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Voelker et al. ("A spike in performance: Training hybrid-spiking neural networks with quantized activation functions." arXiv:2002.03553v2 [cs.LG] 1 Mar 2021) in view of Ruckauer et al. (US20190122110) and further in view of Rouhani et al. (US20210019605 filed 03/21/2019)
Regarding claim 10, Voelker and Ruckauer teaches the method of claim 9, they do not explicitly teach wherein the ANN was trained using a loss function LANN and the ANN was refined using a penalized loss function LIANN that included LANN and one or more penalized terms.
Rouhani teaches wherein the ANN was trained using a loss function LANN (Equation (2) below expresses a first loss term loss1 that accounts for the constraint to maximize the isolation between activations or clusters of outputs from the activation functions applied by the neuron in the hidden layer l. … wherein λ1 may denote a tradeoff hyper-parameter specifying the contribution of the first loss term loss1 during the training of the machine learning model 100 [0061]) and
the ANN was refined using a penalized loss function LIANN that included LANN and one or more penalized terms (This first loss term loss1 may be configured to penalize an activation distribution in which different activations (e.g., outputs from the activation functions applied by the neuron in the hidden layer l) are entangled and difficult to separate [0061]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker and Ruckauer to incorporate the teachings of Rouhani for the benefit of minimizing an error in an output of the first machine learning model by at least minimizing a loss function (Rouhani [0019])
14. Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Voelker et al. ("A spike in performance: Training hybrid-spiking neural networks with quantized activation functions." arXiv:2002.03553v2 [cs.LG] 1 Mar 2021) in view of Bazhenov et al. (US20220374679 filed 07/17/2020)
Regarding claim 12, Voelker teaches the method of claim 2, Voelker does not explicitly teach further comprising: refining the trained SNN using a loss function LSNN.
Bazhenov teaches further comprising: refining the trained SNN using a loss function LSNN ( … a loss function (termed elastic weight consolidation—EWC), which penalizes updates to weights deemed appropriate for previous tasks, made use of synaptic mechanisms of memory consolidation [0040]; However, it should be appreciated that any other plasticity rules can be applied to SNN during sleep phase and any other modifications can be applied to the network to simulate sleep phase [0041]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker to incorporate the teachings of Bazhenov for the benefit of convert the architecture of the ANN to an equivalent Spiking Neural Network (SNN) and simulate a sleep phase in the SNN while using plasticity rules to modify synaptic weights (Bazhenov abstract)
15. Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Voelker et al. ("A spike in performance: Training hybrid-spiking neural networks with quantized activation functions." arXiv:2002.03553v2 [cs.LG] 1 Mar 2021) in view of Bazhenov et al. (US20220374679 filed 07/17/2020) and further in view of Ruckauer et al. (US20190122110)
Regarding claim 13, Voelker and Bazhenov teaches the method of claim 12, they do not explicitly teach wherein the loss function LSNN includes an accuracy term, a latency term, and a power consumption term.
Ruckauer teaches wherein the loss function LSNN includes an accuracy term, a latency term, and a power consumption term (n an example, by implementing the neural network conversion operation of the present disclosure, the implemented SNN may maintain an accuracy loss less than 1% [0062]; … and a latency until a calculated output is available is reduced [0057]; a spiking neural network (SNN) of the present disclosure employing “all-or-none pulses” to transfer information may be used, which may require less power consumption than a corresponding ANN implementation [0046]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker to incorporate the teachings of Ruckauer for the benefit of a neural network conversion operation, in which an activation of a neuron of an ANN is matched to a reciprocal of information about a timing at which a first spike is to be generated by a neuron of an SNN, thus advantageously reducing the number of computations performed by the SNN (Ruckauer [0061])
16. Claims 15 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Voelker et al. ("A spike in performance: Training hybrid-spiking neural networks with quantized activation functions." arXiv:2002.03553v2 [cs.LG] 1 Mar 2021) in view of Bazhenov et al. (US20220374679 filed 07/17/2020) in view of Smith et al (US20170286828) and further in view of Numaoka et al. (US20220366723 filed 07/14/2020)
Regarding claim 15, Voelker and Bazhenov teaches the method of claim 12, they do not explicitly teach the limitation of claim 15.
Smith teaches wherein refining the SNN further comprises: applying, to each of the plurality of neurons, a scaling factor ŋj, wherein H is a set of scaling factors for the plurality of neurons (As specified above, a neuron takes as input normalized volleys containing spikes having times between 0 and Tmax. The neuron produces a single output spike which can take on a range of values that depends on a number of factors to be described later. Tmax and Wmax are related via the scale factor γ=Tmax/Wmax. That is, γ is the ratio of the maximum range of input values to the maximum range of spike times [0164]);
providing first labeled training data to the trained SNN (Using labels associated with the training data, the N output neurons are trained in a supervised manner to spike for the class indicated by the label) [0150]);
receiving first output from the trained SNN for the first labeled training data (At layer 2, overlapping regions from layer 1 are processed. For example a 2×2 set of RF classifier outputs, represented as four 10 line bundles, are merged to form a single 40 line volley. This volley forms the input to a layer 2 classifier [0218]);
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker to incorporate the teachings of Smith for the benefit of a computationally faster training patterns (Smith [0028])
Voelker, Bazhenov and Smith does not explicitly teach calculating a first loss based on the first labeled training data and the first output from the trained SNN using the loss function LSNN; adjusting values of the scaling factors in H based on the loss: applying the adjusted scaling factors to the plurality of neurons of the trained SNN; providing second labeled training data to the trained SNN; receiving second output from the trained SNN for the second labeled training data; and calculating a second loss based on the second labeled training data and the second output from the trained SNN using the loss function LSNN.
Numaoka teaches calculating a first loss based on the first labeled training data and the first output from the trained SNN using the loss function LSNN (A loss function 404 is a function defined using the emotion output and the emotion label as arguments [0107]; The emotion learning processing logic 304 includes an artificial intelligence using a learning model such as …, a spiking neural network (SNN) [0094]);
adjusting values of the scaling factors in H based on the loss: applying the adjusted scaling factors to the plurality of neurons of the trained SNN (Then, learning or training of the neural network is performed so as to minimize the loss function 404 by modifying the coupling weighting coefficient between neurons from the output layer toward the input layer of the full coupling layer 403 using a method such as back propagation [0107]; The emotion learning processing logic 304 includes an artificial intelligence using a learning model such as …, a spiking neural network (SNN) [0094]);
providing second labeled training data to the trained SNN (The emotion learning processing logic 304 inputs the data preprocessed by the learning data preprocessing logic 301 [0125]);
receiving second output from the trained SNN for the second labeled training data (The output layer of the full coupling layer 403 is a node for emotion output [0106]); and
calculating a second loss based on the second labeled training data and the second output from the trained SNN using the loss function LSNN (The emotion learning processing logic 304 includes an artificial intelligence using a learning model such as …, a spiking neural network (SNN) … It is assumed that the artificial intelligence used in the emotion learning processing logic 304 according to the present embodiment includes a mechanism for learning a result of calculation by a loss function … or the like to estimate an optimal solution (output) for a question (input) [0094]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker, Bazhenov and Smith to incorporate the teachings of Numaoka for the benefit of learning or training of the neural network is performed so as to minimize the loss function (Numaoka [0107])
Regarding claim 16, Voelker and Bazhenov teaches the method of claim 12, they do not explicitly teach the limitations of claim 16.
Smith teaches wherein refining the SNN further comprises: setting an initialization value V0 for each of the plurality of neurons, wherein I includes a set of initialization values (Before each input volley, it is assumed that the value of the neuron's body potential is quiescent (initialized at 0) [0098]);
providing first labeled training data to the trained SNN (Using labels associated with the training data, the N output neurons are trained in a supervised manner to spike for the class indicated by the label) [0150]);
receiving first output from the trained SNN for the first labeled training data (At layer 2, overlapping regions from layer 1 are processed. For example a 2×2 set of RF classifier outputs, represented as four 10 line bundles, are merged to form a single 40 line volley. This volley forms the input to a layer 2 classifier [0218]);
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker to incorporate the teachings of Smith for the benefit of a computationally faster training patterns (Smith [0028])
Voelker, Bazhenov and Smith does not explicitly teach calculating a first loss based on the first labeled training data and the first output from the trained SNN using the loss function LSNN; adjusting values of the initialization values in I based on the loss; applying the adjusted initialization values to the plurality of neurons of the trained SNN; providing second labeled training data to the trained SNN; receiving second output from the trained SNN for the second labeled training data; and calculating a second loss based on the second labeled training data and the second output from the trained SNN using the loss function LSNN.
Numaoka teaches calculating a first loss based on the first labeled training data and the first output from the trained SNN using the loss function LSNN (A loss function 404 is a function defined using the emotion output and the emotion label as arguments [0107]; The emotion learning processing logic 304 includes an artificial intelligence using a learning model such as …, a spiking neural network (SNN) [0094]);
adjusting values of the initialization values in I based on the loss; applying the adjusted initialization values to the plurality of neurons of the trained SNN (Then, learning or training of the neural network is performed so as to minimize the loss function 404 by modifying the coupling weighting coefficient between neurons from the output layer toward the input layer of the full coupling layer 403 using a method such as back propagation [0107]; The emotion learning processing logic 304 includes an artificial intelligence using a learning model such as …, a spiking neural network (SNN) [0094]);
providing second labeled training data to the trained SNN (The emotion learning processing logic 304 inputs the data preprocessed by the learning data preprocessing logic 301 [0125]);
receiving second output from the trained SNN for the second labeled training data (The output layer of the full coupling layer 403 is a node for emotion output [0106]); and
calculating a second loss based on the second labeled training data and the second output from the trained SNN using the loss function LSNN (The emotion learning processing logic 304 includes an artificial intelligence using a learning model such as …, a spiking neural network (SNN) … It is assumed that the artificial intelligence used in the emotion learning processing logic 304 according to the present embodiment includes a mechanism for learning a result of calculation by a loss function … or the like to estimate an optimal solution (output) for a question (input) [0094])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker, Bazhenov and Smith to incorporate the teachings of Numaoka for the benefit of learning or training of the neural network is performed so as to minimize the loss function (Numaoka [0107])
17. Claims 21 is rejected under 35 U.S.C. 103 as being unpatentable over Voelker et al. ("A spike in performance: Training hybrid-spiking neural networks with quantized activation functions." arXiv:2002.03553v2 [cs.LG] 1 Mar 2021) in view of Smith et al (US20170286828) and further in view of Ruckauer et al. (US20190122110)
Regarding claim 21, Voelker and Smith teaches the system of claim 20, Voelker and Smith does not explicitly teach the limitation of claim 21.
Ruckauer teaches wherein the at least one processor comprises a neuromorphic processor (the processor 620 may be a specialized computer, or may be representative of one or more processors to control a specialized SNN processor according to the conversion of the ANN. In an example, the specialized processor may be a neuromorphic chip or processor [0105]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Smith to incorporate the teachings of Ruckauer for the benefit of a neural network conversion operation, in which an activation of a neuron of an ANN is matched to a reciprocal of information about a timing at which a first spike is to be generated by a neuron of an SNN, thus advantageously reducing the number of computations performed by the SNN (Ruckauer [0061])
18. Claims 22 is rejected under 35 U.S.C. 103 as being unpatentable over Voelker et al. ("A spike in performance: Training hybrid-spiking neural networks with quantized activation functions." arXiv:2002.03553v2 [cs.LG] 1 Mar 2021) in view of Smith et al (US20170286828) in view of Ruckauer et al. (US20190122110) and further in view of Chen et al. (US20210011161 filed 07/09/2019)
Regarding claim 22, Voelker, Smith and Ruckauer teaches the system of claim 21, Voelker teaches wherein the data comprises image data (each input pixel is presented for two time-steps rather than one), a latency of 112 additional time-steps to accumulate the classification after the image has been presented (resulting in a total of 2 × 282 + 112 = 1680 steps), pg. 5, right col., last para.), the system further comprising:
Voelker, Smith and Ruckauer does not explicitly teach an image data source in communication with the at least one processor, the image data source comprising an array of single-photon avalanche photodiodes (SPADs); and wherein the at least one processor that is further configured to: receive the image data from the image data source.
Chen teaches an image data source in communication with the at least one processor (Vehicle 100 may also include computer system 112 to perform operations, such as operations described therein. As such, computer system 112 may include at least one processor 113 (which could include at least one microprocessor) operable to execute instructions 115 stored in a non-transitory, computer-readable medium, such as data storage 114 [0057]),
the image data source comprising an array of single-photon avalanche photodiodes (SPADs) (In some embodiments, the one or more detectors of the laser rangefinder/LIDAR 128 may include one or more photodetectors. ... In some examples, such photodetectors may even be capable of detecting single photons (e.g., single-photon avalanche diodes (SPADs)). Further, such photodetectors can be arranged (e.g., through an electrical connection in series) into an array [0046]); and
wherein the at least one processor that is further configured to: receive the image data from the image data source (Each of these sensors may communicate environment data to a processor in the vehicle about information each respective sensor receives [0086]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker, Smith and Ruckauer to incorporate the teachings of Chen for the benefit of a LIDAR device which may determine the distances by projecting light pulses onto the environment and detecting corresponding return light pulses reflected from the various points within the environment, the intensity of each of the return light pulses may be measured by the LIDAR device and represented as a waveform that indicates the intensity of detected light over time (Chen [0024]).
19. Claims 23 is rejected under 35 U.S.C. 103 as being unpatentable over Voelker et al. ("A spike in performance: Training hybrid-spiking neural networks with quantized activation functions." arXiv:2002.03553v2 [cs.LG] 1 Mar 2021) in view of Smith et al (US20170286828) and further in view of Deng et al. ("Tianjic: A unified and scalable chip bridging spike-based and continuous neural computation." IEEE Journal of Solid-State Circuits 55.8 (2020): 2228-2246)
Regarding claim 23, Voelker and Smith teaches the system of claim 20, Voelker and Smith does not explicitly teach further comprising: a communications connection to an image data source, configured to communicate a stream of image data from the image data source to the processor; and wherein time step τ is measured from a commencement of computation of the SNN for the stream of image data, and predetermined based upon a refinement of the SNN.
Deng teaches further comprising: a communications connection to an image data source (The SNN recognizes the voice commands from human, the CNN receives resized images from the camera and detects the initial location, pg. 2242, left col., second para.)
configured to communicate a stream of image data from the image data source to the processor (and a neural state machine (NSM) for decision making, as illustrated in Fig. 25, pg. 2242, left col., second para. The Examiner notes the neural state machine as the processor); and
wherein time step τ is measured from a commencement of computation of the SNN for the stream of image data (There are two levels of temporal execution unit in our design: time step and time phase that work synergistically to support a flexible operating pattern …each time step has multiple time phases … For example, by configuring the timing registers, i.e., start_phase, end_phase, #on_phases, and #off_phase, pg. 2235, right col., second para.), and
predetermined based upon a refinement of the SNN (In SNN mode with 1 < Tw ≤ 16, A/S_MEM0 and A/S_MEM1 are merged to be a whole chunk to contain the spike pattern within a historical temporal window (i.e., Tw time phases), which is updated at each time phase, pg 2234, left col., last para.).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker and Smith to incorporate the teachings of Deng for the benefit of object tracking in a video stream (Deng, left col., first para.)
20. Claims 24 and 25 are rejected under 35 U.S.C. 103 as being unpatentable over Voelker et al. ("A spike in performance: Training hybrid-spiking neural networks with quantized activation functions." arXiv:2002.03553v2 [cs.LG] 1 Mar 2021) in view of Smith et al (US20170286828) in view of Deng et al. ("Tianjic: A unified and scalable chip bridging spike-based and continuous neural computation." IEEE Journal of Solid-State Circuits 55.8 (2020): 2228-2246) and further in view of Tang et al. ("Spike counts based low complexity SNN architecture with binary synapse." IEEE Transactions on Biomedical Circuits and Systems 13.6 (2019): 1664-1677).
Regarding claim 24, Voelker, Smith and Deng teaches the system of claim 23, Voelker, Smith and Deng does not explicitly teach wherein the trained SNN is structured so as not to generate the output until time step τ regardless of intervening neuron activations.
Tang teaches wherein the trained SNN is structured so as not to generate the output until time step τ regardless of intervening neuron activations (When the membrane voltage exceeds the predefined membrane threshold Vth, an output spike is generated from the neuron, and its membrane voltage is reset. The excitatory layer neurons are fully-connected with the inhibitory neurons. Through the connections, the lateral inhibition occurs that makes competitions among the excitatory neurons. The winner-take-all (WTA) mechanism that means that once a winner is chosen, other neurons are prohibited from generating output spikes (pg. 1665, left col., last para. to right col., first para.). The Examiner notes spikes represents neuron activations according to instant specification).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker, Smith and Deng to incorporate the teachings of Tang for the benefit of a SNN processor in learning mode, processes 79 input spikes per each time step in 3,024 clock cycles on average and consumes average 1.42 pJ/SOP with a throughput of 4.0 GSOP/s which is one of the lowest energy/ classification and energy/SOP (Tang, pg. 1675, left col., last para.)
Regarding claim 25, Voelker, Smith and Deng teaches the system of claim 23, Voelker, Smith and Deng does not explicitly teach wherein the trained SNN is structured so as to generate the output based upon cumulative activations of the neurons in the output layer from the commencement of computation until the time step τ.
Tang teaches wherein the trained SNN is structured so as to generate the output based upon cumulative activations of the neurons in the output layer from the commencement of computation until the time step τ (In the proposed accumulation based computing scheme, as the accumulated number of input spikes are used per one membrane voltage updates, multiple excitatory neurons generate output spikes at a time (pg. 1669, right col., second to the last para.). The Examiner notes spikes represents neuron activations according to instant specification).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Voelker, Smith and Deng to incorporate the teachings of Tang for the benefit of a SNN processor in learning mode, processes 79 input spikes per each time step in 3,024 clock cycles on average and consumes average 1.42 pJ/SOP with a throughput of 4.0 GSOP/s which is one of the lowest energy/ classification and energy/SOP (Tang, pg. 1675, left col., last para.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MORIAM MOSUNMOLA GODO whose telephone number is (571)272-8670. The examiner can normally be reached Monday-Friday 8:00am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T. Bechtold can be reached on (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.G./Examiner, Art Unit 2148
Prosecution Insights