DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
Claim(s) 1-4, 7, 9, 12, 14-17, 19, 22-24, 26, 28-35 is/are rejected under 35 U.S.C. 103 as being unpatentable over Piyush Khandelwal et al. (hereinafter Khandelwal) (“Efficient Real-Time Inference in Temporal Convolution Networks”, 05/31/2021) in view of Oleg Rybakov et al. (hereinafter Rybakov) (“Streaming keyword spotting on mobile devices”, 07/29/2020) further in view of Kale Poorna (hereinafter Poorna) (US 20230161936 A1, 05/25/2023).
Regarding claim 1, Khandelwal teaches;
implements a convolutional neural network (CNN) as a sliding convolutional neural network (SCNN) having a plurality of layers to generate
([Abstract] It has been recently demonstrated that Temporal Convolution Networks (TCNs) provide state-of-the-art results in many problem domains where the input data is a time-series. TCNs typically incorporate information from a long history of inputs (the receptive field) into a single output using many convolution layers. Real-time inference using a trained TCN can be challenging on devices with limited compute and memory, especially if the receptive field is large. This paper introduces the RT-TCN algorithm that reuses the output of prior convolution operations to minimize the computational requirements and persistent memory footprint of a TCN during real-time inference. We also show that when a TCN is trained using time slices of the input time-series, it can be executed in real time continually using RT-TCN.)
NOTE: The disclosed TCN (which is a type of CNN) can be considered a sliding convolutional neural network, because it discloses a TCN that computes outputs continually as new time-series samples arrive, while reusing overlapping convolution results from prior steps. Thus, Khandelwal teaches an SCNN having a plurality of layers (TCNs have many convolutional layers) to generate results based on a sequence of data samples (processing time-series data as it becomes available is sequential).
(Reasoning as to why it would be obvious for these results to be classification results will be explained later below)
the sequentially processing the sequence of data samples including:
for each received sample of a set of received data samples of the sequence of data samples,
[pg. 4]
PNG
media_image1.png
202
433
media_image1.png
Greyscale
NOTE: Teaches performing the above algorithm for a received sample (next input data, xt) of a set of received data samples (the set of received data samples being the current data samples populating the buffer of the first layer, b[1]) of the sequence of data samples.
[pg. 3]
PNG
media_image2.png
462
508
media_image2.png
Greyscale
NOTE: Algorithm 2 is performed for each received sample xt of the aforementioned set of received data samples of the sequence of data samples.
iteratively updating partial results of an inference of a first layer of the plurality of layers
[pg. 4]
PNG
media_image1.png
202
433
media_image1.png
Greyscale
NOTE: In the above algorithm, when the buffer of the first layer of the plurality of layers (n layers) becomes full (bh[1] = numcols(b[1])), it performs convolution and activation to produce/update a partial result of the inference of the first layer (convolutional output x't). This process is performed when each new data point xt is received and the first layer becomes full, thereby iteratively updating partial results of an inference of the first layer of the plurality of layers.
based on a respective patch of data samples of the sequence of data samples, the respective patch of data samples including the received data sample.
[pg. 4]
PNG
media_image1.png
202
433
media_image1.png
Greyscale
NOTE: The buffer of the first layer b[1] includes data samples of the sequence of data samples, and can therefore be considered a respective patch of data samples of the sequence of data samples, including the received data sample, xt. The partial result x’t of the first layer 1 is determined by a convolution involving b[1]. Thus, Khandelwal teaches the partial results are based on a respective patch of data samples (b[1]) of the sequence of data samples, the respective patch of data samples including the received data sample xt.
iteratively updating partial results of subsequent layers of the plurality of layers of the SCNN based on the iteratively updated partial results of the first layer of the SCNN,
[pg. 4]
PNG
media_image1.png
202
433
media_image1.png
Greyscale
NOTE: The aforementioned iteratively updated partial results of the first layer are propagated to the second layer of the SCNN (the partial results of layer 1 are stored in x’t to be used in the next iteration i = 2 for the second layer). In the next iteration i = 2, the same process is repeated. When the buffer of the second layer b[2] becomes full, the partial results of the second layer of the SCNN are updated in the same manner, where this process is repeated iteratively. The algorithm performs these steps for each layer 1 through n. Thus, the sequentially processing the sequence of data samples includes, for each received sample of the set of received data samples of the sequence of data samples, iteratively updating partial results of subsequent layers of the plurality of layers (1 through n) of the SCNN based on the iteratively updated partial results of the first layer.
Khandelwal fails to teach but Rybakov teaches;
wherein the SCNN sequentially processes the sequence of data samples to generate the classification results,
([Abstract] NN model conversion from non-streaming mode (model receives the whole input sequence and then returns the classification result) to streaming mode (model receives portion of the input sequence and classifies it incrementally) may require manual model rewriting.)
NOTE: Teaches a SCNN (the NN model in streaming mode) sequentially processing the sequence of data samples (receives portion of input sequence) to generate the classification results (classifies incrementally).
wherein the plurality of layers of the SCNN include different types of layers (Note: Convolutional layer and dense layer, see figure 1 below) and a last layer of the plurality of layers of the SCNN is a dense layer (Note: see figure 1 below)
[pg. 1, figure 1]
PNG
media_image3.png
396
687
media_image3.png
Greyscale
OBVIOUSNESS TO COMBINE RYBAKOV WITH KHANDELWAL:
Rybakov and Khandelwal are analogous art to each other and to the present disclosure as they both pertain to a neural network architecture which sequentially processes input data as it is received rather than waiting for the whole sequence.
Specifically, Rybakov pertains to converting a model to a streaming model which reads portions of data to classify incrementally, while Khandelwal pertains to a TCN (temporal convolutional network) architecture which performs real-time inference by buffering intermediate outputs.
Khandelwal states;
([pg. 1] This paper introduces the Real-Time TCN (RT-TCN) algorithm for computing TCN outputs, which retains prior convolution outputs so that convolutions are not recomputed. Specifically, in the example TCN in Figure 1b, RT-TCN only computes the convolution shared between yt−2 and yt once. For some TCN architectures, RT-TCN may also reduce the memory footprint compared to the straightforward approach of buffering the last T inputs. On devices with limited compute and memory, RT-TCN allows executing larger TCN architectures than would be otherwise possible.)
NOTE: This excerpt details that the real time TCN architecture of their disclosure reduces computational requirements and persistent memory footprint of real time inference for temporal CNNs.
Additionally, Rybakov states that one of the NN architectures which they converted to streaming mode was;
([pg. 3] Temporal Convolution ResNet (TC-ResNet))
NOTE: Indicating that the streaming mode presented by Rybakov is applicable to temporal convolutional architectures such as TC-ResNet.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to use the real-time buffering/state-retention technique of Khandelwal in the implementation of Rybakov’s streaming mode to allow the streaming mode models to perform classifications with reduced computational requirements and persistent memory footprint.
Khandelwal and Rybakov fail to teach but Poorna teaches;
A device, comprising: a sensor, which, in operation, generates a sequence of data samples;
([Abstract] For example, an integrated sensor device may be configured to execute instructions with matrix operands and configured with: a sensor to generate measurements of stimuli; … and a controller to store the measurements generated by the sensor into the random access memory as an input to the artificial neural network.)
NOTE: Teaches a device, comprising: a sensor.
([0041] For example, a stream of sensor input data to the artificial neural network (ANN) can be configured in the form of a sequence of input data sets.)
NOTE: Teaches a sensor, which, in operation, generates a sequence of data samples.
and processing circuitry coupled to the sensor, wherein the processing circuitry, in operation, implements
PNG
media_image4.png
594
465
media_image4.png
Greyscale
([0151] For example, the at least one processing unit (111) can be formed on an integrated circuit die of a field-programmable gate array (FPGA) or application specific integrated circuit (ASIC) implementing a deep learning accelerator)
NOTE: Teaches processing circuitry (processing unit 111) coupled to the sensor (processing unit 111 is coupled to the sensor as shown in fig. 7).
([0024] processed by an artificial neural network (ANN) implemented using the deep learning accelerator (DLA).)
NOTE: The ANN is implemented using the DLA, which is implemented using the processing circuitry (processing unit) as previously mentioned. Thus, Poorna teaches the processing circuitry, in operation, implementing an ANN.
OBVIOUSNESS TO COMBINE POORNA WITH KHANDELWAL, RYBAKOV:
Poorna is analogous art to the present disclosure as it pertains to hardware embodiments for implementing a neural network architecture using sensor data as input.
Poorna additionally states;
([0024] At least some embodiments disclosed herein provide a sensor device that has a general-purpose integrated circuit configured to perform computations of artificial neural networks (ANNs) with reduced energy consumption and computation time.)
NOTE: Poorna discloses hardware for implementing an ANN with reduced energy consumption and computation time.
Khandelwal additionally states;
([pg. 1] We believe that these contributions will be particularly of interest to the robotics community, as low-powered sensors and IoT devices that operate on real-world time-series data and use TCN architectures for estimation will benefit from faster inference methods.)
NOTE: Khandelwal discloses that their disclosed TCN architectures would be applicable in the context of robotics using sensor data, which is a very similar field of endeavor to the disclosure of Poorna.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to implement the SCNN of claim 1 (as taught by Khandelwal in view of Rybakov) using the hardware disclosed by Poorna to reduce the energy consumption and computation time of the system.
Regarding claim 2, Khandelwal in view of Rybakov and Poorna teaches;
The device according to claim 1,
(Using the same reasoning from claim 1)
Khandelwal teaches;
wherein sequentially processing the sequence of data samples includes determining when the updated partial results correspond to a complete inference of the SCNN.
[pg. 4]
PNG
media_image1.png
202
433
media_image1.png
Greyscale
NOTE: The aforementioned partial results are propagated to the proceeding layer (so in the case of the first layer 1, the partial results are stored in layer 2). When the following layer becomes full, the updated partial results are propagated to the next layer, and so on. The algorithm eventually determines when the updated partial results correspond to a complete inference of the SCNN when the last layer becomes full (if layer i = n, the produce the final output, ‘o’).
Regarding claim 3, Khandelwal in view of Rybakov and Poorna teaches;
The device according to claim 2,
(Using the same reasoning from claim 2)
Khandelwal teaches;
wherein the set of received samples is a subset of received samples associated with the complete inference of the SCNN.
[pg. 4]
PNG
media_image1.png
202
433
media_image1.png
Greyscale
NOTE: In algorithm 2, the current contents of the first layer input buffer b[1] are a subset of received samples as old samples are removed when the buffer becomes full, meaning b[1] contains only some of the samples of the sequence, i.e. a subset. When b[1] becomes full, convolution with an activation gets performed, and the output is propagated to the next layer in the SCNN. This process continues with subsequent layers until the complete inference is determined. Thus, Khandelwal teaches the set of received samples is a subset of received samples associated with the complete inference of the SCNN.
Regarding claim 4, Khandelwal in view of Rybakov and Poorna teaches;
The device according to claim 3,
(Using the same reasoning from claim 3)
Khandelwal teaches;
wherein the iteratively updating the partial results of the first layer comprises applying a stride parameter.
[pg. 4]
PNG
media_image5.png
86
333
media_image5.png
Greyscale
PNG
media_image6.png
48
397
media_image6.png
Greyscale
NOTE: The aforementioned iteratively updating the partial results of the first layer (the process disclosed in algorithm 2) comprises applying a stride parameter.
Regarding claim 7, Khandelwal in view of Rybakov and Poorna teaches;
The device according to claim 1,
(Using the same reasoning from claim 1)
Khandelwal fails to teach but Rybakov teaches;
wherein the sequentially processing the sequence of data samples includes, for each received sample of the set of received data samples of the sequence of data samples, iteratively updating partial classification results of the SCNN.
([pg. 1] As a result, the convolution will be computed on the new input data only and all previous computations will be buffered, as shown on Fig 1 b. In this case states are internal variables of the model which have to be updated with every prediction.)
NOTE: Rybakov teaches a very similar process to Khandelwal of iteratively updating and buffering states representing previous computations (where the previous computations can be considered partial results) and removing oldest states for each received sample (new input data). Thus, Rybakov teaches iteratively updating partial results.
([Abstract] NN model conversion from non-streaming mode (model receives the whole input sequence and then returns the classification result) to streaming mode (model receives portion of the input sequence and classifies it incrementally) may require manual model rewriting.)
NOTE: Rybakov discloses classifying a set/portion of received data samples incrementally. Thus, Rybakov teaches for each received sample of the set of received data samples of the sequence of data samples (receives portion/set of the input sequence), iteratively updating (using the aforementioned process) partial classification results of the SCNN (classifies incrementally).
OBVIOUSNESS: Using the same reasoning from claim 1.
Regarding claim 9, Khandelwal teaches;
respective
([pg. 3] RT-TCN operates using these 2 principles: • A convolution operation is performed as soon as a dilated kernel width of inputs are available at that layer. • Each layer uses an independent buffer to maintain)
NOTE: Teaches a respective buffer for each of the multiple layers.
Khandelwal fails to teach but Rybakov teaches;
each circular FIFO buffer having a size of a data patch associated with an iterative update operation associated with the respective layer.
[pg. 1]
PNG
media_image3.png
396
687
media_image3.png
Greyscale
([pg. 1] There are two options of implementing streaming inference: with internal state (Fig 1b) and with external state (Fig 1c). A model with internal state receives a new input sample with dimensions 3x1 (grey box 3x1 on Fig 1b), appends it to the ring buffer State1i, and at the same time removes the oldest sample (marked by blue with dashed lines on State1i) from State1i. This way, State1i always has a shape of 3x3… In this case states are internal variables of the model which have to be updated with every prediction.)
NOTE: Teaches a circular FIFO buffer (state1i is a fixed size ring buffer where entries are written in first in first out behavior) having a size of a data patch (3x3) associated with an iterative update operation (states are updated every prediction, i.e., iteratively) associated with the respective layer (as pictured above, state1i is associated with a convolutional layer of the neural network model).
OBVIOUSNESS:
Using the same reasoning from claim 1.
Rybakov fails to teach but Poorna teaches;
wherein the processing circuitry comprises: a memory, which, in operation, maintains a respective
([0032] The deep learning accelerator (DLA) can have local memory, such as registers, buffers and/or caches, configured to store vector/matrix operands and the results of vector/matrix operations.
NOTE: Teaches the processing circuitry (the aforementioned processing unit which is used to implement the deep learning accelerator) comprising: a memory, which, in operation, maintains buffers storing results of vector/matrix operations (which could be convolutions, for example).
OBVIOUSNESS:
From the reasoning provided in claim 1, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to implement the claimed SCNN (as taught by Khandelwal and Rybakov) using the hardware of Poorna to reduce energy consumption and computation time of the system.
Additionally, Poorna states;
([0032] The deep learning accelerator (DLA) can have local memory, such as registers, buffers and/or caches, configured to store vector/matrix operands and the results of vector/matrix operations. Intermediate results in the registers can be pipelined/shifted in the deep learning accelerator (DLA) as operands for subsequent vector/matrix operations to reduce time and energy consumption in accessing memory/data and thus speed up typical patterns of vector/matrix operations in implementing a typical artificial neural network (ANN).)
NOTE: Storing intermediate values improves speed and reduces energy consumption.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to maintain the circular FIFO buffers of each layer using the memory disclosed by Poorna.
Regarding claim 12, Khandelwal in view of Rybakov and Poorna teaches;
The device according to claim 1,
(Using the same reasoning as claim 1)
Khandelwal and Rybakov fail to teach but Poorna teaches;
comprising: an integrated circuit, wherein the sensor and the processing circuitry are embedded in the integrated circuit.
PNG
media_image4.png
594
465
media_image4.png
Greyscale
NOTE: Teaches the device comprising: an integrated circuit (the aforementioned processing units 111), wherein the sensor and the processing circuitry are embedded in the integrated circuit (as pictured).
OBVIOUSNESS: Using the same reasoning from claim 1.
Regarding claim 14,
Claim 14 is a system claim substantially similar to device claim one except for the following limitation, which is taught by Poorna;
A system, comprising: a host device; and a sensing device coupled to the host device, wherein the sensing device, in operation: generates a sequence of data samples;
PNG
media_image4.png
594
465
media_image4.png
Greyscale
NOTE: Teaches a host device (integrated circuit device) and a sensing device coupled to the host device (the aforementioned sensor 102).
([0041] For example, a stream of sensor input data to the artificial neural network (ANN) can be configured in the form of a sequence of input data sets.)
NOTE: Teaches the sensing device generating a sequence (stream) of data samples
The remaining limitations are taught using the same reasoning as in claim 1.
Regarding claim 15, Khandelwal in view of Rybakov and Poorna teaches;
The system according to claim 14,
(Using the same reasoning as in claim 14)
Khandelwal and Rybakov fail to teach but Poorna teaches;
wherein the sensing device comprises: one or more sensors,
([0138] For example, the control unit (113) periodically retrieves measurements from the sensors (102) and stores the measurements into the random access memory (105) through the high bandwidth connection (119).)
NOTE: Teaches the sensing device (sensors 102) comprises one or more sensors (‘sensors’ is plural, indicating one or more sensors).
which, in operation, generate one or more sequences of data samples;
([0041] For example, a stream of sensor input data to the artificial neural network (ANN) can be configured in the form of a sequence of input data sets.)
NOTE: Teaches the sensors generating one or more sequences of data samples.
and processing circuitry coupled to the one or more sensors, wherein the processing circuitry, in operation, implements the
PNG
media_image4.png
594
465
media_image4.png
Greyscale
([0151] For example, the at least one processing unit (111) can be formed on an integrated circuit die of a field-programmable gate array (FPGA) or application specific integrated circuit (ASIC) implementing a deep learning accelerator)
([0024] an artificial neural network (ANN) implemented using the deep learning accelerator (DLA).)
NOTE: Teaches processing circuitry (processing units, which have circuitry) coupled to the one or more sensors (as pictured in fig. 7), wherein the processing circuitry, in operation, implements an ANN (the processing units implement the deep learning accelerator, which implements the ANN).
OBVIOUSNESS:
Using the same reasoning from claim 1, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to implement the SCNN as taught by Khandelwal in view of Rybakov using the hardware disclosed by Poorna to reduce the energy consumption and computation time of the system.
Regarding claim 16,
Claim 16 is a system claim that is substantially similar to device claim 2, and is rejected using the same reasoning.
Regarding claim 17, Khandelwal fails to teach but Rybakov teaches;
wherein the sequentially processing the sequence of data samples includes, for each received sample of the set of received data samples of the sequence of data samples, iteratively updating partial classification results of the SCNN.
([pg. 1] As a result, the convolution will be computed on the new input data only and all previous computations will be buffered, as shown on Fig 1 b. In this case states are internal variables of the model which have to be updated with every prediction.)
NOTE: Rybakov teaches a very similar process to Khandelwal of iteratively updating and buffering states representing previous computations (where the previous computations can be considered partial results) and removing oldest states for each received sample (new input data). Thus, Rybakov teaches iteratively updating partial results.
([Abstract] NN model conversion from non-streaming mode (model receives the whole input sequence and then returns the classification result) to streaming mode (model receives portion of the input sequence and classifies it incrementally) may require manual model rewriting.)
NOTE: Rybakov discloses classifying a set/portion of received data samples incrementally. Thus, Rybakov teaches for each received sample of the set of received data samples of the sequence of data samples (receives portion/set of the input sequence), iteratively updating (using the aforementioned process) partial classification results of the SCNN (classifies incrementally).
OBVIOUSNESS: Using the same reasoning from claim 1.
Regarding claim 19,
Claim 19 is a system claim that is substantially similar to device claim 9, and is rejected using the same reasoning.
Regarding claim 22,
Claim 22 is a method claim that is substantially similar to device claim 1, and is rejected using the same reasoning.
Regarding claim 23,
Claim 23 is a method claim that is substantially similar to device claim 2, and is rejected using the same reasoning.
Regarding claim 24, Khandelwal fails to teach but Rybakov teaches;
wherein the sequentially processing the sequence of data samples includes, for each received sample of the set of received data samples of the sequence of data samples, iteratively updating partial classification results of the SCNN.
([pg. 1] As a result, the convolution will be computed on the new input data only and all previous computations will be buffered, as shown on Fig 1 b. In this case states are internal variables of the model which have to be updated with every prediction.)
NOTE: Rybakov teaches a very similar process to Khandelwal of iteratively updating and buffering states representing previous computations (where the previous computations can be considered partial results) and removing oldest states for each received sample (new input data). Thus, Rybakov teaches iteratively updating partial results.
([Abstract] NN model conversion from non-streaming mode (model receives the whole input sequence and then returns the classification result) to streaming mode (model receives portion of the input sequence and classifies it incrementally) may require manual model rewriting.)
NOTE: Rybakov discloses classifying a set/portion of received data samples incrementally. Thus, Rybakov teaches for each received sample of the set of received data samples of the sequence of data samples (receives portion/set of the input sequence), iteratively updating (using the aforementioned process) partial classification results of the SCNN (classifies incrementally).
OBVIOUSNESS: Using the same reasoning from claim 1.
Regarding claim 26,
Claim 26 is a method claim that is substantially similar to device claim 9, and is rejected using the same reasoning.
Regarding claim 28, Khandelwal in view of Rybakov and Poorna teaches;
The method according to claim 22
(Using the same reasoning from claim 22)
Khandelwal teaches;
wherein the sequentially processing the sequence of data samples comprises: applying a stride parameter.
[pg. 4]
PNG
media_image7.png
682
524
media_image7.png
Greyscale
NOTE: Teaches the sequentially processing the sequence of data samples xt comprising applying a stride parameter (s[ ]).
Regarding claim 29,
Claim 29 is a computer readable medium (CRM) claim that is substantially similar to device claim 1 except for the preamble which is taught by Poorna;
A non-transitory computer-readable medium having contents which cause a sensing device to perform a method,
([0179] A machine readable medium can be used to store software and data which when executed by a data processing system causes the system to perform various methods.)
([0180] Examples of computer-readable media include but are not limited to non-transitory, recordable and non-recordable type media such as volatile and non-volatile memory devices,)
[0154] At block 305, a sensor (102) integrated in the device (e.g., 101 or 191) generates measurements of stimuli, such as acceleration, rotation, vibration, pressure, touch, sound, light, infrared radiation, stress, strain, etc.
NOTE: Teaches a non-transitory computer readable medium having contents which cause a sensing device to perform a method.
The remaining limitations are taught using the same reasoning provided in claim 1.
Regarding claim 30 Khandelwal fails to teach but Rybakov teaches;
wherein the sequentially processing the sequence of data samples includes, for each received sample of the set of received data samples of the sequence of data samples, iteratively updating partial classification results of the SCNN.
([pg. 1] As a result, the convolution will be computed on the new input data only and all previous computations will be buffered, as shown on Fig 1 b. In this case states are internal variables of the model which have to be updated with every prediction.)
NOTE: Rybakov teaches a very similar process to Khandelwal of iteratively updating and buffering states representing previous computations (where the previous computations can be considered partial results) and removing oldest states for each received sample (new input data). Thus, Rybakov teaches iteratively updating partial results.
([Abstract] NN model conversion from non-streaming mode (model receives the whole input sequence and then returns the classification result) to streaming mode (model receives portion of the input sequence and classifies it incrementally) may require manual model rewriting.)
NOTE: Rybakov discloses classifying a set/portion of received data samples incrementally. Thus, Rybakov teaches for each received sample of the set of received data samples of the sequence of data samples (receives portion/set of the input sequence), iteratively updating (using the aforementioned process) partial classification results of the SCNN (classifies incrementally).
Regarding claim 31, Khandelwal in view of Rybakov and Poorna teaches;
The non-transitory computer-readable medium of claim 29,
(Using the same reasoning from claim 29)
Khandelwal and Rybakov fail to teach but Poorna teaches;
wherein the contents comprise instructions executable by processing circuitry of the sensing device.
([0179] A machine readable medium can be used to store software and data which when executed by a data processing system causes the system to perform various methods.)
NOTE: Teaches the contents comprising instructions (software) executable by processing circuitry of the sensing device (executed by a data processing system, such as the aforementioned processing circuitry disclosed by Poorna)
OBVIOUSNESS: Using the same reasoning from claim 1.
Regarding claim 32, Khandelwal fails to explicitly teach but Rybakov teaches;
wherein the plurality of layers of the SCNN include convolutional, pooling, batch normalization ([pg. 3] DSCNN[21] models … using a sequence of 2D convolutional … followed by batch normalization with average pooling … DSCNN … can be automatically converted to streaming mode), and flattening layers ([pg. 2] Stream(cell=tf.keras.layers.Flatten(...)))
OBVIOUSNESS: Using the same reasoning from claim 1.
Regarding claims 33-35
Claims 33-35 are substantially similar to claim 32, and are rejected using the same reasoning.
Claim(s) 13, 21, 27 is/are rejected under 35 U.S.C. 103 as being unpatentable over Khandelwal (“Efficient Real-Time Inference in Temporal Convolution Networks”, 05/31/2021) in view of Rybakov (“Streaming keyword spotting on mobile devices”, 07/29/2020) further in view of Poorna (US 20230161936 A1, 05/25/2023) further in view of Cabrita Condesaa Filipe J. (hereinafter Filipe) (“US 20210182731 A1”, 06/17/2021).
Regarding claim 13, Khandelwal in view of Rybakov and Poorna teach;
The device according to claim 1
(Using the same reasoning from claim 1)
Khandelwal, Rybakov, and Poorna fail to teach but Filipe teaches;
wherein the processing circuitry, in operation, generates one or more control signals based on the classification results.
([0017] In the system 100, the processor 102 may include one or more integrated circuits that implement the functionality of a central processing unit (CPU) and/or graphics processing unit (GPU).)
NOTE: The system includes processing circuitry (processor 102 including integrated circuits).
([0048] Using the generative models 114, the sensor data may be accurately classified into the K classes. Based on the classification of the sensor data, the system 100 may compute one or more control signals for controlling a physical system, such as a computer-controlled machine, like a robot, a vehicle, a domestic appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system.)
NOTE: Teaches the processing circuitry (the processor of the system 100), in operation, generates one or more control signals based on the classification results (compute control signals based on classifications of the sensor data).
OBVIOUSNESS TO COMBINE FILIPE WITH KHANDELWAL, RYBAKOV, AND POORNA:
Filipe is analogous art to the present disclosure as it pertains to classifying sensor data using machine learning models.
Additionally, Filipe states;
([0015] Such techniques provide practical applications by being applicable to received sensor signals from a wide variety of sensors (such as video, radar, LiDAR, ultrasonic and/or motion sensors), and may be used to compute a control signal for controlling a physical system (such as a computer-controlled machine, like a robot, a vehicle, a domestic appliance, a power tool, a manufacturing machine, a personal assistant or an access control system). The described techniques do so by classifying the sensor data and explicitly training on in-distribution samples and finding a proxy for out-of-distribution samples.)
NOTE: Filipe details that the disclosed techniques provide practical applications, such as controlling physical systems using sensor data.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to generate control signals from the classification results of the SCNN to provide practical applications for the system.
Regarding claim 21,
Claim 21 is a system claim that is substantially similar to device claim 13, and is rejected using the same reasoning.
Regarding claim 27,
Claim 27 is a method claim that is substantially similar to device claim 13, and is rejected using the same reasoning.
Response to Arguments
Applicant's arguments filed 7/22/2026 regarding the rejections under 35 USC 103 have been fully considered but they are not persuasive. Regarding amended claim 1, the applicant states that “Khandelwal appears to discuss only convolutional layers, and thus does not disclose implementing a CNN as a SCNN having a plurality of different types of layers, or iteratively updating partial results of subsequent layers of such a plurality of layers of the SCNN based on the iteratively updated partial results of the first layer of the SCNN. Rybakov, Pooma and Filipe do not appear to cure these deficiencies. Accordingly, independent claim 1, as amended, is allowable over the cited references. While the language and scope of independent claims 14, 22, and 29, as amended, are not identical to the language and scope of independent claim 1, as amended, independent claims 14, 22 and 29 are allowable for reasons that will be apparent in view of the discussion herein. The dependent claims, including new claims 32-35, are allowable at least by reason of their dependencies.” Examiner respectfully disagrees. As reflected by the present office action, Claim 1 is rejected by Khandelwal in view of Rybakov further in view of Poorna, where Khandelwal does teach implementing a CNN as a SCNN (see pages 2-3 of the present office action) and iteratively updating partial results of subsequent layers of such plurality of layers of the SCNN based on the iteratively updated partial results of the first layer of the SCNN (see pages 8-9 of the present office action), while Rybakov is relied upon for teaching the plurality of layers including different types of layers and a last layer of the plurality of layers being a dense layer (see pages 9-10 of the present office action). Therefore, the 35 USC 103 rejections of claim 1 stand. Accordingly, the 35 USC 103 rejections of independent claims 14, 22, and 29 as well as the corresponding dependent claims 2-4, 7, 9, 12-13, 15-17, 19, 21, 23-24, 26-28, 30-31 stand, using for the corresponding reasons set forth in the respective rejections. Additionally, Rybakov teaches the limitations introduced in the new claims 32-35 (see page 33 of the current office action). As such, new claims 32-35 are also rejected under 35 USC 103 by Khandelwal in view of Rybakov further in view of Poorna.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Matthew Alan Cady whose telephone number is (571) 272-7229. The examiner can normally be reached Monday - Friday, 7:30 am - 5:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached on (571)272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC)
at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW ALAN CADY/ Examiner, Art Unit 2145
/CESAR B PAULA/ Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights