DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 17, 19, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bigioi et al. (hereinafter Bigioi), U.S. Patent Application Publication 2019/0065410 in view of Carmona-Galan (hereinafter Carmona), A hierarchical vision processing architecture oriented to 3D integration of smart camera chips, further in view of Green et al. (hereinafter Green), U.S. Patent Application Publication 2015/0239482.
Regarding Claim 1, Bigioi discloses a system comprising:
an image sensor configured to generate first image data [“CCD sensors” ¶75; “various image sensors including: a conventional camera (VIS sensor), a NIR sensitive camera, and a thermal imaging camera” ¶78];
a memory device configured to store the first image data [“a shared memory 40’” ¶50; “a memory controller 210 arrange for the transfer of initial image information…into the shared memory 40'” ¶50]; and
a host interface of the image sensor configured to communicate with a host system [“a memory controller 210 arrange for the transfer of initial image information as well as network configuration information from the either the memory 99, 102 into the shared memory 40'” ¶50; “the results of processing by one CNN within a given cluster 92 can be communicated through the SRAM controller 210 for further processing by another CNN in either the same or another cluster 92” ¶63; “channelled through the host CPU” ¶60];
wherein the memory device is stacked with the image sensor [“simple stacking of other die to the DBI bonded processor/memory stack, for example, CCD sensors, MEMS devices can also be achieved using Zibond techniques” ¶75], and the memory device comprises an inference engine configured to generate inference results using the first image data as input to a first artificial neural network (ANN) [“the PCNN cluster 92 and stacked memory die” ¶75; “a programmable CNN (PCNN) engine” ¶12];
wherein the inference engine includes a neural network accelerator [“providing a general purpose neural processor, capable of loading its networks ( e.g. equivalent of the program in a traditional CPU) through a separate memory channel 38 from the input image data 36 and results 39.” ¶12; Note: A neural network accelerator is a processor that is optimized specifically to handle neural network workloads.] configured to perform matrix arithmetic computations on the first image data stored in the memory device [“convolution layers require a lot of multiply accumulate type of instructions” ¶9]; and
wherein the host interface of the image sensor is further configured to output the inference results to the host system [“the results of processing by one CNN within a given cluster 92 can be communicated through the SRAM controller 210 for further processing by another CNN in either the same or another cluster 92” ¶63; “channelled through the host CPU” ¶60] in response to a determination that the inference results are within a predetermined tolerance of other inference results generated from sensor data generated by a sensor device other than the image sensor [“a programmable CNN (PCNN) engine” ¶12; “output neurons, each providing a value for each category of object the network is attempting to identify in an input image” ¶5].
However, Bigioi fails to explicitly disclose a host interface of the image sensor configured to communicate;
wherein the host interface of the image sensor is further configured to output.
Carmona discloses a host interface of the image sensor configured to communicate [Fig. 5; Note: While Bigioi discloses an interface for communicating with the system including the host device, the sensor on the stack that uses the controller does not explicitly recite the sensors are part of that communication. In Carmona, the stack including the sensor is disclosed having a communication pathway (e.g., Data I/O) in tier 2 which uses the sensor interface in tier 3 to send data from the sensor.];
wherein the host interface of the image sensor is further configured to output [Fig. 5; Note: While Bigioi discloses an interface for communicating with the system including the host device, the sensor on the stack that uses the controller does not explicitly recite the sensors are part of that communication. In Carmona, the stack including the sensor is disclosed having a communication pathway (e.g., Data I/O) in tier 2 which uses the sensor interface in tier 3 to send data from the sensor.];].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bigioi and Carmona before him before the effective filing date of the claimed invention, to modify the system of Bigioi to incorporate the communication aspects of Carmona.
Given the advantage of giving the senor the ability to communication the information it acquires in order to process the information, one having ordinary skill in the art would have been motivated to make this obvious modification.
However, Bigioi fails to explicitly disclose in response to a determination that the inference results are within a predetermined tolerance of other inference results generated from sensor data generated by a sensor device other than the image sensor.
Green discloses in response to a determination that the inference results are within a predetermined tolerance of other inference results generated from sensor data generated by a sensor device other than the image sensor [“Data is received from at least two sensors” ¶51; “Resolving conflicts between the received data results is performed when data received from one sensor does not substantially match with data received by the other sensor. In some embodiments, a predetermine tolerance threshold is established for determining whether a conflict exists within the received data.” ¶54; “In some embodiments, a conflict is identified if an object is detected by one sensor but the object is not detected by the other sensor.” ¶54; Note: A person having ordinary skill in the art knows that one sensor outside the predetermined tolerance would be excluded given several other sensors being in agreement and within the tolerance.].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bigioi, Carmona, and Green before him before the effective filing date of the claimed invention, to modify the combination to incorporate conflict resolution to ensure accurate data from among a plurality of devices of Green.
Given the advantage of only processing valid data for higher accuracy, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 17, Bigioi discloses a method comprising:
generating, by an image sensor, image data [“various image sensors including: a conventional camera (VIS sensor), a NIR sensitive camera, and a thermal imaging camera” ¶78];
storing, by a memory device, the image data [“a memory controller 210 arrange for the transfer of initial image information as well as network configuration information from the either the memory 99, 102 into the shared memory 40'” ¶50], wherein the memory device comprises an inference engine configured to generate inference results using the image data as input to a first portion of an artificial neural network (ANN) [“the PCNN cluster 92 and stacked memory die” ¶75; “a programmable CNN (PCNN) engine” ¶12], and wherein the inference engine includes a neural network accelerator [“providing a general purpose neural processor, capable of loading its networks ( e.g. equivalent of the program in a traditional CPU) through a separate memory channel 38 from the input image data 36 and results 39.” ¶12; Note: A neural network accelerator is a processor that is optimized specifically to handle neural network workloads.] configured to perform matrix arithmetic computations on the image data [“convolution layers require a lot of multiply accumulate type of instructions” ¶9]; and
communicating by a host interface of the image sensor with a host system, wherein the host interface of the image sensor is configured to send the inference results to the host system [“the results of processing by one CNN within a given cluster 92 can be communicated through the SRAM controller 210 for further processing by another CNN in either the same or another cluster 92” ¶63; “channelled through the host CPU” ¶60] in response to a determination that the inference results are within a predetermined tolerance of other inference results generated from sensor data generated by a sensor device other than the image sensor [“a programmable CNN (PCNN) engine” ¶12; “output neurons, each providing a value for each category of object the network is attempting to identify in an input image” ¶5].
However, Bigioi fails to explicitly disclose communicating by a host interface of the image sensor;
wherein the host interface of the image sensor is further configured to output.
Carmona discloses communicating by a host interface of the image sensor [Fig. 5; Note: While Bigioi discloses an interface for communicating with the system including the host device, the sensor on the stack that uses the controller does not explicitly recite the sensors are part of that communication. In Carmona, the stack including the sensor is disclosed having a communication pathway (e.g., Data I/O) in tier 2 which uses the sensor interface in tier 3 to send data from the sensor.];
wherein the host interface of the image sensor is further configured to output [Fig. 5; Note: While Bigioi discloses an interface for communicating with the system including the host device, the sensor on the stack that uses the controller does not explicitly recite the sensors are part of that communication. In Carmona, the stack including the sensor is disclosed having a communication pathway (e.g., Data I/O) in tier 2 which uses the sensor interface in tier 3 to send data from the sensor.].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bigioi and Carmona before him before the effective filing date of the claimed invention, to modify the system of Bigioi to incorporate the communication aspects of Carmona.
Given the advantage of giving the senor the ability to communication the information it acquires in order to process the information, one having ordinary skill in the art would have been motivated to make this obvious modification.
However, Bigioi fails to explicitly disclose in response to a determination that the inference results are within a predetermined tolerance of other inference results generated from sensor data generated by a sensor device other than the image sensor.
Green discloses in response to a determination that the inference results are within a predetermined tolerance of other inference results generated from sensor data generated by a sensor device other than the image sensor [“Data is received from at least two sensors” ¶51; “Resolving conflicts between the received data results is performed when data received from one sensor does not substantially match with data received by the other sensor. In some embodiments, a predetermine tolerance threshold is established for determining whether a conflict exists within the received data.” ¶54; “In some embodiments, a conflict is identified if an object is detected by one sensor but the object is not detected by the other sensor.” ¶54; Note: A person having ordinary skill in the art knows that one sensor outside the predetermined tolerance would be excluded given several other sensors being in agreement and within the tolerance.].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bigioi, Carmona, and Green before him before the effective filing date of the claimed invention, to modify the combination to incorporate conflict resolution to ensure accurate data from among a plurality of devices of Green.
Given the advantage of only processing valid data for higher accuracy, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 19, Bigioi, Carmona, and Green disclose the method of claim 17. Bigioi further discloses further comprising:
receiving, by the host interface, sensor data from the host system [“host 80 prepares data for processing, for example, an image or images, and writes the image(s)” ¶41];
wherein the inference engine is further configured to generate the inference results using the sensor data as additional input to the first portion of the ANN [“the results of processing by one CNN within a given cluster 92 can be communicated through the SRAM controller 210 for further processing by another CNN in either the same or another cluster 92” ¶63; “channelled through the host CPU” ¶60].
Regarding Claim 20, Bigioi, Carmona, and Green disclose the method of claim 17. Bigioi further discloses further comprising:
storing the first portion of the ANN in the memory device [“one CNN” ¶63];
wherein the host system stores a second portion of the ANN [“another CNN” ¶63];
wherein generating the inference results comprises using the image data as input to the first portion of the ANN [“input image data” ¶12; Fig. 1]; and
wherein the processing by the host system comprises using the inference results as input to the second portion of the ANN to provide a result for controlling a vehicle [“daisy chained clusters can intercommunicate under the control of the CPU 50/50-A,50-B and the local PCNN CPUs 200 so that the results of processing by one CNN within a given cluster 92 can be communicated through the SRAM controller 210 for further processing by another CNN in either the same or another cluster 92” ¶63].
Claim(s) 2-7, 9, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bigioi, Carmona, and Green in view of Chen et al. (hereinafter Chen), SubMac: Exploiting the subword-based computation in RRAM-based CNN accelerator for energy saving and speedup.
Regarding Claim 2, Bigioi, Carmona, and Green disclose the system of claim 1.
However, Bigioi fails to explicitly disclose wherein the memory device is a neuromorphic memory device, and the neural network accelerator includes a memristor crossbar array configured to perform the matrix arithmetic computations.
Chen discloses wherein the memory device is a neuromorphic memory device [“resistive random access memory (RRAM) for implementing the CNN accelerators” §1 ¶3], and the neural network accelerator includes a memristor crossbar array configured to perform the matrix arithmetic computations [“RRAM crossbar contains an array of the memristor devices” §3.2 ¶1; “Each activation in the receptive field is multiplied with the corresponding weight and accumulated with other products of the neuron to generate an output activation” §3.1 ¶1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bigioi, Carmona, Green, and Chen before him before the effective filing date of the claimed invention, to modify the system of Bigioi to incorporate the neuromorphic memory and crossbar array of Chen.
Given the advantage of being able to both perform logic and store data, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 3, Bigioi, Carmona, Green, and Chen disclose the system of claim 2. Bigioi further discloses wherein the matrix arithmetic computations include matrix multiplication and accumulation operations [“convolution layers require a lot of multiply accumulate type of instructions” ¶9].
Regarding Claim 4, Bigioi, Carmona, Green, and Chen disclose the system of claim 2.
However, Bigioi fails to explicitly disclose wherein the first image data is input to the memristor crossbar array.
Chen discloses wherein the first image data is input to the memristor crossbar array [“image classification” §1 ¶1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bigioi, Carmona, Green, and Chen before him before the effective filing date of the claimed invention, to modify the system of Bigioi to incorporate the memristor crossbar array of Chen.
Given the advantage of being able to both perform logic and store data, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 5, Bigioi, Carmona, Green, and Chen disclose the system of claim 2.
However, Bigioi fails to explicitly disclose wherein the memristor crossbar array comprises memristors, and each memristor is connected between a wordline and a bitline.
Chen discloses wherein the memristor crossbar array comprises memristors, and each memristor is connected between a wordline and a bitline [“memristor at the crosspoint has two terminals, one connected to the horizontal wordline and the other connected to the vertical bitline” §3.2 ¶1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bigioi, Carmona, Green, and Chen before him before the effective filing date of the claimed invention, to modify the system of Bigioi to incorporate the memristor crossbar array of Chen.
Given the advantage of being able to both perform logic and store data, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 6, Bigioi, Carmona, Green, and Chen disclose the system of claim 5.
However, Bigioi fails to explicitly disclose wherein: currents in bitlines of the memristor crossbar array correspond to summation of multiplications of weights and responses of neurons in the first ANN;
the neurons are implemented via programmed resistances of the memristors in the memristor crossbar array; and
voltages of wordlines of the memristor crossbar array represent input to the neurons.
Chen discloses wherein: currents in bitlines of the memristor crossbar array correspond to summation of multiplications of weights and responses of neurons in the first ANN [“weight matrix is stored in the memristor array to perform the in-situ computation” & “branch currents on the same bitline are then accumulated” §3.2 ¶1; Fig. 2];
the neurons are implemented via programmed resistances of the memristors in the memristor crossbar array [“electrical conductance of the memristor depends on the history of the current flowing through the device” & “weight matrix is stored in the memristor array” §3.2 ¶1; Fig. 2]; and
voltages of wordlines of the memristor crossbar array represent input to the neurons [“input to the RRAM crossbar as voltages at the wordlines” §3.2 ¶1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bigioi, Carmona, Green, and Chen before him before the effective filing date of the claimed invention, to modify the system of Bigioi to incorporate the memristor crossbar array of Chen.
Given the advantage of being able to both perform logic and store data, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 7, Bigioi, Carmona, Green, and Chen disclose the system of claim 6.
However, Bigioi fails to explicitly disclose wherein the memristor crossbar array performs multiply- and-accumulate (MAC) operations by converting the voltages on the wordlines to currents on the bitlines.
Chen discloses wherein the memristor crossbar array performs multiply- and-accumulate (MAC) operations by converting the voltages on the wordlines to currents on the bitlines [“memristor at the crosspoint has two terminals, one connected to the horizontal wordline and the other connected to the vertical bitline” §3.2 ¶1; Fig. 2; “Each activation in the receptive field is multiplied with the corresponding weight and accumulated with other products of the neuron to generate an output activation” §3.1 ¶1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bigioi, Carmona, Green, and Chen before him before the effective filing date of the claimed invention, to modify the system of Bigioi to incorporate the memristor crossbar array of Chen.
Given the advantage of being able to both perform logic and store data, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 9, Bigioi, Carmona, and Green discloses the system of claim 1.
However, Bigioi fails to explicitly disclose wherein the memory device is a resistive random-access memory (RRAM).
Chen discloses wherein the memory device is a resistive random-access memory (RRAM) [“Resistive RAM-based accelerators” Abstract].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bigioi, Carmona, Green, and Chen before him before the effective filing date of the claimed invention, to modify the system of Bigioi to incorporate the RRAM of Chen.
Given the advantage of the RRAM crossbars which store the weights to eliminate the need for fetching the weights before computation, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 18, Bigioi, Carmona, and Green discloses the method of claim 17. Bigioi further discloses the matrix arithmetic computations include matrix multiplication and accumulation operations [“convolution layers require a lot of multiply accumulate type of instructions” ¶9].
However, Bigioi fails to explicitly disclose wherein the memory device is a neuromorphic memory device, the neural network accelerator includes a memristor array configured to perform the matrix arithmetic computations, …, and the method further comprises providing the image data as input to the memristor array.
Chen discloses wherein the memory device is a neuromorphic memory device [“resistive random access memory (RRAM) for implementing the CNN accelerators” §1 ¶3], the neural network accelerator includes a memristor array configured to perform the matrix arithmetic computations [“RRAM crossbar contains an array of the memristor devices” §3.2 ¶1; “Each activation in the receptive field is multiplied with the corresponding weight and accumulated with other products of the neuron to generate an output activation” §3.1 ¶1], …, and the method further comprises providing the image data as input to the memristor array [“image classification” §1 ¶1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bigioi, Carmona, Green, and Chen before him before the effective filing date of the claimed invention, to modify the system of Bigioi to incorporate the memristor crossbar array of Chen.
Given the advantage of being able to both perform logic and store data, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bigioi, Carmona, and Green in view of Querlioz et al. (hereinafter Querlioz), Simulation of a Memristor-Based Spiking Neural Network Immune to Device Variations.
Regarding Claim 8, Bigioi, Carmona, and Green discloses the system of claim 1.
However, Bigioi fails to explicitly disclose wherein the first ANN comprises a spiking neural network (SNN).
Querlioz discloses wherein the first ANN comprises a spiking neural network (SNN) [“spiking neural network” Abstract].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bigioi, Carmona, Green, and Querlioz before him before the effective filing date of the claimed invention, to modify the system of Bigioi to incorporate SNN of Querlioz.
Given the advantage of computing performances and robustness to variability, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claim(s) 10, 12, 14, 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bigioi in view of Polikar, Ensemble Based Systems in Decision Making.
Regarding Claim 10, Bigioi discloses a system comprising:
a first camera configured to generate a first image stream [“connection to various image sensors including: a conventional camera (VIS sensor), a NIR sensitive camera, and a thermal imaging camera” ¶78], wherein the first camera comprises a first inference engine configured to generate a first intermediate result using the first image stream as input to a first artificial neural network (ANN) [“one CNN” ¶63];
a second camera configured to generate a second image stream [“connection to various image sensors including: a conventional camera (VIS sensor), a NIR sensitive camera, and a thermal imaging camera” ¶78], wherein the second camera comprises a second inference engine configured to generate a second intermediate result using the second image stream as input to a second artificial neural network (ANN) [“another CNN” ¶63];
a processing device or logic circuit configured to communicate with the first camera and the second camera [“a host device” ¶17]; and
wherein the first inference engine uses the first intermediate result and the second intermediate result as input to the first ANN to generate a first final result [“the results of processing by one CNN within a given cluster 92 can be communicated through the SRAM controller 210 for further processing by another CNN in either the same or another cluster 92” ¶62]; and
wherein the second inference engine uses the first intermediate result and the second intermediate result as input to the second ANN to generate a second final result [“the results of processing by one CNN within a given cluster 92 can be communicated through the SRAM controller 210 for further processing by another CNN in either the same or another cluster 92” ¶62].
However, Bigioi fails to explicitly disclose wherein the processing device or logic circuit is further configured to determine whether the first final result matches the second final result to generate an output including a final result.
Polikar discloses wherein the processing device or logic circuit is further configured to determine whether the first final result matches the second final result to generate an output including a final result [“three versions of majority voting, where the ensemble choose the class (i) on which all classifiers agree (unanimous voting)” §4.1.1 ¶1], wherein the determining of whether the first final result matches the second final result comprises determining whether the first final result is within a predetermined tolerance of the second final result [“three versions of majority voting, where the ensemble choose the class (i) on which all classifiers agree (unanimous voting)” §4.1.1 ¶1; Note: the predetermined tolerance would be the results are the same].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bigioi and Polikar before him before the effective filing date of the claimed invention, to modify the system of Bigioi to incorporate the voting system of Polikar.
Given the advantage of combining outputs from more than one model to increase overall accuracy, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 12, Bigioi and Polikar disclose the system of claim 10. Bigioi further discloses further comprising:
a sensing device or a storage device [“simple stacking of other die to the DBI bonded processor/memory stack, for example, CCD sensors, MEMS devices can also be achieved using Zibond techniques” ¶75] configured to generate an inference result [“the PCNN cluster 92 and stacked memory die” ¶75; “a programmable CNN (PCNN) engine” ¶12], wherein the sensing device or storage device comprises an interface to communicate the inference result to the processing device or logic circuit [“external interface block 95A with one or more serial peripheral interfaces (SPIs) enables the host processors 50 to connect to other processors within a vehicle network (not shown) and indeed a wider network environment” ¶77, 78].
However, Bigioi fails to explicitly disclose wherein the processing device or logic circuit comprises a majority voter configured to provide an output to a host system, and wherein an inference result from the first ANN, an inference result from the second ANN, and an inference result from the sensing device or storage device are input to the majority voter.
Polikar discloses wherein the processing device or logic circuit comprises a majority voter configured to provide an output to a host system, and wherein an inference result from the first ANN, an inference result from the second ANN, and an inference result from the sensing device or storage device are input to the majority voter [“majority voting” §4.1.1 ¶1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bigioi and Polikar before him before the effective filing date of the claimed invention, to modify the system of Bigioi to incorporate the majority voting of Polikar.
Given the advantage of combining outputs from more than one model to increase overall accuracy, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 14, Bigioi and Polikar disclose the system of claim 10. Bigioi further discloses wherein the first camera further comprises:
an image sensor configured to generate the first image stream [“various image sensors including: a conventional camera (VIS sensor), a NIR sensitive camera, and a thermal imaging camera” ¶78]; and
a memory device configured to store the first image stream [“a memory controller 210 arrange for the transfer of initial image information as well as network configuration information from the either the memory 99, 102 into the shared memory 40'” ¶50];
wherein the memory device is stacked with the image sensor [“simple stacking of other die to the DBI bonded processor/memory stack, for example, CCD sensors, MEMS devices can also be achieved using Zibond techniques” ¶75], and the memory device includes the first inference engine [“the PCNN cluster 92 and stacked memory die” ¶75; “a programmable CNN (PCNN) engine” ¶12]; and
wherein the first inference engine includes a neural network accelerator [“providing a general purpose neural processor, capable of loading its networks ( e.g. equivalent of the program in a traditional CPU) through a separate memory channel 38 from the input image data 36 and results 39.” ¶12; Note: A neural network accelerator is a processor that is optimized specifically to handle neural network workloads.] configured to perform matrix arithmetic computations on data stored in the memory device [“convolution layers require a lot of multiply accumulate type of instructions” ¶9].
Regarding Claim 21, Bigioi and Polikar disclose the system of claim 10. Bigioi further discloses communicate, via the host interface, an output to the host system for processing [“signalling between the host device and peripheral processing device” ¶18].
However, Bigioi fails to explicitly disclose wherein the processing device or logic circuit is further configured to:
in response to determining that the first final result matches the second final result, …, wherein the output is based on at least one of the first final result or the second final result.
Polikar discloses wherein the processing device or logic circuit is further configured to:
in response to determining that the first final result matches the second final result, …, wherein the output is based on at least one of the first final result or the second final result [“three versions of majority voting, where the ensemble choose the class (i) on which all classifiers agree (unanimous voting)” §4.1.1 ¶1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bigioi and Polikar before him before the effective filing date of the claimed invention, to modify the system of Bigioi to incorporate the comparison of model outputs of Polikar.
Given the advantage of combining outputs from more than one model to increase overall accuracy, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bigioi and Polikar in view of Naser et al. (hereinafter Naser), U.S. Patent Application Publication 2020/0143179.
Regarding Claim 11, Bigioi and Polikar the system of claim 21.
However, Bigioi fails to explicitly disclose wherein a memory device of the system is configured to store the first image stream in a cyclic manner.
Naser discloses wherein a memory device of the system is configured to store the first image stream in a cyclic manner [“a cyclic buffer 270 is run with an image stream 252 from a camera” ¶34].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bigioi, Polikar, and Naser before him before the effective filing date of the claimed invention, to modify the combination to incorporate the cyclic memory of Naser.
Given the advantage of retaining as much data as possible on limited memory, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bigioi and Polikar in view of Ito et al., (hereinafter Ito) U.S. Patent Application Publication 2022/0036190.
Regarding Claim 13, Bigioi and Polikar disclose the system of claim 10. Bigioi further discloses wherein a host system is configured to receive, from the processing device or logic circuit via a host interface, at least one of the first final result or the second final result, and the host system [“the results of processing by one CNN within a given cluster 92 can be communicated through the SRAM controller 210 for further processing by another CNN in either the same or another cluster 92” ¶63; “channelled through the host CPU” ¶60].
However, Bigioi fails to explicitly disclose further comprising:
a control for at least one of steering, braking, or acceleration of a vehicle;
is further configured to generate input for the control based on the at least one of the first final result or the second final result.
Ito discloses further comprising:
a control for at least one of steering, braking, or acceleration of a vehicle;
is further configured to generate input for the control based on the at least one of the first final result or the second final result [“In the autonomous driving system, a vehicle 10 that performs autonomous driving using a neural network and a server 20 that periodically notifies the vehicle 10 of a parameter of the neural network are connected by wireless communication as illustrated in FIG. 1. The vehicle 10 is provided with: a camera 120 that captures an image of the periphery of the vehicle; a vehicle control unit 110 that controls the autonomous driving of the vehicle 10; and an actuator 130 used for various operations such as an accelerator, a brake, and a steering.” ¶24].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bigioi, Polikar, and Ito before him before the effective filing date of the claimed invention, to modify the combination to incorporate the autonomous driving based on camera images of Ito.
Given the advantage of providing a self-driving vehicle, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bigioi and Polikar, in view of Chen.
Regarding Claim 15, Bigioi and Polikar discloses the system of claim 14.
However, Bigioi fails to explicitly disclose wherein the neural network accelerator includes a memristor crossbar array configured to perform the matrix arithmetic computations.
Chen discloses wherein the neural network accelerator includes a memristor crossbar array configured to perform the matrix arithmetic computations [“RRAM crossbar contains an array of the memristor devices” §3.2 ¶1; “Each activation in the receptive field is multiplied with the corresponding weight and accumulated with other products of the neuron to generate an output activation” §3.1 ¶1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bigioi and Polikar, Chen before him before the effective filing date of the claimed invention, to modify the combination to incorporate the memristor crossbar array of Chen.
Given the advantage of being able to both perform logic and store data, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bigioi and Polikar, in view of Querlioz.
Regarding Claim 16, Bigioi and Polikar discloses the system of claim 10.
However, Bigioi fails to explicitly disclose wherein the first ANN comprises a spiking neural network (SNN).
Querlioz discloses wherein the first ANN comprises a spiking neural network (SNN) [“spiking neural network” Abstract].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bigioi, Polikar, and Querlioz before him before the effective filing date of the claimed invention, to modify the combination to incorporate SNN of Querlioz.
Given the advantage of computing performances and robustness to variability, one having ordinary skill in the art would have been motivated to make this obvious modification.
Examiner’s Note
The Examiner respectfully requests of the Applicant in preparing responses, to fully consider the entirety of the reference(s) as potentially teaching all or part of the claimed invention. It is noted, REFERENCES ARE RELEVANT AS PRIOR ART FOR ALL THEY CONTAIN. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain.” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)). A reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art, including non-preferred embodiments (see MPEP 2123). The Examiner has cited particular locations in the reference(s) as applied to the claim(s) above for the convenience of the Applicant. Although the specified citations are representative of the teachings of the art and are applied to the specific limitations within the individual claim(s), typically other passages and figures will apply as well.
Additionally, any claim amendments for any reason should include remarks indicating clear support in the originally filed specification.
Response to Arguments
Regarding Applicant’s arguments concerning claims 1 and 17, Applicant's arguments with respect to the claims have been considered but are moot because the arguments do not apply to the references being used in the current rejection of the limitations.
Regarding Applicant’s arguments concerning claim 10 and Polikar, Applicant's arguments have been fully considered but have been found unpersuasive. Applicant argues that 1) Polikar does not disclose determining whether the first and second results match, and 2) Polikar does not disclose within a predetermined tolerance. Examiner disagrees for at least the following reasons.
First, Polikar discloses various voting systems between the classifiers of an ensemble in section 4.1.1. One of these systems is the unanimous voting system, where all classifiers match. Accordingly, in order to determine whether it is unanimous, Polikar has to determine if the results match.
Second, the term “predetermined tolerance” is not defined in the claim. Therefore, the results are matching using a predetermined tolerance of being the same.
Accordingly, the rejections are maintained.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT H BEJCEK II whose telephone number is (571)270-3610. The examiner can normally be reached Monday - Friday: 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T. Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/R.B./ Examiner, Art Unit 2148
/MICHELLE T BECHTOLD/ Supervisory Patent Examiner, Art Unit 2148
Prosecution Insights