DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 23 July 2026 has been entered.
Response to Amendment
Applicant’s response, filed 23 July 2026, to the last office action has been entered and made of record.
In response to the cancellation of claims 9 and 18, they are acknowledged and made of record.
In response to the amendments to the claims, they are acknowledged, supported by the original disclosure, and no new matter is added.
Amendments to the independent claims 1, 10, and 19 have necessitated an updated ground of rejection over the applied prior art. Please see below for the updated interpretations and rejections.
Response to Arguments
Applicant's arguments filed 23 July 2026 have been fully considered but they are not persuasive.
In response to Applicant’s arguments on p. 11-15 of Applicant’s reply, that the combined teachings of Balasubramaniyan, Tran, and Matsumoto fail to teach or suggest the limitations of “wherein each of the set of convolution operations is at least one of a dilation convolution, a fast convolution, or a functional safety convolution based on a user-defined configuration of the reconfigurable convolutional cluster engine, wherein: the user-defined configuration, for the dilation convolution, comprises a dilation rate of the input feature map; the fast convolution comprises employing a convolution grid engine (CGRID); the functional safety convolution comprises enabling at least one of a double- module redundancy (DMR), a triple-module redundancy (TMR), and one or more diagnostic features”, the Examiner respectfully disagrees.
Examiner notes the claims are treated with their broadest reasonable interpretations consistent with the specification. See MPEP 2111. Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). Furthermore, the test for obviousness is what the combined teachings of the references would have suggested to those of ordinary skill in the art. See In re Keller, 642 F.2d 413, 208 USPQ871 (CCPA 1981).
The amended independent claims are noted to recite, “wherein each of the set of convolution operations is at least one of a dilation convolution, a fast convolution, or a functional safety convolution based on a user-defined configuration of the reconfigurable convolutional cluster engine”. The broadest reasonable interpretation the recited claim language does not require that the each convolutional operation to be selectively executed in different modes based upon a user defined configuration, but that each of the convolution operations is one of “a dilation convolution”, “a fast convolution”, or “a functional safety convolution” based on a user-defined configuration of the reconfigurable convolutional cluster engine.
Furthermore, for the scenario where the convolution operations is “a functional safety convolution”, the broadest reasonable interpretation for the recited claim features “wherein: the user-defined configuration, for the dilation convolution, comprises a dilation rate of the input feature map; the fast convolution comprises employing a convolution grid engine (CGRID); the functional safety convolution comprises enabling at least one of a double- module redundancy (DMR), a triple-module redundancy (TMR), and one or more diagnostic features” would only necessarily require that the corresponding “functional safety convolution comprises enabling at least one of a double- module redundancy (DMR), a triple-module redundancy (TMR), and one or more diagnostic features”.
Balasubramaniyan and Tran are relied upon to teach a method for implementing a convolution neural network in the context of a reconfigurable convolution engine using a convolution operator system comprised of a set of computing blocks or a plurality of groups of computing blocks for performing convolution operations for input image data according to received kernel value and image feature matrices and generates convolution outputs for calculating the output feature maps for the convolutional neural network (see Balasubramaniyan [0003]-[0005], [0025], [0032]-[0033], [0035]-[0041], and [0050]; see Tran [0061]-[0064]).
Matsumoto is further relied upon to teach a known technique for implementing a fault tolerant cellular neural network architecture, where a triple modular redundancy technique is implemented in the neural network architecture in which three modules perform the same operation in parallel, and the output is decided by majority voting (see Matsumoto sect. 3. Fault Tolerance in Small World Cellular Neural Networks, Fig. 4, and Fig. 5).
As Matsumoto provides a known a triple modular redundancy (TMR) technique is implemented in the field of neural network architecture, in which three modules perform the same operation in parallel and the output is decided by majority voting, the combined teachings of the cited prior art would thus suggest to one of ordinary skill in the art that by applying Matsumoto’s techniques would allow for the method of Balasubramaniyan and Tran to further implement triple modular redundancy in performing the convolution operations of the set of computing blocks, thus enabling triple modular redundancy in the convolution operator system.
Thus, the combined teachings of the cited prior art teachings suggest that the convolutional neural network of Balasubramaniyan and Tran is further configured to implement triple modular redundancy in performing the convolution operations of the set of computing blocks, providing for the broadest reasonable interpretation, in light of the specification, for “wherein each of the set of convolution operations is at least one of a dilation convolution, a fast convolution, or a functional safety convolution based on a user-defined configuration of the reconfigurable convolutional cluster engine, wherein: the user-defined configuration, for the dilation convolution, comprises a dilation rate of the input feature map; the fast convolution comprises employing a convolution grid engine (CGRID); the functional safety convolution comprises enabling at least one of a double- module redundancy (DMR), a triple-module redundancy (TMR), and one or more diagnostic features”.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1-4, 6-8, 10-13, 15-17, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Balasubramaniyan et al. (US 2020/0218960), herein Balasubramaniyan, in view of Tran et al. (US 2020/0019849), herein Tran, and Matsumoto et al. (“Fault Tolerance in Small World Cellular Neural Networks for image processing”), herein Matsumoto.
Regarding claim 1, Balasubramaniyan discloses a method of feature extraction from an input image from a plurality of images in an image sensor pipeline, the method comprising:
determining, by the CNN acceleration device (see Balasubramaniyan Fig. 2 and [0032], where a hardware implementation of the convolutional operator system is disclosed; and see Balasubramaniyan [0003]-[0005], where the disclosed convolutional operator systems and methods are applicable to implementing convolution neural network techniques), a number of logical convolutional operations to be performed, within a reconfigurable convolutional cluster engine, based on a size of an input feature map corresponding to the input image (see Balasubramaniyan [0035]-[0038], where input image data is received, comprising a kernel value and a set of input feature matrices, and a controller allocates the input features to a set of computing blocks or a plurality of groups of computing blocks for performing convolution operations according to the number and size of the input feature matrices);
performing, by the CNN acceleration device, a set of concurrent row wise convolutions on the input feature map, based on the number of logical convolutional operations (see Balasubramaniyan [0035]-[0038], where the plurality of groups may be configured to generate a set of convolution output corresponding to the set of rows, where each group performs convolution operation concurrently on each row of each input feature matrix),
wherein each of the set of concurrent row wise convolutions comprises a set of convolution operations corresponding to a pre-determined kernel size so as to generate a set of corresponding convolution output (see Balasubramaniyan [0035]-[0038], where each computing block may perform convolution operation on each input feature based on the kernel value received),
wherein each of the set of convolution operations is one of a one-dimensional (1D) convolution, a two- dimensional (2D) convolution, or a three-dimensional (3D) convolution (see Balasubramaniyan [0050], where the system and method performs 2D or 3D convolution operations concurrently), and
wherein at least one of the output feature map or the input image is transmitted, based on a user-defined mode, for subsequent storage or processing prior to performing feature extraction from a next input image from the plurality of images in the image sensor pipeline (see Balasubramaniyan [0036]-[0041], where a set of convolution outputs or aggregated convolution output is generated and the convolution output or the aggregated convolution output is further transmitted to an external memory and may be further configured to transmit the output for subsequent convolution operations to generate a convolution result for the image data).
While Balasubramaniyan teaches that a set of convolution outputs or aggregated convolution output is generated and the convolution output and the set of convolution outputs can be aggregated to generate the aggregated convolution output (see Balasubramaniyan [0036]-[0039]); Balasubramaniyan does not explicitly disclose
performing, by the CNN acceleration device, at least one of a maximum pooling or an average pooling operation on the set of corresponding convolution output through one or more pooling elements to generate a set of pooling output, and
generating, by the CNN acceleration device, an output feature map based on the set of pooling output.
Tran teaches in a related and pertinent system and method for accessing redundant non-volatile memory cells for operating a neural memory system used in a deep learning artificial neural network (see Tran Abstract), where the neural network utilizing non-volatile memory array receives an image input and processes the image with synapses of different set of weights which scans portions of the input image with kernels, which multiply the input values with appropriate weights and summing the outputs of the multiplication to determine a single output value by a first neuron for generating a pixel of one of the layers of a feature map, and the process is repeated using different sets of weights to generate a different feature map until all of the feature maps are calculated (see Tran [0061]-[0063]), and an activation function (pooling) is applied to the feature map values which pools values from regions in each feature map to average out the nearby location (or a max function can be used) and reduce the data size before going to the next stage (see Tran [0064]).
At the time of filing, one of ordinary skill in the art would have found it obvious to apply the teachings of Tran to the teachings of Balasubramaniyan, such that when using the convolution operator system of Balasubramaniyan for implementing a convolutional neural network, a set of pooling activation functions are applied to the sets of convolutional outputs to average out the feature map values and reduce the data size before going to the next stage of the convolutional neural network.
This modification is rationalized as an application of a known technique to a known method ready for improvement to yield predictable results.
In this instance, Balasubramaniyan disclose a base method and system for performing convolution operations comprised of a set of computing blocks or a plurality of groups of computing blocks for performing convolution operations for input image data according to received kernel value and image feature matrices.
Tran teaches a known technique of operating a neural memory system used in a deep learning artificial neural network, where an activation function (pooling) is applied to the calculated feature map values from an input image which pools values from regions in each feature map to average out the nearby location (or a max function can be used) and reduce the data size before going to the next stage.
One of ordinary skill in the art would have recognized that by applying Tran’s techniques would allow for the method of Balasubramaniyan to further apply a set of pooling activation functions to the sets of convolutional outputs to average out the feature map values and reduce the data size before going to the next stage of the convolutional neural network, when using the convolution operator system for implementing a convolutional neural network, predictably leading to an improved method and system for implementing a convolutional neural network using the convolution operator system.
While Balasubramaniyan teaches that the embodiments are described in the context of an exemplary reconfigurable convolution engine, and that a user may interact with the convolution operator system via an interface (see Balasubramaniyan [0025] and [0033]); Balasubramaniyan and Tran do not explicitly disclose wherein each of the set of convolution operations is at least one of a dilation convolution, a fast convolution, or a functional safety convolution based on a user-defined configuration of the reconfigurable convolutional cluster engine, wherein: the user-defined configuration, for the dilation convolution, comprises a dilation rate of the input feature map; the fast convolution comprises employing a convolution grid engine (CGRID); the functional safety convolution comprises enabling at least one of a double- module redundancy (DMR), a triple-module redundancy (TMR), and one or more diagnostic features.
Matsumoto teaches in a related and pertinent method for a fault tolerant cellular neural network architecture (see Matsumoto Abstract), where a triple modular redundancy technique is implemented in the neural network architecture in which three modules perform the same operation in parallel and the output is decided by majority voting (see Matsumoto sect. 3. Fault Tolerance in Small World Cellular Neural Networks, Fig. 4, and Fig. 5).
At the time of filing, one of ordinary skill in the art would have found it obvious to apply the teachings of Matsumoto to the teachings of Balasubramaniyan and Tran, such that triple modular redundancy is implemented in performing the convolution operations of the set of computing blocks.
This modification is rationalized as an application of a known technique to a known method ready for improvement to yield predictable results.
In this instance, Balasubramaniyan and Tran disclose a base method for implementing a convolution neural network using a convolution operator system comprised of a set of computing blocks or a plurality of groups of computing blocks for performing convolution operations for input image data according to received kernel value and image feature matrices and generates convolution outputs for calculating the output feature maps for the convolutional neural network.
Matsumoto teaches a known technique for implementing a fault tolerant cellular neural network architecture, where a triple modular redundancy technique is implemented in the neural network architecture in which three modules perform the same operation in parallel and the output is decided by majority voting.
One of ordinary skill in the art would have recognized that by applying Matsumoto’s techniques would allow for the method of Balasubramaniyan and Tran to further implement triple modular redundancy in performing the convolution operations of the set of computing blocks, providing for the broadest reasonable interpretation of the claims that each of the set of convolution operations be one of a dilation convolution, a fast convolution, or a functional safety convolution, where the functional safety convolution comprises enabling at least one of a double-module redundancy (DMR), a triple-module redundancy (TMR), and one or more diagnostic features, predictably leading to an improved method and system for implementing a convolutional neural network with fault tolerant convolution operations.
Regarding claim 2, please see the above rejection of claim 1. Balasubramaniyan, Tran, and Matsumoto disclose the method of claim 1, wherein the reconfigurable convolution cluster engine comprises a set of Mini Parallel Rolling Engines (MPREs), wherein each MPRE is configured to perform the concurrent row wise convolution operation on the input feature map, and wherein the number of MPRE is based on a number of lines in the input feature map (see Balasubramaniyan [0036], where the set of computing blocks may operate concurrently to produce convolution output corresponding to each row of each input feature matrix; see Balasubramaniyan [0042], where the number convolution operator systems correspond to the number of rows of the input feature matrix, and each convolution operator system may generate a convolution result for the received row of the input feature matrix).
Regarding claim 3, please see the above rejection of claim 2. Balasubramaniyan, Tran, and Matsumoto disclose the method of claim 2, wherein each of the set of MPREs comprises a set of Convolution Multiply and Accumulate-XtendedGen2 (CMAC-XG2) elements, wherein each CMAC-XG2 is configured to perform a convolution operation corresponding to the pre-determined kernel size, and wherein the number of CMAC-XG2 is based on a number of pixels in each of the line in the input feature map (see Balasubramaniyan [0036], where each computing block may perform convolution operation on each input feature based on the kernel value received, where, if the received image data has a width of 128, and the number of computing blocks available are 128, then the controller allocates each input feature to each computing block; see Tran [0061]-[0064], where the synapses of different set of weights scans portions of the input image with kernels, which multiply the input values with appropriate weights and summing the outputs of the multiplication to determine a single output value by a first neuron for generating a pixel of one of the layers of a feature map, and an activation function pools values from consecutive regions in each feature map).
Regarding claim 4, please see the above rejection of claim 3. Balasubramaniyan, Tran, and Matsumoto disclose the method of claim 3, wherein each of the set of CMAC-XG2 comprises at least one of a Double Module Redundancy (DMR) or a Triple-Module Redundancy (TMR) (see Matsumoto sect. 3. Fault Tolerance in Small World Cellular Neural Networks, Fig. 4, and Fig. 5, where a triple modular redundancy technique is implemented in the neural network architecture, in which three modules perform the same operation in parallel and the output is decided by majority voting).
Regarding claim 6, please see the above rejection of claim 1. Balasubramaniyan, Tran, and Matsumoto disclose the method of claim 1, wherein the reconfigurable convolution cluster engine further comprises an input feature map memory to store the input image and an output feature map memory to store the output feature map (see Balasubramaniyan [0040]-[0041], where a set of convolution outputs or aggregated convolution output is generated and the convolution output or the aggregated convolution output is further transmitted to an external memory and may be further configured to transmit the output for subsequent convolution operations to generate a convolution result for the image data; see Balasubramaniyan [0043]-[0044], where the method may be described in the context of computer executable instructions and can be implemented in a distributed computing environment located in both local and remote computer storage media, including memory storage devices; suggesting memory storage devices that store an input feature map and output feature map).
Regarding claim 7, please see the above rejection of claim 1. Balasubramaniyan, Tran, and Matsumoto disclose the method of claim 1, wherein the reconfigurable convolution cluster engine comprises a kernel memory space capable for holding a set of network parameters associated to a network layer (see Tang [0065], where a memory array stores the weights utilized by the synapses of the different neural network layers).
Regarding claim 8, please see the above rejection of claim 1. Balasubramaniyan, Tran, and Matsumoto disclose the method of claim 1, wherein the reconfigurable convolution cluster engine comprises a kernel controller to enable the parallel convolution operation by loading the network parameters into the one or more CMAC-XG2 elements simultaneously (see Balasubramaniyan [0035]-[0038], where the controller may allocate a plurality of groups comprising one or more computing blocks and configured to generate a set of convolution output corresponding to the set of rows, where each group performs convolution operation concurrently on each row of each input feature matrix).
Regarding claim 10, it recites a system performing the method of claim 1. Balasubramaniyan, Tran, and Matsumoto teach a system performing the method of claim 1. Please see above for detailed claim analysis, with the exception to the following further limitations:
a processor; and a memory communicatively coupled to the processor, wherein the memory stores processor-executable instructions, which, on execution, cause the processor to perform the method of claim 1 (see Balasubramaniyan [0043]-[0044], where the method may be described in the context of computer executable instructions and can be implemented in a distributed computing environment where functions are performed by remote processing devices that are linked through a communication network and the computer executable instructions are located in both local and remote computer storage media, including memory storage devices)
Please see the above rejection for claim 1, as the rationale to combine the teachings of Balasubramaniyan, Tran, and Matsumoto are similar, mutatis mutandis.
Regarding claim 11, see above rejection for claim 10. It is a system claim reciting similar subject matter as claim 2. Please see above claim 2 for detailed claim analysis as the limitations of claim 11 are similarly rejected.
Regarding claim 12, see above rejection for claim 11. It is a system claim reciting similar subject matter as claim 3. Please see above claim 3 for detailed claim analysis as the limitations of claim 12 are similarly rejected.
Regarding claim 13, see above rejection for claim 12. It is a system claim reciting similar subject matter as claim 4. Please see above claim 4 for detailed claim analysis as the limitations of claim 13 are similarly rejected.
Regarding claim 15, see above rejection for claim 10. It is a system claim reciting similar subject matter as claim 6. Please see above claim 6 for detailed claim analysis as the limitations of claim 15 are similarly rejected.
Regarding claim 16, see above rejection for claim 10. It is a system claim reciting similar subject matter as claim 7. Please see above claim 7 for detailed claim analysis as the limitations of claim 16 are similarly rejected.
Regarding claim 17, see above rejection for claim 10. It is a system claim reciting similar subject matter as claim 8. Please see above claim 8 for detailed claim analysis as the limitations of claim 17 are similarly rejected.
Regarding claim 19, it recites a non-transitory computer readable medium storing computer-executable instructions for performing the method of claim 1. Balasubramaniyan, Tran, and Matsumoto teach a non-transitory computer readable medium storing computer-executable instructions for performing the method of claim 1 (see Balasubramaniyan [0043]-[0044], where the method may be described in the context of computer executable instructions and can be implemented in a distributed computing environment where functions are performed by remote processing devices that are linked through a communication network and the computer executable instructions are located in both local and remote computer storage media, including memory storage devices). Please see above for detailed claim analysis.
Please see the above rejection for claim 1, as the rationale to combine the teachings of Balasubramaniyan, Tran, and Matsumoto are similar, mutatis mutandis.
Regarding claim 20, see above rejection for claim 19. It is a non-transitory computer readable medium claim reciting similar subject matter as claim 2. Please see above claim 2 for detailed claim analysis as the limitations of claim 20 are similarly rejected.
Claims 5 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Balasubramaniyan, Tran, and Matsumoto as applied to claims 4 and 13 above, and further in view of Buyuktosunoglu et al. (US 2018/0358110), herein Buyuktosunoglu.
Regarding claim 5, please see the above rejection of claim 4. Balasubramaniyan, Tran, and Matsumoto do not explicitly disclose the method of claim 4, further comprising validating the each of the set of CMAC- XG2 through safety diagnostics registers and Built-In Self-Test (BIST).
Buyuktosunoglu teaches in a related and pertinent self-evaluating memory array for a neural network (see Buyuktosunoglu Abstract), where the neural network may additionally run a self diagnosing voltage test that identifies which memory cells in the neural network operate properly at the given voltage, and may include any built-in self-test to run a pattern of voltages through the memory array (see Buyuktosunoglu [0066]-[0077]).
At the time of filing, one of ordinary skill in the art would have found it obvious to apply the teachings of Buyuktosunoglu to the teachings of Balasubramaniyan, Tran, and Matsumoto, such that the neural network further implements a built in self test to run self diagnosing voltage tests on the memory cells and registers of the convolutional operator system to identify which memory cells and registers operate properly given the voltages applied.
This modification is rationalized as an application of a known technique to a known method ready for improvement to yield predictable results.
In this instance, Balasubramaniyan, Tran, and Matsumoto disclose a base method for implementing a convolution neural network using a convolution operator system comprised of a set of computing blocks or a plurality of groups of computing blocks for performing convolution operations for input image data according to received kernel value and image feature matrices and generates convolution outputs for calculating the output feature maps for the convolutional neural network.
Buyuktosunoglu teaches a known technique for implementing a self diagnosing voltage test that identifies which memory cells in the neural network operate properly at the given voltage, and may include any built-in self-test to run a pattern of voltages through the memory array.
One of ordinary skill in the art would have recognized that by applying Matsumoto’s techniques would allow for the method of Balasubramaniyan and Tran to further implement a built in self test to run self diagnosing voltage tests on the memory cells and registers of the convolutional operator system to identify which memory cells and registers operate properly given the voltages applied, predictably leading to an improved method and system for implementing a convolutional neural network where the memory cells and registers of the convolutional operator system are identified to operate properly according to given voltages.
Regarding claim 14, see above rejection for claim 13. It is a system claim reciting similar subject matter as claim 5. Please see above claim 5 for detailed claim analysis as the limitations of claim 14 are similarly rejected.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TIMOTHY WING HO CHOI whose telephone number is (571)270-3814. The examiner can normally be reached 9:00 AM to 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, VINCENT RUDOLPH can be reached at (571) 272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TIMOTHY CHOI/Examiner, Art Unit 2671
/VINCENT RUDOLPH/Supervisory Patent Examiner, Art Unit 2671
Prosecution Insights