DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-9 are rejected under 35 U.S.C. 101 because the claims are not directed to one of the four statutory categories.
Regarding claim 1, in step 1 of the 101 analysis set forth in MPEP 2106, the claim recites An artificial neural network, comprising:. An artificial neural network is interpreted as a computer program. Therefore, the claimed invention in claim 1 is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because a computer program is interpreted as “software per se” which is not one of the four statutory categories (MPEP 2106.03). Applicant is encouraged to amend the claim into one of the four statutory categories.
Regarding claims 2-8, the claims are rejected for at least their dependence on claim 1.
Regarding claim 9, in step 1 of the 101 analysis set forth in MPEP 2106, the claim recites A computer program product, comprising instructions which, when executed by processing circuitry, cause the processing circuitry to implement an artificial neural network, the artificial neural network comprising. A computer program product is interpreted as a computer program. Therefore, the claimed invention in claim 9 is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because a computer program is interpreted as “software per se” which is not one of the four statutory categories (MPEP 2106.03). Applicant is encouraged to amend the claim into one of the four statutory categories.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-13, and 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Gao, et al., US Pre-Grant Publication US20190279072A1 (“Gao”) in view of He, et al., Non-Patent Literature “BD-NET: A Multiplication-less DNN with Binarized Depthwise Separable Convolution” (“He”).
Regarding claim 1, Gao discloses:
An artificial neural network, comprising: at least one unit cell, the at least one unit cell including: a first binary…convolution layer configured to receive an input tensor and to generate a first tensor; (Gao, ⁋68, “Referring to FIG. 1, eight convolution layers are included in the binary convolutional neural network models [An artificial neural network, comprising: at least one unit cell, the at least one unit cell including:]…Five sub-structures may be divided out from the model shown in FIG. 1, and FIG. 1 in which the sub-structures are divided becomes to the structure shown in FIG. 2A in which each dashed block represents a sub-structure…the head layer of the third sub-structure is a binary convolution layer 4 [a first binary…convolution layer configured to receive an input tensor and to generate a first tensor;], the middle layer is a batch normalization layer 4, and the tail layer is a quantization layer 4”).
a first batch normalization layer configured to receive the first tensor and to generate a second tensor; (Gao, ⁋68, “the head layer of the third sub-structure is a binary convolution layer 4, the middle layer is a batch normalization layer 4 [a first batch normalization layer configured to receive the first tensor and to generate a second tensor;], and the tail layer is a quantization layer 4”).
…a second binary…convolution layer configured to receive a third tensor and to generate a fourth tensor; and a second batch normalization layer configured to generate an output tensor based on the fourth tensor. (Gao, ⁋68, “the head layer of the fourth sub-structure is a binary convolution Layer 5 […a second binary…convolution layer configured to receive a third tensor and to generate a fourth tensor;], the middle layers are a pooling layer 5 and a batch normalization layer 5 sequentially, and the tail layer is a quantization layer 5 [and a second batch normalization layer configured to generate an output tensor based on the fourth tensor.]”).
While Gao teaches a system for quantizing a convolutional neural network into a binary version and suggests merging elements between layers, Gao does not explicitly teach:
a first two-dimensional convolution layer
a concatenation layer configured to generate a third tensor by concatenating the input tensor and the second tensor;
a second two-dimensional convolution layer
He teaches:
a first two-dimensional convolution layer (He, pg. 131 col. 1, “The operation of depthwise separable convolution is described in the form of data flow in Fig. 1. Considering the input tensor is in the dimension of h×w×p, which denotes height, width and channel respectively. Note that, in the depthwise convolution layer, each channel of input tensor performs convolution with m kernels in the size of kh×kw (i.e. kernel-height and width) [a first two-dimensional convolution layer]”).
a concatenation layer configured to generate a third tensor by concatenating the input tensor and the second tensor; (He, pg. 131 col. 1, “We found that larger m could effectively compensate accuracy degradation in weight binarization at the cost of model size, which will be explicitly investigated in the next section. Those generated feature maps are concatenated along its depth dimension as tensor in size of h×w×(p·m)1 [a concatenation layer configured to generate a third tensor by concatenating the input tensor and the second tensor;], which is taken as the input of pointwise convolution layer.”).
a second two-dimensional convolution layer (He, pg. 131 col. 2, “On the contrary to the distinctive depthwise convolution, pointwise layer is just normal spatial convolution layer with 1×1 convolution kernel size [a second two-dimensional convolution layer].”).
Gao and He are both in the same field of endeavor (i.e. quantized convolutional neural networks). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Gao and He to teach the above limitation(s). The motivation for doing so is that using a depthwise separable convolution reduces computational costs of a convolutional neural network (cf. He, pg. 131 col. 1, “Recently, the depthwise separable convolution [7] has been widely used in many state-of-the-art deep neural networks, such as MobileNet [8] and Xception [9], which replaces the standard convolution layers to reduce DNN computational cost and memory usage.”).
Regarding claim 2, Gao in view of He teaches the artificial neural network according to claim 1. He further teaches wherein the first binary two-dimensional convolution layer is configured to perform a depthwise convolution on the input tensor. (He, pg. 131 col. 1, “The operation of depthwise separable convolution is described in the form of data flow in Fig. 1. Considering the input tensor is in the dimension of h×w×p, which denotes height, width and channel respectively. Note that, in the depthwise convolution layer, each channel of input tensor performs convolution with m kernels in the size of kh×kw (i.e. kernel-height and width) [wherein the first binary two-dimensional convolution layer is configured to perform a depthwise convolution on the input tensor.]”, and He, pg. 131 col. 2, “In this work, we only make the weights of depthwise convolution and the input tensor of pointwise convolution in binary (i.e.+1and-1) [binary]”).
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of He with the teachings of Gao for the same reasons disclosed in claim 1.
Regarding claim 3, Gao in view of He teaches the artificial neural network according to claim 1. He further teaches wherein the second binary two-dimensional convolution layer is configured to perform a pointwise convolution on the third tensor generated by the concatenation layer. (He, pg. 131 col. 1, “We found that larger m could effectively compensate accuracy degradation in weight binarization at the cost of model size, which will be explicitly investigated in the next section. Those generated feature maps are concatenated along its depth dimension as tensor in size of h×w×(p·m)1, which is taken as the input of pointwise convolution layer [wherein the second binary two-dimensional convolution layer is configured to perform a pointwise convolution on the third tensor generated by the concatenation layer.].”, and He, pg. 131 col. 2, “In this work, we only make the weights of depthwise convolution and the input tensor of pointwise convolution in binary (i.e.+1and-1) [binary]”).
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of He with the teachings of Gao for the same reasons disclosed in claim 1.
Regarding claim 4, Gao in view of He teaches the artificial neural network according to claim 1. He also teaches the two-dimensional convolution layer as seen in claim 1. Gao further teaches wherein the at least one unit cell includes a pooling layer between the second binary…convolution layer and the second batch normalization layer. (Gao, ⁋68, “the head layer of the fourth sub-structure is a binary convolution Layer 5 [the second binary…convolution layer], the middle layers are a pooling layer 5 [wherein the at least one unit cell includes a pooling layer between] and a batch normalization layer 5 sequentially [and the second batch normalization layer.], and the tail layer is a quantization layer 5”).
Regarding claim 6, Gao in view of He teaches the artificial neural network according to claim 1. Gao further teaches further comprising at least one classification layer configured to: receive the output tensor generated at an output of the at least one unit cell; classify a data tensor received at an input of the neural network based on the output tensor generated at the output of the at least one unit cell; and output a signal indicative of the classification of the data tensor. (Gao, ⁋23, “The task requirement herein refers to the purpose of modeling the binary convolutional neural network model, that is, the network model is used to perform what kind of data processing. For example, if the task requirement is picture classification processing, the binary convolutional neural network model is used to perform data processing for classifying the pictures [classify a data tensor received at an input of the neural network based on the output tensor generated at the output of the at least one unit cell; and output a signal indicative of the classification of the data tensor.]”, and Gao, ⁋69 and see Figure 2A, “Loss layer (Softmax WithLoss) [further comprising at least one classification layer configured to: receive the output tensor generated at an output of the at least one unit cell;]”).
Regarding claim 7, Gao in view of He teaches the artificial neural network according to claim 1. Gao further teaches further comprising at least one detection layer configured to: receive the output tensor generated at an output of the at least one unit cell; detect elements in a data tensor received at an input of the neural network based on the output tensor generated at the output of the at least one unit cell; and output a signal indicative of the detected elements. (Gao, ⁋23, “The task requirement herein refers to the purpose of modeling the binary convolutional neural network model, that is, the network model is used to perform what kind of data processing. For example, if the task requirement is picture classification processing, the binary convolutional neural network model is used to perform data processing for classifying the pictures [detect elements in a data tensor received at an input of the neural network based on the output tensor generated at the output of the at least one unit cell; and output a signal indicative of the detected elements.]”, and Gao, ⁋69 and see Figure 2A, “Loss layer (Softmax WithLoss) [further comprising at least one detection layer configured to: receive the output tensor generated at an output of the at least one unit cell;]”).
Regarding claim 8, Gao in view of He teaches the artificial neural network according to claim 1. Gao further teaches wherein the at least one unit cell includes a plurality of successive unit cells. (Gao, ⁋68, “Five sub-structures may be divided out from the model shown in FIG. 1, and FIG. 1 in which the sub-structures are divided becomes to the structure shown in FIG. 2A in which each dashed block represents a sub-structure [wherein the at least one unit cell includes a plurality of successive unit cells.]”).
Regarding claim 9, the claim is similar to claim 1 and rejected under the same rationales. Gao teaches the additional limitations A computer program product, comprising instructions which, when executed by processing circuitry, cause the processing circuitry to implement an artificial neural network, (Gao, ⁋41, “The processor unit 101 may be a CPU or a GPU. The memory unit 102 includes a random access memory (RAM) and a read only memory (ROM). The RAM may be used as a main memory, a work area and the like of the processor unit 101. The ROM may be used to store the control program of the processor unit 101, and additionally, may also be used to store files or other data to be used when the control program is operated [A computer program product, comprising instructions which, when executed by processing circuitry, cause the processing circuitry to implement an artificial neural network,].”).
Regarding claim 10, the claim is similar to claim 1 and rejected under the same rationales. Gao teaches the additional limitations A device comprising: a computer-readable memory configured to store instructions for implementing an artificial neural network; and processing circuitry configured to implement the artificial neural network by executing the instructions stored in the computer-readable memory, (Gao, ⁋41, “The processor unit 101 may be a CPU or a GPU. The memory unit 102 includes a random access memory (RAM) and a read only memory (ROM). The RAM may be used as a main memory, a work area and the like of the processor unit 101. The ROM may be used to store the control program of the processor unit 101, and additionally, may also be used to store files or other data to be used when the control program is operated [A device comprising: a computer-readable memory configured to store instructions for implementing an artificial neural network; and processing circuitry configured to implement the artificial neural network by executing the instructions stored in the computer-readable memory,].”).
Regarding claims 11-13 and 15-17, the claims are similar to claims 2-4 and 6-8 and are rejected under the same rationales.
Regarding claims 18-20, the claims are similar to claims 1-3 and are rejected under the same rationales.
Claims 5 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Gao, et al., US Pre-Grant Publication US20190279072A1 (“Gao”) in view of He, et al., Non-Patent Literature “BD-NET: A Multiplication-less DNN with Binarized Depthwise Separable Convolution” (“He”) and further in view of Ye, et al., US Pre-Grant Publication US20210295114A1 (“Ye”).
Regarding claim 5, Gao in view of He teaches the artificial neural network according to claim 1. While the combination teaches a system for quantizing a convolutional neural network into a binary version using depthwise separable convolution, the combination does not explicitly teach further comprising at least one attribute extraction layer configured to extract attributes from an input data tensor and to generate the input tensor based on the extracted attributes, the at least one unit cell being configured to receive as input the input tensor based on the extracted attributes.
Ye teaches further comprising at least one attribute extraction layer configured to extract attributes from an input data tensor and to generate the input tensor based on the extracted attributes, the at least one unit cell being configured to receive as input the input tensor based on the extracted attributes. (Ye, ⁋8, “the image is input into the backbone network, and feature extraction is performed on the image and at least one feature tensor is output through the backbone network [further comprising at least one attribute extraction layer configured to extract attributes from an input data tensor and to generate the input tensor based on the extracted attributes,]. Each feature tensor that is output by the backbone network is input into a feature fusion subnetwork, and a fusion feature tensor corresponding to the feature tensor is obtained through the feature fusion subnetwork. The fusion feature tensor is input into a classification subnetwork and a bounding box regression subnetwork [the at least one unit cell being configured to receive as input the input tensor based on the extracted attributes.].”).
Gao, in view of He, and Ye are both in the same field of endeavor (i.e. image processing). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Gao, in view of He, and Ye to teach the above limitation(s). The motivation for doing so is that efficiently extracting features from images reduces the number of resources used by a system over time (cf. Ye, ⁋3-4).
Regarding claim 14, the claim is similar to claim 5 and is rejected under the same rationales.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Gabriel, et al., US20200257960A1 teaches a system that quantizes a convolutional neural network with multiple sections of layers.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS S WU whose telephone number is (571)270-0939. The examiner can normally be reached Monday - Friday 8:00 am - 4:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached at 571-431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.S.W./Examiner, Art Unit 2148 /MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148
Prosecution Insights