DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-5, 7-9, 14 is/are rejected under 35 U.S.C. 102(a)(1) as being unpatentable over YeonWoo Jeong et al. (hereinafter Jeong) (“Optimal channel selection with discrete QCQP”, 2022-02-24)
Regarding claim 1, Jeong teaches;
A computer-implemented method of processing data using a Neural Network (NN) implemented in hardware,
([Abstract] Reducing the high computational cost of large convolutional neural networks is crucial when deploying the networks to resource constrained environments… We also propose a quadratic model that accurately estimates the actual inference time of the pruned network which allows us to adopt inference time as a resource constraint option.)
NOTE: Jeong discloses using a Neural Network for inference (i.e. processing data) to improve computational cost of the hardware on which it is implemented.
the NN comprising a plurality of layers,
([pg. 4] Concretely, we model the inference time of the l-th layer convolution operation with respect to the number of its input channels)
NOTE: l-th output layer indicates a plurality of layers of the NN.
each layer being configured to operate on activation data input to the layer so as to form output data for the layer, said data being arranged in data channels,
[pg. 6]
PNG
media_image1.png
236
1163
media_image1.png
Greyscale
NOTE: Jeong teaches each layer being configured to operate on activation data input to the layer (input feature maps) so as to form output data for the layer (output feature maps), said data being arranged in data channels (the first dimension of the feature maps is the channel dimension, C).
the method comprising: for an identified channel of output data for a layer,
([pg. 3] we propose a channel selection method based on the QCQP (Quadratic Constrained Quadratic Program) with importance evaluation respecting both the input and the output channels… We first propose our discrete QCQP formulation of channel pruning for sequential convolutional neural networks (CNNs). Then, we present two extended versions of our formulation…)
NOTE: Jeong discloses a method which selects the optimal channel(s) (which includes output channels) to prune.
operating on activation data input to the layer such that the output data for the layer does not include the identified channel;
[pg. 6]
PNG
media_image2.png
366
1054
media_image2.png
Greyscale
NOTE: Jeong discloses ‘remaining output channels’ of the output data (output feature maps) of the layer, indicating that the channels selected for pruning are not included in the output data. Thus, Jeong teaches operating on activation data (each layer operates on an input feature map to produce an output feature map) such that the output data for the layer (the output feature map) does not include the identified channel (a channel selected for pruning is not included in the output feature map).
and prior to an operation of the NN configured to operate on the output data for the layer, inserting a replacement channel into the output data for the layer in lieu of the identified channel in dependence on information indicative of the structure of the output data for the layer were the identified channel to have been included.
[pg. 13]
PNG
media_image3.png
386
1630
media_image3.png
Greyscale
NOTE: Jeong discloses a skip addition operation of the NN which adds the input feature map of the s+1-th convolution layer to the output feature map of the t-th layer. Jeong discloses that the channel dimensions of the feature maps must be the same, so they utilize zero padding (i.e. inserting a replacement channel of 0s into the input or output feature map) to fix mismatching channel dimensions before performing the skip addition. These mismatching dimensions can be from the aforementioned channel pruning operations. Thus, Jeong teaches prior to an operation of the NN (the skip addition) configured to operate on the output data for the layer (the skip addition operates on the output feature map of the t-th layer), inserting a replacement channel into the output data for the layer in lieu of the identified channel (replacing the channel(s) selected to be pruned via zero padding before performing the skip addition) in dependence on information indicative of the structure of the output data for the layer were the identified channel to have been included (the zero padding / channel replacement is dependent on the original dimensionality / structure of the output feature map, i.e., the dimensionality of the output feature map having the identified channel being included rather than pruned).
Regarding claim 2, Jeong teaches;
wherein the replacement channel is a channel consisting of a plurality of zero values.
PNG
media_image3.png
386
1630
media_image3.png
Greyscale
[pg. 13]
NOTE: Jeong teaches the replacement channel is a channel consisting of a plurality of zero values (missing channels of the feature maps are replaced using zero padding, i.e., a plurality of zero values).
Regarding claim 3, Jeong teaches;
comprising performing the operation of the NN in dependence on the replacement channel.
PNG
media_image3.png
386
1630
media_image3.png
Greyscale
[pg. 13]
NOTE: Jeong teaches performing the operation (skip addition) of the NN in dependence on the replacement channel, because the skip addition is performed using the zero-padded feature map, i.e. the feature map including the aforementioned replacement channel.
Regarding claim 4, Jeong teaches;
wherein the operation is a summation operation configured to sum two or more sets of activation data, one of those sets of activation data being the output data for the layer.
PNG
media_image4.png
27
1275
media_image4.png
Greyscale
[pg. 13]
NOTE: Jeong teaches that the operation (skip addition) is a summation operation configured to sum two or more sets of activation data (the skip addition sums an input feature map / activation data with an output feature map / activation data), one of those sets of activation data being the output data for the layer (output feature map of the layer).
Regarding claim 5, Jeong teaches;
wherein each layer is configured to combine respective weight data with activation data input to the layer so as to form output data for the layer, the weight data being arranged in one or more output channels each responsible for forming respective output channels of the output data for the layer,
[pg. 2]
PNG
media_image5.png
622
929
media_image5.png
Greyscale
NOTE: Jeong teaches that each layer is configured to combine respective weight data with activation data input to the layer so as to form output data for the layer (the convolution for each layer combines weight data W with input activation data / feature map X^l-1 to generate an output feature map X^l), the weight data W being arranged in one or more output channels (the weight data has C_l output channels) each responsible for forming respective output channels of the output data X for the layer (as shown in the example convolution operation, the j-th output channel of W is responsible for producing the corresponding j-th output channel of the generated output feature map X).
the method comprising not including the output channel of the weight data that is responsible for forming the identified channel such that the output data for the layer does not include the identified channel.
[pg. 3]
PNG
media_image6.png
49
811
media_image6.png
Greyscale
PNG
media_image7.png
217
811
media_image7.png
Greyscale
NOTE: Jeong discloses a method of identifying channels to be pruned, then builds a binary mask over the convolutional weights, then deactivates / does not include values of weights that correspond to the channels that are specified to be pruned. As previously taught, the j-th output channel of the weight tensor of a layer l is responsible for generating the corresponding j-th output feature map channel. Thus, by not including the output channel j of the weight data, the output feature map will also not include channel j. Using this reasoning, Jeong teaches not including the output channel of the weight data (deactivating / not including weights of pruned channel j) that is responsible for forming the identified channel such that the output data for the layer does not include the identified channel (output feature map also will not include channel j).
Regarding claim 7, Jeong teaches;
wherein at least one subsequent layer of the NN is also configured to operate on the output data for the layer.
[pg. 2]
PNG
media_image8.png
157
980
media_image8.png
Greyscale
NOTE: Jeong teaches that each layer l generates an output feature map X^l using the output feature map of the previous layer, X^l-1. Thus, Jeong teaches at least one subsequent layer of the NN is also configured to operate on the output data for the layer, since the output of each layer is operated on by subsequent layers.
Regarding claim 8, Jeong teaches;
wherein the operation of the NN is also configured to operate on output data for another layer of the NN.
[pg. 2]
PNG
media_image9.png
165
1012
media_image9.png
Greyscale
NOTE: In the disclosed embodiment, the output feature map of the l-th layer, X^l, is generated using the output feature map of the previous layer, X^l-1, as input data. Thus, Jeong teaches a sequential NN where the input data for each layer is also the output data of the previous layer.
[pg. 13]
PNG
media_image4.png
27
1275
media_image4.png
Greyscale
NOTE: Jeong teaches the skip addition operation of the NN operates on (sums) the output feature map of the t-th layer, and the input feature map of the s+1-th convolution layer. As previously taught, the input feature map of an s+1-th layer can also be the output feature map of the s-th layer. Thus, Jeong teaches the skip addition operation of the NN is also configured to operate on output data for another layer of the NN (the skip addition operation sums both the output data of the t-th layer, and the output data of the s-th layer).
Regarding claim 9, Jeong teaches;
wherein the operation of the NN is configured to combine two or more sets of data having the same structure.
PNG
media_image10.png
365
1590
media_image10.png
Greyscale
[pg. 13]
NOTE: Jeong teaches the skip addition operation of the NN is configured to combine two or more sets of data (the skip addition sums / combines input and output feature maps) having the same structure (the feature maps must have the same channel dimensions).
Regarding claim 14, Jeong teaches;
wherein a channel is an array of values.
[pg. 6]
PNG
media_image11.png
199
1015
media_image11.png
Greyscale
NOTE: Jeong discloses feature maps / activation data having C x H x W dimensions, where C is the channel dimension and H and W are spatial dimensions. Thus, each individual channel of the feature map comprises an H x W array of values. Therefore, Jeong teaches that a channel is an array of values.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jeong as applied to claim 1 above, and further in view of Mart van Baalen et al. (hereinafter Baalen) (“Bayesian Bits: Unifying Quantization and Pruning”, 2020-10-27).
Regarding claim 6, Jeong fails to teach but Baalen teaches;
wherein it is identified in a training phase of the NN that the output channel of the weight data that is responsible for forming the identified channel is quantisable with a bit width of zero.
([pg. 6] We include pruning by performing group sparsity on the output channels of the weight tensors only, as pruning an output channel of the weight tensor corresponds to pruning that specific activation.)
NOTE: Baalen teaches pruning of output channels of the weight tensors of a Neural Network, which in turn, prunes the corresponding channel of the activation data.
([pg. 2] learnable gates: by placing a gate on each of the quantized residual error tensors, the effective bit width can be controlled, thus allowing for data-dependent optimization of the bit width of each tensor, which we learn jointly with the (quantization) scales and network parameters.)
NOTE: Baalen teaches learnable gates that control the bit widths during an optimization / training phase.
([pg. 3] If one of the gates zi takes the value of zero, it completely de-activates the addition of all of the higher bit width residuals, thus controlling the effective bit width of the quantized value xq. Actually, we can take this a step further and consider pruning as quantization with a zero-bit width… we can then perform, e.g., structured pruning by employing a separate quantizer of this form for each filter in a convolutional layer.)
NOTE: Baalen frames filter / output channel pruning as quantization with a zero-bit width during the aforementioned optimization / learning phase. Thus, Baalen teaches that is identified in an optimization / training phase of the NN that a pruned filter / output channel of the weight data that is responsible for forming the identified channel (the corresponding channel of the activation data) is quantizable with a bit width of zero.
OBVIOUSNESS TO COMBINE BAALEN WITH JEONG:
Baalen is analogous art to the present disclosure as it pertains to quantization and channel removal using zero-bit width quantization.
Jeong already provides a channel pruning CNN framework.
([Abstract, Baalen] By starting with a power-of-two bit width, this decomposition will always produce hardware-friendly configurations, and through an additional 0-bit option, serves as a unified view of pruning and quantization.)
Baalen teaches joint mixed-precision quantization and pruning through optimization that includes a 0-bit option that unifies pruning and quantization.
Baalen further states;
([pg. 1] To reduce the computational cost of neural network inference, quantization and compression techniques are often applied before deploying a model in real life.)
NOTE: Baalen indicates that quantization reduces computational cost of neural network inference.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate Baalen’s zero-bit quantization / pruning technique into Jeong’s channel selection framework so that Jeong’s pruned weight-output channels are represented and identified as 0-bit channels, to unify channel pruning with quantization, thereby reducing the computational cost of the neural network.
Claim(s) 10-13, 16-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jeong as applied to claim 1 above, and further in view of Edvard Olav Valter Fagerholm et al. (hereinafter Fagerholm) (US 20200234129 A1, 2020-07-23).
Regarding claim 10, Jeong teaches;
A bit mask, each bit of the bit mask representing a data channel, a first bit value being indicative of a data channel included in the output data and a second bit value being indicative of a data channel not included in the output data.
[pg. 3]
PNG
media_image12.png
315
836
media_image12.png
Greyscale
NOTE: Jeong teaches a channel activation, which uses bits 0 and 1 to mask data channels of the network, which can be considered a bitmask. The channel activation / bit mask uses bits to represent which channels remain in the pruned network (0 indicates that the channel is not included, 1 indicates that the channel is included). Thus, Jeong teaches a bit mask (channel activation), each bit of the bit mask representing a data channel, a first bit value, 1, being indicative of a data channel included in the output data and a second bit value, 0, being indicative of a data channel not included in the output data.
Jeong fails to teach but Fagerholm teaches;
wherein the information comprises a
([0036] Scatter operation S.sub.1 inserts zeros into tensor x corresponding to elements of tensor W previously zeroed out via mask M and subsequently removed)
NOTE: Fagerholm teaches using information of a mask M to insert a replacement element in place of a previously removed element of a tensor W.
OBVIOUSNESS TO COMBINE FAGERHOLM WITH JEONG;
Fagerholm is analogous art to the present disclosure as it pertains to replacing removed neural network data elements using a masking technique.
Jeong teaches the base CNN channel-selection and channel pruning framework with a bitmask (channel attention) indicating present and removed channels.
Fagerholm teaches a method of replacing pruned data elements using information from a mask in a scatter operation (as previously taught).
([0018] Evaluation of the first function based on tensor w generates a smaller output tensor, denoted x. Because a subsequent node in the neural network expects the given node to provide an output with the larger dimensions of tensor X, a scatter operation is inserted in the subsequent node in order to add zeros into tensor x, thereby expanding tensor x to produce tensor X (or an equivalently dimensioned tensor).)
NOTE: Fagerholm discloses that subsequent neural network operations may require an output tensor having dimensionality of the original / un-pruned output tensor. Thus, replacing the pruned / removed elements to generate an output tensor having the original dimensionality allows future operations using the output tensor to be performed.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to use the channel bitmasks of Jeong as the mask in the scatter operation of Fagerholm to replace removed data channels of the neural network, thereby enabling subsequent operations which require output data having specific channel dimensions.
Regarding claim 11, Jeong teaches;
wherein: the first bit value is 1 and the second bit value is 0; or the first bit value is 0 and the second bit value is 1.
[pg. 3]
PNG
media_image12.png
315
836
media_image12.png
Greyscale
NOTE: Jeong teaches the first bit of the channel activation / bitmask being 1 and the second bit being 0.
Regarding claim 12, Jeong teaches;
a second bit value of the bitmask indicating removed channels
[pg. 3]
PNG
media_image12.png
315
836
media_image12.png
Greyscale
NOTE: Jeong teaches a second value (0) of the bitmask / channel activation indicating removed / pruned channels.
comprising inserting the replacement
([0036] Scatter operation S.sub.1 inserts zeros into tensor x corresponding to elements of tensor W previously zeroed out via mask M and subsequently removed)
NOTE: Fagerholm teaches comprising inserting the replacement element into the output data for the layer where indicated by the mask.
OBVIOUSNESS:
Using the same reasoning from claim 10, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to use the channel bitmasks of Jeong as the mask in the scatter operation of Fagerholm to replace removed data channels of the neural network, thereby enabling subsequent operations which require output data having specific channel dimensions.
Regarding claim 13, Jeong teaches;
the information being indicative of the structure of the output data for the layer including the identified channel.
[pg. 3]
PNG
media_image12.png
315
836
media_image12.png
Greyscale
NOTE: Jeong teaches channel activations, which indicate which data channels remain in the network after pruning. Thus, Jeong teaches information (the channel activations / bitmask) being indicative of the structure of the output data for the layer including the identified channel.
Jeong fails to teach but Fagerholm teaches;
Using said information to insert replacement data elements into data having missing elements
([0036] Scatter operation S.sub.1 inserts zeros into tensor x corresponding to elements of tensor W previously zeroed out via mask M and subsequently removed)
NOTE: Fagerholm teaches using information of a mask M to insert a replacement element in x in place of a previously removed element of a tensor W.
OBVIOUSNESS:
Using the same reasoning from claim 10, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to use the channel bitmasks of Jeong as the mask in the scatter operation of Fagerholm to replace removed data channels of the neural network, thereby enabling subsequent operations which require output data having specific channel dimensions.
Regarding claim 16,
Claim 16 is a non-transitory CRM claim that is substantially similar to claim 1, (where all similar limitations are taught using the same reasoning from claim 1) with one added limitation, which is taught by Fagerholm;
A non-transitory computer readable storage medium having stored thereon computer readable instructions that, when executed at a computer system, cause the computer system to perform a computer-implemented method of
([103-109] Some embodiments include a non-transitory computer-readable medium storing program instructions that, when executed by at least one processor, cause the at least one processor to [perform the methods of the disclosure])
OBVIOUSNESS:
Fagerholm is analogous art to the present disclosure as it pertains to replacing removed neural network data elements using a masking technique.
Jeong teaches the base CNN channel-selection and channel pruning framework with a bitmask (channel attention) indicating present and removed channels.
Fagerholm teaches a method of pruning / removing data elements of a neural network and replacing the pruned data elements using information from a mask in a scatter operation (as previously taught).
Fagerholm teaches a very similar process to Jeong, and Fagerholm additionally provides hardware (the previously taught non-transitory CRM, processors, etc.) capable of performing the disclosed operations.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to implement the channel pruning and associated processes using the hardware of Fagerholm, to provide a physical medium capable of carrying out the methods of the system.
Regarding claim 17,
Claim 17 is a device claim that is substantially similar to claim 1, (where all similar limitations are taught using the same reasoning from claim 1) with one added limitation, which is taught by Fagerholm;
the computing-based device comprising:at least one processor configured to:
([103-109] Some embodiments include a non-transitory computer-readable medium storing program instructions that, when executed by at least one processor, cause the at least one processor to [perform the methods of the disclosure])
NOTE: Fagerholm teaches a processor configured to perform the embodiments of the disclosure.
OBVIOUSNESS:
Using the same reasoning from claim 16, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to implement the channel pruning and associated processes using the hardware of Fagerholm, to provide a physical medium capable of carrying out the methods of the system.
Regarding claims 18-20,
Claims 18-20 are device claims directly corresponding to method claims 2-4, and are therefore rejected using the same reasoning.
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jeong as applied to claim 1 above, and further in view of Oh Jin Wook (hereinafter Wook) (KR 102256288 B1, 2021-05-27).
Regarding claim 15, Jeong fails to teach but Wook teaches;
wherein the NN is implemented using a neural network accelerator.
([Abstract] Disclosed is a pruning-based training method for acceleration hardware of an artificial neural network. The pruning-based training method for acceleration hardware of an artificial neural network comprises: a step in which a processing unit identifies a SIMD width of a processing element included in an accelerator; a step in which the processing unit initializes the weights of the artificial neural network; a step in which the processing unit groups weights arranged in an input channel and an output channel of the artificial neural network into weight groups according to the identified SIMD width; a step in which the processing unit prunes the plurality of weights included in the artificial neural network by being aware of the weight groups; and a step in which the processing unit updates the plurality of pruned weights.)
NOTE: Wook teaches a NN implemented using a NN accelerator.
OBVIOUSNESS TO COMBINE WOOK WITH JEONG:
Wook is analogous art to the present disclosure as it pertains to a pruning method using a neural network implemented using a neural network accelerator.
Jeong teaches the base CNN channel-selection and channel pruning framework, and Wook teaches a neural network implemented using a neural network accelerator.
Jeong additionally states;
([pg. 2] Inference operations are performed on an AI accelerator, which is an acceleration hardware specially designed to accelerate artificial intelligence applications.)
NOTE: Jeong explains that AI / NN accelerator used to implement the disclosed neural network improves the efficiency during inference.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to implement the neural network of Jeong on the NN accelerator of Wook to improve the efficiency of the neural network during inference.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
CONCLUSION
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Matthew Alan Cady whose telephone number is (571) 272-7229. The examiner can normally be reached Monday - Friday, 7:30 am - 5:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached on (571)272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC)
at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW ALAN CADY/ Examiner, Art Unit 2145
/CESAR B PAULA/ Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights