5The present application, filed on or after 16 March 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This office action is in response to Applicant’s submission filed on 17 September 2024. THIS ACTION IS NON-FINAL.
Status of Claims
Claims 26-45 are pending.
Claims 1-25 are cancelled.
Claims 26-45 are rejected under 35 U.S.C. 112(b) as indefinite.
Claims 26,33, and 40 are objected to.
Claims 26-45 are rejected under 35 U.S.C. 103 as unpatentable.
Claim Objections
Claims 26, 33 and 40 are objected to because of the following informalities:
(a) “the context information comprising channels values” should read “channel values” (see instant specification ¶[0018], FIG. 1, “the context information includes channel values”).
(b) “the modified activation map having less channels” should read “fewer channels” (count noun).
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
A claim is indefinite if, when read in light of the specification, it fails to inform, with reasonable certainty, those skilled in the art about the scope of the invention. Nautilus, Inc. v. Biosig Instruments, Inc., 110 USPQ.2d 1688, U.S. Supreme Court (2014).
Claims 26–45 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Regarding claims 26, 33 and 40, the preamble-level limitation recites “the first network layer preceding a second network layer in the neural network,” and the final limitation again introduces “processing … the modified activation map in a second network layer, the second network layer arranged after the first network layer.” The second use of the indefinite article introduces a new element, so it is unclear whether the layer that processes the modified activation map is the same “second network layer” recited earlier or a different layer. In addition, “during the inference of the neural network” lacks antecedent basis; the claims earlier recite “an inference process.”, the claim is therefore indefinite. For examination, the two recitations are interpreted as the same second network layer and “the inference” as “the inference process.” Applicant may consider amending to “processing, during the inference process, the modified activation map in the second network layer.” Claims 27–32, 34–39 and 41–45 are rejected for incorporating these deficiencies by dependency.
Regarding claims 26, 33 and 40, the limitation “the modified activation map having less channels than the activation map” is unclear in scope. The only disclosed way of “modifying” the activation map is to multiply it element-wise by an importance score vector containing zero values for less important channels (instant specification ¶[0019], FIG. 1, multiplier 32; ¶[0021], FIG. 2, multiplier 52; ¶[0028], FIG. 5, block 94), and dependent claims 32 and 39 expressly require that “the multiplier output[s] the modified activation map.” Element-wise multiplication yields a tensor with the same channel dimension as the input, in which some channels are zero. It is therefore unclear whether “having less channels” requires the channel dimension of the tensor to be physically reduced, or is met by zeroing (masking) channel values while the tensor keeps its original number of channels. A person of ordinary skill in the art would not be reasonably apprised of the metes and bounds of the claims, the claim is therefore indefinite. For examination, the limitation is interpreted as requiring that fewer channels of the modified activation map carry non-zero values to be considered by the second network layer (consistent with “exclude … from consideration by the second network layer,” ¶[0018]).
Regarding claims 30, 37 and 44, the claims recite “the one or more fully-connected layers performing the importance classification and outputting importance score vector.” Without an article, it is unclear whether this refers to “an importance score vector” recited in the parent independent claim or to a further vector, the claim is therefore indefinite. For examination it is interpreted as “the importance score vector.”
Regarding claims 29, 31, 36, 38, 43 and 45, the claims recite that the pooling layer (claims 29, 36, 43) or the one or more fully-connected layers (claims 31, 38, 45) are “external to the neural network.” The specification describes the pooling (downsample) layer and fully-connected layers as components of a “branch path 20 located (e.g., positioned and/or connected) between the first network layer 22 and the second network layer 24” (¶[0018], FIG. 1), and states that method 80 “may generally be implemented in a neural network containing one or more branch paths” (¶[0024], FIG. 4). Because the specification treats the branch path as located inside, and contained in, the neural network, the boundary of “the neural network” for purposes of being “external” to it cannot be determined, the claim is therefore indefinite. For examination, “external to the neural network” is interpreted as “located in a branch path/sub-network separate from the main (backbone) data path of the first and second network layers.”
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 26–45 are rejected under 35 U.S.C. 103 as being unpatentable over Hu et al., “Squeeze-and-Excitation Networks,” arXiv:1709.01507v1, Sept. 5, 2017 [hereafter Hu] in view of Lin et al., “Runtime Neural Pruning,” NIPS 2017 [hereafter Lin].
With regards to claim 26, Hu teaches
“A method, comprising: generating, during an inference process of a neural network, an activation map from a first network layer in the neural network, the first network layer preceding a second network layer in the neural network;” (Hu,
PNG
media_image1.png
211
635
media_image1.png
Greyscale
FIG.1-3, Section 3 Squeeze-and-Excitation Blocks, ‘For simplicity of exposition, in the notation that follows we take Ftr to be a standard convolutional operator … We can then write the outputs of Ftr as U = [u1, u2, . . . , uC]’; Section 4 Model and Computational Complexity, ‘we also benchmark CPU inference time for each model: for a 224×224 pixel input image, ResNet-50 takes 164ms, compared to 167ms for SE-ResNet-50’; Section 1 Introduction, ‘The feature maps U are then reweighted to generate the output of the SE block which can then be fed directly into subsequent layers’; the convolution Ftr is a first network layer whose output feature maps U (W×H×C) are an activation map; the SE-network is run on test images at inference, and U precedes the subsequent layers, i.e., a second network layer).
“generating, during the inference process, an importance score vector using context information from the first network layer by performing an importance classification, the context information comprising channels values associated with the first network layer;” (Hu, FIG.1-3, Section 3.1 Squeeze: Global Information Embedding, ‘we propose to squeeze global spatial information into a channel descriptor. This is achieved by using global average pooling to generate channel-wise statistics’; Section 3.2 Exemplars: SE-Inception and SE-ResNet, ‘s = Fex(z,W) = σ(g(z,W)) = σ(W2δ(W1z))’; Section 3.2 Discussion, page 4, ‘The activations act as channel weights adapted to the input-specific descriptor z. In this regard, SE blocks intrinsically introduce dynamics conditioned on the input’; Section 1, page 1, ‘learn to use global information to selectively emphasise informative features and suppress less useful ones’; the channel-wise statistics z are context information comprising channel values of U; the per-channel gate vector s, computed for each input at run time, is an importance score vector; the sigmoid gating that decides which channels are emphasised and which are suppressed is an importance classification).
“modifying the activation map using the importance score vector ….” (Hu, FIG.1-3, Section 3.2 Exemplars: SE-Inception and SE-ResNet, ‘The final output of the block is obtained by rescaling the transformation output U with the activations: x̃c = Fscale(uc, sc) = sc · uc … Fscale(uc, sc) refers to channel-wise multiplication between the feature map uc ∈ R^(W×H) and the scalar sc’; U is modified channel-by-channel using the vector s).
“processing, during the inference of the neural network, the modified activation map in a second network layer, the second network layer arranged after the first network layer in the neural network.” (Hu, FIG.1-3, Section 3 Squeeze-and-Excitation Blocks, ‘We propose to achieve this by explicitly modelling channel interdependencies to recalibrate filter responses in two steps, squeeze and excitation, before they are fed into next transformation’; Section 1 Introduction, ‘the output of the SE block which can then be fed directly into subsequent layers’; the recalibrated feature map X̃ is processed by the next transformation, i.e., a second network layer arranged after the first, during the forward pass at inference).
Hu does not explicitly detail “the modified activation map having less channels than the activation map.”
However, Lin teaches “the modified activation map having less channels than the activation map;” (Lin, FIG.1,
PNG
media_image2.png
396
657
media_image2.png
Greyscale
Abstract, ‘we propose a Runtime Neural Pruning (RNP) framework which prunes the deep neural network dynamically at the runtime … The agent judges the importance of each convolutional kernel and conducts channel-wise pruning conditioned on different samples’; Section Runtime Neural Pruning, ‘The goal is to find and prune the redundant convolutional kernels in Ki+1, given feature maps Fi … to reduce computation and achieve maximum performance simultaneously’; Section 3.2, page 4, ‘For convolution kernel Ki with ni output channels, we determine which output channels are calculated and which to prune’; Figure 1, page 3, ‘calculated / pruned feature maps Fi’; Section 3.1, page 3, ‘Note that our framework infers through standard convolutional layer after pruning’; Algorithm 1,
PNG
media_image3.png
399
630
media_image3.png
Greyscale
Lin decides at inference, per input and from the current feature maps, which channels are important and prunes the rest, so that the feature map passed on has fewer (computed, non-zero) channels than the full feature map, and the next convolutional layer processes the pruned feature map).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Hu and Lin before him or her, to modify the neural network structure adjustment process of Hu to include channel reduction as shown in Lin.
The motivation for doing so would have been to support runtime neural network pruning (Lin, Abstract).
With regards to claim 27, Hu in view of Lin teaches
“The method of claim 26, wherein generating the importance score vector comprises: aggregating, during the inference process, the context information from the first network layer;” (Hu, FIG.1-2, Section 1 Introduction, ‘The features U are first passed through a squeeze operation, which aggregates the feature maps across spatial dimensions W × H to produce a channel descriptor’; Section 3.2 Discussion, page 4, ‘channel weights adapted to the input-specific descriptor z’; the squeeze is performed on each input’s feature maps, i.e., during inference).
“and generating the importance score vector from the aggregated context information.” (Hu, FIG.1-2, Section 3.2, page 3, ‘To make use of the information aggregated in the squeeze operation, we follow it with a second operation which aims to fully capture channel-wise dependencies … s = Fex(z,W) ….’; s is generated from the aggregated descriptor z).
With regards to claim 28, Hu in view of Lin teaches
“The method of claim 27, wherein aggregating the context information comprises: inputting the context information into a pooling layer, the pooling layer averaging at least some of the channel values and generating the aggregated context information.” (Hu, FIG.1-2,
PNG
media_image4.png
282
397
media_image4.png
Greyscale
Section 3.1, page 3, ‘This is achieved by using global average pooling to generate channel-wise statistics. Formally, a statistic z ∈ R^C is generated by shrinking U through spatial dimensions W × H, where the c-th element of z is calculated by: zc = Fsq(uc) = 1/(W × H) Σi Σj uc(i, j) …’; Figures 2 and 3, page 4, ‘Global pooling’; the global average pooling layer averages the values of each channel to produce z).
With regards to claim 29, Hu in view of Lin teaches
“The method of claim 28,”
Hu does not explicitly detail “wherein the pooling layer is external to the neural network”.
However, Lin teaches “wherein the pooling layer is external to the neural network.” (Lin, FIG.1, Section 3, page 3, ‘RNP consists of two sub-networks, the backbone CNN network and the decision network which decides how to prune the convolution kernels conditioned on the input image and current feature maps’; Section 3.2, page 4, ‘Given feature map Fi, we first extract a dense feature embedding pFi with global pooling’; the global pooling is part of the separate decision network, not of the backbone CNN, i.e., external to the neural network as interpreted in ¶6 above).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Hu and Lin before him or her, to modify the neural network structure adjustment process of Hu to include more detailed processing as shown in Lin.
The motivation for doing so would have been to support runtime neural network pruning (Lin, Abstract).
With regards to claim 30, Hu in view of Lin teaches
“The method of claim 27, wherein generating the importance score vector from the aggregated context information comprises: inputting the aggregated context information into one or more fully-connected layers, the one or more fully-connected layers performing the importance classification and outputting importance score vector.” (Hu, FIG.1-2, Section 3.2, page 3, ‘we parameterise the gating mechanism by forming a bottleneck with two fully-connected (FC) layers around the non-linearity, i.e. a dimensionality-reduction layer with parameters W1 with reduction ratio r … a ReLU and then a dimensionality-increasing layer with parameters W2’; Section 4, page 4, ‘two small fully connected layers in the excitation phase’; z is input to the two FC layers, which output the per-channel gate vector s).
With regards to claim 31, Hu in view of Lin teaches
“The method of claim 30,”
Hu does not explicitly detail “wherein the one or more fully-connected layers are external to the neural network”.
However, Lin teaches “wherein the one or more fully-connected layers are external to the neural network.” (Lin, FIG.1, Section 3.2, page 4, ‘we use the encoder E (a fully connected layer) to project the pooled feature into a fixed-length embedding … The decoder (also a fully connected layer) produces the Q-value for decision’; Section 3, page 3, ‘RNP consists of two sub-networks, the backbone CNN network and the decision network’; the FC encoder and decoder belong to the separate decision network, external to the backbone neural network).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Hu and Lin before him or her, to modify the neural network structure adjustment process of Hu to include more detailed processing as shown in Lin.
The motivation for doing so would have been to support runtime neural network pruning (Lin, Abstract).
With regards to claim 32, Hu in view of Lin teaches
“The method of claim 26, wherein modifying the activation map using the importance score vector comprises: inputting the activation map and the importance score vector into a multiplier, the multiplier outputting the modified activation map.” (Hu, FIG.1-2, Section 3.2, page 3, ‘x̃c = Fscale(uc, sc) = sc · uc … Fscale(uc, sc) refers to channel-wise multiplication between the feature map uc … and the scalar sc’; Figures 2 and 3, page 4, ‘Scale’; the Scale operation receives U and s and outputs the recalibrated map X̃). In the combination, the importance scores of pruned channels are zero, so the multiplier output has fewer non-zero channels (see claim 26 rejection).
With regards to claim 33, Hu in view of Lin teaches
“One or more non-transitory computer-readable media storing instructions executable to perform operations, the operations comprising:” (Hu, FIG.1-2, Section 4 Model and Computational Complexity, ‘both timings are performed on a server with 8 NVIDIA Titan X GPUs … we also benchmark CPU inference time for each model’; Hu, Section 1, page 2, ‘Our models and related materials have been made available to the research community’).
The remaining limitations of claim 33 are identical to the limitations of claim 26 and are taught by Hu in view of Lin as mapped therein, with the same rationale and motivation to combine.
Claims 34, 35, 36, 37, 38 and 39 recite the same further limitations as claims 27, 28, 29, 30, 31 and 32, respectively, and are rejected for the reasons set forth therein.
With regards to claim 40, Hu in view of Lin teaches
“An apparatus, comprising: a computer processor for executing computer program instructions; and a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations comprising:” (Hu, FIG.1-2, Section 4 Model and Computational Complexity, ‘a server with 8 NVIDIA Titan X GPUs … we also benchmark CPU inference time’).
The remaining limitations of claim 40 are identical to the limitations of claim 26 and are taught by Hu in view of Lin as mapped therein, with the same rationale and motivation to combine.
Claims 41, 42, 43, 44 and 45 recite the same further limitations as claims 27, 28, 29, 30 and 31, respectively, and are rejected for the reasons set forth therein.
Additional Relevant Art
The prior art made of record is considered pertinent to applicant’s disclosure and is recorded on Form PTO-892. Applicant is required under 37 C.F.R. § 1.111 (c) to consider these references fully when responding to this action, with particular attention paid to:
Huang et al., US-PATENT NO.11868878B1 [hereafter Huang] shows neural network with activation map.
Vantrease et al., US-PGPUB NO.20190294413A1 [hereafter Vantrease] shows neural network evaluation with score vectors.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TSU-CHANG LEE whose telephone number is 571-272-3567. The fax number is 571-273-3567.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas, can be reached 571-272-2589.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TSU-CHANG LEE/
Primary Examiner, Art Unit 2128
